Make the Web Readable Again: Deliver Article and News Digests to Your Favourite e-reader

are a largely underestimated feature of the web nowadays — at least outside the circles of geeks. Not only they are much more omnipresent than many think (every single respectable news website provides at least one feed); not only they empower users with the possibility of creating their own news feeds and boards through , without relying on the mercy of a cloud-run algorithm; but their structured nature (under the hood an RSS feed ) offers the possibility to build automation pipelines that deliver the content we want wherever we want, whenever we want, in whichever format we want. is a popular option to build custom logic on RSS feeds. It makes it very intuitive to build relatively complex rules such as “send me a weekly digest with The Economist articles published in the latest issue”, or “ ”, or “ ”. RSS feeds aggregators is just a structured XML IFTTT send a Telegram message with the digest from NYT every day at 6am send a notification to my mobile whenever XKCD publishes new comics In my opinion, however, it’s thanks to internet-connected e-readers, such as the or , as well as web-services like and that can convert a web page in a clean print-friendly format, that RSS feeds can really be used at their best. Kindle Mobiscribe Mercury Instapaper It’s great to have our news sources neatly organized in an aggregator, as well as the possibility to configure push notifications upon the publication of new articles, or daily/weekly/monthly digests delivered wherever we like. But these features solve only the first part of the problem — . The second part of the problem — — comes when we click on a link, delivered on whichever device and in whichever format we like, and we start reading the actual article. content distribution content consumption Such an experience nowadays happens mostly on laptop screens or, worse, tiny smartphone screens, where we are expected to hectically scroll through often non-mobile optimized content filled with ads and paywalls, while a myriad of other notifications demand for their share of our attention. Reading lengthy content on a smartphone screen is arguably a bad experience as much as it is browsing the web on a Kindle. Wouldn’t it be great if we could get our favourite content automatically delivered to our favourite reading device, properly formatted and in a comfortably readable size, without all the clutter and distractions, and without having a backlit screen always in front of our eyes? In this article we’ll see how to do it by using several technological tools (an e-reader, a Kindle account, Mercury API, Instapaper), and how to connect all those pieces together through . platypush Configuring your Kindle account I’ll assume in the first section that you have a Kindle, a linked Amazon account, and a Gmail account that we’ll use to programmatically send documents to the device via email (we’ll see later in this story how to leverage Instapaper with other devices). First, you’ll have to create an email address associated to your Kindle that will be used to remotely deliver documents: Head to the and login with your account. Amazon content and device portal Click on the second tab (“Your Devices”) and click on the context menu next to the device where your content should be delivered. You’ll see the email address associated to your device. Copy it or click on “Edit” to change it. Click on the third tab (“Settings”) and scroll to the bottom to the section titled “Personal Document Settings”. Scroll to the bottom to the section named “Approved Personal Document E-mail List” and add your Gmail address as a trusted source. To check that everything works you can now try and send a PDF document to your Kindle from your personal email address: if the device is connected to a WiFi then the document should automatically downloaded within a few seconds. Install platypush with the required dependencies Those who follow me may already have read some of my articles on platypush, the automation platform I’ve built over the past few years — those unfamiliar can get started with on the topic. Among the features it offers there are also all the ingredients we need for the purpose of this article. We need in particular to build an automation pipeline that: my first Medium article Periodically checks a list of RSS sources for new content; Pre-processes the new items by simplifying the web page (through the or Instapaper) and optionally exporting them to PDF; Mercury parser Programmatically sends emails to your device(s) with the new content. First, install platypush with the required extras (any device with any compatible OS will do: a RaspberryPi, an unused laptop, a remote server…): install pip 'platypush[http,pdf,rss,google]' You’ll also need to install npm and mercury-parser: Postlight used to provide a web API for its parser before, : but they have discontinued it choosing to make instead the project open source apt-get install nodejs npm npm install @postlight/mercury-parser # Supposing you're on Debian or Debian-derived OS Second, link platypush to your Gmail account to send documents via email: Create a new project on the ; Google developers console Click on “Credentials” from the context menu -> OAuth Client ID; Once generated you can see your new credentials in the “OAuth 2.0 client IDs” section. Click on the “Download” icon to save them to a JSON file; Copy the file to your platypush device/server under e.g. ; ~/.credentials/client_secret.json Run the following command on the device to authorize the application: python -m platypush.plugins.google.credentials \ \ ~/.credentials/client_secret.json \ --noauth_local_webserver "https://www.googleapis.com/auth/gmail.modify" Copy the link in your browser, login with your Google account if required and authorize the application. Now that you’ve got everything in place it’s time to configure platypush to process your favourite feeds. Create a rule to automatically send articles to your Kindle The backend is a flexible component that can be configured to poll and process updates from many web resources — JSON, RSS, Atom etc. http.poll Suppose that you want to check for updates on RSS feed twice a day and deliver a digest with the new content to your Kindle. You’ll want to create a configuration that looks like this in your file: The NYT Daily ~/.config/platypush/config.yaml backend.http.poll: requests: - # This poll will handle an RSS feed type: platypush.backend.http.request.rss.RssUpdates # RSS feed URL and title url: http://feeds.podtrac.com/zKq6WZZLTlbM title: NYT - The Daily # How often we want to check for updates # 12h = 43200 secs poll_seconds: 43200 # We want to convert content to PDF digest_format: pdf # We want to parse and extract the content from # the web page using Mercury Parser extract_content: True Create now an event hook that reacts to a NewFeedEvent and sends the processed content to your Kindle via email: event.hook.OnPdfDigest: if: type: platypush.message.event.http.rss.NewFeedEvent digest_format: pdf then: action: google.mail.compose args: sender: your-personal@gmail.com to: your-kindle@kindle.com subject: ${title} feed digest body: Your ${title} RSS feeds delivered directly to your Kindle files: - ${digest_filename} Restart platypush. As soon as the application finds items in the target feed that haven’t yet been processed it’ll parse them, convert them to PDF, trigger a that will be captured by your hook, and the resulting PDF will be delivered to your Kindle. You can add more monitored RSS sources by simply adding more items in the request attribute of the backend. Now enjoy reading your articles from a proper screen, delivered directly to your e-reader once or twice a day — tiny smartphone screens, paywalls, popups and ads feel so much old-fashioned once you dive into this new experience! NewFeedEvent http.poll Sending content to your e-reader on the fly RSS feeds are awesome but they aren’t the only way we consume discover and consume content today. Many times we scroll through our favourite social media timeline, bump into an interesting article, start reading it on our tiny screen, and we’d like to keep reading it later when we are on a bigger screen. Several tools and products have spawned to provide a solution to the “parse it, save it and read it later” problem — among those , and itself. Evernote Pocket Instapaper Most of them, however, are still affected by the same issue: either they don’t do a good job at actually parsing and extracting the content in a more readable format (except for Instapaper; Pocket only saves a link to the original content, while Evernote’s content parsing capabilities have quite some room for improvement, to say the least), or they’re still bound to the backlit screen of the smartphone or computer that runs them. Wouldn’t it be cool to bump into an interesting article while we scroll our Facebook timeline on our Android device, and with a single click deliver it to our Kindle in a nice and readable format? Let’s see how to implement such a rule in platypush. First, we’ll need something that runs on our mobile device to programmatically communicate with the instance of platypush installed on our Raspberry/computer/server. I consider one of the best applications suited for this purpose: with Tasker (and the other related apps developed by joaoapps) it’s possible to automate anything on your Android device and create sophisticated rules that connect it to anything. Tasker There are many ways for Tasker to communicate with platypush (direct RPC over HTTP calls, using with an external MQTT server to dispatch messages, using an intermediate IFTTT hook, or Pushbullet…), as well as there are many ways for platypush to communicate back to Tasker on your mobile device (using with the to send custom events, using IFTTT with any service connected to your mobile, using the , or, again, Pushbullet…). We’ll use in this article because it doesn’t require as many configuration steps as other techniques. Join AutoRemote platypush plugin Join API Pushbullet Install , and on your Android device; Tasker AutoShare Pushbullet Go to your and click “Create Access Token” to create a new access token that will be used by platypush to listen for the messages sent to your account. Enable the Pushbullet backend on platypush by adding these lines to : Pushbullet account page ~/.config/platypush/config.yaml backend.pushbullet: token: YOUR-TOKEN device: platypush-device Also add a procedure to your config.yaml that given a URL as input extracts the content, converts it to PDF and sends it to your Kindle: procedure.send_web_page_to_kindle: # Some apps don't share only the link, but also some # text such as "I've found this interesting article # on XXX". The following action strips out extra content # from the input and only extracts the URL. - action: variable.mset args: url: '${__import__("re").sub(r"^.*(https?://[^\s]*).*", r"\1", ${url})}' # Extract the content and generate a PDF - action: http.webpage.simplify args: url: ${url} outfile: ~/outfile.pdf # Use the parsed title as a name for the PDF - action: variable.mset args: OUTFILE: /home/user/${title}.pdf - action: shell.exec args: cmd: 'mv ~/outfile.pdf "${OUTFILE}"' # Send the file to your Kindle email address - action: google.mail.compose args: sender: your-personal@gmail.com to: your-kindle@kindle.com subject: ${title} body: 'Original URL: ${url}' files: - ${OUTFILE} # Remove the temporary PDF - action: shell.exec args: cmd: 'rm "${OUTFILE}"' Restart platypush and check from that your new virtual device, “platypush-device” in the example above, has been created; Pushbullet On your mobile open AutoShare, select “Manage Commands” and create a new command named e.g. “Send to Kindle”; Open Tasker and create a new profile; select Event -> AutoShare -> Configuration -> Command -> Command Filter -> “Send to Kindle”. It means that this profile will be triggered whenever you select “Send to Kindle” from a share link dialog; In the task associated to this trigger tap the “plus” icon to add a new action and select “Push a notification” (the action with the green Pushbullet icon next to it); Select “platypush-device” as a target device and paste the following JSON as message: { : , : , : { : }} "type" "request" "action" "procedure.send_web_page_to_kindle" "args" "url" "%astext" In the example above is a special variable in Tasker that contains the text shared by the source app (in this case the link sent to AutoShare); %astext Open your browser and go to the web link of an article that you’d like to send to your Kindle, select “Share” -> AutoShare command -> Send to Kindle; The parsed article should be delivered to your e-reader in optimized PDF format within seconds. Using Instapaper on other Android-based e-readers I’ve briefly mentioned Instapaper through this article already. I really love both the service as the app; I consider it somehow an implementation of what Evernote should have been and it’s never been. Just browse to an article on the web, click “Share to Instapaper”, and within one click that web page will be parsed into a readable format, all the clutter and ads removed, and added to your account. What makes Instapaper really interesting though is the fact that its Android app is really minimal (yet extremely well designed), and it runs well also on devices that run older versions of Android or aren’t that powerful. That wouldn’t be such a big deal itself if it weren’t that products like the are slowly hitting the market — and I hope its example will be followed by others. Mobiscribe The Mobiscribe can be used both as e-reader and e-ink notepad, but what really makes it interesting is that it runs Android — even if it’s an , a more recent version should arrive sooner or later. The presence of an Android OS is what makes this e-reader/tablet much more interesting than other similar products (like , that has opted instead of its own OS): even if it’s an old version of Android that runs on an under-powered device, it’s still possible to install some apps on it — and Instapaper is one of them. ancient Android KitKat modified release reMarkable It makes it very easy to enhance your reading experience: simply browse the web, add articles to your Instapaper account, and deliver them on-the-fly to your e-reader. If you want, you can also use the in platypush to programmatically send content to your Instapaper account instead of your Kindle. Just create a procedure like this: Instapaper API procedure.instapaper_add: - action: http.request.get args: url: https://www.instapaper.com/api/add params: url: ${url} username: your_instapaper_username password: your_instapaper_password And call it by sending a JSON request on whichever backend you like: { : , : , : { : }} "type" "request" "action" "procedure.instapaper_add" "args" "url" "https://custom-url/article" You can also change the content of the event.hook.OnPdfDigest shown before so it saves digests to your Instapaper account instead of sending them to your Kindle via email. Conclusions The amount of information and news channels available on the web has increased exponentially in the last years, but the methods to distribute and consume such content, at least when it comes to flexibility, haven’t improved much. The exponential growth of social media and platforms like Google News means that few large companies nowadays decide which content should appear in front of your eyes, how that content should be delivered to you, and where you can consume it. Technology should be about creating more opportunities and flexibility, not reducing them, so such a dramatic centralization shouldn’t be acceptable for a power user. Luckily, decades-old technologies like RSS feeds can come to rescue, allowing us to tune what we want to read and build automation pipelines that distribute the content wherever and whenever we like. Also, e-readers are becoming more and more pervasive, thanks also to the drop in the price of e-ink displays in the last few years and to more companies and products entering the market. Automating the delivery of web content to e-readers can really create a new and more comfortable way to stay informed — and find another great use case for our Kindle, other than downloading novels to read on the beach.