Make the Web Readable Again: Deliver Article and News Digests to Your Favourite e-reader

RSS feeds are a largely underestimated feature of the web nowadays — at least outside the circles of geeks. Not only they are much more omnipresent than many think (every single respectable news website provides at least one feed); not only they empower users with the possibility of creating their own news feeds and boards through aggregators, without relying on the mercy of a cloud-run algorithm; but their structured nature (under the hood an RSS feed is just a structured XML) offers the possibility to build automation pipelines that deliver the content we want wherever we want, whenever we want, in whichever format we want. IFTTT is a popular option to build custom logic on RSS feeds. It makes it very intuitive to build relatively complex rules such as “send me a weekly digest with The Economist articles published in the latest issue”, or “send a Telegram message with the digest from NYT every day at 6am”, or “send a notification to my mobile whenever XKCD publishes new comics”.

In my opinion, however, it’s thanks to internet-connected e-readers, such as the Kindle or Mobiscribe, as well as web-services like Mercury and Instapaper that can convert a web page in a clean print-friendly format, that RSS feeds can really be used at their best.

It’s great to have our news sources neatly organized in an aggregator, as well as the possibility to configure push notifications upon the publication of new articles, or daily/weekly/monthly digests delivered wherever we like. But these features solve only the first part of the problem — content distribution. The second part of the problem — content consumption — comes when we click on a link, delivered on whichever device and in whichever format we like, and we start reading the actual article.

Such an experience nowadays happens mostly on laptop screens or, worse, tiny smartphone screens, where we are expected to hectically scroll through often non-mobile optimized content filled with ads and paywalls, while a myriad of other notifications demand for their share of our attention. Reading lengthy content on a smartphone screen is arguably a bad experience as much as it is browsing the web on a Kindle.

Wouldn’t it be great if we could get our favourite content automatically delivered to our favourite reading device, properly formatted and in a comfortably readable size, without all the clutter and distractions, and without having a backlit screen always in front of our eyes? In this article we’ll see how to do it by using several technological tools (an e-reader, a Kindle account, Mercury API, Instapaper), and how to connect all those pieces together through platypush.

Configuring your Kindle account

I’ll assume in the first section that you have a Kindle, a linked Amazon account, and a Gmail account that we’ll use to programmatically send documents to the device via email (we’ll see later in this story how to leverage Instapaper with other devices).

First, you’ll have to create an email address associated to your Kindle that will be used to remotely deliver documents:

• Head to the Amazon content and device portal and login with your account.
• Click on the second tab (“Your Devices”) and click on the context menu next to the device where your content should be delivered.
• You’ll see the email address associated to your device. Copy it or click on “Edit” to change it.
• Click on the third tab (“Settings”) and scroll to the bottom to the section titled “Personal Document Settings”.
• Scroll to the bottom to the section named “Approved Personal Document E-mail List” and add your Gmail address as a trusted source.

To check that everything works you can now try and send a PDF document to your Kindle from your personal email address: if the device is connected to a WiFi then the document should automatically downloaded within a few seconds.

Install platypush with the required dependencies

Those who follow me may already have read some of my articles on platypush, the automation platform I’ve built over the past few years — those unfamiliar can get started with my first Medium article on the topic. Among the features it offers there are also all the ingredients we need for the purpose of this article. We need in particular to build an automation pipeline that:

• Periodically checks a list of RSS sources for new content;
• Pre-processes the new items by simplifying the web page (through the Mercury parser or Instapaper) and optionally exporting them to PDF;
• Programmatically sends emails to your device(s) with the new content.

First, install platypush with the required extras (any device with any compatible OS will do: a RaspberryPi, an unused laptop, a remote server…):

pip install 'platypush[http,pdf,rss,google]'

You’ll also need to install npm and mercury-parser: Postlight used to provide a web API for its parser before, but they have discontinued it choosing to make instead the project open source:

# Supposing you're on Debian or Debian-derived OS
apt-get install nodejs npm
npm install @postlight/mercury-parser

Second, link platypush to your Gmail account to send documents via email:

• Create a new project on the Google developers console;
• Click on “Credentials” from the context menu -> OAuth Client ID;
• Once generated you can see your new credentials in the “OAuth 2.0 client IDs” section. Click on the “Download” icon to save them to a JSON file;
• Copy the file to your platypush device/server under e.g.

  ~/.credentials/client_secret.json

  ;
• Run the following command on the device to authorize the application:

python -m platypush.plugins.google.credentials \
    "https://www.googleapis.com/auth/gmail.modify" \
    ~/.credentials/client_secret.json \
    --noauth_local_webserver

• Copy the link in your browser, login with your Google account if required and authorize the application.

Now that you’ve got everything in place it’s time to configure platypush to process your favourite feeds.

Create a rule to automatically send articles to your Kindle

The http.poll backend is a flexible component that can be configured to poll and process updates from many web resources — JSON, RSS, Atom etc.

Suppose that you want to check for updates on The NYT Daily RSS feed twice a day and deliver a digest with the new content to your Kindle. You’ll want to create a configuration that looks like this in your

~/.config/platypush/config.yaml

backend.http.poll:
    requests:
        -
            # This poll will handle an RSS feed
            type: platypush.backend.http.request.rss.RssUpdates
            # RSS feed URL and title
            url: http://feeds.podtrac.com/zKq6WZZLTlbM
            title: NYT - The Daily
            # How often we want to check for updates
            # 12h = 43200 secs
            poll_seconds: 43200
            # We want to convert content to PDF
            digest_format: pdf
            # We want to parse and extract the content from
            # the web page using Mercury Parser
            extract_content: True

Create now an event hook that reacts to a NewFeedEvent and sends the processed content to your Kindle via email:

event.hook.OnPdfDigest:
    if:
        type: platypush.message.event.http.rss.NewFeedEvent
        digest_format: pdf
    then:
        action: google.mail.compose
        args:
            sender: [email protected]
            to: [email protected]
            subject: ${title} feed digest
            body: Your ${title} RSS feeds delivered directly to your Kindle
            files:
                - ${digest_filename}

Restart platypush. As soon as the application finds items in the target feed that haven’t yet been processed it’ll parse them, convert them to PDF, trigger a NewFeedEvent that will be captured by your hook, and the resulting PDF will be delivered to your Kindle. You can add more monitored RSS sources by simply adding more items in the request attribute of the

http.poll

Sending content to your e-reader on the fly

RSS feeds are awesome but they aren’t the only way we consume discover and consume content today. Many times we scroll through our favourite social media timeline, bump into an interesting article, start reading it on our tiny screen, and we’d like to keep reading it later when we are on a bigger screen. Several tools and products have spawned to provide a solution to the “parse it, save it and read it later” problem — among those Evernote, Pocket and Instapaper itself.

Most of them, however, are still affected by the same issue: either they don’t do a good job at actually parsing and extracting the content in a more readable format (except for Instapaper; Pocket only saves a link to the original content, while Evernote’s content parsing capabilities have quite some room for improvement, to say the least), or they’re still bound to the backlit screen of the smartphone or computer that runs them.

Wouldn’t it be cool to bump into an interesting article while we scroll our Facebook timeline on our Android device, and with a single click deliver it to our Kindle in a nice and readable format? Let’s see how to implement such a rule in platypush.

First, we’ll need something that runs on our mobile device to programmatically communicate with the instance of platypush installed on our Raspberry/computer/server. I consider Tasker one of the best applications suited for this purpose: with Tasker (and the other related apps developed by joaoapps) it’s possible to automate anything on your Android device and create sophisticated rules that connect it to anything.

There are many ways for Tasker to communicate with platypush (direct RPC over HTTP calls, using Join with an external MQTT server to dispatch messages, using an intermediate IFTTT hook, or Pushbullet…), as well as there are many ways for platypush to communicate back to Tasker on your mobile device (using AutoRemote with the platypush plugin to send custom events, using IFTTT with any service connected to your mobile, using the Join API, or, again, Pushbullet…). We’ll use Pushbullet in this article because it doesn’t require as many configuration steps as other techniques.

• Install Tasker, AutoShare and Pushbullet on your Android device;
• Go to your Pushbullet account page and click “Create Access Token” to create a new access token that will be used by platypush to listen for the messages sent to your account. Enable the Pushbullet backend on platypush by adding these lines to

  ~/.config/platypush/config.yaml

  :

backend.pushbullet:
    token: YOUR-TOKEN
    device: platypush-device

• Also add a procedure to your config.yaml that given a URL as input extracts the content, converts it to PDF and sends it to your Kindle:

procedure.send_web_page_to_kindle:
    # Some apps don't share only the link, but also some
    # text such as "I've found this interesting article
    # on XXX". The following action strips out extra content
    # from the input and only extracts the URL.
    - action: variable.mset
      args:
          url: '${__import__("re").sub(r"^.*(https?://[^\s]*).*", r"\1", ${url})}'
    # Extract the content and generate a PDF
    - action: http.webpage.simplify
      args:
          url: ${url}
          outfile: ~/outfile.pdf
    # Use the parsed title as a name for the PDF
    - action: variable.mset
      args:
          OUTFILE: /home/user/${title}.pdf
    - action: shell.exec
      args:
          cmd: 'mv ~/outfile.pdf "${OUTFILE}"'
    # Send the file to your Kindle email address
    - action: google.mail.compose
      args:
          sender: [email protected]
          to: [email protected]
          subject: ${title}
          body: 'Original URL: ${url}'
          files:
              - ${OUTFILE}
    # Remove the temporary PDF
    - action: shell.exec
      args:
          cmd: 'rm "${OUTFILE}"'

• Restart platypush and check from Pushbullet that your new virtual device, “platypush-device” in the example above, has been created;
• On your mobile open AutoShare, select “Manage Commands” and create a new command named e.g. “Send to Kindle”;
• Open Tasker and create a new profile; select Event -> AutoShare -> Configuration -> Command -> Command Filter -> “Send to Kindle”. It means that this profile will be triggered whenever you select “Send to Kindle” from a share link dialog;
• In the task associated to this trigger tap the “plus” icon to add a new action and select “Push a notification” (the action with the green Pushbullet icon next to it);
• Select “platypush-device” as a target device and paste the following JSON as message:

{"type":"request", "action":"procedure.send_web_page_to_kindle", "args": {"url":"%astext"}}

• In the example above

  %astext

  is a special variable in Tasker that contains the text shared by the source app (in this case the link sent to AutoShare);
• Open your browser and go to the web link of an article that you’d like to send to your Kindle, select “Share” -> AutoShare command -> Send to Kindle;
• The parsed article should be delivered to your e-reader in optimized PDF format within seconds.

Using Instapaper on other Android-based e-readers

I’ve briefly mentioned Instapaper through this article already. I really love both the service as the app; I consider it somehow an implementation of what Evernote should have been and it’s never been. Just browse to an article on the web, click “Share to Instapaper”, and within one click that web page will be parsed into a readable format, all the clutter and ads removed, and added to your account.

What makes Instapaper really interesting though is the fact that its Android app is really minimal (yet extremely well designed), and it runs well also on devices that run older versions of Android or aren’t that powerful. That wouldn’t be such a big deal itself if it weren’t that products like the Mobiscribe are slowly hitting the market — and I hope its example will be followed by others.

The Mobiscribe can be used both as e-reader and e-ink notepad, but what really makes it interesting is that it runs Android — even if it’s an ancient Android KitKat modified release, a more recent version should arrive sooner or later. The presence of an Android OS is what makes this e-reader/tablet much more interesting than other similar products (like reMarkable, that has opted instead of its own OS): even if it’s an old version of Android that runs on an under-powered device, it’s still possible to install some apps on it — and Instapaper is one of them.

It makes it very easy to enhance your reading experience: simply browse the web, add articles to your Instapaper account, and deliver them on-the-fly to your e-reader. If you want, you can also use the Instapaper API in platypush to programmatically send content to your Instapaper account instead of your Kindle. Just create a procedure like this:

procedure.instapaper_add:
    - action: http.request.get
      args:
          url: https://www.instapaper.com/api/add
          params:
              url: ${url}
              username: your_instapaper_username
              password: your_instapaper_password

And call it by sending a JSON request on whichever backend you like:

{"type":"request", "action":"procedure.instapaper_add", "args": {"url":"https://custom-url/article"}}

You can also change the content of the event.hook.OnPdfDigest shown before so it saves digests to your Instapaper account instead of sending them to your Kindle via email.

Conclusions

The amount of information and news channels available on the web has increased exponentially in the last years, but the methods to distribute and consume such content, at least when it comes to flexibility, haven’t improved much. The exponential growth of social media and platforms like Google News means that few large companies nowadays decide which content should appear in front of your eyes, how that content should be delivered to you, and where you can consume it.

Technology should be about creating more opportunities and flexibility, not reducing them, so such a dramatic centralization shouldn’t be acceptable for a power user. Luckily, decades-old technologies like RSS feeds can come to rescue, allowing us to tune what we want to read and build automation pipelines that distribute the content wherever and whenever we like.

Also, e-readers are becoming more and more pervasive, thanks also to the drop in the price of e-ink displays in the last few years and to more companies and products entering the market. Automating the delivery of web content to e-readers can really create a new and more comfortable way to stay informed — and find another great use case for our Kindle, other than downloading novels to read on the beach.