I’ve always been a supporter of well-curated newsletters. They give me an opportunity to get a good overview of what happened in the fields I follow within a span of a day, a week or a month. However, not all the newsletters fit this category. Some don’t think three times before selling email addresses to 3rd-parties — and within the blink of an eye your mailbox can easily get flooded with messages that you didn’t request. Others may sign up your address for other services or newsletters as well, and often they don’t often much granularity to configure which communications you want to receive. Even in the best-case scenario, the most privacy-savvy user may still think twice before signing up for a newsletter — you’re giving your personal email address to someone else you don’t necessarily trust, implying “yes, this is my address and I’m interested in this subject”. Additionally, most of the newsletters spice up their URLs with tracking parameters, so they can easily measure user engagement — something you may not necessarily be happy with. Moreover, the customization junkie may also have a valid use case for a more finely tuned selection of content in his newsletter — you may want to group some sources together into the same daily/weekly email, or you may be interested only in some particular subset of the subjects covered by a newsletter, filtering out those that aren’t relevant, or customize the style of the digest that gets delivered. Finally, a fully automated way to deliver newsletters through 5 lines of code and the tuning of a couple of parameters is the nirvana for many companies of every size out there. Feed up the newsletter Those who read my articles in the past may know that I’m an avid consumer of RSS feeds. Despite being a 21-year-old technology, they do their job very well when it comes to deliver the information that matters without all the noise and trackers, and they provide a very high level of integration being simple XML documents. However, in spite of all the effort I put to be up-to-date with all my sources, a lot of potentially interesting content inevitably slips through — and that’s where newsletters step in, as they filter and group together all the content that was generated in a given time frame and periodically deliver it to your inbox. My ideal solution would be something that combines the best aspects of both the worlds: the flexibility of an RSS subscription, combined with a flexible way of filtering and aggregating content and sources, and get the full package delivered at my door in whichever format I like (HTML, PDF, MOBI…). In this article I’m going to show how to achieve this goal with a few tools: One or more sources that you want to track and that support RSS feeds (in this example I’ll use the MIT Technology Review , but the procedure works for any RSS feed). RSS feed An email address. to do the heavy-lifting job — monitor the RSS sources at custom intervals, trigger events when a source has some new content, create a digest out of the new content, and deliver the full package to a list of email addresses. Platypush Let’s cover these points step by step. Installing and configuring Platypush Those who have already read my previous articles may have heard of — the automation platform I’ve been building in the past few years. For those who aren’t familiar, an advised read is my that illustrates some of its capabilities and the paradigm behind it. Platypush first Medium post We’ll be using the backend configured with one or more objects to poll our RSS sources at regular intervals and create the digests, and either the plugin or the plugin to send the digests to our email. http.poll RssUpdates mail.smtp google.mail You can install Platypush on any device where you want to run your logic — a RaspberryPi, an old laptop, a cloud node, and so on. We will install the base package with the module. rss Optionally, you can install it with the pdf module as well (if you want to export your digests also to PDF) or the module (if you want to send the newsletter from a GMail address instead of an SMTP server). google The first option is to install the latest stable version through : pip pip install pip install 'platypush[rss]' # Or 'platypush[rss,pdf,google]' The other option is to install the latest git version: git git@github.com/BlackLight/platypush.git platypush
pip install pip install clone cd '.[rss]' # Or '.[rss,pdf,google]' Monitoring your RSS feeds Once the software is installed, create the configuration file if it doesn't exist already and add the configuration for the RSS monitor: ~/.config/platypush/config.yaml # Generic HTTP endpoint monitor backend.http.poll: requests: # Add a new RSS feed to the pool - type: platypush.backend.http.request.rss.RssUpdates url: https://www.technologyreview.com/feed/ # URL to the RSS feed title: MIT Technology Review # Title of the feed (shown in the head of the digest) poll_seconds: 86400 # How often we should monitor this source (24*60*60 secs = once a day) digest_format: html # Format of the digest (HTML or PDF) You can also add more sources to the requests object, each with its own configuration. Also, you can customize the style of your digest by passing some valid CSS to these configuration attributes: http.poll # Style of the body element body_style: 'font-size: 20px; font-family: "Merriweather", Georgia, "Times New Roman", Times, serif' # Style of the main title title_style: 'margin-top: 30px' # Style of the subtitle subtitle_style: 'margin-top: 10px; page-break-after: always' # Style of the article titles article_title_style: 'font-size: 1.6em; margin-top: 1em; padding-top: 1em; border-top: 1px solid #999' # Style of the article link article_link_style: 'color: #555; text-decoration: none; border-bottom: 1px dotted font-size: 0.8em' # Style of the article content article_content_style: 'font-size: 0.8em' The attribute determines the output format of your digest - you may want to choose if you want to deliver a summary of the articles in a newsletter, or if you want instead to deliver the full content of each item as an attachment to an email address. digest_format html pdf Bonus point: since you can send PDFs to a Kindle if you , this mechanism allows you to deliver the full digest of your RSS feeds to your Kindle's email address. configured an email address The object also provides native integration with the to automatically scrape the content of a web page - I covered some of these concepts in my on how to parse RSS feeds and send the PDF digest to your e-reader. RssUpdates Mercury Parser API past article The same mechanism works well for newsletters too. If you want to parse the content of the newsletter as well, all you have to do is configure the Platypush plugin. Since the Mercury API doesn't provide a Python binding, this requires a couple of JavaScript dependencies: http.webpage apt-get install nodejs npm npm install [-g] @postlight/mercury-parser pip install # Install Node and NPM, e.g. on Debian: # Install the Mercury Parser API # Make sure that the Platypush PDF module dependencies # are installed if you plan HTML->PDF conversion 'platypush[pdf]' Then, if you want to parse the full content of the items and generate a PDF digest out of them, change your configuration to something like this: http.poll backend.http.poll: requests: - type: platypush.backend.http.request.rss.RssUpdates url: https://www.technologyreview.com/feed/ title: MIT Technology Review poll_seconds: 86400 digest_format: pdf # PDF digest format extract_content: True # Extract the full content of the items : Extracting the full content of the articles in an RSS feed has two limitations — a practical one and a legal one: WARNING Some websites may require user login before displaying the full content of an article. Some websites perform such checks client-side — and the parser API can usually circumvent them, especially if the full content of an article is actually just hidden behind a client-side paywall. Some websites, however, implement their user checks server-side too before sending the content to the client — and in those cases the parser API may return only a part of the content or no content at all. Always keep in mind that parsing the full content of an article behind a paywall may represent a violation of intellectual property under some jurisdictions. Configuring the mail delivery When new content is published on a subscribed RSS feed Platypush will generate a and it should create a copy of the digest under . NewFeedEvent ~/.local/share/platypush/feeds/cache/{date:time}_{feed-title}.[html|pdf] The in particular is the link you need to create your custom logic that sends an email to a list of addresses when new content is available. NewFeedEvent First, configure the Platypush mail plugin you prefer. When it comes to sending emails you primarily have two options: The plugin — if you want to send emails directly through an SMTP server. Platypush configuration: mail.smtp mail.smtp: username: you@gmail.com password: your-pass server: smtp.gmail.com port: 465 ssl: True The plugin— if you want to use the native GMail API to send emails. If that is the case then first make sure that you have the dependencies for the Platypush Google module installed: google.mail pip install 'platypush[google]' In this case you’ll also have to create a project on the and download the OAuth credentials: Google Developers console Click on “Credentials” from the context menu > OAuth Client ID. Once generated, you can see your new credentials in the “OAuth 2.0 client IDs” section. Click on the “Download” icon to save them to a JSON file. Copy the file to your Platypush device/server under e.g. . ~/.credentials/google/client_secret.json Run the following command on the device to authorize the application: python -m platypush.plugins.google.credentials \ \
    ~/.credentials/google/client_secret.json \
    --noauth_local_webserver "https://www.googleapis.com/auth/gmail.modify" At this point the GMail delivery is ready to be used by your Platypush automation. Connecting the dots Now that both the RSS parsing logic and the mail integration are in place, we can glue them together through the event. The new advised way to configure events in Platypush is through native Python scripts - the custom YAML-based syntax for events and procedure was becoming too cumbersome to maintain and write (although it’s still supported), and I feel like going back to a clean and simple Python API may be a better option. NewFeedEvent Create and initialize the Platypush scripts directory, if it doesn’t existing already: mkdir -p ~/.config/platypush/scripts ~/.config/platypush/scripts
touch __init__.py cd # Initialize the root Python module Then, create a new on : hook NewFeedEvent rss_news.py $EDITOR os typing List platypush.event.hook hook platypush.message.event.http.rss NewFeedEvent platypush.utils run maillist = os.path.expanduser( ) open(maillist, ) f: [addr.strip() addr f.readlines() addr.strip() addr.strip().startswith( )] open(event.args[ ], ) f:
        run(action= ,
            from_= ,
            to=get_addresses(),
            subject= ,
            body=f.read(),
            body_type= ) import from import from import from import from import # Path to your mailing list - a text file with one address per line '~/.mail.list' -> List[str]: def get_addresses () with 'r' as return for in if and not '#' # This hook matches: # - event_type=NewFeedEvent # - digest_format='html' # - source_title='MIT Technology Review' @hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review') : def send_mit_rss_feed_digest (event: NewFeedEvent, **_) # The digest output file is stored in event.args['digest_filename'] with 'digest_filename' 'r' as 'mail.smtp.send' 'you@yourdomain.com' f' feed digest' {event.args.get( )} "source_title" 'html' If you opted for the native GMail plugin you may want to go for: open(event.args[ ], ) f:
        run(action= ,
            sender= ,
            to=get_addresses(),
            subject= ,
            body=f.read()) @hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review') : def send_mit_rss_feed_digest (event: NewFeedEvent, **_) # The digest output file is stored in event.args['digest_filename'] with 'digest_filename' 'r' as 'google.mail.compose' 'you@gmail.com' f' feed digest' {event.args.get( )} "source_title" If instead you want to send the digest in PDF format as an attachment: run(action= ,
        from_= ,
        to=get_addresses(),
        subject= ,
        body= ,
        attachments=[event.args[ ]) run(action= ,
        sender= ,
        to=get_addresses(),
        subject= ,
        body= ,
        files=[event.args[ ]) @hook(NewFeedEvent, digest_format='html', source_title='MIT Technology Review') : def send_mit_rss_feed_digest (event: NewFeedEvent, **_) # mail.smtp plugin case 'mail.smtp.send' 'you@yourdomain.com' f' feed digest' {event.args.get( )} "source_title" '' 'digest_filename' # google.mail case 'google.mail.compose' 'you@gmail.com' f' feed digest' {event.args.get( )} "source_title" '' 'digest_filename' Finally, create your file with one destination email address per line and start platypush either from the command line or as a service. You should receive your email with the first batch of articles shortly after startup, and you'll receive more items if a new batch is available after the configured period. ~/.mail.list poll_seconds Previously published at https://medium.com/@automationguru/how-to-automatically-deliver-customized-newsletters-from-rss-feeds-with-platypush-8c540a557fa

Blink

Glue

Google

Build A Smart Baby Monitor Using a RaspberryPi and Tensorflow

Human Detection System Using RaspberryPi, Thermal Camera and Machine Learning

Book a call

Nominated for 2022 - HackerNoon Contributor of the Year - Proof Of Stake

Nominated for 2022 - HackerNoon Contributor of the Year - Git

Nominated for 2022 - HackerNoon Contributor of the Year - Distributed Systems

Too Long; Didn't Read

Make resilience your competitive advantage

Automating Newsletters Generation from RSS Feeds Using Platypush

Automating Newsletters Generation from RSS Feeds Using Platypush

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Build A Smart Baby Monitor Using a RaspberryPi and Tensorflow

10 Indications That You Should Invest in Automation Via APIs

10 Commandments for AI-Assisted Social Media Marketers

11 Best Automation Testing Tools to Try in 2021

12 Use Cases of AI and Machine Learning In Finance

Build A Smart Baby Monitor Using a RaspberryPi and Tensorflow

10 Indications That You Should Invest in Automation Via APIs

10 Commandments for AI-Assisted Social Media Marketers

11 Best Automation Testing Tools to Try in 2021

12 Use Cases of AI and Machine Learning In Finance

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps