Defeat All Paywalls By Building Your Own Google Chrome Extension [DIY]

They’ve been telling us internet will make us free, that we will have the knowledge at the reach of our browser, infinite possibilities and all that stuff… until you hit the (pay)wall:

When you find this message, you have 2 options, you pay the member fee or you can use all your knowledge to bypass it¹… Let’s go that way.

There are various types of paywall control techniques:

• By cookie (we will focus on this one today)
  
• By redirecting to another site
  
• By hiding content using JavaScript
  

Paywall by cookie

Medium use cookies to track us, so the first option would be to delete the cookie that triggers this paywall (field sid in the medium cookie by the way). The problem is I would have to login again then, because maybe after reading it i want to bookmark it o follow the author (my problem is with the paywall not the content).

But why bother, just open the article in private/incognito mode and voilà, no paywall at all.

So how can we achieve this with less friction? Let’s say we build a Chrome extension that can detect we were stopped by the paywall and automatically open the site in private mode, pretty simple no?

If you want a quick guide in how to make a default extension see this Google’s guide, i will focus on the paywall bypass, download the template app and let’s start with that.

The extension

Our app consist of mainly of :

• manifest.json holds the permissions and declare the other files
  
• background.js for communicating the extension
  
• hidden.js detects the paywall warning and triggers the private window
  
• display.js we will use it after in this series

Let’s dive into manifest:

{
  "name": "Gandalf",
  "version": "1.0",
  "description": "You shall not redirect",
  "manifest_version": 2,
  "content_security_policy": "script-src 'self' https://ajax.googleapis.com; object-src 'self'",
  "background": {
   "scripts": ["app/background.js"] <---- 1
  },
  "permissions": [ <---- 2
    "webRequest",
    "webRequestBlocking",
    "tabs",
    "<all_urls>"
  ],
  "browser_action": { <---- 3 
      "default_title": "You shall not pass",   
      "default_popup": "popup.html"
  },
  "icons": {
    "16": "logo-small.png",
    "48": "logo-small.png",
    "128": "logo-small.png"
  },
  "content_scripts": [
    {
      "matches": ["*://*.uy/*"],
      "js": ["app/display.js"] 
    },
    {
      "matches": ["<all_urls>"],
      "js": ["app/hidden.js"] <---- 4
    }
  ]
}

1. Link to the script that holds the extension logic, add listeners and define some constants.
   
2. The extension need some permissions, webRequest and webRequestBlocking for blocking a request (for the next chapter), tabs for creating a new window or tab. <all_urls> let us operate in all websites, you can change this if you only want specific sites.
   
3. Just a title for our extension and popup.html is what we render when someone click on the extension icon.
   
4. Content scripts can change the site content, this one apply to all urls and invoke hidden.js.

So we have defined that for all urls, hidden.js will be executed. This script can access the site body, check if paywall is present and send a message (internal messaging is the way our content scripts communicates with the extension).

var paywallId = ""; // We need an id to know if paywall is shown
window.onload = function() { // When site loads this will be called
  var content = document.getElementById(paywallId); // Find paywall
  if (content != null){
    // Send a message to the background with current url
    chrome.runtime.sendMessage({
      incognito: true, 
      url: window.location.href
      }, 
      function(response) {
        console.log("Opening page in incognito", response);
      }
    );
  }
}

For medium we can use the id paywall-background-color to check if we have to open in incognito; now its time to check the background script:

window.chrome.runtime.onMessage.addListener(
  function(request, sender, sendResponse) {
    // Check if the message is ours
    if (request.incognito == true){
      // Create incognito window with the requested url
      openIncognito = chrome.windows.create({"url": request.url, "incognito": request.incognito});
      sendResponse({msg: "Time to read"});
    }
  });

Now that we have our extension ready, we can load it to the browser and start reading non-stop.

Next in the series we will try to beat the redirection type of paywalls.

Resources

• Google’s extension develop guide
  
• Template app repo

[1] All the extension is doing is removing some friction from the user (that could always copy the link, enter in private mode and paste it), by no means we are hacking or altering medium behaviour.