Turn any Website into a MicroService/Serverless Endpoint.

All via a single command via the Universal JavaScript Console -> :service

Serverless and Microservice-based applications are all the rage these days. Serverless is defined here:

Serverless Architectures_Serverless architectures replace a managed server with a collection of third party services and FaaS_martinfowler.com

and Microservices here:

Microservices_"Microservices" - yet another new term on the crowded streets of software architecture. Although our natural…_martinfowler.com

Basically, these technologies both refer to the ability to host and run code in third party hosting environments without having to worry about all the devops usually associated with maintaining your own servers. Two good examples of this are:

Amazon Lambda:

AWS Lambda - Serverless Compute_AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume …_aws.amazon.com

and Microsoft Azure Functions:

Azure Functions-Serverless Architecture | Microsoft Azure_React to events across your stack with Microsoft Azure Functions, a serverless architecture, event driven experience…_azure.microsoft.com

While these environments are both awesome and I highly recommend them for production work, they involve quite a bit of effort to get up and running. Other startups offer alternative means to do serverless computing at substantially lower “friction”. Some good examples in this space are:

now_realtime global deployments_zeit.co

Gomix_Combining automated deployment, instant hosting & collaborative editing, Gomix gets you straight to coding so you can…_gomix.com

Serverless - The Serverless Application Framework powered by AWS Lambda and API Gateway_Build web, mobile and IoT applications powered exclusively by AWS Lambda and API Gateway_serverless.com

Webtask_All you need is code! Run code with an HTTP call. No provisioning. No deployment._webtask.io

All these solutions offer considerable functionality with a minimum of setup and configuration.

There are times, however when you just want to use an existing website or webpage as the basis for some functionality. This is especially true when you are trying to scrape data from an existing website or leverage some cool JavaScript functionality that some other developer has already built and deployed. Other times, you have a brilliant idea for an app that you want to build and you just want to prototype it fast! Another scenario is that you want to host some private web content but you don’t have a web server or the means to make it available via the browser. Whatever the reason, you just want something fast, simple and available everywhere.

I have a solution that I built for myself and I am pretty sure other developers have a similar set of problems that this could be used for. It is based on the Universal JavaScript Console that I built and discussed in this article:

Universal JavaScript Console for Browser, Electron, NW.js, Node.js and Windows_I love JavaScript and I love the productivity and flexibility that comes from a good command line interface. If you…_hackernoon.com

Please take the time to go read this article because the functionality we are going to be using is dependent upon having this JavaScript Console available in your browser. It is as simple as creating a bookmarklet , which is available here:

Console_Edit description_s3.amazonaws.com

Just navigate to this Web Site and drag the bookmarklet to your bookmarks toolbar, as illustrated here:

Once you have the bookmarklet installed, you can navigate to any website and inject the JavaScript Console into that page.

NOTE: Using the bookmarklet is OPTIONAL and you can simple navigate to the Console URL and use it in “standalone mode”

In our first example, we are going to use the standalone mode to demonstrate the Serverless/MicroService functionality. So navigate to this URL:

Console_Edit description_s3.amazonaws.com

You should have a window that looks like this:

One of the commands is :service , which will create a microservice endpoint for this page. Execute the command in the console and you should get something like this:

In a couple of seconds, you should see your service created with a description of the various endpoints. Basically, this page (i.e. the JavaScript Console) is now available both as an HTTPS endpoint as well as a Microservice Endpoint that can be accessed via any computer or programming language.

Basically, your web page is now available as a web server!

To test the functionality, open another browser tab and navigate to the Service Endpoint shown in your console. In my case, this is

https://webpage.mybluemix.net/service/iyn3kmoh-27ekgyqpxih/

When you first navigate to this URL, you will get a message that looks like this:

Don’t worry, your service is not broken, you just need to provide a service command. When you execute the :service command, it creates a microservice that does a JavaScript eval on any command you specify on the URL. For example, if you navigate to

https://webpage.mybluemix.net/service/iyn3kmoh-27ekgyqpxih/location

you should see something like this:

Basically whatever command(or code string)you specify, the Console will try and eval it and send back a result. The result is text and is JSON serialized. Please note that not everything can be serialised to JSON. For example, if you try this,

https://webpage.mybluemix.net/service/iyn3kmoh-27ekgyqpxih/document

you will not get back the results you might expect since the document object cannot be JSON serialized.

So remember that the entire object model of the page is accessible (i.e. the window object) so you don’t want to publish your service URL on the public internet!

You can disable this immediately by typing the following code in the console:

client.rpc.unprovide(serviceId)

The default microservice endpoint is defined as:

client.rpc.provide(serviceId, (req, res) => {var x = '';try {x = eval(d);} catch (e) {x = 'Error: ' + e.message};res.send(x)});

You can provide your own functionality by overriding this by typing your own implementation in the console, like this:

client.rpc.unprovide(serviceId);

//echo the command backclient.rpc.provide(serviceId, (req, res) => {res.send(d)});

Here is another example that returns the current date:

client.rpc.provide(serviceId, (req, res) => {if(req=="date")res.send(Date());elseres.send(req +" is an invalid command");});

You can have as many endpoints as you like just by parsing whatever gets sent in the req variable. For example, you could just use a switch statement as follows:

client.rpc.provide(serviceId, (req, res) => {var result="";switch (req) {case "date":result=Date();break;case "hello":result="Hello World!";break;default:result=req +" is an invalid command";}res.send(result);});

The functionality for the service implementation is based on the awesome DeepStream server and associated JavaScript client library.

Javascript Client_The entry point API documentation for the deepstream.io js client_deepstream.io

In addition to the RPC functionality, the service implementation also creates a DeepStream Record, which can be accessed via its endpoints or the record variable in the console.

To set a value via its endpoint, call the URL provided when you invoked the :service command:

https://webpage.mybluemix.net/set/iyn3kmoh-27ekgyqpxih/name/mike

where name is the key and mike is the value

To retrieve the value, call this URL:

https://webpage.mybluemix.net/get/iyn3kmoh-27ekgyqpxih/name

To do this in the console, just use the record variable:

record.set("name","mike");

record.get("name");

Again, this functionality is provided by DeepStream so you can look at the documentation here:

Record_The API docs for deepstream records_deepstream.io

All of the functionality provided by DeepStream client library is available via the client variable in the console. This includes publish and subscribe capability, which we will discuss in another article.

NOTE: Records are persisted based on the service id (serviceId in the console) so they are available even when your console session has ended. If you return to the console on the same machine to the same URL, it should reuse the same service id (it stores it in localStorage).

Create a service based on an existing Web Page

In this example, we are going to inject the Console into an existing Web Page and use the service functionality to create an endpoint for that page. First of all, you need to have the Inject Console Bookmarklet as describe above or in this article

Universal JavaScript Console for Browser, Electron, NW.js, Node.js and Windows_I love JavaScript and I love the productivity and flexibility that comes from a good command line interface. If you…_hackernoon.com

Once you have done this, navigate to

Hacker News_Edit description_news.ycombinator.com

and inject the console via the Bookmarklet. Once the console has finished loaded, you can type the following in the console to ensure you are connected up to the Hacker News page:

page.location

Remember that page is a global reference to the window object on the host page.

Next, execute the service command:

:service

This will take a few seconds but you should get back the information about your newly created service endpoint:

To verify that your endpoint works and is hooked up to the host page, open a new browser tab and navigate to the URL specified in your console:

Now that your endpoint is working, you can scrape the page by doing something as simple as:

https://webpage.mybluemix.net/service/iyn392du-9c7cbhv5vdr/page.document.body.innerText

You should get something that looks like the following:

Of course, you can get get more sophisticated by overriding your endpoint implementation, as I demonstrated above. Also, remember that you can pull in any JavaScript library you wish by using the :load script command in the console. By example, here is the code to leverage the awesome AlaSql library to scrape the page and present the results as JSON:

:load alasql

hackernews=JSON.stringify(alasql("select href,innerText INTO JSON({headers:true}) from ? where className='storylink'",[[...page.document.all]]));

Then, you should be able to access the result simply via its URL

https://webpage.mybluemix.net/service/iyn392du-9c7cbhv5vdr/hackernews

And get this:

NOTE: You may have to refresh the URL a couple of times to get the result!

As you can see, this provides some very powerful, yet simple to access functionality that makes prototyping and hacking super productive and, dare I say, FUN!

Because we are using the DeepStream Server and client library to enable this functionality, this also means we can access our service functionality OUTSIDE the browser, including in NodeJS, Java and native mobile applications. This is beyond the scope of this article but I’ll have another followup article that talks about this. I’ll also have another article on the server implementations that I have running in IBM Bluemix to support this.

IBM Bluemix - Cloud infrastructure, platform services, Watson, & more PaaS solutions_Build and run your apps with the IBM Bluemix hybrid cloud development platform to access a catalog of infrastructure…_www.ibm.com

Also worth noting is that since our JavaScript Console is “Universal”, the service functionality comes for free in Electron, NWJS, NodeJS as well as native Windows applications. Please refer to my previous article on this.

Ultimately, this functionality does come with some cost as I have to provide the servers to deploy it. If you like this functionality or any of my articles, please consider a small donation to feed the hampsters :-)

Pay Micheal R Parsons using PayPal.Me_Go to paypal.me/michaelparsons and type in the amount. Since it's PayPal, it's easy and secure. Don't have a PayPal…_www.paypal.me

thanks and please recommend and share this and my other articles:

Mike Parsons - Medium_and optionally export it to the clipboard or Excel, JSON, HTML, Tab or Comma delimited file formats! In an earlier…_medium.com