Mock it until you have it

Mock the HTTP and HTTPS websites and REST APIs so that you don’t depend on these services or internet while working on any related development or automated testing.

When developing a new client facing application either desktop, web or mobile, on of the biggest challenge is availability of right web services of API which might be part of your own server or some other third party web services if you are integrating your application to third party system. This problem also arises if application and the web services which are required to be consumed are being developed in parallel due to unavoidable situations.

This problem is not just limited to development, its also a challenge in automated/manual testing. Any testing is only useful if its reliable. Having this in mind, it can be really problematic if required third party services/website changes, are down or some other issue or even if you are consuming your own web services, availability of right ad complete data every time during test itself can be big challenge.

And if sometimes your test failed due to data contract changes in any of these services, its really hard to find out the exact cause.

Fortunately, there are many tools available which can help you to mock the web services so that you no longer wait or get stuck due to unavailability or changes in the services.

Betamax (its officially dead)and Flashback are two of them to name. Betamax had few of its own challenges like it require internet connection always to work which could be a problem if you are doing something security critical and you are working in isolated system.

In this article, we will learn how to use Flashback to much the web services. Flashback is developed by LinkedIn which was open sourced in June 2017.

What is Flashback

Flashback s Java based cross platform tool written to mock HTTP and HTTPS services/ REST APIs. Basically it records the HTTP and HTTPS requests and stores locally and later it plays back the same request by matching the requests based on various request matching rules. Each such recording of request is called Scene.A match rule associates an incoming request with a previously-recorded request, which is then used to generate a response. HTTP requests generally contain a URL, method, headers, and body. Flashback allows match rules to be defined for any combination of these components.

You can read more about Flashback and how it works here: https://github.com/linkedin/flashback

Mocking a HTTP request

Mocking a HTTP request is very simple using Flashback. Flashback works exactly like Man in Middle proxy which intercepts and then records all the web traffic. Flashback has two parts of it:

Flashback Admin Server — Its centrally controls the Flashback proxies created for recording and replaying HTTP and HTTPS scenes.

Flashback Proxy server — It actually intercepts the HTTP/HTTPS traffic and record and play them back. Save these recording as scene and then plays them back based on various match rules and provided configuration.

Record HTTP Scene

Before we actually mock a service, lets us first setup Flashback for mocking HTTP services. First of all make sure that http server is enabled in your machine if you are using the MAC.

sudo apachectl start

1. Clone the Flashback repo

git clone https://github.com/linkedin/flashback.git

2. On terminal, change your current working directory to this clone repo directory and start Flashback admin server by running below script:

./startAdminServer.sh -port 1234

3. Once admin server started, you can interact with it using REST APIs it exposes. Next step is to start a Flashback HTTP proxy server to record the HTTP scene. Simply hit the below API either using Curl or any of your favourite REST client, I have used Postman for this demo:

POST http://localhost:1234/admin?action=startFlashback

Request Body:

{"sceneMode": "record","sceneName": "test1","matchRule": "matchMethodBodyUri","scenePath": "/tmp","proxyHost": "localhost","proxyPort": "5555"}

In body, we pass the various config parameter, you can refer here for more details of parameters.

4. Everything is done now, our proxy is running to record the traffic but before that you need to configure this HTTP proxy settings on the device or MAC where you have run this proxy and where you want to record the traffic (typically the same machine). Sample setting below on MAC.

Once configured save settings and restart you browser or any other client you are using.

If you are using any REST client, you can also choose to configure proxy setting in that REST client only rather than configuring in the system but make sure you have this proxy for all your request.

5. Simply hit any HTTP website or REST API and once you are done, stop the Flashback proxy server. Flashback saves the Scenes on disk only once its stopped.

POST http://localhost:1234/admin?action=shutDownFlashback

6. To verify if everything went well, jus go to your ‘scenePath’ location which you configure while starting proxy server and you should see a file with same name as ‘sceneName’, just open it and you will find all the HTTP websites/APIs you have visited.

Playback HTTP Scene

To replay a previously stored scene, use the same basic setup as is used when recording; the only difference is that you set the “sceneMode” to “playback” in step 3 above:

POST http://localhost:1234/admin?action=startFlashback

Request Body:

{"sceneMode": "playback","sceneName": "test1","matchRule": "matchMethodBodyUri","scenePath": "/tmp","proxyHost": "localhost","proxyPort": "5555"}

One way to verify that the response is from the scene, not the external source, is to disable your internet connectivity temporarily when you go through Steps 1 through 6. Another way is to modify your scene file and see if the response is the same as what you have in the file. Make sure you still have system proxy setting configured in you machine as mentioned in step 4 above.

There are various configuration parameter available for Flashback Admin server to start or stop HTTP to HTTPS proxy service which you would need as you go. Most of these parameters are not directly provided anywhere on git page of Flashback. Only way is to dig through he code and find out. Don’t worry you need not to do that I have found a few which you can use directly. Lets go through these parameters:

Mocking a HTTPS request

Mocking a HTTPs transaction is similar to HTTP transactions but it required few more configuration. HTTPS transactions are encrypted and Flashback can’t encode the transactions by default. In order to enable Flashback, we need to create a Certificate Authority (CA) certificate. This certificate will be used by Flashback to intercept the HTTPS traffic by violating man in middle attack.

There are two way you can obtain and provide certificate to Flashback.

• Purchase a Intermediate CA certificate from any of trusted CA.
• Create your own CA and certificate and add it to you machine’s trusted store. You need to add it to CA trusted store as its your own and will not be identified and will not trusted by operating system as its not issue by trusted CA. Hence all communication will be blocked. Once you add your certificate to system trusted store, its become trustable for the OS.

I will now be covering on how to created your own CA certificate, you can follow below guide blindly step by step to create your own certificate:

https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html

Once you obtain the certificate, its just matter for some configuration to mock HTTPS traffic. If you follow the steps in above like to create the certificate, you will get one certificate ‘ca.cert.pem’ and one private key ‘ca.key.pem’ also a password (say its ‘secretpassword’) for these certificate for certificate and private key. Keep them handy, we would need them soon.

There are two way you can provide the certificate to Flashback either directly input as .pem file or create the .p12 file.

To create the .p12 file, run the below command after obtaining both of above files:

openssl pkcs12 -export -out intermediateCA.p12 -inkey private/ca.key.pem -in certs/ca.cert.pem -name “secretpassword”

It will generate you intermediateCA.p12 certificate. Now we are ready with all the prerequisites.

Record and Replay HTTPS Scene

1. Simply hit the below API

POST http://localhost:1234/admin?action=startFlashback

Request Body:

{"sceneMode": "record","sceneName": "httpsTestScene","matchRule": "matchMethodBodyUri","scenePath": "/tmp","proxyHost": "localhost","proxyPort": "5555","caCertPath":"<Path to your intermediateCA.p12 file>/intermediate.p12","caCertPwd":"secretpassword", //or the password you have chosen"caAlias": "<CA alias name you provided when creating the CA certificate>","caKeyPwd": "secretpassword", //or the password you have chosen"caCertOU": "","caCertO": "","caCertCN": "","caCertL": "","caCertCC": ""}

Refer here for more details of parameters.

2. Make sure you system proxy is configured as below, you can remove the HTTP proxy if you want which we configured earlier.

If you are using any REST client, you can also choose to configure proxy setting in that REST client only rather than configuring in the system but make sure you have this proxy for all your request.

3. Hit any HTTPS websites or web services

4. Stop the Flashback Proxy server:

POST http://localhost:1234/admin?action=shutDownFlashback

5. Replay the HTTPS recorded Scene:

POST http://localhost:1234/admin?action=startFlashback

Request Body:

{"sceneMode": "playback","sceneName": "httpstest2","matchRule": "matchMethodBodyUri","scenePath": "/tmp","proxyHost": "localhost","proxyPort": "5555","caCertPath":"<Path to your intermediateCA.p12 file>/intermediate.p12","caCertPwd":"secretpassword","caAlias": "HTS","caKeyPwd": "secretpassword","caCertOU": "","caCertO": "","caCertCN": "","caCertL": "","caCertCC": ""}

6. Make sure you system proxy is configures as in Step 2. And hit the web service or REST api from your favourite REST client or browser. Congratulations you have done it!

Additional Configuration parameters for Flashback Admin Server

Here is the description and other possible value used for various Flashback REST APIs. You can choose them based on your need. ‘matchRule’ would be the possibly most used.

sceneMode-> Should be either record or playbacksceneName-> Name of the scene, you can pass any valuematchRule-> Rule to match a request URL while replaying the scene. Possible match rules are matchEntireRequest, matchMethodUri, matchMethodBodyUri, matchMethodUriBodyWithAnyBoundaryscenePath-> Where to store the recorded scene with 'sceneName'proxyHost-> HTTP or HTTPS proxy hostname to startproxyPort-> HTTP or HTTPS proxy port on which proxy should startcaCertPath-> CA certificate path, required only for HTTPS proxycaCertPwd-> CA certificate password, required only for HTTPS proxycaAlias-> CA certificate Alias name, required only for HTTPS proxycaKeyPwd-> CA private key password, required only for HTTPS proxycaCertOU-> optional, required only for HTTPS proxycaCertO-> optional, required only for HTTPS proxycaCertCN-> optional, required only for HTTPS proxycaCertL-> optional, required only for HTTPS proxycaCertCC-> optional, required only for HTTPS proxy

Disclaimer: Please note that this article may include few of the content from official Flashback git page to avoid rewriting.

If you have any question, please post below. Happing Mocking!