Running Selenium and Headless Chrome on AWS Lambda Layers

Selenium and Headless Chrome on AWS Lambda Layers

AWS has extended the timeout limit for Lambda functions from 5 to 15 minutes, also AWS released new Lambda layers feature at re:Invent 2018, with these new features, we can now move Selenium tests to server-less frameworks without any performance issues!

After many attempts of using different version of chrome drivers and binaries — I eventually find a way to get it work — ChromeDriver was able to run and interact with Headless Chrome inside a Lambda Layer.

I created Serverless Framework (≥1.34.0) project to publish and use Lambda Layers with Selenium and Headless Chrome, thus team is able to do UI test using Python without running Selenium on server or local machine.

Selenium and Headless Chrome

Incompatible versions of serverless-chrome, chromedriver, and Selenium can cause Chrome not reachable error. These are the versions that play well together in through my testing eventually:

File Structure

There are 2 sub sls projects in root directory, seleniumLayer is Selenium Lambda Layer which stores selenium libs, Headless Chromium driver and binary. lambda is normal lambda function to do the UI Testing

root
── /seleniumLayer/  # lambda layers
  ├── /selenium  lambda layer of selenium lib
  │  └──/python/      # python libs
  │   └── /lib/    
  │     └── /python3.6/*    
  ├── /chromedriver/    # lambda layer of headless Chrome 
  │ ├── /chromedriver   # chrome driver
  │ └── /headless-chromium # headless chrome binary
  └── /serverless.yaml     
── /lambda/            # lambda function
  ├── /handler.py      # source code of lambda function 
  └── /serverless.yaml # serverless config

Install Selenium library

Lambda runtimes include paths in the /opt directory to ensure that your function code has access to libraries that are included in layers.

To include libraries in a layer, place them in python/lib/python3.6/site-packages/

# download Selenium 2.37 to layer directory
$ pip3.6 install -t seleniumLayer/selenium/python/lib/python3.6/site-packages selenium==2.37

Install Headless Chrome Driver and Binary

Go to root directory, install chrome binary and driver

# download chrome driver
$ mkdir -p seleniumLayer/chromedriver
$ cd seleniumLayer/chromedriver
$ curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
$ unzip chromedriver.zip
$ rm chromedriver.zip

# download chrome binary
$ curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-41/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip
$ unzip headless-chromium.zip
$ rm headless-chromium.zip

Copy following code to /seleniumLayer/serverless.yaml

service: selenium-layer

provider:
  name: aws
  runtime: python3.6
  region: ap-southeast-2
  timeout: 900

layers:
  selenium:
    path: selenium
    CompatibleRuntimes: [
      "python3.6"
    ]
  chromedriver:
    path: chromedriver
    description: chrome driver layer
    CompatibleRuntimes: [
      "python3.6"
    ]
resources:
  Outputs:
    SeleniumLayerExport:
        Value:
          Ref: SeleniumLambdaLayer
        Export:
          Name: SeleniumLambdaLayer
    ChromedriverLayerExport:
       Value:
         Ref: ChromedriverLambdaLayer
       Export:
         Name: ChromedriverLambdaLayer

Lambda Function

and Copy following code to /lambda/handler.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def hello(event, context):
    options = Options()
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')

    driver = webdriver.Chrome('/opt/chromedriver',chrome_options=options)

    driver.get('https://www.neaminational.org.au/')
    body = f"Headless Chrome Initialized, Page title: {driver.title}"

    driver.close();
    driver.quit();

    response = {
        "statusCode": 200,
        "body": body
    }

    return response

Copy following code to /lambda/serverless.yaml

service: selenium-lambda

provider:
  name: aws
  runtime: python3.6
  region: ap-southeast-2
  timeout: 900


functions:
  hello:
    handler: handler.hello
    layers:
      - ${cf:selenium-layer-dev.SeleniumLayerExport}
      - ${cf:selenium-layer-dev.ChromedriverLayerExport}

Deploy Lambda Layers

Go to /seleniumLayer directory

 $ sls deploy

Deploy Lambda Function

Go to /lambda directory

$ sls deploy

Start Testing

Go to /lambda directory

$ sls invoke --function hello

You should get Response as below

{
"statusCode": 200,
"body": "Headless Chrome Initialized, Page title Home | Neami    National"
}

I hope you have found this article useful, You can find complete project in my GitHub repo:

yai333/Selenium-UI-testing-with-AWS-Lambda-Layers

Learn more

How To Add NodeJs Library Dependencies in a AWS Lambda Layer With Serverless Framework