In this article, I’ll show you how to create with Python3.7 + Serverless lambda Assets Monitor My name is Uria Franko and I’m a free-lancer developer Here is the plan Setup Serverless framework Get AWS credentials Write backend of the with python3.7 Assets Monitor Deploy backend to Lambda Schedule running Before we start In order to go through this tutorial, make sure you have: Installed Python 3.7 Installed Node.js v6.5.0 or later AWS account with admin rights (You can create one for free ) right here Step 1: Serverless framework Let’s start from the beginning. In order to easily write and deploy our lambda we will use this awesome framework called . It’s written on NodeJS so we need npm to install it. Let’s go: Serverless npm -g serverless install After this, let’s create a new project from template: serverless create --template aws-python3 --path my-assets-monitor It will create a new folder with two files: my-assets-monitor — a template for your Python code handler.py — configuration file serverless.yml Step 2: AWS Credentials This is the best part of this process, because once you’ve got credentials, you’ll never deal with AWS again. It’s super easy: Log in to your AWS Console, go to and click on “Add User” blue button. My Security Credentials > User Specify username (something like “serverless-admin”) and choose only “Programmatic access” checkbox. On the second page, choose “Attach existing policies directly” and look for “Administrator Access”. Once you created the user, copy your “Access key ID” and “Secret access key”. This is what you actually need to continue. (Tip: save them where you can find them ;) ) Congrats. You’ve got your keys. Open your terminal in and execute: serverless config credentials --provider aws --key xxxxxxxxxxxxxx --secret xxxxxxxxxxxxxx Step 3: Write Assets Monitor Code I’m not going to teach you how to write in Python, so just copy this code and paste to your `handler.py` file: json os sys concurrent.futures ThreadPoolExecutor urllib.parse urlparse

here = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(here, )) requests bs4 BeautifulSoup mailer Mailer

mailer = Mailer(os.environ[ ], os.environ[ ], os.environ[ ],)
internal_urls = set()
external_urls = set() ThreadPoolExecutor(workers) ex:
        res = ex.map(func, args) list(res) parsed = urlparse(url) bool(parsed.netloc) bool(parsed.scheme) mailer
    resp = requests.get(url) resp.status_code > :
        mailer.assets.append(url) :
        response = requests.get(url, timeout= )
        response.raise_for_status()
        soup = BeautifulSoup(response.content, ) requests.exceptions.ConnectTimeout err:
        errors = [ ]
        mailer.send_errors(errors) requests.exceptions.ConnectionError err:
        errors = [err]
        mailer.send_errors(errors) requests.exceptions.HTTPError err:
        errors = [ ]
        mailer.send_errors(errors) soup urls = set()
    url_parsed = urlparse(url)
    domain_name = url_parsed.netloc
    soup = request_url(url) soup : a_tag soup.findAll([ , , , ]):
        source = a_tag.name == a_tag.name == :
            source = href = a_tag.attrs.get(source) href == href href: parsed_href = urlparse(href) parsed_href.netloc == : href[ ] == :
                href = url_parsed.scheme + + domain_name + href :
                href = url_parsed.scheme + + domain_name + + href is_valid(href): href internal_urls: domain_name href: href external_urls:
                external_urls.add(href) urls.add(href)
        internal_urls.add(href) urls crawled_links = get_all_website_links(os.environ[ ]) crawled_links :
        response = { : , : } response
    multi_threading(check_status, crawled_links, )
    mailer.send_mail()
    response = { : ,
    } response import import import from import from import "./vendored" import from import from import 'TARGET_URL' 'SOURCE_EMAIL' 'DESTINATION_URL' : def multi_threading (func, args, workers) with as return : def is_valid (url) return and : def check_status (url) global if 399 : def request_url (url) try 5 "html.parser" except as 'Connection timed out to your target' return False except as return False except as f'Your target raised <strong> </strong> status code' {response.status_code} return False return : def get_all_website_links (url) if is False return False for in "a" "link" "img" "script" 'src' if "a" or "link" 'href' if "" or is None or '#' in continue if "" if 0 "/" "://" else "://" "/" if not continue if in continue if not in if not in continue return : def main (event, context) 'TARGET_URL' if is False "statusCode" 500 "body" "Error raised trying to get the target" return 20 "statusCode" 200 return Now create new file in the same directory and call it mailer.py, now copy this code and paste it in: boto3 os aws_access_key_id = os.environ[ ]
        aws_secret_access_key = os.environ[ ]
        self.client = boto3.client( , aws_access_key_id = aws_access_key_id,
                                   aws_secret_access_key = aws_secret_access_key,
                                   region_name = )

        self.target_email = source_email target_email :
            self.target_email = target_email
        self.base_url = base_url
        self.source_email = source_email
        self.assets = [] subject = body = link self.assets:
            body += self.send(subject, body) subject = body = err errors:
            body += self.send(subject, body) self.client.send_email(
            Source = self.source_email,
            Destination = { : [
                    self.target_email,
                ],
            },
            Message = { : { : subject, : }, : { : { : body, : }
                }
            },
            ReplyToAddresses = [
                self.source_email,
            ],
        ) import import : class Mailer : def __init__ (self, base_url, source_email, target_email = None) 'AWS_KEY' 'AWS_SECRET' 'ses' 'us-east-1' if is not None : def send_mail (self) "Assets Monitor Asset Failure" f"""
        <h2>There's a problem with one of your assets!</h2>
        <h4>Base URL: <a href= > </a></h4>
        """ {self.base_url} {self.base_url} for in f'<a href=" "><p> </p></a><br>' {link} {link} : def send_errors (self, errors) "Assets Monitor Website Failure" f"""
        <h3>There's a problem with your monitored website:</h3>
        <h4> </h4>
        """ {self.base_url} for in f"<p> </p><br>" {err} : def send (self, subject, body) 'ToAddresses' 'Subject' 'Data' 'Charset' 'UTF-8' 'Body' 'Html' 'Data' 'Charset' 'UTF-8' If you are lazy enough, just fork or clone it from my GitHub . repo Also, you will need to create requirements.txt file and write: requests
bs4 and execute this command to install it locally: pip install -r requirements -t vendored .txt Step 4: Deploy to AWS Lambda Pretty much like on Heroku, all you need is one configuration file “serverless.yml”. Go ahead and edit yours to make it look like this: service: my-asset-monitor

provider:
  name: aws
  runtime: python3 stage: dev
  region: us-east environment:
    AWS_KEY: AWS_SECRET: TARGET_URL: SOURCE_EMAIL: DESTINATION_EMAIL: functions:
  post:
    handler: handler.main
    events:
      - schedule:
          name: asset-checker-schedule
          description: rate: rate( minutes) .7 -1 "" "" "" "" "" 'Schedule asset checking every 30 minutes' 30 Notice that you need to change AWS_KEY -> Your AWS key that we created AWS_SECRET -> Your AWS key that we created TARGET_URL -> The website that you want to monitor (Ex: 'https://google.com') SOURCE_EMAIL -> Your verified email that you can get/verify here DESTINATION_URL -> The email that the notifications will be sent to You can also change the rate(30 minutes) to what ever you want... you can find the rates syntax here And the magic happens once you execute this in your terminal: serverless deploy It will pack all your files into .zip archive and upload to AWS, then it will create AWS CloudEvent to schedule and run the monitor automatically The serverless backend of your monitor is now live and running. Now if any asset or link in your website will be broken (Not found/ Server error / etc) you will get an email that's looks like that: Also if there's any error accessing your target you will get an email like that: Keep in mind At the beginning it will cost you nothing, because The AWS Lambda free usage tier includes 400,000 GB-seconds of compute time per month. But if you have any other lambda function or you multiply the monitors it can get to a point where they charge you... Our current usage is 60/30 * 60 * 24 * 30 = 86,400 calls each calls is between 1-5 seconds depends on the assets amount and a usage of 256MB So 86,400 * 3 (Average)  / 4 (GB-sec / 256MB = 3.9xxx) = 66,355 seconds of usage per month Serverless approach is very convenient in some cases because it allows you to build very scalable solution. It also change your thinking paradigm about managing and paying for servers. Serverless framework gives you very simple tool to deploy your function with no needing to know AWS or any other cloud provider. AWS gives you nice free tier period for services so you can build your MVPs totally for free, go live and start paying only if you reach a certain amount of users.

Google

Heroku

Super

Target

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Assets Monitor as a Function

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Uria Franko

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Assets Monitor as a Function

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Uria Franko

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES