is the most popular platform to organize community events. With more than 52 million members and over 300,000 Meetup groups, the site facilitates over 100,000 meetups per week around the world. Meetup The COVID-19 pandemic has shifted these in-person community gatherings to virtual events. With this shift, I have noticed that groups I personally attend have become less engaged over time. For example, In North America, the AWS meetup groups had at least one meetup in the past 12 months, while only 36% of groups have had one meetup in the past 3 months. This data indicates that "zoom fatigue" is growing as events have stayed virtual. As a freshly minted , I wanted to gather data on the best way to re-engage with the meetup groups. Before anything gets done at AWS, data is needed. To collect meetup data I developed a mechanism to capture relevant data and built a dashboard with to display insights. Some highlights include… Developer Advocate Streamlit The team now has visibility into which user groups has been active in the past 12, 6 or, 3 months Tracking active meetups: The team now has a map to see where our groups are located, and how big their footprint is. Visualizing where members are: : The team now has a table ordered by last event timestamp to see which groups recently had an event Keeping up with events In this post, I will describe how I built this solution and how I leveraged cloud resources. Overview of the solution While this solution could be deployed in a cloud provider of your choosing, I will focus on how I leveraged AWS. First, I started by developing python code to scrape meetup data. Second, I used Amazon Location Service to geocode the meetup cities into longitude and latitude coordinates. Afterward, the final data is stored in an Amazon S3 bucket. I used Amazon EventBridge to set up a daily job to trigger a lambda function to call my python code. Then, I used Streamlit to create a dashboard to display the data. Finally, the entire pipeline is deployed by using AWS The following diagram illustrates this architecture. Serverless Application Model (SAM) Collecting Meetup data I used and the requests library to scrape the content from the AWS User group website. The script first gets the meetup URL for each user group through the . Based on the presence of certain div attributes, it stores the relevant meetup URL and name in a list to be scrapped. Next, the function iterates through the list and parses the information on each individual meetup page, number of members, and meetup location. The raw data is saved as a CSV for further processing. BeautifulSoup get_user_group_data function get_meetup_info The following shows a sample of the script meetup_json = {} page = requests.get(meetup_url) usergroup_html = page.text soup = BeautifulSoup(usergroup_html, "html.parser") # Get Meetup Name meetup_name = soup.findAll("a", {"class": "groupHomeHeader-groupNameLink"})[0].text # Meetup location meetup_location = soup.findAll("a", {"class": "groupHomeHeaderInfo-cityLink"})[ 0 ].text # Number of members meetup_members = ( soup.findAll("a", {"class": "groupHomeHeaderInfo-memberLink"})[0] .text.split(" ")[0] .replace(",", "") ) # Past events past_events = ( soup.findAll("h3", {"class": "text--sectionTitle text--bold padding--bottom"})[ 0 ] .text.split("Past events ")[1] .replace("(", "") .replace(")", "") ) Geocoding user groups In order to plot each meetup group on a map, we need the longitude and latitude for each city in the meetup group. I was able to use Amazon Location Service to geocode each city name into longitude and latitude coordinates using a place index. For more information about creating a place index, see . Here is an example of a block of python code that uses a place index for geocoding. Amazon Location Service Developer Guide import boto3 def get_location_data(location: str): """ Purpose: get location data from name Args: location - name of location Returns: lat, lng - latitude and longitude of location """ client = boto3.client("location") response = client.search_place_index_for_text( IndexName="my_place_index", Text=location ) print(response) geo_data = response["Results"][0]["Place"]["Geometry"]["Point"] # Example output for Arlington, VA: 'Results': [{'Place': {'Country': 'USA', 'Geometry': {'Point': [-77.08628999999996, 38.89050000000003]}, 'Label': 'Arlington, VA, USA', 'Municipality': 'Arlington', 'Region': 'Virginia', 'SubRegion': 'Arlington County'}} lat = geo_data[1] lng = geo_data[0] print(f"{lat},{lng}") return lat, lng Using SAM to Orchestrate the Deployment After testing the script locally, the next step was to create a mechanism to run the script daily and store the results in S3. I used SAM to create a serverless application that does this. From a terminal window, initialize a new application sam init Change directory: cd ./sam-meetup Update dependencies update my_app/requirements.txt requests pandas bs4 Add in your code to the example my_app/app.py import json import logging import get_meetup_data def lambda_handler(event, context): logging.info("Getting meetup data") try: get_meetup_data.main() except Exception as error: logging.error(error) raise error return { "statusCode": 200, "body": json.dumps( { "message": "meetup data collected", } ), } Update template.yml Globals: Function: Timeout: 600 Resources: S3Bucket: Type: 'AWS::S3::Bucket' Properties: BucketName: MY_BUCKET_NAME GetMeetupDataFunction: Type: AWS::Serverless::Function Properties: CodeUri: my_app/ Handler: app.lambda_handler Policies: - S3WritePolicy: BucketName: MY_BUCKET_NAME Runtime: python3.9 Architectures: - x86_64 Events: GetMeetupData: Type: Schedule Properties: Schedule: 'rate(24 hours)' Name: MeetupData Description: getMeetupData Enabled: True Run sam build Deploy the application to AWS: sam deploy --guided For more detailed information on developing SAM applications visit Getting started with AWS SAM Visualizing data with Streamlit To gain insights from the meetup data, I chose to use Streamlit to host our dashboard. Streamlit is an open-source python module that allows you to build beautiful visualizations with ease. Streamlit works well on your laptop or in the cloud making it great for building sharable dashboards. The following are example screenshots from the dashboard. Here is a snippet of code used to render the map from Streamlit using streamlit-folium import streamlit as st from streamlit_folium import folium_static import folium def show_map(df): """ Purpose: Show map Args: df - raw data Returns: N/A """ # center of US m = folium.Map(location=[38.500000, -98.000000], zoom_start=4) for index, row in df.iterrows(): # add marker for meetup group meetup_name = row["meetup_name"] meetup_members = row["meetup_members"] lat = row["latitude"] lng = row["longitude"] is_meetup_active_6_months = row["is_meetup_active_6_months"] popup_html = f"<ul><li>Meetup Name:{meetup_name}</li><li>Meetup Members:{meetup_members} </li><li>is_meetup_active_6_months: {is_meetup_active_6_months} </li></ul>" curr_color = "" if is_meetup_active_6_months: curr_color = "blue" else: curr_color = "red" folium.Circle( [lat, lng], popup=popup_html, tooltip=meetup_name, radius=float(meetup_members) * 20, color=curr_color, fill=True, fill_color=curr_color, ).add_to(m) # call to render Folium map in Streamlit folium_static(m) Conclusion In this post, we discussed how I built a serverless data pipeline to analyze meetup data with Streamlit. I was able to successfully highlight the following: Geocode addresses using Amazon Location Service Use Amazon EventBridge and AWS Lambda to transform and load the data daily to S3 Visualize and share the data stored using Streamlit Automate and orchestrate the entire solution using SAM My team plans on using this solution to strategize on how best to engage our meetup community. If you are interested in participating in your local AWS community, check out the list of meetups located here: https://aws.amazon.com/developer/community/usergroups/ Disclaimer I currently work for AWS as a Senior Developer Advocate. The opinions expressed in this blog are mine.
