If you have ever dealt with traditional Intrusion detection systems (IDS), you know the struggle: Heavy appliances, constant rule tuning, as well as the inability to stop false positives. And then the bills kept coming—it got really expensive even when traffic went down. I’ve experienced this before and it was frustrating. The good news? We don’t have to use IDS those ways anymore. By leveraging AWS’s serverless stack and a dash of machine learning, you can create a lightweight Intrusion detection systems that scale up when you need it, cost next to nothing when idle, and actually learn when to flag suspicious behavior. Does this sound too good to be true? Let’s breakdown how to approach this step-by-step. Why Go Serverless for Intrusion Detection? Why Go Serverless for Intrusion Detection? Before we give the high-level architecture, let’s answer a simple question: why bother going serverless at all? Elastic by design – No servers to patch, no EC2 instances to resize. If an event comes in, AWS Lambda just scales with traffic.
Pay-per-use – You only pay when an event is processed. Idle time costs basically-zero.
AI-powered – ML models can identify anomalies that static IDS signatures may miss.
Easy to integrate – AWS provides many sources of event data such as Cognito, IAM, and CloudTrail; or log data from an app you developed. Elastic by design – No servers to patch, no EC2 instances to resize. If an event comes in, AWS Lambda just scales with traffic. Elastic by design Pay-per-use – You only pay when an event is processed. Idle time costs basically-zero. Pay-per-use AI-powered – ML models can identify anomalies that static IDS signatures may miss. AI-powered Easy to integrate – AWS provides many sources of event data such as Cognito, IAM, and CloudTrail; or log data from an app you developed. Easy to integrate Think about this: Instead of maintaining a bulky IDS hardware appliance, you’re letting AWS take care of the scaling while you focus your efforts on developing, and refining, the intelligence. Architecture Overview Architecture Overview The following is the high-level flow of a serverless IDS (Intrusion Detection System). Event Source – Authentication logs (either from Cognito, IAM, or custom apps) are sent to EventBridge or dropped into an S3 bucket.
Lambda Function – The function picks up events, extracts the useful features (login time, geo, device fingerprint), and calls a SageMaker ML endpoint.
SageMaker Model – The model scores the event “legit” vs. “suspicious” based on the features that it sees.
DynamoDB Table – Holds the scored events with the risk detail.
Alerting Layer – If an event is deemed suspicious, it generates an SNS notification or an integration with AWS Security Hub. Event Source – Authentication logs (either from Cognito, IAM, or custom apps) are sent to EventBridge or dropped into an S3 bucket. Event Source Lambda Function – The function picks up events, extracts the useful features (login time, geo, device fingerprint), and calls a SageMaker ML endpoint. Lambda Function SageMaker Model – The model scores the event “legit” vs. “suspicious” based on the features that it sees. SageMaker Model DynamoDB Table – Holds the scored events with the risk detail. DynamoDB Table Alerting Layer – If an event is deemed suspicious, it generates an SNS notification or an integration with AWS Security Hub. Alerting Layer ASCII flow for the visual thinkers: Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) Auth Event → EventBridge → Lambda → SageMaker (score) → DynamoDB (log + risk) → SNS / Security Hub (alert) It looks simple but what's happening in these components is where the magic occurs. Developing the Anomaly Detection Model Developing the Anomaly Detection Model The core of this IDS is the ML model. A natural choice for a starting point—and of all models this is specifically built for anomaly detection—is Random Cut Forest (RCF). Random Cut Forest (RCF) For example, you may detect unusual login times (such as 3 AM logins from an unfamiliar device located within a foreign country)—RCF will perform well in this case. This is how you could setup your model in SageMaker: import boto3, sagemaker 
from sagemaker import RandomCutForest
session = sagemaker.Session()
role = ""
rcf = RandomCutForest(    
    role=role,    
    instance_count=1,    
    instance_type="ml.m5.large",    
    num_samples_per_tree=512,    
    num_trees=50
) import boto3, sagemaker 
from sagemaker import RandomCutForest
session = sagemaker.Session()
role = ""
rcf = RandomCutForest(    
    role=role,    
    instance_count=1,    
    instance_type="ml.m5.large",    
    num_samples_per_tree=512,    
    num_trees=50
) Training data should be in CSV format, and should include your features [hour, geo, user_hash, device_hash]. train_data = "s3://my-bucket/auth-train.csv"
rcf.fit(s3_input(train_data))
predictor = rcf.deploy(    
  initial_instance_count=1,    
  instance_type="ml.m5.large",    
  endpoint_name="ids-anomaly-detector"
) train_data = "s3://my-bucket/auth-train.csv"
rcf.fit(s3_input(train_data))
predictor = rcf.deploy(    
  initial_instance_count=1,    
  instance_type="ml.m5.large",    
  endpoint_name="ids-anomaly-detector"
) The model returns an anomaly score. The consequence is a higher score shows the login appears increasingly "weird". Handling Events Using Lambda Handling Events Using Lambda Think of the Lambda function acting as the connector for the whole process. It takes the event in its raw form; it will take the pieces of information associated with the event and send those to your ML model for scoring and then record the score to DynamoDB. And it also has the capability to send an alert, if the score is high enough. This is a simplified version of a handler that I would start with: import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { SageMakerRuntimeClient, InvokeEndpointCommand } from "@aws-sdk/client-sagemaker-runtime";
const ddb = new DynamoDBClient();
const sm = new SageMakerRuntimeClient();
const TABLE = process.env.TABLE;
const ENDPOINT = process.env.ENDPOINT;
export const handler = async (event) => {  
  // Depending on the source of the log, it might have come wrapped in `detail`  
  const record = event.detail || event;  
  // Create features for the model, in this case, hour of login, geo hash, and device hash  
  const features = [record.hour, record.geoHash, record.deviceHash];  
  // Get the score of the event with SageMaker   
  const resp = await sm.send(    
      new InvokeEndpointCommand({      
            EndpointName: ENDPOINT,      
            Body: JSON.stringify(features),      
            ContentType: "application/json",    
          })  
      );  
  const { score } = JSON.parse(new TextDecoder().decode(resp.Body));  
  const risk = score > 3 ? "suspicious" : "legit";  
  // Write the results to DynamoDB  
  await ddb.send(    
      new PutItemCommand({      
        TableName: TABLE,      
        Item: {        
            eventId: { S: record.eventId },        
            userId: { S: record.userId },        
            score: { N: score.toString() },        
            risk: { S: risk },        
            ts: { S: new Date().toISOString() },      
          },    
        })  
    );  
  // If something seems suspicious, log it (or send it to SNS / Security Hub)  
  if (risk === "suspicious") {    
    console.warn("⚠️ Suspicious activity detected:", record);  
  }  
  return { status: "ok", risk };
}; import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { SageMakerRuntimeClient, InvokeEndpointCommand } from "@aws-sdk/client-sagemaker-runtime";
const ddb = new DynamoDBClient();
const sm = new SageMakerRuntimeClient();
const TABLE = process.env.TABLE;
const ENDPOINT = process.env.ENDPOINT;
export const handler = async (event) => {  
  // Depending on the source of the log, it might have come wrapped in `detail`  
  const record = event.detail || event;  
  // Create features for the model, in this case, hour of login, geo hash, and device hash  
  const features = [record.hour, record.geoHash, record.deviceHash];  
  // Get the score of the event with SageMaker   
  const resp = await sm.send(    
      new InvokeEndpointCommand({      
            EndpointName: ENDPOINT,      
            Body: JSON.stringify(features),      
            ContentType: "application/json",    
          })  
      );  
  const { score } = JSON.parse(new TextDecoder().decode(resp.Body));  
  const risk = score > 3 ? "suspicious" : "legit";  
  // Write the results to DynamoDB  
  await ddb.send(    
      new PutItemCommand({      
        TableName: TABLE,      
        Item: {        
            eventId: { S: record.eventId },        
            userId: { S: record.userId },        
            score: { N: score.toString() },        
            risk: { S: risk },        
            ts: { S: new Date().toISOString() },      
          },    
        })  
    );  
  // If something seems suspicious, log it (or send it to SNS / Security Hub)  
  if (risk === "suspicious") {    
    console.warn("⚠️ Suspicious activity detected:", record);  
  }  
  return { status: "ok", risk };
}; Look at the steps—nice and clean right?extract → score → store → alert! extract → score → store → alert! DynamoDB Schema Design DynamoDB Schema Design You should not have to be fancy with your DynamoDB table. A minimal schema is all that is needed: PK: eventId
Attributes: userId, score, risk, timestamp
Optional: GSI on userId if you need to access login history quickly. PK: eventId PK Attributes: userId, score, risk, timestamp Attributes Optional: GSI on userId if you need to access login history quickly. Optional Sometimes simple is all you need. Lessons Learned (The Hard Way) Lessons Learned (The Hard Way) Cold Starts vs Latency Cold Starts vs Latency You will be just fine dealing with cold starts in lambda for most use cases.
If you need ultra-fast latency latency, consider autoscaling and/or deploy compiled models (with neo) inside of lambda. You will be just fine dealing with cold starts in lambda for most use cases. If you need ultra-fast latency latency, consider autoscaling and/or deploy compiled models (with neo) inside of lambda. Feature Engineering is King Feature Engineering is King Basic simple features like login hour, geo delta, or device fingerprinting will give incredible accuracy boosts.
Do NOT just throw the raw logs into the model—curate features as it relates to your use case. Basic simple features like login hour, geo delta, or device fingerprinting will give incredible accuracy boosts. Do NOT just throw the raw logs into the model—curate features as it relates to your use case. False Positives Hurt Trust False Positives Hurt Trust Set your threshold carefully.
Do NOT auto-block suspicious events. Report them to an analyst instead. Set your threshold carefully. Do NOT auto-block suspicious events. Report them to an analyst instead. Cost Efficiency Cost Efficiency DynamoDB on-demand + lambda is ridiculously cheap.
If you have sudden bursts of traffic, using brace complete inference with SageMaker async inference will save a ton of money. DynamoDB on-demand + lambda is ridiculously cheap. If you have sudden bursts of traffic, using brace complete inference with SageMaker async inference will save a ton of money. Security Hygiene Security Hygiene Lock down lambdas IAM role strictly (white list it). Similar to above; only allow it the capability to invoke SageMaker and write to DynamoDB. Lock down lambdas IAM role strictly (white list it). Similar to above; only allow it the capability to invoke SageMaker and write to DynamoDB. No wildcards in IAM policies, ever. Frequently Asked Questions Frequently Asked Questions Question: Can I implement this without using SageMaker?

Yes. If your model is not large, it can be packaged with Lambda. However, if your model is large, SageMaker provides the flexibility to train and auto-scale the model to match your traffic needs.


Question: If my application is not in AWS, can I still log the events to EventBridge?

Yes, you can send logs into EventBridge via APIs or dump logs into S3. Moreover, our pipeline does not have to be 100% AWS-native.


Question: What is the cost of this approach?


With moderate traffic:

Lambda—pennies per million invokes
DynamoDB—on-demand can be a few dollars
SageMaker endpoint—approximately $0.05/hr for very small instances

That is very affordable compared to traditional IDS solutions.



Question: Won’t anomaly detection create too many false positives for normal traffic?

It certainly can. This is the reason why thresholds and feedback loops are important. Add a regular analyst review of the alerts, and you can tune your model iteratively until appropriate thresholds are set. Question: Can I implement this without using SageMaker?

Yes. If your model is not large, it can be packaged with Lambda. However, if your model is large, SageMaker provides the flexibility to train and auto-scale the model to match your traffic needs. Question: Can I implement this without using SageMaker? Yes. If your model is not large, it can be packaged with Lambda. However, if your model is large, SageMaker provides the flexibility to train and auto-scale the model to match your traffic needs. Yes. If your model is not large, it can be packaged with Lambda. However, if your model is large, SageMaker provides the flexibility to train and auto-scale the model to match your traffic needs. Question: If my application is not in AWS, can I still log the events to EventBridge?

Yes, you can send logs into EventBridge via APIs or dump logs into S3. Moreover, our pipeline does not have to be 100% AWS-native. Question: If my application is not in AWS, can I still log the events to EventBridge? Yes, you can send logs into EventBridge via APIs or dump logs into S3. Moreover, our pipeline does not have to be 100% AWS-native. Yes, you can send logs into EventBridge via APIs or dump logs into S3. Moreover, our pipeline does not have to be 100% AWS-native. Question: What is the cost of this approach?


With moderate traffic:

Lambda—pennies per million invokes
DynamoDB—on-demand can be a few dollars
SageMaker endpoint—approximately $0.05/hr for very small instances

That is very affordable compared to traditional IDS solutions. Question: What is the cost of this approach? With moderate traffic:

Lambda—pennies per million invokes
DynamoDB—on-demand can be a few dollars
SageMaker endpoint—approximately $0.05/hr for very small instances

That is very affordable compared to traditional IDS solutions. With moderate traffic:

Lambda—pennies per million invokes
DynamoDB—on-demand can be a few dollars
SageMaker endpoint—approximately $0.05/hr for very small instances

That is very affordable compared to traditional IDS solutions. With moderate traffic: Lambda—pennies per million invokes
DynamoDB—on-demand can be a few dollars
SageMaker endpoint—approximately $0.05/hr for very small instances Lambda—pennies per million invokes DynamoDB—on-demand can be a few dollars SageMaker endpoint—approximately $0.05/hr for very small instances That is very affordable compared to traditional IDS solutions. Question: Won’t anomaly detection create too many false positives for normal traffic?

It certainly can. This is the reason why thresholds and feedback loops are important. Add a regular analyst review of the alerts, and you can tune your model iteratively until appropriate thresholds are set. Question: Won’t anomaly detection create too many false positives for normal traffic? It certainly can. This is the reason why thresholds and feedback loops are important. Add a regular analyst review of the alerts, and you can tune your model iteratively until appropriate thresholds are set. It certainly can. This is the reason why thresholds and feedback loops are important. Add a regular analyst review of the alerts, and you can tune your model iteratively until appropriate thresholds are set. Summary Summary So take-away summary: you can build a intrusion detection system with AWS Lambda, DynamoDB and SageMaker that is serverless, scalable and intelligent. No server to babysit.
Costs are tied to usage.
ML implementation of the models can learn and adapt to new attack patterns.
Alerts will be native to your entire AWS infrastructure security ecosystem. No server to babysit. Costs are tied to usage. ML implementation of the models can learn and adapt to new attack patterns. Alerts will be native to your entire AWS infrastructure security ecosystem. In all honesty, it feels good to depart from the old static IDS world, where we fought against false positives and spent our weekends applying patches to physical boxes. Now, question for you to consider: if you had this set up in place, how would you extend the functionality? Would you add a new capability to support blocking in the flows of your system, or monitor for a while before taking the active blocking step? That is the nice part about building it—when you build it, you choose the level of balance between a fully automated process and one that actively seeks a human to be appraised of what is happening.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Session

The 5-Second Data Delay That Nearly Broke Us (And How We Fixed It in Milliseconds)

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Building a Serverless Intrusion Detection System on AWS

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The 5-Second Data Delay That Nearly Broke Us (And How We Fixed It in Milliseconds)

The Noonification: Anomaly Detection with Privileged Information—Part 1 (12/13/2023)

The Noonification: Craig Wright is Satoshi Nakamoto (12/24/2023)

3 Types of Anomalies in Anomaly Detection

A Zero-Knowledge Anomaly Detection Approach for Robust Federated Learning

The 5-Second Data Delay That Nearly Broke Us (And How We Fixed It in Milliseconds)

The Noonification: Anomaly Detection with Privileged Information—Part 1 (12/13/2023)

The Noonification: Craig Wright is Satoshi Nakamoto (12/24/2023)

3 Types of Anomalies in Anomaly Detection

A Zero-Knowledge Anomaly Detection Approach for Robust Federated Learning

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps