Amazon Kinesis is the real-time stream processing service of AWS. Whether you got video, audio, or IoT streaming data to handle, Kinesis is the way to go. Kinesis is a serverless managed service that . Often, you will use it when SQS or SNS is too low-level. integrates nicely with other services like Lambda or S3 But as with all the other services on AWS, is a professional tool that comes with its share of complications. This article will discuss the most common issues and explain how to fix them. So, let's get going! Kinesis 1. What Limits Apply when AWS Lambda is Subscribed to a Kinesis Stream? If your Kinesis stream only has one shard, the Lambda function won't be called in parallel even if multiple records are waiting in the stream. To scale up to numerous parallel invocations, you . need to add more shards to a Kinesis Stream Kinesis will strictly serialize all your invocations per shard. This is a nice feature for controlling your parallel Lambda invocations. But it can slow down overall processing if the function takes too long to execute. If you aren't relying on previous events, you can use more shards, and Lambda will automatically scale up to more concurrent invocations. But keep in mind that Lambda itself has a soft limit on 1,000 concurrent invocations. You can reach out to AWS to get this limit lifted. There isn't an explicitly defined hard limit above that, but AWS mentions its multiples of 10,000. 2. Data Loss with Kinesis Streams and Lambda If you call put_record in a loop to publish records from a Lambda function to a Kinesis stream, this can fail mid-loop. To fix this, make sure you ; otherwise, your function will crash and only partially publish the list of records. catch any errors the put_record method throws If one Lambda invocation is responsible for publishing multiple records to a Kinesis stream, you have to make sure a crash of the Lambda function doesn't lose data. Depending on your use case, this could mean you need to use retries or another queue in front of your Lambda function. You can also try to catch any errors instead of crashing and then put the missing records somewhere else to ensure they don't get lost. 3. InvokeAccessDenied Error When Pushing Records from Firehose to Lambda You're trying to push a record from Kinesis Firehose to a Lambda function but get an error. This is usually a permission issue with IAM roles. To fix this, make sure to . assign your firehose the correct IAM role In the Resource section of your policy document, you need to make sure all your Lambda functions' ARNs are listed. You achieve this with either a wildcard in the ARN or an array of ARNs. But there can be many other permission problems that prevent invocation. Some of them are: Missing the "Action": ["lambda:InvokeFunction"] Having an "Effect": "Deny" somewhere Assigning the wrong role to the firehose 4. Error When Trying to Update the Shard Count You tried to update the shard count too often in a given period. The UpdateShardCount method has rather tight limits. To get around this issue, you can , with more generous quotas. call other functions like and SplitShard MergeShards Often, you don't know how many shards are sufficient to handle your load, so you have to . AWS limits how you meddle with the shard count. To quote the docs here, you can't: update their numbers over time Scale more than ten times per rolling 24-hour period per stream Scale up to more than double your current shard count for a stream Scale down below half your current shard count for a stream Scale up to more than 10000 shards in a stream Scale a stream with more than 10000 shards down unless the result is less than 10000 shards Scale up to more than the shard limit for your account If you use other methods, you can get around some of the limitations, which give you more flexibility around sharding. 5. Shard is Not Closed You interacted too soon after you created a new stream. Creating a new stream can take up to 10 minutes to complete. You can or ensure that you retry a few times to fix this. set timeouts after creating a stream Creating new streams or shards isn't an instant action. It happens very quickly, but you might have to wait for minutes in the worst case. As with any distributed system, you have to keep latencies in mind. Otherwise, your logs will be littered with errors. Summary If you have to , it's best to go for . process your data or media in real-time Kinesis on AWS Sadly, it's not as straightforward as SQS and SNS, but it's also more flexible than those services. Your best course of action is to learn about the limitations of the service so you aren't littered with avoidable error messages. Also, make sure to so they don't crash with half your data not processed yet. program your Lambda functions robustly Monitoring Kinesis with Dashbird will . Additionally, Dashbird will evaluate all your Kinesis logs according to . So, it's not just metrics and errors en masse, but to improve your architecture with AWS best practices. Dashbird monitor all your Kinesis streams out of the box the Well-Architected Framework actionable information

Amazon

Getting Down and Dirty with Metric-based Alerting for AWS Lambda

5 Common Amazon SQS Issues

Serverless af

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Serverless

Nominated for 2022 - HackerNoon Contributor of the Year - Aws

Too Long; Didn't Read

Make resilience your competitive advantage

5 Common Amazon Kinesis Issues

5 Common Amazon Kinesis Issues

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10 Things in Engineering We Don't Spend Enough Time On

Amazon Kinesis: The AWS Data Streaming Solution

AWS Decoupling: The Big Comparison

AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them

Choosing Between Enterprise Messaging and Event Streaming For Your Architecture

10 Things in Engineering We Don't Spend Enough Time On

Amazon Kinesis: The AWS Data Streaming Solution

AWS Decoupling: The Big Comparison

AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them

Choosing Between Enterprise Messaging and Event Streaming For Your Architecture

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps