Serverless Allergy Checker with Amazon Rekognition, Lex, Polly, DynamoDB, S3 and Lambda

This post is part of a series of posts about AWS services in my previous blog . We are living in an age where software and and big companies . And improvements in AI field is unprecedented. mobile is eating the world turning to AI-first In this fast changing world, our responsibility is to provide solutions that use technology for helping people. is a very good example of . The Pollexy Project using the technology to help people After reading that use case, I have been thinking about finding a good use case for , an image recognition service using deep learning technology. Amazon Rekognition lets users to detect objects and scenes in pictures. It can also be used to index faces in images and search faces. Amazon Rekognition Our twins like playing in grass, but they have grass allergy and we can’t spend too much time playing in grass due to their allergy. While I was doing research on allergies, I have learned that in some extreme cases, . That struck me lot. drug allergies could cause anaphylactic reactions And I have decided to use Amazon Rekognition for saving and retrieving pictures of the patients that have drug allergy. This system can be used in emergency services to easily and quickly recording and checking patient allergy information. Also I have decided to use Amazon Lex and Polly for allowing emergency service personnel to use the application hands free by their voice. In this way, wearable devices like Google Glass can be used to let personnel use their hands freely and use their voice to record and search patient information. The Application The application is used to save patient information and to check information of a patient. The application is a that is made of static web files has no server-side component. Static files of the web application is hosted on S3 and the other functionality is implemented using the managed services of . AWS services is used directly from the browser using AWS JavaScript SDK. serverless application AWS The services used are below: is used to store the pictures of patients and static files of the web application S3 is used to store patient information including name, allergies and picture url DynamoDB is used to index and search the face of patients Rekognition is used to index patient face when a patient record is added to Lambda DynamoDB is used to allow users to add and search patients only by their voice Lex is used to give voice responses to the user Polly The diagram below shows the steps of the save patient use case that will be used when a new patient information will be recorded. Once Lex bot determined the patient name and allergen, the picture of the patient is taken in the browser and sent to S3. Then the patient record is inserted into DynamoDB. A DynamoDB trigger fetchs the picture and calls Rekognition to index face of the patient. Saving patient allergy information use case When we want to check whether a patient has allergy or not, we can take the picture of the patient in the browser and search the face of the patient in the Rekognition face collection. The steps of the check use case are shown below. Checking patient allergy information use case The HTML5 user interface and corresponding JavaScript code for recording audio to be sent to Lex, playing response from Lex and chat interface are used from and can be found at . That application was using Java Spring server-side to process commands and calls AWS services using Java SDK, but in this application all AWS services is called browser using JavaScript SDK. my previous post my GitHub repo Preparation You should have an AWS account and to create the required AWS resources. install AWS CLI Keep in mind that not all services are supported in all regions. I will use us-east-1, US East (N. Virginia) region because Amazon Lex is supported only in this region for now. The code can be found . here For simplicity, AWS access key id and secret key used to access AWS services using JavaScript SDK is put in HTML file. In production you should definitely . Note: keep your AWS credentials private Steps 1. Get the code 2. Create and configure IAM policy, role and user 3. Create Rekognition collection 4. Create and configure DynamoDB table 5. Create Lambda DynamoDB trigger 6. Create and configure the S3 bucket 7. Create the Lex bot 8. Modify and review the application 9. Test the application Let’s start. 1. Get the code Clone from GitHub repo . here 2. Create and configure IAM policy, role and user To access AWS services, we will create a role that has specific permissions for accesing AWS services to provide a minimum permission set according to least privilege principle. Please do not use your AWS root account or another account that have wider permissions. The AWS CLI commands are below. First we create a user, then create a policy and attach that policy to the new user. Please replace with your AWS account id. AWS_ACCOUNT_ID After the access key is created, output will be like below. because this credentials will be used by JavaScript SDK to call AWS services in the browser. We will put these credentials to HTML file. Note the access key id and secret key 3. Create Rekognition collection Rekognition uses face collections to store face information that are extracted from images and use this collection to search faces also. We can create the collection with the command below. 4. Create and configure DynamoDB table We will store patient information in the table with a partition key named like the picture below. Other fields are , , and . Patient patientId name allergen imageUrl s3File Patient table We can create the table and enable streaming on it with the commands below. Patient Note the in output, we will use that stream as event source for our Lambda trigger. It will be like . LatestStreamArn arn:aws:dynamodb:us-east-1:AWS_ACCOUNT_ID:table/Patient/stream/2017–10–09T19:41:06.419 5. Create Lambda DynamoDB trigger After we created the Patient table, we will associate a Lambda function that will be executed when a record is added to the table. First we create a Lambda execution role, then attach the policy we have created earlier to this role. This gives the required permissions to the Lambda function to call other AWS services, in this case method of . indexFaces Rekognition Please replace with your AWS account id and with the stream arn created in the previous step. Also set the value of to the S3 bucket name you want to use. AWS_ACCOUNT_ID PATIENT_TABLE_STREAM_ARN S3_BUCKET_NAME Lambda function runtime is and the code is in file. The code is shown below. The code creates a Rekognition object when loading and use this object to index faces when a patient is added to the the table. Patient picture url is put in field. Node.js PatientFaceIndexerLambda.js Patient s3File The code calls method with the S3 bucket name of the patient picture, our face collection id and a as . When we search a face with an image, Rekognition will return the matching face with this and we could find the other information with this . indexFaces patientId ExternalImageId ExternalImageId patientId 6. Create and configure the S3 bucket We will create a S3 bucket to store pictures of the patients. Normally, we don’t need to store patient images to be able to search, but to show the original image when a matching face is found we will store the original pictures. We should so we can upload from different domains. For simplicity I have enabled all domains, but you should limit the domains that access to the bucket in production usage. CORS settings are in file, please set them accordingly. Also replace with name you want to use. enable CORS s3-bucket-cors.json bucketName The application mark pictures as after uploading to S3 to enable them to be shown in browsers. In production, please set bucket pemissions accordingly as the are related to . public-read most recent data leakages misconfigured S3 buckets 7. Create the Lex bot Now all resources are ready except the Lex bot. Actually, to use the Rekognition, S3 and DynamoDB we don’t need to create a bot, but the Lex bot will enable users to use the system with their voice easily while using their hands for other emergency operations. Create a Lex bot named with two intents, and . For more information on creating a Lex bot, you can see or AWS Lex tutorial . AllergyChecker AddPatient CheckPatient my post on Lex Exercise 2: Create a Custom Amazon Lex Bot intent will be used for adding a patient to the system with a patient name and allergen. This intent will use and slots. When Lex has elicited the slots, the web application will take the picture of the patient and save these information to table. AddPatient PatientName Allergen Patient AddPatient intent The slot type of slot is . I have used built-in Amazon.Person but I have experienced problems and created a custom slot type. The custom slot behaved better. Also I had to use English names because currently Lex only supports English. PatientName AllergicPersonName AllergicPersonName custom slot type Also slot uses a custom slot named . Allergen Allergen Allergen custom slot type intent will be used for checking patient by taking a picture. This intent has no slots. When this intent is ready for fulfillment, the web application will take the picture of the patient and will call to find the patient with a matching face. If the patient is found the allergen info will be shown with the original patient picture. CheckPatient Rekognition CheckPatient intent 8. Modify and review the application Now all AWS resources are ready. We are ready to modify our application. Replace the configuration variables shown below in index.html. Adding a patient to face collection is implemented in the Lambda method we have created early. Rekognition PatientFaceIndexer Checking a patient using is implemented in method in . method is called with face collection id, image and a face match threshold. We want to return the faces with a . If a match is found, method shows the patient name, allergen and patient picture. Rekognition checkPatient index.html searchFacesByImage Rekognition match confidence greater than 80% handleCheckPatientResponse 9. Test the application Open in your browser. You should use a WebRTC compliant browser to be able to record audio and video. You can check your browser . index.html here Once the application is loaded, you can say ‘ ’ or ‘ ’ to use the application as shown in the video below. add check Next Steps First, I want to remind you about deleting the resources you have created for this post. Please be very strict about the security of resources you have created. I have fulfilled the intents in the web app. In real applications, Lambda functions can be used for slot validation and fulfillment. Also for simplicity, audio is recorded for 4 seconds and if all the recorded data is silence, the audio is not sent to Lex. In real applications, we can create a to analyze the recorded audio data in real-time and stop the recording if the silence is detected for a specific duration. For more information, see . **ScriptProcessorNode** Web Audio API docs For simplicity, I have not used a user authentication and authorization mechanism, but in production usage you should definitely use one like to provide security. This way you can keep your AWS access key and secret key private and not put your credentials in JS or HTML files. Amazon Cognito For now, Lex supports only English speech commands. I hope it will support other languages and regions. Summary In this post, I have developed an allergy checker application that use to save and find patient allergy information with their face. Also I have shown the usage of the and to accept voice commands and give response with natural speech. Amazon Rekognition Amazon Lex Amazon Polly I have used to call AWS services. AWS JavaScript SDK The code can be found . here For more information on AWS services, you can see my posts: Uploading Images to Amazon S3 Directly from the Browser Using S3 Direct Uploads Implementing Object Persistence for Our Digital Cards with Amazon DynamoDB Serverless User Activation Mechanism with Amazon Lambda, Amazon Step Functions and DynamoDB Triggers Nobody Understands You? There’s An App For That, with Amazon Lex Look Who’s Talking!. Creating a Web Application That Talks with Amazon Polly Storing User Sessions in Redis Using Amazon ElastiCache I would like to hear your comments about the application and different use cases for . Rekognition If you liked this post, please share, like or follow.