Your Kid Can Code a Fruit Detector with This MIT App Inventor AWS AI Services Extension

I am using Amazon Web Services (AWS) for more than 2 years. I have always been thinking about finding ways for enabling anyone to use AWS cloud services. Especially I am trying to encourage kids to learn simple AI services that AWS provides.

This easy to use AI services includes a chat-bot generation service called Amazon Lex, a text-to-speech service called Amazon Polly, an image and video recognition service called Amazon Rekognition and a language translation service called Amazon Translate.

Visual block based programming tools are an important tool for teaching coding to kids. One of the most popular block based programming environment is Scratch. Scratch is a joint project between Google and MIT Media Lab and based on a Google project called Blockly, declared in 2012.

In my previous post, I have blogged about my Scratch Extension that enables kids to use text-to-speech and translation services Amazon Polly and Amazon Translate easily.

MIT App Inventor is an another popular block based programming environment for beginners that is used for creating mobile applications for Android platform (iOS support is in the road map) using visual blocks. It is also created by Google, then developed and maintained by MIT. Many different projects are built over App Inventor like MIT spin-out Thunkable, AppyBuilder, Kodular, DroidMaker, Hybro Studio.

App Inventor provides an extension mechanism that allows developers to enable additional functionality for users like controlling robot kits, etc. You can find a rich collection of extensions here.

In this post, I will talk about my App Inventor AWS AI Services extension that enables easy usage of Amazon Polly, Amazon Translate and Amazon Rekognition. Also I will talk about a sample application that detects the fruit from a taken picture and says the name of the fruit after translating to an another language.

The demo of the sample app is shown below.

2. Design the user interface

By using the tool palette on the left, design the user interface as shown below. You can download the sample app here and upload it to Thunkable by clicking Apps menu and selecting ‘Upload app project (.aia) from my computer’.

3. Import the extension

Click Extension on the tool palette and select ‘Import extension’. Click to ‘URL’ button on the new window and paste the URL of extension. You can copy extension URL here. Then click ‘Import’.

Enter extension URL and click ‘Import’ to import the extension

After the extension is imported you should see the extension components in the tool palette under Extension like below.

Imported extension components

4. Add extension components

Before using the extension components, you should add them to the screen. This components won’t shown in the screen as other visible components but they should be listed below the phone screen and in the Components tree on the right side.

5. Design the behaviour with blocks

So far we have designed the user interface and added required extension components. Now we will create the code with blocks. Click ‘Blocks’ button below ‘Apps’ menu. By using the blocks for methods and events design the behavior as shown below.

Prepare the code with visual blocks

6. Prepare an AWS Access Key

This extension requires an AWS Access Key ID and AWS Secret Key to access AWS services. Create an IAM user and give Polly, Translate and Rekognition permissions. Then create an AWS Access Key ID and AWS Secret Key for that user. For more information see here. Please note that to try this extension easily you can give full access to these services but you should consider using AWS IAM Best Practices especially “Grant Least Privilege” practice.

The extension ask AWS credentials at first usage and save it for later usage. While live testing the app with Thunkable Classic Live for Android the credentials are saved into the Thunkable Classic Live for Android app. Any other Thunkable app you try can access this credentials, so be careful.

If you generate an Android app apk for the Thunkable app, the credentials are saved into the app you created. This is more secure compared to generic live app.

Please keep your AWS credentials private and be careful.

7. Test the app

App Inventor platforms have an option for quick testing the application before generating the Android app. This feature is called Live Testing. You can should install the Thunkable Classic Live for Android app to your device to live test your app.

After you installed the app on your device, click ‘Test’ on Thunkable and select ‘Thunkable Live’. Then on your Thunkable Live app click ‘Scan QR Code’ and scan the code generated. After the application is downloaded you should see the app in Thunkable Live app and you can test.

If you want to install the app to your device as an Android app, you can generate an APK to install your device. Click ‘Export’ on Thunkable and select the option you want.

Next Steps

The extension requires entering the AWS credentials to the app which is ok for simple tests. It would be great to not putting the credentials into the app. By using Amazon Cognito, the services can be called without sharing the credentials. If I have time, I would add Cognito integration.

A few months ago, Amazon announced streaming transcription for Amazon Transcribe service. This can be useful for real-time transcription and translation use cases.

Integrating Amazon Lex can be useful for using chat-bots in App Inventor apps.

Conclusion

In this post, I have shown how my App Inventor AWS AI Services extension can be used easily. I have shown the steps starting with an empty app and adding the extension.

You can find the extension here. Also you can download Fruit Detector sample app from here.

You can find more information about the blocks on AWS AI Services App Inventor Extension Wiki.

If you liked this post, you might find my other posts useful also.

AI Is Hard? It’s Child’s Play With This AWS AI Services Scratch Extension_In my previous post, I have mentioned that we should prepare our children for the future. In the future, to succeed our…_hackernoon.com

Speak, Translate and See with this AWS AI Services Chrome Extension_I have been learning AWS services for more than a year. On the one hand, I have been learning how to use them, on the…_hackernoon.com

You Can’t Fail To Learn A New Language With These AWS AI Services: Translate, Rekognition, Polly_With the improvement of AI, machines are doing more and more jobs that only humans did before. According to a report…_hackernoon.com

Serverless Allergy Checker with Amazon Rekognition, Lex, Polly, DynamoDB, S3 and Lambda_This post is part of a series of posts about AWS services in my previous blog._hackernoon.com

Please clap, share or follow.

Thanks for reading.