This is part 2 of the two-part article on deploying ML models on mobile. We saw how to convert our ML models to TfLite format here. For those of you who came here first, I recommend you click on the above link to get the whole picture. If you just want the android part ,the demo app we are building has a GAN model generating handwritten digits and a classifier model predicting the generated digit.
Tl;dr : Find the code for the android app here.
Part 2: The Android Story
Now that we have the TfLite model ready along with information about inputs and outputs to the model, we need to add it to our Android Studio project.
Step 1 : Create a new Android Studio Project.
Step 2 : Go to the location
Project -> app -> src -> mainassetsStep 3 : Paste your .tflite file(s) in the assets directory. The structure will look something like this -
Let us proceed to adding dependencies in android studio.. Add the following in
buid.gradle(Module:app)android{
aaptOptions {
noCompress "tflite"
noCompress "lite"
}
}
dependencies {
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'
}To load the models in Android, we must import and use TfLite interpreter.
TfLite Interpreter is used for the following:
TfLite Interpreter is used for the following:
- Loading the model
- Transform data
- Running inference
- Interpreting output
The model can be loaded as follows:
Create an Interpreter object and use the function loadModelFile() to load the TfLite model as a MappedByteBuffer. Here, the
MODEL_FILEInterpreter tflite_gan;
String gan_model="gan_model.tflite";
try {
tflite_gan = new Interpreter(loadModelFile(context, gan_model));
}
catch (IOException e) {
e.printStackTrace();
}
}
private MappedByteBuffer loadModelFile(Context c, String MODEL_FILE) throws IOException {
AssetFileDescriptor fileDescriptor = c.getAssets().openFd(MODEL_FILE);
FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}For the GAN model, let us take the input and output ByteBuffers for easy pixel manipulation for image display.
First let us generate random latent points of a float array of [1x100] to provide as input ,as shown in the model details in part 1 of the article.
private float[][] generateLatentPoints(int n, int dim){
Random rn = new Random();
float[][] gan_input=new float[1][100];
for(int i =0; i<100; i ++){
float random_number=rn.nextFloat();
gan_input[0][i]=random_number;
}
return gan_input;
}Next, we will copy this data into a ByteBuffer and also allocate a ByteBuffer to store the output. As discussed, the size allocation is based on the model information. The last line in the code runs the ML model
float[][] input = generateLatentPoints(N,DIM);
ByteBuffer input_data = ByteBuffer.allocate( 100 * BYTE_SIZE_OF_FLOAT);
input_data.order(ByteOrder.nativeOrder());
ByteBuffer gan_output = ByteBuffer.allocate(1 *28 * 28 * 1 * BYTE_SIZE_OF_FLOAT);
gan_output.order(ByteOrder.nativeOrder());
input_data.clear();
gan_output.clear();
input_data.rewind();
for (int i=0;i<100;i++) {
input_data.putFloat(input[0][i]);
}
tflite_gan.run(input, gan_output);In the above code,
tflite_gan.run(input,output) gan_output.rewind();
Bitmap bitmap = Bitmap.createBitmap(IMAGE_WIDTH, IMAGE_HEIGHT, Bitmap.Config.RGB_565);
int [] pixels = new int[IMAGE_WIDTH * IMAGE_HEIGHT];
for (int i = 0; i < IMAGE_WIDTH*IMAGE_HEIGHT; i++) {
int a = 0xFF;
float r = gan_output.getFloat() * 255.0f;
pixels[i] = a << 24 | (((int) r)<<8);
}
//This is a colored image which you can display
bitmap.setPixels(pixels, 0, 28, 0, 0, IMAGE_WIDTH, IMAGE_HEIGHT);All you have to do is set this image to an ImageView or wherever you want to display it on your app.
Yay! you just deployed a GAN model on android.
Let us proceed to MNIST classifier model.
The process remains same for loading and running the TfLite model. Replace the MODEL_NAME with the classifier model name. The input to the classifier is the output from GAN, which is a 28x28 image. We have to process this image in Black and White which the classifier is trained to identify. This can be done by following -
//Create a Black & white image by taking HSV values from RGB Image
Bitmap bwBitmap = Bitmap.createBitmap( bitmap.getWidth(), bitmap.getHeight(), Bitmap.Config.RGB_565 );
float[] hsv = new float[ 3 ];
for( int col = 0; col < bitmap.getWidth(); col++ ) {
for( int row = 0; row < bitmap.getHeight(); row++ ) {
Color.colorToHSV( bitmap.getPixel( col, row ), hsv );
if( hsv[ 2 ] > 0.5f ) {
bwBitmap.setPixel( col, row, 0xff000000 );
} else {
bwBitmap.setPixel( col, row, 0xffffffff );
}
}
}
return bwBitmap;Again, we shall provide input as a BYteBuffer. This has to be processed. Let us do so by setting 0 for white and 255 for black pixels.
public ByteBuffer Process(Bitmap bitmap){
ByteBuffer inputBuffer = ByteBuffer.allocateDirect(BYTE_SIZE_OF_FLOAT * DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE);
inputBuffer.order(ByteOrder.nativeOrder());
int[] pixels = new int[28 * 28];
// Load bitmap pixels into the temporary pixels variable
bitmap.getPixels(pixels, 0, 28, 0, 0, 28, 28);
for (int i = 0; i < pixels.length; ++i) {
// Set 0 for white and 255 for black pixels
int pixel = pixels[i];
int channel = pixel & 0xff;
inputBuffer.putFloat(0xff - channel);
}
return inputBuffer;
}We have reached the last step of the process - getting the predicted digit from the classifier based on the most probable value. The output contains a 2D array with index against a probability.Iterating through it, we get the most probable value and that index will be our answer. Since we know that the output is a [1x10] float array, we will save it in the same form and iterate through it.
public float Classify(ByteBuffer input){
float[][] mnistOutput = new float[DIM_BATCH_SIZE][NUMBER_LENGTH];
tflite_mnist.run(input, mnistOutput);
float min_probability= Float.MIN_VALUE;
float digit = 0;
for (int i = 0; i < mnistOutput[0].length; i++) {
float value = mnistOutput[0][i];
if(value>min_probability){
digit=i;
}
}
return digit;
}That digit returned right there is your predicted output!
You can deploy as many models as you like in the same way. The key is to know and understand the input and outputs required by each of them and you are good to go. We can also get multiple outputs from the Interpreter using the following piece of code.
import org.tensorflow.lite.Interpreter;
Interpreter tflite;
tflite.runForMultipleInputsOutputs(inputs,outputs)Make sure the data structure used to save your outputs is able to accommodate multiple arrays. For instance, you can define the output as a HashMap of arrays as shown.
output1 = new float[1][28][28][1];
output2 = new float[1][14][14][34];
Map<Integer, Object> outputs = new HashMap<>();
outputs.put(0, out1);
outputs.put(1, out2); So finally the app looks something like this -
The code for the whole app is mentioned right the beginning. We have come to the end of the story.
For more resources on TfLite on android, visit the official quick-start site.
Explore some more example applications with slightly more complexities involving camera activities etc, here.
For exploring more on GANs, visit here.