Massive Scale-Out of Deep Learning (DL) Models for Computer Vision to Android and iOS Devices using Flutter Framework
Jong Hang Siong
I founded OTONOCO in Singapore to design and build SaaS and Mobile Apps that incorporates Generative and Agentic AI to solve complex problems in the industry
What this Article is About
This article discusses scaling out deployment of deep learning for computer vision to billions of mobile devices without having to increase cloud infrastructure. This approach has the overriding advantages of:
A common approach to expose deep learning (DL) models to the outside world is to make them accessible to multiple applications via API.? This process is called model serving.? A Model Serving system usually consists of Endpoints / Access Points. An endpoint is responsible for receiving HTTP requests from the the outside world. These requests are then passed on to the model for inference. The resulted predictions are sent back to the requestors for further processing and delivery.
This article presents an alternative way of massively deploying DL models to mobile devices to perform inferencing in situ.
The Cost of Scaling Cloud Infrastructure
In standard endpoint inferencing, when the number of requestors increases, there is a need to scale the Cloud infrastructure vertically or horizontally. Scaling vertically involves increasing memory, storage and CPUs. Scaling horizontally involves increasing the number of containers, virtual machines, networking, SDatabases, etc.
The Economies of Scale of the New Approach
This article presents a technical tour on how to bring deep learning models for computer vision to billions of mobile devices running on iOS, Android, Linux, MacOS and Windows where deep learning models reside in those devices and inferencing is carried in situ. This tour covers mainly iOS and Android devices. This approach offers a MASSIVE ECONOMIES OF SCALE for deploying AI models:
The main development tool used is Flutter. Flutter is an open-source UI software development framework created by Google. It is used for building natively compiled applications for mobile, web, and desktop from a single codebase. Developers enjoy the benefits of writing Single Codebase once and deploy it on multiple platforms.
What You Need
This article assumes readers already have considerable amount of experience for the following:
This article intends to fill the knowledge gaps of encapsulating DL models in Flutter for iOS and Android devices by addressing the following needs:
The process starts with training of DL image classification models.? There are 2 ways of training DL models – Google Teachable Machine and Google MediaPipe.? Teachable Machine is a graphical user interface web based software and MediaPipe is a DL library used in conjunction with Google Colab.
Create Deep Learning Models for Image Classification
Image Preparation
We have prepared 1000+ images across 4 different medical image modalities – HISTOPATHOLOGY, XRAY, FUNDUS, MRI.
teachable_imagedir
?
Images were obtained from the following sources:
?Google Teachable Machine
Teachable Machine is a web-based tool created by Google that allows users to train machine learning models without the need to write code or machine learning skills. Users can train models to recognize and classify images, sounds, human poses through a web browser.
Go to Google Teachable Machine website and select “Get Started”.
Select Image Project à New Image Project (Standard Image Model).
The following user interface allows you to upload images. Edit the ‘Class 1’ field by entering the appropriate name of the image class.? Add more classes by clicking on ‘Add a class’ button.
Drag and drop the images into the correct class:
Upon completion of uploading images, a preview is visible:
Expand the Advance tab to tune your model according to your appetite. You can start with low Epochs and high learning rate.
After tuning, start the training process:
Evaluate Teachable Machine Model
Select some images and upload it to the trained model to be inferred to ensure the model is working properly.? Upload new images to the model in Teachable to determine if the model predicts the class accurately:
Select the TensorFlow Lite tab. Select Floating point and click on Download my model. Wait for it to be converted to TFLite format before downloaded into your computer.? Unzip the downloaded file to ensure there are 2 files in it – labels.txt and model_unquant.tflite.
Integrate Metadata (Labels) into Teachable TFLite Model
The Zip file produced by Teachable model contains 2 files – labels.txt and model_unquanted.tflite.? We will inject the label file into the model using Google Colab as shown below:
To do this, go to Google Colab (https://colab.research.google.com/). Create a new notebook.
In the notebook, start with installing tflite library:
The following uses tflite package to consume the model and label files, combine them into a new TFLite model.
Google MediaPipe in Colab
Google MediaPipe is the second way of creating image classification DL model in Colab.? Before building a new DL model, compress the image directories into a zip and upload to your Google drive.
Create a new notebook Colab, install mediapipe_model_maker package.? Mount the Google drive containing the zip file onto Colab:
领英推荐
! pip install mediapipe_model_maker
Unzip the mounted zip file and ensure all the subdirectories are present.
Use Matplotlib to visualize randomly sampled images:
Prepare Data Splits
ImageClassifier class in mediapipe package comes with useful tools to split data into training, validation and test sets for creating DL models and to evaluate their performances. The following code snippets shows 2 stages of splits. The first stage split the data into 80% training and and the remainder was subjected to splits of 70% to validation set and 30% to holdout set.
After splitting the data, build an image classifier model according to the following code snippet.? MediaPipe package comes with 4 DL backbones – 3 EfficientNets and 1 MobileNet. ?
Use the holdout set to evaluate the performance of the model by accuracy.? Use model.export_model() function to convert the model to TFLite format and download it for further use.
With the availability of models created in Teachable Machine and MediaPipe, we can now prepare to assemble an App using Flutter framework in Android Studio.
Assemble Flutter App
Android Studio
Create a new Flutter project in Android Studio.? Go to the Terminal at the bottom and run the following to install required Flutter libraries:
Open the pubspec.yaml file and ensure the following entries have been added. They were automatically added upon successful installation of these libraries:
Test iPhone Simulator
Open a new Terminal, run the following to launch an iPhone Simulator:
$ open -a Simulator
In Android Studio, check the Device drop down to ensure iPhone is available:
Apple Inc’s Privacy Policy on Photos
ImagePicker class is used in this app to acquire photos from Gallery as well as Camera.? Apple Inc has mandated all third party App developers to explicitly request permission from phone’s owner to use photos and camera.
Open Info.plist file and add the following XML tags:
Refer to Apple Photo Privacy and Developer Documentation for more information.
Add TFLite Asset
In the Android Studio project root, create a new directory assets and subdirectory ml:
Copy the TFLite models built earlier to assets/ml directory. Go to pubspec.yaml and create a new entry as follow:
Double-click on a TFLite model to see the model details for verification:
Configure Xcode Runner
To test your App using iPhone Simulator, you need to configure Xcode.? Right-click on the ios directory in Android Studio, go to the bottom-most option, select ‘Open iOS/MacOS Modules in Xcode’.? Select Runner and change the Minimum Deployment to 12.0.
Perform the same on RunnerTests and Pods.
Enter ‘architecture’ in the filter text field to search for Excluded Architectures section. Edit Debug, Profile and Release fields by entering arm64.
In Android Studio, expand android directory, open build.gradle file, change minSdkVersion to 21:
Dart Code
In main.dart file in Android Studio, create loadModel() function to store ImageLabeler class and instantiate it in initState() function:
The getModelPath() function contains utilities provided by path and path_provider libraries necessary to locate and read the TFLite DL models.
_selectPhotoFromGallery() function facilitate the acquisition of photos from the phone gallery or the camera for DL model to infer in order to determine if the image provided belongs to one of the HISTOPATHOLOGY, XRAY, MRI or FUNDUS type.
inferImageModality() function is responsible to perform the inference and generate the results containing the predicted class and percentage of confidence:
Further Reading