A Slacker’s Guide to Programming
Build a food classifier app with Google Cloud Vision
Using machine learning to combat fake food and prevent stomach upset.
Introduction
What would you say if I told you that there is an app on the market...you know what, we're past that part!
In this article, we will build a food classifier app—more specifically, an app to identify Singapore local delicacies.
We Singaporeans are obsessed with food. Let's one-up this obsession with machine learning.
We will be using Google Cloud Vision API to analyse food images and React Native as the app framework.
Requirements
- Node.js and npm
- An iOS or Android Device
- A developer API Key from Google Cloud Platform
- One curry puff
Initial Setup
Refer to our previous article on how to generate a React Native boilerplate.
Refer to the Google Cloud Vision documentation on how to obtain an API key and to enable the Cloud Vision API.
So what is Google Cloud Vision API?
Google Cloud Vision API is an image analysis service on the Google Cloud Platform.
The algorithms behind the machine learning models and how the neural network functions are extremely complicated.
The good news is that Google Cloud Vision API enables developers to integrate vision detection capabilities for various applications in an easy-to-use REST API.
Feel free to try the API at Cloud Vision’s home page.
Say Cheese
We will be using Expo's Camera component for this article.
The application will function in the following manner.
- Obtaining camera access permission from the user.
- Capture the image using Expo's Camera component and convert it to a base64 string.
- Send a post request to the Google Cloud Vision API endpoint with the request data for image analysis.
- Display a Success or Fail UI depending on the image analysis results.
Obtaining camera access permission from the user
Let's do the necessary imports. Open App.js and paste the following code.
import React from 'react';
import {
Image,
Text,
View,
StyleSheet,
TouchableOpacity
} from 'react-native';
import {
Constants,
Camera,
Permissions,
} from 'expo';
import {Ionicons} from '@expo/vector-icons';Next, we set a null state for hasCameraPermissions and prompt the user for camera access on load.
...
export default class CameraExample extends React.Component {
state = {
hasCameraPermission: null,
type: Camera.Constants.Type.back,
};
async componentDidMount() {
const {status} = await Permissions.askAsync(Permissions.CAMERA);
this.setState({
hasCameraPermission: status === 'granted',
});
}
...To keep it simple, we will show a message if the user denies camera access for our application.
...
render() {
const {hasCameraPermission} = this.state;
if (hasCameraPermission === null) {
return <View/>;
} else if (hasCameraPermission === false) {
return <Text>No access to camera</Text>;
}
...Capture the image using Expo's Camera component and convert it to a base64 string
We will create a function and name it takePicture.
The takePicture function will capture an image, convert the image to a base64 string and pass the string value to another function call detectLabels.
The detectLabels function will take the string value and send a POST request to Google Cloud Vision API for image analysis.
Include this function in App.js.
...
takePicture = async => {
if (this.camera) {
this.camera.takePictureAsync({
base64: true,
quality: 0,
skipProcessing: true
}).then(image => {
//detectLabels Function
});
}
}
...Expo has a set of vector icons call Ionicons that are installed by default when we created our application.
We will use an icon(ios-radio-button-on) from that library as the camera button and pass the takePicture function to the onPress event handler.
...
render() {
...
<View>
<TouchableOpacity onPress={this.takePicture}>
<Ionicons name="ios-radio-button-on" size={70} color="white"/>
</TouchableOpacity>
</View>
...Send a post request to the Google Cloud Vision API endpoint with the request data for image analysis
Here's when things get interesting.
We use Google Cloud Vision API to detect labels for the captured image via the REST API.
For the request data, set type as LABEL_DETECTION and maxResults to 10.
const requestData = {
"requests": [
{
"image": {
"content": base64
},
"features": [
{
"type": "LABEL_DETECTION",
"maxResults": 10
}
]
}
]
}Once Cloud Vision has finished analysing the image, it will return the response as a JSON object with various entity annotations.
Very cool!
{
"responses": [
{
"labelAnnotations": [
{
"mid": "/m/02q08p0",
"description": "Dish",
"score": 0.9934035,
"topicality": 0.9934035,
"boundingPoly": {}
},
{
"mid": "/m/02wbm",
"description": "Food",
"score": 0.9903261,
"topicality": 0.9903261,
"boundingPoly": {}
},
{
"mid": "/m/01ykh",
"description": "Cuisine",
"score": 0.93498266,
"topicality": 0.93498266,
"boundingPoly": {}
},
{
"mid": "/m/0hz4q",
"description": "Breakfast",
"score": 0.62847126,
"topicality": 0.62847126,
"boundingPoly": {}
},
{
"mid": "/m/09jn47",
"description": "Curry puff",
"score": 0.6007812,
"topicality": 0.6007812,
"boundingPoly": {}
}
]
}
]
}We need a way to display the results back to our user. Let's build the UI next.
Display a Success or Fail UI depending on the image analysis results
Initialise an empty string for the description state.
We will use this.state.description to show the results of the analysis.
...
export default class CameraExample extends React.Component {
state = {
hasCameraPermission: null,
type: Camera.Constants.Type.back,
description: "",
};
...
render() {
...
<Text>{this.state.description}></Text>
...
}
...
Let's make the text more prominent.
Create a ClassifierText component for the description in App.js.
...
class ClassifierText extends React.Component {
render() {
const correct = <View>
<Text style={styles.correct}>{this.props.description}</Text><Ionicons name="ios-checkmark-circle" size={70} color="white" style={styles.resultsIcon}/></View>;
const wrong = <View>
<Text style={styles.wrong}>{this.props.description}</Text><Ionicons name="ios-close-circle" size={70} color="white" style={styles.resultsIcon}/></View>;
if (this.props.isMatched == null) {
return (<View></View>)
}
return (
<View style={styles.classifier}>
{this.props.isMatched ? correct : wrong}
</View>);
}
}
...
const styles = StyleSheet.create({
classifier: {
position: 'absolute',
top: 0,
left: 0,
right: 0
},
resultsIcon: {
position: 'absolute',
top: 105,
left: 0,
right: 0,
width: '100%',
textAlign: 'center'
},
correct: {
width: '100%',
height: 140,
paddingTop: 50,
color: 'white',
textAlign: 'center',
fontSize: 40,
fontWeight: 'bold',
backgroundColor: 'green'
},
wrong: {
width: '100%',
height: 140,
paddingTop: 50,
color: 'white',
textAlign: 'center',
fontSize: 40,
fontWeight: 'bold',
backgroundColor: 'red'
},
});
An app that no one use is as good as dead.
Include a call to action text for the button to provoke a response from the user.
render() {
...
<Text style={styles.intro}>Touch to SEEFOOD</Text>
...
}
...
Wow, this front-end design looks great. Nice Work!
Time to demo the app.
Let's start with a curry puff.
Let's try a packet of Singapore Economic Bee Hoon.
What about Durian? (It's not durian season yet so a picture will suffice.)
Conclusion
TL;DR: Watch the following video.
DAMMIT JIAN YANG!
Disclaimer: No curry puffs were sponsored for this demo. I paid for them out of my own pocket.
