Capacitor ML Kit Face Detection Plugin

Unofficial Capacitor plugin for ML Kit Face Detection.[^1]

Use Cases

The Face Detection plugin is typically used whenever an app needs to locate faces in an image, for example:

• Photo organization: Detect which photos contain faces and where they are located, for example to crop thumbnails around them.
• Selfie effects: Use the detected face landmarks and contours to position stickers or other virtual elements on faces.
• Smile detection: Use the classification results to determine the probability that a person is smiling or has their eyes open.
• Face tracking: Maintain a consistent ID for each face across consecutive frames using the tracking option.

Compatibility

Installation

You can use our AI-Assisted Setup to install the plugin. Add the Capawesome Skills to your AI tool using the following command:

npx skills add capawesome-team/skills --skill capacitor-plugins

Then use the following prompt:

Use the `capacitor-plugins` skill from `capawesome-team/skills` to install the `@capacitor-mlkit/face-detection` plugin in my project.

If you prefer Manual Setup, install the plugin by running the following commands and follow the platform-specific instructions below:

npm install @capacitor-mlkit/face-detection
npx cap sync

Attention: This plugin only supports CocoaPods for iOS dependency management. Swift Package Manager (SPM) is not supported for the ML Kit SDK, see this comment.

Android

Dependencies

You need to add the following meta data in the application tag in your AndroidManifest.xml:

<meta-data android:name="com.google.mlkit.vision.DEPENDENCIES" android:value="face"/>
<!-- To use multiple models: android:value="face,model2,model3" -->

Variables

If needed, you can define the following project variable in your app’s variables.gradle file to change the default version of the dependency:

• $mlkitFaceDetectionVersion version of com.google.mlkit:face-detection (default: 16.1.7)
• $playServicesMlkitFaceDetectionVersion version of com.google.android.gms:play-services-mlkit-face-detection (default: 17.1.0)

This can be useful if you encounter dependency conflicts with other plugins in your project.

iOS

Minimum Deployment Target

Make sure to set the deployment target in your ios/App/Podfile to at least 15.5:

platform :ios, '15.5'

Configuration

No configuration required for this plugin.

Demo

A working example can be found here: robingenz/capacitor-mlkit-plugin-demo

Usage

The following example shows how to detect faces in an image.

Detect faces in an image

Detect human faces in an image at a local path. You can configure the accuracy/speed trade-off and enable landmarks, contours, classification (smiling and eyes open probabilities), and face tracking. Only available on Android and iOS:

import {
  FaceDetection,
  PerformanceMode,
  LandmarkMode,
  ContourMode,
  ClassificationMode,
} from '@capacitor-mlkit/face-detection';

const processImage = async () => {
  const { faces } = await FaceDetection.processImage({
    path: 'path/to/image.jpg',
    performanceMode: PerformanceMode.Fast,
    landmarkMode: LandmarkMode.All,
    contourMode: ContourMode.All,
    classificationMode: ClassificationMode.All,
    minFaceSize: 0.1,
    enableTracking: false,
  });
  return faces;
};

API

• processImage(...)
• Interfaces
• Enums

processImage(...)

processImage(options: ProcessImageOptions) => Promise<ProcessImageResult>

Detects human faces from the supplied image.

Only available on Android and iOS.

Returns: Promise<ProcessImageResult>

Since: 5.1.0

Interfaces

ProcessImageResult

Face

Represents a face detected by FaceDetector.

Rect

Rect holds four integer coordinates for a rectangle.

FaceLandmark

Represent a face landmark. A landmark is a point on a detected face, such as an eye, nose, or mouth.

Point

Point holds two coordinates

FaceContour

Represent a face contour. A contour is a list of points on a detected face, such as the mouth.

ProcessImageOptions

Enums

LandmarkType

ContourType

PerformanceMode

LandmarkMode

ContourMode

ClassificationMode

FAQ

Which platforms are supported by this plugin?

The processImage(...) method is only available on Android and iOS. The Web platform is not supported by the underlying ML Kit Face Detection SDK.

Does the plugin recognize who is in the picture?

No, the plugin performs face detection, not face recognition. It detects the position of faces in an image along with attributes such as landmarks, contours, head rotation, and smiling probability, but it does not identify individuals. With the enableTracking option you can only maintain a consistent ID for the same face across consecutive frames.

What is the difference between landmarks and contours?

A landmark is a single point on a detected face, such as an eye, nose, or mouth. A contour is a list of points that outlines a facial feature, such as the face oval, eyebrows, eyes, or lips. Landmarks are enabled via the landmarkMode option and contours via the contourMode option. Note that contours are only returned for up to 5 faces.

How can I detect whether a person is smiling or has their eyes open?

Set the classificationMode option to All. The detected faces then contain the smilingProbability, leftEyeOpenProbability, and rightEyeOpenProbability properties, each a value between 0.0 and 1.0.

What is the difference between this plugin and the Face Mesh Detection plugin?

This plugin detects faces with bounding boxes, landmarks, contours, and classifications on Android and iOS. The ML Kit Face Mesh Detection plugin returns a dense mesh of 468 3D points and triangle information for detected faces, but is only available on Android.

Can I use this plugin with Ionic, React, Vue or Angular?

Yes, the plugin is framework-agnostic. It works in any Capacitor app regardless of the web framework, including Ionic with Angular, React, or Vue, as well as plain JavaScript projects.

Related Plugins

• ML Kit Face Mesh Detection: Detect face meshes with 468 3D points in images.
• ML Kit Selfie Segmentation: Segment selfies from the background with ML Kit Selfie Segmentation.
• ML Kit Subject Segmentation: Separate subjects from the background with ML Kit Subject Segmentation.
• ML Kit Pose Detection: Detect body poses alongside faces in the same scene.

Terms & Privacy

This plugin uses the Google ML Kit:

• Terms & Privacy
• Android Data Disclosure
• iOS Data Disclosure

Newsletter

Stay up to date with the latest news and updates about the Capawesome, Capacitor, and Ionic ecosystem by subscribing to our Capawesome Newsletter.

Changelog

See CHANGELOG.md.

License

See LICENSE.

[^1]: This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries.