@cloudflare/realtimekit-virtual-background

RealtimeKit Video Background Transformer

A package that adds video background & blur to participant video feeds in your RealtimeKit Meetings.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

• About the Project
  • Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Contributing
• License

About The Project

This package provides the foundation to add video background and blur to a participant's video feed in your RealtimeKit Meeting with your custom UI implementation.

If you are using the rtk-meeting component directly and prefer abstraction over this package, refer to UI Kit Addons.

Built With

• RealtimeKit
• Typescript

Getting Started

Installation

npm install @cloudflare/realtimekit-virtual-background

Run the example

1. Run npm install in this repository.
2. Replace YOUR_AUTH_TOKEN in index.html with a valid RealtimeKit participant token.
3. Run npm run dev and open http://localhost:3002.
4. Enable the camera from the setup screen to try the virtual background.

Usage

Disable the default per frame rendering of video middleware to improve speed and quality by letting this middleware control it on its own.

await meeting.self.setVideoMiddlewareGlobalConfig({
  disablePerFrameCanvasRendering: true
});

A videoBackgroundTransformer object can be created using the RealtimeKitVideoBackgroundTransformer.init({meeting: meeting}) method.

const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
  meeting,
});

Types of middlewares exposed by videoBackgroundTransformer:

1. createStaticBackgroundVideoMiddleware expects an imageUrl as a parameter and then creates the image the background for the current user.

meeting.self.addVideoMiddleware(
    await videoBackgroundTransformer.createStaticBackgroundVideoMiddleware(imageUrl)
);

2. createBackgroundBlurVideoMiddleware expects blurStrength (0-100) as a parameter (50% by default) and blurs the background of the user by the given blurStrength.

meeting.self.addVideoMiddleware(
  await videoBackgroundTransformer.createBackgroundBlurVideoMiddleware(50)
);

Note: Some browsers or their old versions might not have support for WebGL or the browser APIs that this package uses. We would recommend checking the support beforehand using:

if(RealtimeKitVideoBackgroundTransformer.isSupported()){
  const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
    meeting: meeting,
  });
  meeting.self.addVideoMiddleware(
    await videoBackgroundTransformer.createStaticBackgroundVideoMiddleware(`REPLACE_THIS_WITH_IMAGE_URL`)
  );
}

Note: Image URLs must allow CORS to avoid tainting the canvas. You can find such images on https://unsplash.com/ & https://imgur.com.

If in case you want to tweak the segmentation for better, sharper results, Please pass the desired segmentation config while initialising RealtimeKitVideoBackgroundTransformer.

const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
  meeting,
  segmentationConfig: {
    model: 'mlkit', // 'meet' | 'mlkit'
    backend: 'wasmSimd',
    inputResolution: '256x256', // '256x144' for meet
    pipeline: 'webgl2', // 'webgl2' | 'canvas2dCpu' // canvas2dCpu gives sharper blur, webgl2 is faster.
    targetFps: 35,
  }
});