tfjs-wechat: running TensorFlow.js inside a WeChat Mini Program

A plugin that hosts TensorFlow.js inside the Mini Program runtime

tfjs-wechat is the official repository for the TensorFlow.js WeChat Mini Program plugin. TensorFlow.js is the Google project for machine learning in JavaScript with hardware accelerated training and deployment, and this plugin wraps that library so that third party Mini Programs can call it. The mechanism is a WeChat plugin, which is a bundle of code that runs outside the Mini Program's own sandbox and is invoked through a plugin id. The repository is TypeScript, and the working directory holds a src directory for the plugin code, a demo directory with a mobilenet example and a posenet example, a doc directory, a karma configuration and a cloudbuild file. The npm package is published as tfjs-wechat at version 0.2.0, with a plugin version of 0.0.6 referenced by consumers in their app.json.

Adding the plugin, then declaring it in app.json

Using the plugin takes two steps on the WeChat side. First, in the Mini Program management console, under the settings for third party services and plugin management, you add the plugin by searching for it with the appid wx6afed118d9e81df9. The README notes that this plugin needs no application process, so once it is added it can be used directly. Second, the Mini Program declares the plugin in its app.json:

{
  ...
  "plugins": {
    "tfjsPlugin": {
      "version": "0.0.6",
      "provider": "wx6afed118d9e81df9"
    }
  }
  ...
}

The provider value is the appid of the plugin, and the version is the plugin version, which is a separate number from the npm package version of the plugin source.

The 2MB package limit is why you import tfjs in pieces

The README is explicit that the umbrella package is a poor fit for a Mini Program, because of the 2MB app size limit. TensorFlow.js v2 ships a combined package containing six separate npm packages: tfjs-core as the base package, tfjs-converter for GraphModel import and execution, tfjs-layers for creating, importing and running LayersModel, tfjs-backend-webgl, tfjs-backend-cpu, and tfjs-data for data pipelines. The recommendation is to load only the subpackages you need. For a Mini Program that only imports and runs a GraphModel, the minimum set is tfjs-core, tfjs-converter, tfjs-backend-webgl and tfjs-backend-cpu, and tfjs-layers is added only if you need to create, import or train a LayersModel. The README then shows what that looks like in a package.json:

{
  "name": "yourProject",
  "version": "0.0.1",
  "main": "dist/index.js",
  "license": "Apache-2.0",
  "dependencies": {
    "@tensorflow/tfjs-core": "3.5.0"，
    "@tensorflow/tfjs-converter": "3.5.0"，
    "@tensorflow/tfjs-backend-webgl": "3.5.0"
  }
}

The Mini Program npm tooling then compiles those packages into the Mini Program, and the README points at the WeChat developer tools documentation for that step, along with a requirement to use a developer tools build of at least version 1.02.1907022.

The fetch polyfill and the configPlugin call

Mini Programs have no fetch, so if you use tf.loadGraphModel or tf.loadLayersModel the fetch function has to be filled in. The README gives two ways. If you use npm you can load the fetch-wechat package, which appears in the dependency list as fetch-wechat at version 0.0.3, or you can copy the file directly into your JavaScript source directory from a CDN. Then the plugin is configured in app.js in the onLaunch function:

var fetchWechat = require('fetch-wechat');
var tf = require('@tensorflow/tfjs-core');
var webgl = require('@tensorflow/tfjs-backend-webgl');
var plugin = requirePlugin('tfjsPlugin');
//app.js
App({
  onLaunch: function () {
    plugin.configPlugin({
      // polyfill fetch function
      fetchFunc: fetchWechat.fetchFunc(),
      // inject tfjs runtime
      tf,
      // inject webgl backend
      webgl,
      // provide webgl canvas
      canvas: wx.createOffscreenCanvas()
    });
  }
});

The plugin is given three things: the polyfilled fetch function, the TensorFlow.js runtime, and a WebGL backend attached to an offscreen canvas. The WebGL backend needs a canvas because WeChat Mini Programs have no DOM, and an offscreen canvas is the closest equivalent.

Model caching, with a 10MB ceiling on both storage paths

Two caching paths are documented, and both are capped. The plugin exposes a localStorage handler through the localStorageIO member, which the app passes into globalData, and the model loader then tries the cached copy first and falls back to fetching from the network and saving on failure. The alternative is a file storage handler, exposed the same way as fileStorageIO, and the README notes that the Mini Program also limits local files to 10MB, and that saving the model as a binary file uses less space than the base64 string localStorage uses. The size limit is the constraint that decides which path to use, and for a model under 10MB either works. The mechanism is the same in both cases, load with the handler and fall back on exception, and the difference is only where the bytes land.

Model URLs need a mirror reachable from China

The last substantive section of the README is about where the model files come from, and it is a practical trap rather than a technical one. The tfjs-models library provides trained models for image recognition, speech recognition, human pose estimation, object detection and text classification, and by default those model files are stored on Google Cloud, which means users in China cannot read them directly. The fix is to pass a modelUrl parameter when calling the model APIs, pointing at a Google mirror hosted in China. The base url for Google Cloud is https://storage.googleapis.com and the China mirror base url is https://www.gstaticcnapps.cn, and the url paths are otherwise identical, so a model path such as the posenet mobilenet float 050 model-stride16 json file resolves to the same path on the mirror. The demo directory in the repository contains a mobilenet example and a posenet example, which are the two cases the README illustrates.

Two documented problems: the WASM backend and the canvas lifetime

The README does not hide its rough edges, and there are two. The first is the WebAssembly backend. The README explains the appeal, that WeChat Mini Programs support WebAssembly on Android phones, that the TensorFlow.js WASM backend suits mid-range Android devices whose GPU is weaker than their CPU, and that a WASM backend generally uses less energy. The example shows setting the wasm path and switching the backend on Android. The warning underneath is unambiguous: the WASM backend is broken because of a bundle incompatible with the WeChat npm loader, with a note that it will be updated when it is fixed. The second problem is the offscreen canvas. Because the newest WeChat OffscreenCanvas becomes invalid when the page navigates, configuring tfjs in the onLaunch function of app.js causes the Mini Program to exit or operations to fail after a page jump, and the recommendation is to call configPlugin in the onLoad of the page that uses TensorFlow.js instead, with a note that the December WeChat version fixes it. The example shows the onLoad form with a per-page backend name.

Editorial conclusion

tfjs-wechat fits a team shipping on-device inference inside a WeChat Mini Program, who has accepted the 2MB package budget and the need to route model downloads through a mirror reachable from China. It does not fit a team that needs a stable, supported plugin, since the WebAssembly backend is documented as broken and the offscreen canvas lifetime has a known workaround. Before building on it, read the two notes at the end of the README, check the WeChat developer tools version requirement of 1.02.1907022 or later, and decide whether the plugin version 0.0.6 and the package version 0.2.0 in the manifest are both current enough for your release cycle.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. tensorflow/tfjs-wechat on GitHub