PythonMonkey: SpiderMonkey inside the Python runtime

One process, two virtual machines

PythonMonkey is a Mozilla SpiderMonkey JavaScript engine embedded into the Python Runtime, using the Python engine to provide the JavaScript host environment. The homepage is pythonmonkey.io, the primary language is C++, and the licence in the project metadata is MIT, though the reported licence is one GitHub cannot classify.

The framing that matters is in the goals list. The same process runs both the JavaScript and the Python virtual machines, with no serialisation and no pipes. Everything else follows from that.

So there is no bridge process and no message passing. A JavaScript array and a Python list are the same object viewed from two languages, and the implementation is at the C API level in both directions: JavaScript Array and Object methods are implemented on Python List and Dictionaries using the cPython C API, and the inverse is done through the Mozilla SpiderMonkey JavaScript C++ API.

The project reached MVP as of September 2024 and is under maintenance by Distributive, and external contributions and feedback are welcome. The primary language being C++ rather than Python is the giveaway that this is a native extension with a real engine inside it, not a wrapper around a subprocess.

The shortest possible demonstration is three lines:

$ pip install pythonmonkey

from pythonmonkey import eval as js_eval

js_eval("console.log")('hello, world')

Evaluating a JavaScript expression in Python and calling the result, with no bridge in between.

How the data actually crosses the boundary

The Data Interchange section is the technical core, and the rules are specific enough to be worth reading as a contract.

Strings share immutable backing stores whenever possible, when the engine chooses UCS-2 or Latin-1 internal string representation, which keeps memory consumption under control and makes it possible to move very large strings between JavaScript and Python library code without memory-copy overhead. TypedArrays share mutable backing stores.

JS objects are represented by Python dicts through a Dict subclass for optimal compatibility, and similarly JS arrays and Python lists. JS Date objects are represented by Python datetime.datetime objects. Intrinsics, meaning boolean, number, null and undefined, are passed by value. JS functions are automatically wrapped so that they behave like Python functions, and the reverse also holds.

And then the part that makes this more than a type mapping. Python Lists are represented by JS true arrays and support all Array methods through a JS API Proxy, and similarly for Python Dicts and JS objects. That means a Python list handed into JavaScript is a real JavaScript array with push, map and filter working on it, not a wrapped object that emulates them.

The goal list states the intent: Python Lists and Dicts behave as JavaScript Arrays and Objects, and vice versa, fully adapting to the given context. The word adapting is the load-bearing part, because it is what distinguishes a proxy-backed bridge from a copy.

Two other properties are worth noting because they are unusual for this kind of project. The Python host environment supplies the event loop, including EventEmitter and setTimeout, and it supplies XMLHttpRequest. So the JavaScript side is not a bare engine with no host, it has a working asynchronous environment, and that is what makes running an npm package realistic rather than theoretical.

The CommonJS layer, and what the roadmap says is done

The roadmap is a long list of items marked done, and it is the clearest statement of what the project actually supports.

The coercion items are all done: JS intrinsics coerce to Python intrinsics, JS strings coerce to Python strings, JS objects coerce to Python dicts for own properties only, JS functions coerce to Python function wrappers, JS exceptions propagate to Python, Python strings and intrinsics coerce to JS strings and intrinsics, Python dicts coerce to JS objects, and Python functions coerce to JS function wrappers.

The module system items are the interesting ones. There is a NodeJS and NPM compatible CommonJS module system, a Python require function that returns a coerced dict of module exports, and a CommonJS module system .py loader that loads Python modules for use by JavaScript. So JavaScript can require a Python module and Python can require a JavaScript package, in both directions.

The typed array items are done in both directions too, Python lists coerce to JS arrays and JS arrays coerce to Python lists.

And the last entry is the one that shows what the project was built for. PythonMonkey can run the dcp-client npm package from Distributive, which is a real package from the maintaining organisation running on this runtime.

The eval function is also done: an eval() function in Python which accepts JS code and returns JS to Python coerced values.

The examples directory matches the two directions. There is use-python-module.py and use-python-module/ for calling Python from JavaScript, and use-require.py and use-require/ for the other way.

Building it, and the six build types

The build instructions are for people who want a local version, and the toolchain list is long because the project compiles SpiderMonkey from source.

You need bash, cmake, Doxygen in the 1.9 series and graphviz if you want the docs, llvm, rust, python3.8 or later with header files, SpiderMonkey latest from mozilla-central, npm, Poetry, and poetry-dynamic-versioning. The setup.sh script can install these automatically.

The build is a single command:

poetry install

which automatically compiles the project and installs it with its dependencies into the Poetry virtual environment, and setting the BUILD_DOCS environment variable builds the docs as well.

The build types are set with the BUILD_TYPE environment variable and are case-insensitive. Release strips symbols and uses maximum optimisations, and is the default. DRelease is the same except symbols are not stripped. Debug uses minimal optimisations. Sanitize is Debug with AddressSanitizer enabled. Profile is Debug with profiling enabled. And None does not compile at all, which is useful if you only want to build the docs.

That list is a good map of what the maintainers actually do, because the profiling and sanitiser builds only exist if someone uses them regularly.

Testing follows the same two-language split. You compile the project, install development dependencies with `poetry install --no-root --only=dev`, then run the Python tests with `poetry run pytest./tests/python` and the JavaScript tests with `poetry run bash./peter-jr./tests/js/`. The peter-jr script at the top level is the JavaScript test runner.

The Makefile at the top level is not part of the PythonMonkey build, and its header says it is a workflow helper for Wes, but it is a convenient summary, with a build target that calls build.py, a test target that runs both suites, a clean target and a debug target that prints the job and CPU counts.

Installing, and the detail about npm

Installing the released version is one command:

$ pip install pythonmonkey

And there is a nightly build from a separate index:

$ pip install --extra-index-url https://nightly.pythonmonkey.io/ --pre pythonmonkey

A local build is available in the Poetry virtualenv once compiled, and you run it with `poetry run python`, after which the module can be imported and used. The README gives the session transcript, and the shape of the API is visible in it: `pm.eval` takes a JavaScript arrow function as a string and returns a Python callable, so the result is called directly.

There is a detail that will save you an afternoon, and it is called out in a note above the library section. npm, meaning Node.js, is required during installation only, to populate the JavaScript dependencies. It is not a runtime requirement, which is not obvious from a project that ships a JavaScript engine.

The packaging in the project manifest shows how much is being shipped. On Linux and macOS the wheel includes pythonmonkey.so and the libmozjs shared libraries, and on Windows it includes pythonmonkey.pyd and the mozjs DLLs. There is also a py.typed marker file, added for pep561, which means the package ships type information and type checkers will use it.

The build script is Python, in build.py, and the version is derived from the git tag and substituted into CMakeLists.txt so the C++ and Python versions never disagree.

One small piece of housekeeping the README bothers to document, and it is a real trap. Installing pythonmonkey also installs the pminit package as a dependency, and pip uninstall does not remove dependencies, so a clean removal is `pip uninstall pythonmonkey pminit`.

Editorial conclusion

Adopt PythonMonkey if your Python service needs to call an existing JavaScript library in process, because sharing one runtime removes the pipe and the serialisation cost that a subprocess or a REST boundary would add. Do not adopt it for a script that only needs to evaluate a few expressions, since the wheel is a large native extension carrying the whole JavaScript engine. Before you install, know that it needs a C++ toolchain when building from source, that npm is required during installation only to populate the JavaScript dependencies, and that the latest release is v1.4.1, so read the changelog before pinning.

Frequently asked questions

Official sources

1. Distributive-Network/PythonMonkey on GitHub
2. Issues
3. Project website
4. README
5. Releases