PyCall: calling Python from Ruby in the same process, and the GVL problem that comes with it

Two runtimes in one process

PyCall provides the features to directly call and partially interoperate with Python from the Ruby language. You can import arbitrary Python modules into Ruby modules, and call Python functions with automatic type conversion from Ruby to Python.

The mechanism is an embedded interpreter, not a service boundary. The Python library is loaded into the Ruby process through the CPython C API, and the objects that come back are wrappers around Python object pointers. That is a materially different proposition from shelling out to a Python script and parsing its output, or from running a Python microservice and calling it over HTTP, because the call is in-process, the data does not have to be serialised, and the failure modes are the failure modes of a single process rather than of a distributed system.

The cost of that choice is that there are now two runtimes in one process, each with its own global lock, and the whole design of the gem is shaped by that fact. The rest of this article is largely about the consequences.

The supported versions are stated up front: Ruby 2.4 or higher, and Python 3.7 or higher. The licence is MIT, with the licence text in `LICENSE.txt` at the repository root, and the project lives with the other red-data-tools projects, which are the ones that give R packages access to Python libraries.

The syntax mapping, and the one place it gets in the way

The goal is that a Python call reads like Ruby, and the readme gives four mappings.

A Python constructor call, `classname(x, y, z)`, is mapped to `classname.new(x, y, z)`. A Python callable object, `obj(x, y, z)`, is mapped to `obj.(x, y, z)`, which is Ruby syntax for calling the object itself. Python keyword arguments, `func(x=1, y=2, z=3)`, are rewritten to `func(x: 1, y: 2, z: 3)`, using Ruby's keyword argument syntax. And the callable attribute of an object, `obj.meth(x, y, z=1)`, is written as `obj.meth(x, y, z: 1)`.

The first three are pure wins. The fourth has a documented cost. Because a callable attribute is mapped to the instance method of the corresponding wrapper object, you cannot get the callable attribute back. The readme says you have to write `PyCall.getattr(obj, :meth)` to get the `obj.meth` object, even though you can write `obj.meth` in Python.

That is a real asymmetry rather than a stylistic one. In Python, `obj.meth` is a bound method value you can pass around, store, or compare. In Ruby through PyCall, the method call is interpreted, and reaching the underlying object requires the explicit accessor. It is the kind of gap that only matters in code that treats callables as first class values, which is exactly the code a functional-style Python user writes.

The worked example in the readme is the shortest possible demonstration, calling Python's `math.sin` and comparing it with Ruby's own:

require 'pycall'
math = PyCall.import_module("math")
math.sin(math.pi / 4) - Math.sin(Math::PI / 4)   # => 0.0

The comment that type conversions from Ruby to Python are automatically performed for numeric, boolean, string, arrays and hashes is the other half of the story, and it is what makes the mapping above usable without hand written marshalling.

Two global locks in one process

The locking situation is the defining constraint of the gem, and the readme deals with it directly in two separate places.

First, the official position: PyCall does not support multi-threaded use officially. The reasoning is given. CPython's C-API has GIL acquiring and releasing functions such as `PyGILState_Ensure` and `PyGILState_Release`, and programmers can call CPython's C-APIs from outside of Python threads if they manage the GIL's state by these functions. But the maintainers do not want to officially support multi-threaded use of PyCall, because creating the feature that enables stable multi-threaded use in any situation is too difficult, and they want to avoid incurring the costs of supporting such use cases.

That is an unusually candid statement of what a project will not do, and it is the right one to plan around.

Second, the escape hatch for the other lock. Ruby has its own GVL, the RubyVM GVL, and while a Python call is in progress the Ruby side can be blocked too. PyCall provides the `PyCall.without_gvl` method for the case where you call a Python function that takes a long time to run. When PyCall performs a Python function call, it checks the current context, and releases the RubyVM GVL when the current context is inside a `without_gvl` block.

PyCall.without_gvl do
  # In this block, all Python function calls are performed without
  # the GVL acquisition.
  pyobj.long_running_function()
end

# Outside of PyCall.without_gvl block,
# all Python function calls are performed with the GVL acquisition.
pyobj.long_running_function()

So a long Python call inside `without_gvl` will not block other Ruby threads, but a long Python call outside it will. That is a sharp edge, and it is the kind of thing that shows up as a latency bug under load rather than at development time.

libpython, and the pyenv trap

PyCall needs Python's shared library, for example `libpython3.7m.so`, and the readme singles out pyenv users because of it.

The note is that pyenv does not build the shared library by default, so you have to specify the `--enable-shared` option at installation time:

$ env PYTHON_CONFIGURE_OPTS='--enable-shared' pyenv install 3.7.2

If you skip that, the gem cannot find libpython at load time, and the error you get is `PyCall::PythonNotFound`. The readme explains how to investigate, by setting the `PYCALL_DEBUG_FIND_LIBPYTHON` environment variable to 1, which prints the search it performs:

$ PYCALL_DEBUG_FIND_LIBPYTHON=1 ruby -rpycall -ePyCall.builtins
DEBUG(find_libpython) find_libpython(nil)
DEBUG(find_libpython) investigate_python_config("python3")

The debug output shows the candidate library names it tries, the paths it searches, and the dlopen result. That is a small thing, but it is the difference between a five minute fix and an afternoon, because the underlying problem, a Python built without a shared library, is not something the error message would tell you.

Choosing which interpreter to use is simpler. You can set the `PYTHON` environment variable to the path of the `python` executable, and when it is not specified, pycall.rb tries `python3` first and then `python`.

Use the wrapper gems for numpy, pandas and matplotlib

There is a section of the readme titled Special notes for specific libraries, and it is worth reading before you write any code.

For matplotlib, use red-data-tools/matplotlib.rb instead of just importing it by `PyCall.import_module("matplotlib")`. The same advice is given for numpy, where the recommendation is red-data-tools/numpy.rb, and for pandas, where it is red-data-tools/pandas.rb.

The reason is the same in all three cases, and it is the reason the project exists in the first place. Importing a raw scientific Python library into Ruby gives you the Python API, in which the awkward parts are the Python parts: a plotting call that opens a window and blocks, a NumPy array whose `shape` and `dtype` are Python objects, a DataFrame whose methods return Python objects rather than Ruby ones. The wrapper gems from red-data-tools are the same authors' answer to that problem, and they build on PyCall to give Ruby-flavoured interfaces for the three libraries that dominate data work in Python.

That framing also tells you what PyCall itself is and is not. It is the interop layer, the thing that removes the process boundary, not the ergonomic layer. If your Python dependency is one of the three above, you want the wrapper, and PyCall is underneath it. If it is something bespoke, you are on your own with the raw import.

The object system underneath is worth knowing about too. PyCall wraps pointers of Python objects in `PyCall::PyPtr` objects, and that class has two subclasses, `PyCall::PyTypePtr` specialised for type objects, and `PyCall::PyRubyPtr` for the objects that wrap pointers of Ruby objects, which is the mechanism for passing Ruby data back into Python.

Where PyCall is the wrong tool

Four limits are clear, and they come from the design rather than from a missing feature.

The first is threads, and it is the serious one. The project does not support multi-threaded use officially, and the reason is a cost decision rather than a technical impossibility. A service with worker threads calling into Python is outside what this gem supports, and a team that needs that has to design around it, for instance by keeping the Python calls on a single thread.

The second is the shared library requirement. A Python installation without libpython, which is the pyenv default, does not work, and the fix is to rebuild the interpreter with `--enable-shared`. That is a change to the machine, and in a locked-down or containerised environment it may not be available to you.

The third is the interpreter coupling. The gem links against a specific CPython C API, so the Python version your application can use is bounded by the version the gem was built for, and the readme's floor of 3.7 is a statement about the past as much as the present. In an environment with several interpreters, the `PYTHON` variable and the shared library path have to be coordinated.

The fourth is the callable attribute gap described earlier, which is small but bites exactly the code that treats callables as values.

There is also a question of what the alternative is, and it is worth naming plainly. If the Python code you need is a well behaved library with a stable API, running it as a subprocess or behind an HTTP boundary removes both the GVL problem and the type conversion problem, at the cost of serialisation and a process to supervise. PyCall is for the case where in-process is worth that trade.

On maintenance, the record is good. v1.5.1 was published in August 2023, v1.5.2 in May 2024, and v1.5.3 on 2026-08-21, with the last push to the repository on the same day.

Editorial conclusion

PyCall fits a Ruby team that needs to call a Python library in the same process, for instance to reuse a model or a data tool, and that wants the call site to look like ordinary Ruby rather than a subprocess or an HTTP client. It does not fit a multi threaded service, because the project states that it does not officially support multi threaded use, and it is the wrong choice if your environment cannot provide a shared libpython. Before you adopt it, check that your Python installation ships the shared library, especially under pyenv, where the documentation requires the `--enable-shared` configure option at install time.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. red-data-tools/pycall.rb on GitHub
5. Releases