TensorWatch: Querying a Live Training Loop from a Jupyter Notebook

The problem TensorWatch targets: metrics you did not know you needed

Most training instrumentation forces a decision before the run starts. You pick the loss, the accuracy, the learning rate, and those are the numbers you get. If the run looks wrong three hours in, the value you actually want (a weight histogram, an intermediate activation, the shape of a batch) was never captured, and your options are to restart with more logging or to accept the blind spot. TensorWatch approaches this from the opposite direction. The README describes the traditional model as what-you-see-is-what-you-log, and positions the project's lazy logging mode as the alternative: execute an arbitrary query against the live training process, get a stream back, and attach whichever visualizer you want to that stream. The audience is the person already inside a notebook, iterating on a model, who wants to inspect a running process rather than a finished one. It is a debugging tool, and the README states that plainly in its own security notice.

Watcher, streams and clients: the mechanism the README exposes

The architecture visible in the documentation has three parts. A Watcher owns a log file and hands out streams. In the quick start, tw.Watcher(filename='test.log') creates the watcher, w.create_stream(name='metric1') creates a named stream, and every s.write((i, i*i)) appends a tuple to it. The second part is the transport. Messages between a WatcherClient and a Watcher travel over ZeroMQ, are serialized with Python's pickle module, and are HMAC-SHA256 signed. The third part is the query path. In lazy logging mode, create_stream(expr=...) sends a Python expression from the client to the server, and the server evaluates it with Python's eval() inside evaler.py. That is the whole trick: the expression is evaluated in the training process, so it can reach live objects that a normal logger would have to serialize ahead of time. The result comes back as a stream, and the visualizer is a separate choice. The README also notes a bundled hiddenlayer utility set for network diagrams, which is where the yaml.SafeLoader default comes from.

Getting a first run working: pip, a log file, and a generated notebook

Installation is a single command, pip install tensorwatch. The README states Python 3.x support and says the project is tested with PyTorch 0.4 through 1.x, with most features expected to work on TensorFlow eager tensors. Network diagrams depend on graphviz, and the README warns that on some platforms you may need to install it manually from the graphviz download page rather than relying on a package manager. The quick start is short enough to reproduce: create a Watcher with a filename, create a stream by name, call w.make_notebook(), then write tuples in a loop. Running that script produces a test.ipynb file next to your script; you start jupyter notebook, open the file, and use Cell > Run all to get a live line graph as values are written. Two things are worth noticing in that flow. The notebook is generated from the script rather than written by hand, and the viewer is a separate process from the writer, which is exactly why the security notice spends so much space on the ZMQ port and the HMAC key.

Lazy logging is eval() over a socket, and the README says so

This is the part that determines whether TensorWatch fits your environment. The README's security notice labels lazy logging as CWE-94, code injection, and states that any authenticated client able to connect to the Watcher's ZMQ port can execute arbitrary Python code in the Watcher process. It calls this behavior by design, which is honest and also a hard boundary. The mitigations listed are real: messages are HMAC-SHA256 signed so only processes sharing the key can send expressions, and the Watcher binds to 127.0.0.1 by default. The user responsibilities are equally specific. Do not expose TensorWatch ports to untrusted networks or users. In multi-process setups where the Watcher and WatcherClient are separate, set ZmqWrapper._hmac_key to a shared secret before calling initialize(), or export TENSORWATCH_HMAC_KEY as a hex-encoded secret. The README gives this example: export TENSORWATCH_HMAC_KEY=$(python -c "import os; print(os.urandom(32).hex())"). The same notice also covers pickle: incoming ZMQ messages are HMAC-verified before deserialization, and a RestrictedUnpickler allowlists builtins, collections, numpy, torch, pandas, tensorwatch and pickle internals. FileStream uses pickle.load() on .log and .pkl files, and the README tells you to treat those files like executable scripts. If your data files arrive from anyone you do not fully trust, this is the wrong tool for reading them.

Where TensorWatch stops being the right choice

The README is direct that TensorWatch is a development and debugging tool, not designed for production, multi-tenant, or adversarial environments. Take that at face value. A hosted notebook service where other users share the host is a poor fit, because the security model assumes you control who can reach the ZMQ port and who holds the HMAC key. The default localhost binding is a mitigation, not a guarantee, and the README explicitly warns against changing it to 0.0.0.0 in untrusted environments. There is a second limitation that has nothing to do with security. The project describes itself as under heavy development, and the release data available shows no tagged releases, so you are tracking the master branch rather than pinning to a version. For a debugging aid you run locally that is tolerable. For anything you need to reproduce months later, an untagged moving dependency is a cost. A third constraint is the stack: the testing claim covers PyTorch 0.4 to 1.x and eager TensorFlow tensors, and the README says most features should also work there, which is weaker language than the PyTorch statement. If your training code is not PyTorch, verify the specific feature you need before building a workflow around it.

TensorBoard and TensorWatch differ in when the query is written

TensorBoard is the obvious comparison, and the difference is the timing of the decision. With TensorBoard you decide what to record at the point you write the training code, then read the recorded events afterwards. The record is fixed once the run starts. TensorWatch keeps an open channel to the live process and lets the query arrive later, which is the entire point of lazy logging mode and the reason eval() is in the design rather than being an oversight. That flexibility is paid for in trust: an expression channel into a running process is a code execution channel, and TensorBoard's write-then-read model does not have that property. The trade-off is not close in either direction. If you want durable, comparable runs and a dashboard you can hand to someone else, the write-then-read model is the safer shape. If you are in the middle of a run and need to ask a question you did not anticipate, TensorWatch is built for exactly that moment. The README also positions the project as extensible, saying you can build your own visualizations, UIs and dashboards, so the visualizer is meant to be replaceable rather than fixed.

Maintenance, releases and the MIT licence

TensorWatch ships under the MIT licence, which is permissive and places few obligations on how you reuse the code. That is the easy part. The harder part is that the repository shows no retrieved releases, and the README states the project is under heavy development toward being a platform for debugging machine learning. Practically, that means your upgrade path is the master branch, and a fix you depend on may arrive without a version number attached to it. The security notice reads like documentation that was revised after the fact: it describes the RestrictedUnpickler as significantly stronger than the previous blocklist approach, which tells you the deserialization boundary has already changed once. If you adopt TensorWatch, the thing to watch on upgrade is not the visualization code but the security surface, specifically the HMAC handling in ZmqWrapper, the allowlist in RestrictedUnpickler, and the default bind address. Those are the parts whose behavior you are relying on, and they are the parts most likely to shift. Nothing here is legal advice; the MIT text governs what you may do with the code.

Editorial conclusion

Adopt TensorWatch if you are debugging a training run interactively in Jupyter on a machine you control, and you want to query the live process instead of pre-declaring every metric. Do not adopt it for production serving, multi-tenant notebooks, or shared hosts where other users can reach a ZMQ port, and do not treat its .log files as safe input from third parties. Before you commit, verify three things: that graphviz is installed, that TENSORWATCH_HMAC_KEY is set to the same hex secret in every process that talks to your Watcher, and that the Watcher's bind address is still 127.0.0.1.

Official sources

1. Issues
2. License: MIT
3. microsoft/tensorwatch on GitHub
4. README