TCAV: Testing Neural Networks Against Human Concepts, Not Pixels
Code for the TCAV ML interpretability project
At a glance
- What is it?
- Google's Testing with Concept Activation Vectors asks whether a trained model relies on a human-named concept such as color or gender. The repository ships the method, a pip package, and an InceptionV3 walkthrough, but the last tagged release is 0.2 from November 2018.
- Who is it for?
- Adopt TCAV when you need a class-level answer about human-named concepts and can supply labelled example sets for each concept. Do not adopt it as a general-purpose explanation tool: it produces a directional sensitivity score, not a causal claim, and its value depends entirely on the quality of the concept examples you collect.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 56 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What TCAV Answers That a Saliency Map Cannot
Most interpretability tooling explains one prediction by scoring input features. A saliency map highlights pixels; a feature attribution method assigns weights to columns. TCAV takes a different unit of analysis. According to the README, it shows the importance of high level concepts such as color, gender or race for a prediction class, and it produces an explanation that is generally true for a class of interest rather than for one image. That is the whole pitch, and it is a real distinction. If a product manager asks why the model classifies a category the way it does, a per-pixel heatmap is not an answer a non-specialist can act on. A number saying the model's decisions for that class are sensitive to a concept the team named is.
The intended audience follows from that. The README states that the consumer of the explanation may not know machine learning well, and that TCAV is designed to make sense to anyone who understands the high level concept. So the tool is aimed at teams that must explain model behaviour to people outside the modelling group, and at researchers who want a quantitative handle on concepts the training labels never mentioned. The README makes the striking claim that race or gender can be shown as important for InceptionV3 classifications even though neither label was part of the training input. That is the kind of result that motivates the method, and also the kind that demands care in interpretation.
How Concept Activation Vectors Are Built From Examples
The mechanism rests on the idea that a concept has a direction in the model's internal activation space. TCAV learns concepts from examples. The README is explicit: to learn a gender concept, TCAV needs a couple of examples of female and something not female. Those labelled examples are pushed through the network up to a chosen bottleneck layer, and a linear classifier separates the concept examples from the non-concept examples. The normal to that separating boundary is the Concept Activation Vector. Once you have a direction, you can measure how sensitive the model's output for a target class is to movement along it.
The constructor in the README shows the data flow directly. You pass a TensorFlow session, a target, a list of concepts, a list of bottlenecks, an activation generator, a list of alphas, a cav_dir for cached vectors, and num_random_exp. The run method returns results. The alphas are the step sizes used when probing sensitivity, and the bottleneck list means the same concept can be tested at more than one layer. The num_random_exp parameter controls how many random concept sets are used as a baseline, which is how TCAV turns a raw sensitivity number into something you can compare against chance. The README does not spell out the scoring formula, so treat the returned results as directional evidence rather than a calibrated probability.
Two design choices are worth naming. First, the network is not modified or retrained, per the README. TCAV is a post hoc probe, so it can be applied to a model you already trust and cannot afford to rebuild. Second, concepts are defined by you, not discovered by the method. That places the entire burden of validity on the example sets. A concept vector learned from six poorly chosen images is still a vector, and TCAV will still return a number for it.
Installing tcav and Running the InceptionV3 Walkthrough
Installation has one wrinkle that the README calls out directly. TensorFlow is required but is deliberately not listed in the package's install_requires, because a user may want either the tensorflow or the tensorflow-gpu package. So the install is two commands, not one:
pip install tensorflow pip install tcav
The rest of the Python dependencies are installed automatically with the tcav package. The README's requirements list names tensorflow, numpy, Pillow, matplotlib, scikit-learn and scipy. After installing, the README points to Run TCAV.ipynb for a step by step guide. That notebook is the fastest way to see the full pipeline before wiring TCAV into your own code.
The minimal usage pattern from the README is:
mytcav = tcav.TCAV(sess, target, concepts, bottlenecks, act_gen, alphas, cav_dir=cav_dir, num_random_exp=2) results = mytcav.run()
Every argument is a decision. The target is the class you want explained. The concepts list determines which human ideas you are testing, and each concept needs its own example directory of positive and negative images. The act_gen argument is the activation generator that feeds examples through the model. If your model is not one of the bundled architectures, this is the component you will have to write, and the README does not document its interface beyond the constructor call. Budget time for reading the source of the model module rather than expecting a tutorial.
For non-image work, the repository includes a discrete data path. The README gives the directory and a notebook:
cd tcav/tcav_examples/discrete/ tcav/tcav_examples/discrete/kdd99_discrete_example.ipynb
That KDD99 notebook is the reference for tabular or otherwise non-image models, and it is the example to copy if your inputs are not pictures.
Verify With the Unit Tests Before Trusting Your Own Setup
The README lists four test modules that can be run directly:
python -m tcav.cav_test python -m tcav.model_test python -m tcav.tcav_test python -m tcav.utils_test
Run all four against your installed environment before you build anything on top of the library. This matters more than usual here. The package's last tagged release is 0.2, dated November 2018, and the 0.1 release landed a week earlier. The README's own instructions for cutting a new version describe building wheels for both Python 2 and Python 3 with python setup.py bdist_wheel --python-tag py3 and --python-tag py2, then uploading with twine upload dist/*. A packaging process that still emits a Python 2 wheel tells you what era the release tooling belongs to. The repository has seen pushes since then, but the pip package you install is the 2018 artefact unless you build from source.
The practical consequence is version drift. TensorFlow's Python API changed substantially after 2018, and the README does not pin a TensorFlow version. If the tests fail on import errors, the mismatch is between your TensorFlow and a library written against an older one, not a flaw in the TCAV idea. Decide early whether you are pinning an old TensorFlow to match tcav or patching tcav to match a current TensorFlow. Both are real work, and the README does not choose for you.
Where TCAV Breaks Down or Misleads
The sharpest limitation is the one the method's own framing invites. A TCAV score tells you the model's output for a class is sensitive to a direction in activation space that a linear classifier built from your examples happens to find. It does not tell you the model uses that concept the way a human would. If your positive and negative example sets differ in some incidental way, say background or lighting, the concept vector will encode that difference, and the sensitivity score will be real while the interpretation is wrong. The README offers no guidance on how many examples per concept are enough, no suggested train and test split for the concept classifier, and no diagnostic for a concept vector that is separating on the wrong axis. Those are gaps, not details.
Second, the result is class-level by construction. That is the selling point, and it is also a constraint. If you need to explain one specific misclassification, TCAV is the wrong instrument; a feature attribution method is the right one. The two answer different questions and neither substitutes for the other.
Third, the bottleneck choice is underdetermined. The constructor accepts a list of bottlenecks, which implies the answer can vary by layer, but the README does not say how to choose or how to reconcile disagreement between layers. If layer A says the concept matters and layer B says it does not, you are on your own.
Finally, there is no causal claim anywhere in the material. A high sensitivity score is an association between a direction and an output, measured by perturbation. Treating it as proof that removing the concept would change predictions is an overreach the documentation does not support.
TCAV Against SHAP and Integrated Gradients: Different Questions
The natural alternatives are feature attribution libraries such as SHAP and Integrated Gradients. The difference is the unit of explanation, not the quality of the implementation. SHAP and Integrated Gradients attribute a prediction to input features: pixels, tokens, columns. They give you a per-instance decomposition and, when aggregated, a per-feature summary. TCAV attributes a class's behaviour to a user-defined concept measured in an internal layer. If your question is which pixels drove this one prediction, SHAP is the direct answer and TCAV cannot answer it at all. If your question is whether the model's decisions for a class depend on a concept that never appeared in the labels, TCAV is the only one of the three designed for it.
There is a second, subtler difference. Feature attribution methods need no additional labelled data beyond the input you are explaining. TCAV needs a curated example set per concept, plus a negative set, plus a choice of layer, plus a random-experiment count for the baseline. That is a real data collection cost, and it is the price of getting an explanation in human terms. Teams that underestimate this cost end up with concept vectors built from whatever images were lying around, which is the failure mode described above. The honest comparison is not which method is better but which one matches the question you have been asked.
Maintenance Cost and the Apache-2.0 Licence
The maintenance picture is mixed and should be read carefully. The repository is not archived, and its last push is dated 2026-07-22, so someone is still touching the code. But the release history tells a different story: 0.1 in November 2018, 0.2 a week later, and nothing tagged since. Anyone installing from PyPI gets the 2018 API. Anyone building from the default branch gets whatever has accumulated since, with no version number to cite. For a method published at ICML, that split between an active repository and a frozen package is the main operational risk.
Upgrade cost is therefore concentrated in TensorFlow compatibility rather than in TCAV's own API, which has been stable enough that the README's constructor example is the same one you would write today. The dependency list is short and all of it is common: numpy, Pillow, matplotlib, scikit-learn, scipy. None of those are exotic pins. The expensive part is the activation generator for a custom model, which the README does not document, and the concept example sets, which are your data problem and not the library's.
On licensing, the repository is Apache-2.0. That is a permissive licence with an explicit patent grant and a requirement to preserve notices and state changes. It does not, by itself, settle what happens when you publish concept-based explanations of a model trained on data you do not own, and it says nothing about the privacy questions raised by probing for concepts like race or gender. Those are questions for your own review, not for the licence text.
Editorial conclusion
Adopt TCAV when you need a class-level answer about human-named concepts and can supply labelled example sets for each concept. Do not adopt it as a general-purpose explanation tool: it produces a directional sensitivity score, not a causal claim, and its value depends entirely on the quality of the concept examples you collect. Before committing, verify which TensorFlow version the installed tcav release expects, run the four unit test modules listed in the README, and confirm your bottleneck layer choice supports the concept separation you need. If a single-image saliency map answers your question, TCAV is the wrong tool.
Community notes