DataProfiler: schema, statistics and PII labels from one Python call
What's in your data? Extract schema, statistics and entities from datasets
At a glance
- What is it?
- Capital One's DataProfiler loads a file into a pandas DataFrame and emits a profile dictionary covering column types, descriptive statistics and entity labels. The interesting part is the labeler, and that is also the part that costs you TensorFlow.
- Who is it for?
- Adopt DataProfiler if you need one pass over a tabular file that returns both column statistics and per-column sensitive-data labels, and you are willing to carry the ML dependency set. Do not adopt it if your only goal is descriptive statistics on a clean numeric table, because pandas describe plus a dtype map gets you most of that without TensorFlow in the image.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap DataProfiler fills between describe() and a data catalog
A pandas DataFrame gives you dtypes and, through describe(), a handful of numeric summaries. What it does not give you is a per-column judgement about what the values mean. A column of nine-digit strings is just an object dtype to pandas. DataProfiler's stated purpose is to close that gap: the README describes it as a Python library for "data analysis, monitoring, and sensitive data detection" that loads a file, identifies "the schema, statistics, entities (PII / NPI) and more", and returns all of it as a dictionary.
The intended audience is fairly narrow. This is for engineers and analysts who have to answer questions like which columns in this export contain personal data, or which columns are effectively categorical versus free text, before the data moves somewhere else. The topics list on the repository points the same direction: gdpr, pii, npi, privacy and security sit alongside csv, avro and pandas. If you are profiling a numeric sensor table, the entity recognition has nothing to do and you are paying for a model you will not use.
It is worth being clear about what the output is. A profile is not a validation report and not a quality score. It is a dictionary of measurements and predictions, and the README is explicit that the predictions come from a pre-trained deep learning model rather than from rules you wrote.
What actually happens between Data() and profile.report()
The README's getting-started example is three statements. Data("your_file.csv") auto-detects and loads the file, and the README lists CSV, AVRO, Parquet, JSON, Text and URL as the supported inputs. The resulting object exposes the loaded table through data.data, described as "a compatible Pandas DataFrame", so anything you would normally do to a DataFrame is available before you profile anything.
Profiler(data) is where the work happens. The README says it calculates statistics and entity recognition. The output shape is documented in detail. For structured data you get a global_stats block with samples_used, column_count, row_count, row_has_null_ratio, row_is_null_ratio, unique_row_ratio, duplicate_row_count, file_type and encoding, plus a profile_schema mapping column names to lists of integers and a times dictionary recording how long each phase took. Alongside it, data_stats is a list with one entry per column, each carrying column_name, data_type, data_label, a categorical flag, an order field, sample values, and a statistics block.
That statistics block is wider than most people expect. It includes null_count, null_types and null_types_index (which records the row indices for each kind of null), min, max, mode, median, median_absolute_deviation, mean, variance, stddev, skewness, kurtosis, num_zeros, num_negatives, a histogram with bin_counts and bin_edges, quantiles, unique_count, unique_ratio, categorical_count, gini_impurity, unalikeability, and a precision block with a margin_of_error and confidence_level. The times key appears at both the global and column level.
Unstructured input produces a different shape entirely. Instead of a list, data_stats is a dictionary keyed by data_label, containing entity_counts and entity_percentages at three granularities: word_level, true_char_level and postprocess_char_level. The presence of both a true and a postprocessed character count is a hint that the raw model output and the reported output are not the same thing, and the README does not explain the difference.
The install extras decide whether you get a labeler at all
This is the part most reviews skip, and it changes the library's behavior rather than just its dependency footprint. The README gives four install paths. pip install DataProfiler installs the base package. pip install DataProfiler[full] installs everything. DataProfiler[ml] adds the ML dependencies without report generation. DataProfiler[reports] is described as the slimmer option for when "the ML requirements are too strict (say, you don't want to install tensorflow)", and the README states plainly that this variant "disables the default sensitive data detection / entity recognition (labler)".
Read that again before you plan anything. If you install the reports extra to avoid TensorFlow, the data_label fields in your profile will not be populated by the pre-trained model. Every downstream decision that depends on knowing which columns hold PII now depends on which extra you installed, and nothing in the profile dictionary itself announces which mode produced it. Two teams running the same script in two environments can get different profiles from the same file.
The README also notes that new entities can be added to the existing pre-trained model, or an entirely new pipeline inserted for entity recognition. That is the extension point for anything the shipped model does not recognize, and it is worth checking the documentation before assuming your domain-specific identifiers are covered out of the box.
Loading and profiling a file, and the report formats
The minimal path, taken directly from the README, is:
import json from dataprofiler import Data, Profiler
data = Data("your_file.csv") print(data.data.head(5)) profile = Profiler(data) readable_report = profile.report(report_options={"output_format": "compact"}) print(json.dumps(readable_report, indent=4))
The only configuration key shown in the README is output_format, set to "compact" for a readable report. The README does not enumerate the other accepted values, so if you need a different serialization you will have to check the API documentation at capitalone.github.io/DataProfiler rather than guess.
One configuration detail is documented and easy to miss. The README notes that "currently the correlation matrix update is toggled off" and that it will be reset in a later update, but that users can still enable it with the is_enable option set to True. So global_stats lists correlation_matrix and chi2_matrix as part of the format, yet the correlation matrix is not computed unless you opt in. Anyone reading the profile schema and assuming the key is populated will be wrong. The chi2_matrix is listed without any such caveat, but the README does not say what it is computed over.
Where DataProfiler is the wrong tool
The dependency situation is the first real limitation. Entity recognition pulls in TensorFlow, and the library's own README frames the ML requirements as potentially "too strict" for some environments. If you are deploying into a constrained container, a Lambda-style runtime, or an environment with a pinned scientific stack, that is a genuine obstacle rather than an inconvenience, and the workaround documented in the README removes the feature you probably came for.
The second limitation is that profiling is not free at scale. The profile is dense: per-column histograms, quantiles, null type indices, and a times entry per column. On a wide table that is a large dictionary, and the per-column times suggest the library is measuring where the cost lands, which implies the cost is real. The README gives no guidance on row limits or how statistics are sampled, though global_stats does expose samples_used, so you can at least see after the fact whether the profile was computed over the whole file or a subset.
The third is that this is not a data quality framework. There is no pass/fail, no expectation language, no schema contract. You get measurements, and you decide what counts as a problem. If you need assertions that fail a pipeline, DataProfiler is an input to that system, not a replacement for it.
Finally, there is no stated support policy in the material. Releases 0.13.2 and 0.13.3 landed five days apart in March 2025, then 0.13.4 arrived in July 2025. That cadence tells you nothing about backwards compatibility of the profile dictionary, which is the thing your downstream code will be coupled to.
How it compares to Great Expectations and to plain pandas
The closest comparison in spirit is Great Expectations, and the difference is in what each one produces. Great Expectations is built around expectations: you declare what a column should look like, it validates the data against those declarations, and the output is a pass or fail with a data docs report. DataProfiler does the inverse. It does not ask you what the data should be. It looks at the data and tells you what it found, including a machine-learning guess about whether a column holds personal or sensitive information. If your problem is "we do not know what is in this file", DataProfiler answers it. If your problem is "this file must match the contract we agreed with the upstream team", Great Expectations is the shape of tool you want, and DataProfiler would only be the step that helps you write the contract in the first place.
Against plain pandas the comparison is sharper. A dtype map plus describe() gives you count, mean, std, min, quartiles and max for numeric columns, and value counts for categorical ones. DataProfiler adds the label predictions, the null typing with row indices, gini impurity and unalikeability, the histogram bins, the precision block with a margin of error, and the timing instrumentation. Whether that is worth a TensorFlow install is a question only your use case answers, but it is the right question to ask rather than assuming the library is strictly a superset of what you already have.
One thing DataProfiler does that neither alternative does by default is handle unstructured text through the same entry point. The unstructured profile reports entity counts and percentages at word and character level, which is a different kind of output from anything pandas or Great Expectations produces.
Licence, maintenance and upgrade exposure
DataProfiler is Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved and any modified files carry prominent change notices. It also includes an explicit patent grant. That is a permissive licence and it is the same one used across much of the Python data stack. This is a description of the licence text, not legal advice; if your organization has a licence review process, route it through that process.
The maintenance cost that matters here is not the library, it is the profile dictionary. Your downstream code will read specific keys: data_stats, column_name, data_label, statistics.quantiles, null_types_index. Those keys are the interface. The README documents the format in detail but says nothing about a compatibility guarantee, and the release history shows three patch releases in 2025 with no major version bump. A change to the shape of statistics or to the set of entity labels the model emits would surface as a patch release, and you would find it when your parser breaks.
The practical exposure is the model. Entity labels come from a pre-trained model that ships with the package. Upgrading the package can change which labels a column receives even when your data has not changed, because the model changed. If you are storing profiles over time to compare them, pin the version and store it alongside the profile, or your historical comparisons will silently mix model generations. The README mentions that new entities can be added to the model or a new pipeline inserted, which gives you a route to stability, but it also means two installations can diverge in behavior without diverging in version number.
Editorial conclusion
Adopt DataProfiler if you need one pass over a tabular file that returns both column statistics and per-column sensitive-data labels, and you are willing to carry the ML dependency set. Do not adopt it if your only goal is descriptive statistics on a clean numeric table, because pandas describe plus a dtype map gets you most of that without TensorFlow in the image. Before committing, install DataProfiler[reports] and DataProfiler[full] into separate virtualenvs, run Profiler on one of your own files, and diff the data_label fields in the two outputs. That diff is the entire decision.
Community notes