# omniparser turns CSV, EDI and fixed width files into JSON with a schema

> omniparser is a Go library that streams input in CSV, fixed length, XML, EDI, JSON or a custom format and transforms it into JSON according to a schema written in JSON, with XPath for record filtering and extraction and a JavaScript engine for custom functions. The design constraint is that the input is never loaded whole into memory.

**jf-tech/omniparser** — omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

- Repository: https://github.com/jf-tech/omniparser
- Stars: 1,085 · Forks: 81
- Language: Go
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/jf-tech-omniparser

## The format list is the pitch

omniparser is a native Go ETL parser that ingests input data of various formats in streaming fashion and transforms data into desired JSON output based on a schema written in JSON. The formats named in the description are CSV, txt, fixed length or fixed width, XML, EDI with X12 and EDIFACT, JSON, and custom formats. The schema files live alongside the sample input files in the example directories, and each example directory also holds a `.snapshots` subdirectory with the corresponding output files, which is what makes the examples usable as regression tests. The minimum Go version is 1.16, and the module is `github.com/jf-tech/omniparser` with the licence MIT.

## XPath is how a record is found and shaped

The documentation is unusually clear that XPath is not optional vocabulary. The XPath based record filtering and data extraction page says that xpath queries are essential to omniparser schema writing, and asks the reader to learn the concept and the tricks in depth. That is a real design choice rather than a convenience. In a fixed width file there is no element structure to select on, and in EDI the segment and element structure has to be addressed somehow, and XPath gives one addressing scheme for every format. A separate documentation page covers the IDR, the in-memory data representation of ingested data, which is the model the transforms operate on, so a schema written for one input format can be reasoned about without knowing the format's internals.

## JavaScript is the extension point, and it replaced the old one

The most interesting change in the project's history is the move to JavaScript for custom functions. The custom function documentation covers how `custom_func` is used and specifically the javascript and javascript_with_context variants, and the release history records that full ES6 feature support in the JavaScript custom function arrived with v1.0.5 in July 2024. The change of direction is visible in the deprecation list: `custom_parse` was deprecated in favour of `custom_func`, and most of the older custom functions were deprecated or removed in favour of using javascript. The JavaScript engine in the dependency list is goja, a JavaScript interpreter written in Go, and the custom function reference is a separate document listing all the built-in ones. The 1.0 release also changed how custom functions are packaged for extensions, so that an extension only sees the custom functions it actually includes.

## The schema changed shape in 1.0 and the version is explicit

The major restructure in 1.0 upgraded the schema version to `omni.2.1` because of a number of incompatible schema changes, and the README lists them. The `result_type` key became `type`, `ignore_error_and_return_empty_str` became `ignore_error`, and `keep_leading_trailing_space` became `no_trim`. The custom function convention also changed, since strings were previously always used for both the input and the result parameter type and now all types are supported on both sides. There is a note that custom_parse is still usable for back-compatibility even though it has been removed from the public documentation and the samples. The current release is v1.0.5 from 2024-07-03, and the last push to the master branch was on 2026-10-11, so the code is touched more often than it is released.

## There is a command line and a server mode

The changelog records the addition of a command line interface with a one-off transform command and a long-running HTTP server mode, and the deployment story follows from that. The Docker image is built on the Go 1.16 base image, compiles a binary with the git commit and build epoch stamped into it through linker flags, runs the help command as a build check, and sets the container command to the server mode. The HTTP side is handled by the chi router, and the schema is validated against a JSON Schema document with the gojsonschema library, so a schema with a typo is caught rather than silently producing empty output. The repository also carries a `cli.sh` and a `cli.ps1` wrapper, a `regen.sh` script, a Dockerfile and a `docker.sh`, and the online playground that was documented has been marked as not functioning because the free Docker hosting it used went away, with the README pointing at cloning and running the wrapper instead.

## Why it exists, in the author's own terms

The Why section is the most persuasive part of the README, because it is specific about the gap. No good ETL transform and parser library exists in Go. Looking at other languages does not help much either. Smooks is described as dead, with EDI parsing and transform that is too heavyweight and needs code generation. BeanIO cannot deal with EDI input. Jolt cannot deal with anything other than JSON input. And JSONata still only does JSON to JSON transform. On top of that, many parsers and transforms do not support streaming reads and load the entire input into memory, which the README calls not acceptable in some situations. That last point is the design principle underneath the streaming claim, and it is the one that matters for a pipeline that handles files larger than memory.

## Conclusion

Adopt omniparser if you ingest flat files or EDI into a JSON pipeline and want the mapping described in data rather than in code, since the schema is a JSON document and the parsing is streaming. Do not adopt it for a one-off conversion, because writing a schema for a single file is more work than writing a script. Verify first which schema version your files need, since the 1.0 release renamed several schema keys and moved the schema version to omni.2.1, and check whether the Go version in your build is new enough, because the minimum is 1.16 and the library has not moved past it.

## FAQ

### What is omniparser?

It is a native Go ETL parser that ingests CSV, txt, fixed length, XML, EDI with X12 and EDIFACT, JSON and custom formats in a streaming fashion, and transforms the data into JSON output based on a schema written in JSON. The minimum Go version is 1.16 and the licence is MIT.

### Why does omniparser use XPath in its schemas?

The documentation says xpath queries are essential to omniparser schema writing. XPath gives one addressing scheme for selecting and extracting records across formats, including fixed width files and EDI segments that have no element structure of their own.

### How do I write custom transforms in omniparser?

With the `custom_func` mechanism, and since v1.0.5 the JavaScript custom function with full ES6 support, running on the goja interpreter. The older `custom_parse` hook is deprecated but still usable for back-compatibility, and most of the original built-in custom functions were deprecated in favour of JavaScript.

### What changed in omniparser 1.0?

The schema version moved to omni.2.1 with incompatible changes, so `result_type` became `type`, `ignore_error_and_return_empty_str` became `ignore_error` and `keep_leading_trailing_space` became `no_trim`, custom functions now support all types for their input and result parameters, and extensions only see the custom functions they include.

### Does omniparser load the whole input file into memory?

No. The ingestion is streaming, and the README names the lack of streaming reads in other parsers as one of the reasons the project exists, calling it not acceptable in some situations. There is also an IDR node cache for allocation recycling.

## Sources

- [Issues](https://github.com/jf-tech/omniparser/issues)
- [jf-tech/omniparser on GitHub](https://github.com/jf-tech/omniparser)
- [License: MIT](https://github.com/jf-tech/omniparser/blob/master/LICENSE)
- [README](https://github.com/jf-tech/omniparser/blob/master/README.md)
- [Releases](https://github.com/jf-tech/omniparser/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/jf-tech-omniparser