NL2Bash is the dataset and model release behind natural language to Bash translation

Four components, and a paper behind them

This repository contains the data and source code release of the paper NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System.

That phrasing matters. It is a research release, not an application, and everything in it is there so that somebody else could reproduce the results or build on the models.

The components are listed as four items.

The first is a set of about 10,000 Bash one-liners collected from websites such as StackOverflow, paired with their English descriptions written by Bash programmers.

That pairing is the corpus, and its quality comes from where both halves come from. The commands are real, because they were written by people solving real problems. The descriptions are real too, because they were written by Bash programmers rather than generated. A synthetic corpus, where descriptions are produced from commands by a template, would be much larger and much less useful, because the resulting descriptions would have the shape of a command rather than the shape of a request.

The second component is TensorFlow implementations of the standard Seq2Seq and CopyNet models, plus a stage-wise natural language to Bash model using argument filling heuristics, which is the approach from Lin et al. 2017.

The third is a Bash command parser which parses a Bash command into an abstractive syntax tree, developed on top of bashlex.

And the fourth is a set of domain-specific natural language processing tools, including a regex-based sentence tokenizer and a domain specific named entity recognizer.

Those last two components are the ones that make the project more than a translation dataset. The parser gives the system a structured view of a command rather than a string, which is what lets the template accuracy metric exist at all. And the sentence tokenizer and entity recognizer are there because general-purpose NLP tools handle shell commands badly, since a description like compress all the files in the current directory is full of tokens that a general tokenizer treats as ordinary words.

There is also a hosted model to try at tellina.rocks, which is the pretrained system rather than the code.

One licensing note is worth catching, because it is unusual. The dataset in data/bash is separately licensed under the MIT license, announced on 24 April 2020, while the repository as a whole is GPL-3.0.

So the code and the data have different terms, and the data is the more permissive of the two.

The corpus, and the discipline of the splits

The corpus contains a diverse set of Bash utilities and flags: 102 unique utilities, 206 unique flags and 15 reserved tokens.

Those three numbers define the problem space. A hundred and two utilities is not the whole of Unix but it is a substantial fraction of what people actually type, and the 206 flags are the options that distinguish one invocation of a utility from another. The reserved tokens are the placeholders in the templates.

The splitting rule is the part that shows the care taken. The set of about 10,000 natural language and Bash command pairs is split into train, dev and test sets such that neither a natural language description nor a Bash command appears in more than one split.

That is stricter than the usual split. It means there is no leakage of a description into the test set, and there is no leakage of a command either. Both halves matter, because a description that appears in training and test would inflate the score, and so would a command that appears in both, since the model would be reproducing a memorised command rather than generalising.

The counts are given as a table. There are 8,090 training pairs, 609 dev pairs and 606 test pairs. The number of unique natural language descriptions is 7,340 in training, 549 in dev and 547 in test. And the number of unique commands is 6,400 in training and 599 in dev.

The gap between 8,090 pairs and 7,340 unique descriptions shows that descriptions repeat, which is what you would expect from Stack Overflow where several people ask the same thing in slightly different words. And the gap between 8,090 pairs and 6,400 unique commands shows the same thing on the command side, with the same command appearing with different descriptions.

The last row of the table is the one that needs the definition, and the definition is given: a command template is defined as a Bash command with all of its arguments replaced by their semantic types.

So a template is the shape of a command with the specific values removed, and there are 4,002 unique templates in training. That number is lower than the number of unique commands, which tells you how much of the variation in the corpus is in the arguments rather than in the structure. It is also why the template metric exists as a separate measure: a model can get the structure right and the arguments wrong, and that is a different failure from getting the structure wrong.

Two metrics, and why they disagree

The leaderboard has two tables, and the difference between them is the most informative part of the results.

The first is manual evaluation, where top-k full command accuracy and top-k command template accuracy are judged by human experts, with the exact procedures in section 4 of the paper.

Sub-token CopyNet, which is the paper's own model, gets F-Acc-Top1 of 0.36 and Top3 of 0.45, meaning 36 percent and 45 percent of the time the top one or top three predictions are the exact command a human judged correct. On templates it gets T-Acc-Top1 of 0.49 and Top3 of 0.61.

Tellina, the earlier stage-wise system, gets 0.27 and 0.32 on full command accuracy but 0.53 and 0.62 on templates.

So on the metric that matters most, the full command, the newer model is clearly ahead. On the template metric the older model is ahead.

The second table is automatic evaluation, using character-based BLEU and a self-defined template matching score as approximations of the true translation accuracy.

On BLEU, Sub-token CopyNet scores 50.9 at Top1 and 58.2 at Top3, against 48.6 and 53.8 for Tellina. On template matching, CopyNet gets 0.574 and 0.634 while Tellina gets 0.625 and 0.698.

So the automatic metrics rank the models the same way as the manual ones, and the copy model wins on the command-level metrics and loses on the template-level ones in both cases.

The reason the template scores are lower is worth understanding. Template accuracy rewards getting the structure right, and the automatic template matching score is a looser proxy, so it does not distinguish a model that produces the right structure with wrong arguments from one that produces a completely different command. In other words, the metric that flatters the older model is the one that forgives the most.

That is a normal pattern in the field and it is worth naming because it explains the shape of the table. The automatic metric is a cheap approximation used to compare systems at scale, and the human evaluation is the ground truth. Here they agree on the ordering of the two systems, which is a good sign for the automatic metric, but the manual numbers are the ones to quote.

The manual evaluation also comes with a warning. If you plan to run manual evaluation yourself, refer to the notes on manual evaluation for issues you should pay attention to. That is the project telling you that reproducing those numbers has gotchas, which is helpful and unusual.

Reproducing the pipeline

The run instructions are organised as a sequence of make targets, and they follow the structure of the paper.

First, install TensorFlow 2.0, following the official instructions. The code has been tested on Ubuntu 16.04 with CUDA 10.0 and cuDNN 7.6.3, which dates the project precisely.

The guidance on hardware is worth quoting. The experiments can be reproduced on machines with or without GPUs. However, training with CPU only is extremely slow and we do not recommend it. Inference with CPU only is slow but tolerable.

That is an honest split of the two use cases. Running the pretrained model on a laptop is a reasonable thing to do, and retraining it is not.

Then the environment and dependencies. Once TensorFlow is installed, you set the Python path and the main experiment dependencies:

export PYTHONPATH=`pwd`

(sudo) make

The backtick around pwd in the export line is the shell command substitution, so the Python path is set to the current directory, which is how the scripts find the modules in the repository without installing anything.

Then you change directory:

cd scripts

And then the data step:

make data

That one target does the whole preprocessing: it cleans the raw corpus, applies filtering, creates the train, dev and test splits, and preprocesses the data into the formats the TensorFlow models take. The documentation points at the data directory for the utility scripts if you want to change the processing workflow.

Training is one target:

make train

And the evaluation is a shell script with four variants, one for manual and one for automatic, and one for dev and one for test. The automatic dev evaluation is:

./bash-run.sh --data bash --prediction_file <path_to_your_model_output_file> --eval

and the manual one differs by passing the manual_eval flag instead of the eval flag, with the test flag added for the test set.

The one thing to get right when you use your own model output is the file format, and it is specified precisely. The i-th line of the file contains predictions for example i in the dataset, and each line contains top-k predictions separated by three pipe characters.

That format is what lets the evaluation script align your predictions with the dataset without any index information of its own. Getting it right is the difference between an evaluation run and a confusing failure.

Where this sits today

The honest way to place this project is as a 2018 research artefact that is still the reference point for its problem, rather than as a tool you would use now.

The paper is from 2018 and the code targets TensorFlow 2.0 with CUDA 10.0 and cuDNN 7.6.3, on Ubuntu 16.04. Those are three releases old and well past their support windows. The last push to the repository was on 2026-09-02, so somebody is still touching it, but the code is not being ported forward to current TensorFlow.

What has changed is the field around it. The task of turning a natural language request into a shell command is now handled by language models that were trained on an enormous amount of shell, and the accuracy figures in the leaderboard would be far below what a current model achieves. A 36 percent exact-command accuracy sounds low, and by the standards of 2026 it is.

But that comparison is not quite fair, and the reason is instructive. The task here is translation in the strict sense, where the output must be a command that appears in the corpus, and the metric is exact match. A language model produces a fluent, plausible command that is not in the corpus, and it would score zero on this metric while being more useful in practice. The dataset measures a narrower property than usefulness.

What has not been superseded is the dataset and the tooling. A corpus of ten thousand human-written pairs with a strict splitting discipline is still useful for anyone training or evaluating a small model, and the parser that turns a command into a syntax tree is still the thing you need to compute a template-level metric.

And the second component worth noting is the parser, because it is the part that is useful outside machine learning. A Bash command parser built on bashlex that produces an abstractive syntax tree is a tool for static analysis of shell scripts, for linting, or for anything that needs to understand a command rather than match its text.

The project is NewLisp as the primary language according to the repository metadata, which is almost certainly a misclassification given the TensorFlow and Python content, and it is GPL-3.0 licensed with the dataset under MIT.

Editorial conclusion

Adopt NL2Bash if you are working on natural language interfaces for the shell and want the standard corpus, the baseline models and the evaluation harness rather than building a dataset yourself. Skip it if you need a tool that translates English into runnable commands today, since this is a research artefact from 2018 with no maintained application. Verify first which TensorFlow version you have, since the instructions call for 2.0 and the tested stack is CUDA 10.0 era, and that you have a GPU if you intend to retrain the models rather than only run inference.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. README
4. TellinaTool/nl2bash on GitHub