Sentence-VAE: A PyTorch Re-Implementation of Bowman et al. 2015 That Stops at Four Epochs

What Sentence-VAE Is Actually For

This repository is a PyTorch re-implementation of Generating Sentences from a Continuous Space by Bowman et al., 2015. The problem that paper addresses is representing a whole sentence as a point in a continuous latent space, so that sentences can be sampled from a prior and interpolated between. The README states the scope plainly: this implementation does not support LSTM units at the moment, only RNN and GRU. That single sentence tells you who the repository is for. It is for someone who wants to read, modify or extend a compact PyTorch model of the original architecture, not for someone who wants a general purpose text generation library. The repository carries no homepage, no releases and no declared licence in the supplied material, which is consistent with a research code drop rather than a maintained package. The topic tags (deep-learning, generative-model, neural-network, nlp, ptb, pytorch, vae) place it squarely in the research re-implementation category.

Encoder, Latent Sample, Decoder: The Data Flow

The architecture follows the paper's structure, and the README links to an image of it. An encoder network reads the token sequence and produces a distribution over a latent vector; a sample is drawn from that distribution; a decoder network conditions on the sample and reconstructs the sentence. Two regularisation knobs act on the decoder input rather than the encoder output. Word dropout replaces input words with the <unk> token at a probability given by --word_dropout, and embedding dropout applies the same idea at the embedding level via --embedding_dropout. These are the mechanisms the paper proposed to stop the decoder from ignoring the latent code, and their presence here is the main reason to read this implementation rather than a generic autoencoder. The KL term is weighted during training by an annealing schedule, chosen with --anneal_function as either logistic or linear. For the logistic schedule, --k sets the steepness and --x0 sets the midpoint, the point at which the weight equals 0.5. For the linear schedule, --x0 acts as the denominator. The README also includes a KL weight plot alongside the ELBO, NLL and KL training curves, which shows the schedule is treated as a first class part of the training loop rather than an afterthought.

Getting the Penn Tree Bank Data and Running Training

Training expects Penn Tree Bank text. The README directs you to download it from Tomas Mikolov's webpage, or to use the included dowloaddata.sh script (the spelling in the repository is as written there). The code expects to find at least ptb.train.txt and ptb.valid.txt in the data directory you specify. The base command is python3 train.py. The data related flags are --data_dir for the directory holding PTB and any auxiliary files, --create_data to regenerate those auxiliary files from the source text, --max_sequence_length to cut off long sentences, and --min_occ, which replaces any word occurring fewer times than the threshold with <unk>. Adding --test also measures performance on the test split. Optimisation flags are -ep/--epochs, -bs/--batch_size and -lr/--learning_rate. Model shape flags are -eb/--embedding_size, -rnn/--rnn_type (either 'rnn' or 'gru'), -hs/--hidden_size, -nl/--num_layers, -bi/--bidirectional and -ls/--latent_size. Logging uses -v/--print_every, and -tb/--tensorboard_logging with -log/--logdir for TensorBoard output. Checkpoints go to the directory passed with -bin/--save_model_path. For sampling and interpolation, the README gives python3 inference.py -c $CHECKPOINT -n $NUM_SAMPLES, where -c is the checkpoint path and -n the number of samples.

What the Reported Numbers and Samples Show

The README reports that training was stopped after four epochs, and that the true ELBO was optimised for approximately one epoch. Averaged over each split, the reported figures are NLL 99.821 and KL 7.944 on train, NLL 103.220 and KL 7.346 on validation, and NLL 103.967 and KL 7.269 on test. Those are the repository's own numbers, not an independent benchmark, and the gap between train and validation NLL is small enough that the model is underfit rather than overfit. The samples tell the more useful story. Drawing from z ~ N(0, I) produces sentences such as 'in the n of the n of the u . s . companies are n't likely to be reached for comment', where 'n' stands in for the unknown token. Several of the listed samples are dominated by that token. The interpolation examples are cleaner: the README shows a path from 'the company said it will be n with the exception of the company' through intermediate points to 'but in a statement that they were n't paid by the end of the past few weeks'. Interpolation between two drawn samples produces more coherent text than sampling from the prior, which is a known symptom of posterior collapse and is visible here without any instrumentation.

The LSTM Gap and Other Limits You Should Weigh

The most concrete limitation is stated in the README itself: no LSTM support, only RNN and GRU. If your comparison baseline or your existing pipeline is an LSTM sentence VAE, this repository cannot reproduce it without code changes. The second limitation is the training budget. Four epochs with the ELBO effectively optimised for about one epoch is a demonstration run, and the token-level samples reflect that. Anyone reading the sample block as evidence of generation quality will be misled; read the interpolation block instead, and read it as a qualitative illustration rather than a metric. Third, the repository has no releases and no declared licence in the supplied material, so you cannot tell from the metadata alone what terms apply to reuse. Fourth, the README does not describe the checkpoint format that inference.py expects beyond the -c path argument, so you should inspect the training save path and the inference loader together before assuming a checkpoint from a different run will load. Finally, the data pipeline is tied to Penn Tree Bank text files with fixed names; adapting it to another corpus means either renaming files or changing the loader.

Where It Sits Next to a General NLP Toolkit

The natural alternative is a general purpose sequence modelling library, where a sentence VAE is one model among many and you get tokenisers, dataset loaders, checkpoint formats and export paths maintained by a larger team. The difference in approach is not subtle. This repository is a single paper reproduction: the training script, the inference script, and the flags map directly onto the paper's hyperparameters, including the annealing function and the word dropout rate. A general toolkit would give you the same mathematical object behind a configuration file, with the annealing schedule and dropout implemented as library components you do not read. If your goal is to understand why the KL weight schedule matters, or to change how word dropout interacts with the decoder, the single file reproduction is easier to modify. If your goal is to ship an encoder that produces sentence embeddings for a downstream task, the toolkit is the better starting point, because you will spend your time on the task rather than on data loading and checkpoint plumbing.

Maintenance Cost and Licence Status

The repository is not archived, and the last push recorded in the supplied metadata is 2026-06-10, so it is not abandoned in the sense of being frozen. There are no releases, which means there is no version to pin and no changelog to read when the code changes under you. Upgrades are therefore a matter of tracking the master branch and re-reading train.py when it moves. The licence field is unknown in the supplied material. That matters more than usual for a repository that re-implements a published paper: the paper's terms and the code's terms are separate questions, and no licence file is declared here. If you intend to redistribute the code or ship a model trained with it, resolve the licence question before you build anything on top, and treat the absence of a declared licence as a reason to ask rather than to assume permissive terms. This is a description of the metadata, not legal advice.

Editorial conclusion

Adopt this repository if you want a compact PyTorch reference for the Bowman et al. 2015 sentence VAE and are willing to read the code rather than rely on the README. Skip it if you need LSTM encoders, a maintained library, or a model that produces fluent text: the README's own samples are dominated by the <unk> token. Before using it, check the repository for a licence file, since no licence is declared in the supplied material, and confirm which checkpoint the inference script expects via the -c flag.

Official sources

1. Issues
2. README
3. timbmg/Sentence-VAE on GitHub