Hardware Target Game Database, file layouts for flash cart ROM packs

What an SMDB file actually contains

The Hardware Target Game Database is described as an archival research initiative whose goal is to allow users to build real-hardware optimized ROM packs based on suggested file and folder layouts.

The reasoning behind it is practical. Most flash carts and optical drive emulators require specific ROMs and specific fixes, so compiling a complete, working setup is a monumental task that is often beyond the capabilities of any one person. The project invests thousands of hours in its source data bases, with the goal of complete, working, real-hardware compatible arrangements of the highest quality dumps.

The sharing mechanism is a plain text file. An SMDB file generated by the parse_pack script is a simple archival text record describing exact files, using hash values, and the location of those files within a folder hierarchy. One record per line, with six tab-separated columns per record.

The columns are the SHA256 value, the folder and file name in Unix and Linux format, the SHA1, the MD5, the CRC32, and the file size in bytes, which the README flags as a new feature not yet available in all SMDBs.

That last column is worth noticing. File size is the one field that is not a cryptographic hash, and it is the field that lets you tell two files with the same content apart, or spot a truncated download. The fact that it is a later addition, and not in every database, tells you the files were built incrementally over years by different people.

The four hash columns together are more than redundancy. SHA256 is what identifies a file unambiguously, and the older SHA1, MD5 and CRC32 values are there for compatibility with the tools and databases that existed when the records were first written. That is the archival nature of the project showing through in the format itself.

The four tools and their commands

The build_pack, parse_pack, verify_pack and base_sorter scripts are written in Python 3, and the README's instruction for running one is to install Python if needed and open a Windows console or a Linux and Unix terminal. There is a specific note for Windows users about trailing backslashes in arguments: a folder name ending in a backslash escapes the closing quote and breaks the argument list, so use a doubled backslash or none.

parse_pack makes SMDBs. It takes the folder to be parsed and the output text file:

```DOS .bat "C:\XXX\parse_pack.py" -f "C:\XXX\Folder to be parsed" -o "C:\XXX\SMDB.txt"

build_pack builds a pack against an existing SMDB, taking the input folder of unorganised ROMs, the database, an output folder and a file listing the missing ROMs:

DOS .bat "C:\XXX\build_pack.py" -i "C:\XXX\Folder with unorganized ROMs" -d "C:\XXX\SMDB.txt" -o "C:\XXX\Output folder for rebuilt pack" -m "C:\XXX\Missing.txt"

verify_pack checks that a pack folder matches its SMDB, writing a file listing ROMs in incorrect locations, extra files and ROMs that are missing:

DOS .bat "C:\XXX\verify_pack.py" -f "C:\XXX\Folder to be parsed" -d "C:\XXX\SMDB.txt" -m "C:\XXX\Mismatch.txt" ```

base_sorter is for starting from nothing. It sorts an unsorted ROM pack with no available SMDB, which is how you bootstrap a new database, and it takes a folder plus options: a minimum file count for the alphabetical group directories defaulting to 149, a file type or extension to limit sorting to, a disc option that puts discs into a folder, and a debug flag.

Two advanced options on build_pack deserve a note because they are the difference between a pack that eats your storage and one that does not. The file_strategy option takes copy, hardlink or smart. The default physically duplicates files, while hardlink and smart avoid duplication and save storage, only on filesystems that support it: use hardlink when source and destination are on the same filesystem and smart when they are on different ones. The README also notes that copying to a FAT32 or exFAT SD card automatically converts hardlinks into normal files, which is exactly the case where you would have wanted them, because those cards cannot hold hardlinks.

The skip_existing option avoids overwriting files already in the destination, and drop_initial_directory skips the first directory level of the SMDB pack so you can rename it, which is how a pack built under ./NES/EverDrive N8/ can instead be built directly under ./NES/.

How the code is organised

The repository layout separates the shared logic from the command line front ends, and the pyproject.toml explains the arrangement.

The shared code lives in a package called htgdb, and it holds the argument parsing, the hashing, the progress reporting and the SMDB reading. The four tools are described as thin front-ends over it, each exposing a main() entry point, and they stay as top-level modules so they can also be run directly as scripts without installing anything.

That is why the pyproject lists both a packages entry containing htgdb and a py-modules entry containing parse_pack, build_pack, verify_pack and base_sorter, and why the project scripts section registers the four as parse-pack, build-pack, verify-pack and base-sorter with dashes.

The repository also has an EverDrive Pack SMDBs directory holding the actual databases, a patches directory, a tests directory with its own README explaining how to run the suite and measure coverage, and a pytest.ini with a .coveragerc. The pyproject requires Python 3.8 or newer and works on Linux, macOS and Windows.

The credits section names the authors and the contributors, and the wiki link points to a page with more information for pack builders. A separate repository by another author provides a graphical user interface for build_pack and parse_pack, which the README suggests to anyone having difficulty with the command line options.

One thing about the design is worth naming, because it is unusual: the SMDB files are the actual product, and the tools are the means of producing and checking them. The databases can be shared, diffed and reviewed as text, and the tooling is deliberately small enough that you can read it and trust what it does with your files.

What the project does not do, and how to use it

The boundary of this project is deliberate and worth stating plainly. It contains no game content. What it contains is a description of how files should be arranged, identified by hash, and the tools to arrange them. The README is explicit that users dump their legally acquired files into one folder and that the tools analyse them by hash comparison, rename and sort them into packs, and that this lets creators share file and folder setups without sharing the files themselves.

That framing is the whole design. A layout is a small piece of information that can be described in text and verified, while the content is not. It is the same reason that a checksum database is useful even though it contains nothing useful on its own.

So the sensible way to use it is as a verification and organisation tool for a library you already have. The workflow the README describes is: gather your files into one folder, run build_pack against the SMDB for your flash cart, and read the missing list to see what you still need. Then run verify_pack to confirm the result matches the expected layout, which catches files in the wrong place as well as extras and absences.

The base_sorter route is for the case where no SMDB exists for your cart yet, or where you want to start a new one. Sorting an unsorted folder and then running parse_pack over the result is the way to produce a database from your own collection.

Two practical cautions. The file size column is not present in every SMDB, so a database from an older contributor will give you slightly different information than a newer one. And the drop_initial_directory option changes the depth of the produced tree, so it is worth running build_pack and verify_pack with the same setting, since a mismatch there looks like a layout failure but is only a naming difference.

For anyone deciding whether this fits, the question is whether you maintain a flash cart library large enough that the layout is a problem. If you have a handful of files, the layout does not matter and the tools are overhead. If you have thousands across several carts, having a text description of the expected arrangement and a way to check it against reality is genuinely useful.

Editorial conclusion

Adopt the SMDB format if you maintain a flash cart library and want to check that your folder layout matches what the hardware expects, or to share a layout with someone else without sharing files. Skip it if you are looking for a source of game content, because the project deliberately contains none of it. Verify first by running build_pack against a folder of files you own and reading the missing list it produces, then run verify_pack on the result to confirm the layout matches the SMDB.

Frequently asked questions

Official sources

1. frederic-mahe/Hardware-Target-Game-Database on GitHub
2. Issues
3. License: GPL-3.0
4. README