Skip to content
abedegnoPublic

About

Toolkit and Claude Code skills for byte-matching decompilation of old DOS games, from the Ultima Underworld II decompilation

Resources

Stars

12 stars

Watchers

1 watching

Forks

Latest commit

 

History

208 Commits

Folders and files

Repository files navigation

Exhume

Exhume is a toolkit and a set of Claude Code skills for byte-matching decompilation: writing C and assembly source that the original compiler, assembler and linker turn back into the same executable, byte for byte. It began with old DOS programs (Borland's 16-bit tools) and now also covers embedded firmware: for ARM Cortex-M microcontrollers built with Keil's Arm Compiler 5 (armcc), and for TI C2000 DSPs built with TI's C2000 code generation tools (cl2000). Each toolchain is a profile (profiles/README.md); the method, the gate and the skills are shared.

It is for preservation and for owners of old programs and devices: people who want a program's source back in a form that provably is the program, as the starting point for study, fixes, ports and changes to firmware they own. It was extracted from the decompilation of Ultima Underworld II (DOS, 1993), its first case study, made in the repository UW2Decomp and now the uw2/ folder of Underworld Exhumed: all of that game's code rebuilt from source with Borland's 1990 tools, linked into an EXE byte-identical to the original, that runs, and that shows a source change on screen; and then, from the same sources, a native port (built for macOS, Linux and Windows, and verified on the first two) that replays sessions recorded in DOS with the same game state and screen at every checkpoint (docs/case-study-uw2.md).

Underworld Exhumed holds both Ultima Underworld games, UW1 in uw1/ and UW2 in uw2/, each with its full history from the repositories UW1Decomp and UW2Decomp, and it includes Exhume as a git submodule, the way docs/port.md recommends. Where these pages say what a project's files are, they mean those folders; where they record what was done and proved (docs/case-study-uw2.md, the examples' results, commit hashes), they keep the name UW2Decomp, which the work was done under. The commits they cite are in Underworld Exhumed's history.

What is in it

  • tools/: the machinery. A headless DOS runner for the original toolchain (dosrun.mjs, in emu2, DOSBox-X, DOSBox Staging or js-dos behind one interface in dosbackend.mjs, with dosbatch.py for many sources at once and setup-emu2.sh to build emu2), builds (build.py), per-function matching with fixups masked (match.py), verification of everything matching masks (verify.py), merge gates (merge.py, rebuild-symbols.py), the map of a binary from an IDA listing and a symbol-bearing sibling build (doslist.py, locate.py, callgraphs.py, anchors.py, align.py, files.py, targets.py, callpairs.py, sibling.py), an assembly draft generator (asmgen.py), a build queue with per-file budgets for sandboxed agents (buildd.py, remote.sh), EXE comparison (exediff.py) and segment class experiments on objects (omfclass.py), booting a build, taking screenshots and searching its memory (rungame.mjs), the modding build's snapshot and changed-source compiles (modding.py), a scan for addresses written as numbers (addrscan.py), a screenshot comparison for bisecting (pngdiff.py), and a first look at an unknown EXE (fingerprint.py). The gate that proves the whole tree in one command (gate.py, with repocheck.py for CI, install-hooks.sh and templates/), sources in subsystem directories found by segment (sources.py) and hashed with their headers (srcdeps.py). The readability pass's tools: a declaration inventory (declinv.py), struct reconciliation (structrec.py), a header generator (headergen.py), raw offset and accessor macro tools (rawoffsets.py, accessors.py), warnings compared (buildwarn.py), and comment edits proved to change only comments (comments.py). Everything project-specific is read from the project's exhume.toml (docs/config.md). The port's tools (docs/port.md): the compile measurement, build and link stubs (portcheck.py, portbuild.py, portstubs.py, with portcfg.py reading the config), explicit widths, the promotion audit and the struct layout check against Turbo C (widths.py, intaudit.py, layoutcheck.py), the static recompiler for hand-written assembly (asm2c.py), record and replay in DOS and the port with golden references (replay.py, golden.py, replaydos.mjs), routine fuzzing of the original's bytes in Unicorn against the port (fuzzasm.py, fuzzhost.c), the sound drivers against a game's own AIL drivers (ailcheck.py), the test tiers and coverage (test.py, coverage.py), players' packages and their icon (package.py, icons.py with render-icon.mjs, setup-port.sh), and CI from an encrypted bundle (citemplates.py with templates/ci/, ci-assets.sh, setup-libs.sh, setup-sound.sh, requirements.txt).
  • runtime/: the porting harness a port builds against in place, never copied (runtime/README.md): the portable.h macros the shared sources use, the record and replay harness and the black box, the portability layer (compat.h and Borland's stand-in headers), the SDL-free platform API with an SDL3 backend, the paragraph map and far heap, EMS memory, the VGA, Borland's C library, finding the game and its settings, the machine the static recompiler's output runs on, and a sound library for Miles AIL 2 games. A project writes a few small bindings headers for it; UW2's are in Underworld Exhumed's uw2/src/port and uw2/src/include.
  • profiles/armcc5/: Arm Compiler 5 (armcc, armasm, armlink, fromelf from Keil MDK, run as MDK-Lite under Wine, in Docker or on the host) for Cortex-M firmware. Its tools (profiles/armcc5/tools/): the toolchain runner and object reader (armtc.py), the image and its map (ehex.py, armmap.py, armmapgen.py, with Ghidra drafts from ghidra/), library identification (armlibid.py), per-function matching by linking each source at its real addresses (armmatch.py, armmerge.py), the RW image and its compression (rwdata.py, lz77.py), the whole-image gate in pieces under the Lite link limit with its data stage and parallel Wine workers (armgate.py, armdata.py), the modded build (armmod.py), proof by object placement for a source the Lite link cannot take, too large or with two sections of one name (objplace.py), per-function scores of a whole file without linking (fnscore.py), drafts in C from an -O0 image alone (o0c.py), the compiler-build finder and the searches for stubborn functions (armbuilds.py, armvariant.py, armbrute.py, armpermute.py, armoutline.py, armreduce.py, armrepeats.py), the register allocator's own trace, turned on in a running compile, and a replay of it that checks itself against the trace, explains a tie and scores every declaration order without compiling (armtrace.py, armreplay.py), the readability tools for an image whose link order follows names (armnames.py, armmovefile.py, armmovefunc.py, armstruct.py, armgroups.py), and testing beyond the bytes (docs/firmware-testing.md): differential tests of two builds function by function, a Cortex-M system emulator with STM32F1-class peripheral models and a plugin interface (emu/), and an open-toolchain build with clang and GNU ld in the original's RAM layout (openbuild/). What matching taught is in compiler.md, linker.md and match-brief.md.
  • profiles/ti-c2000/: TI's C2000 code generation tools (cl2000 from TI's Windows build under Wine; lnk2000, hex2000, ofd2000, dis2000 and ar2000 from a native build) for C28x firmware with the FPU. Its tools (profiles/ti-c2000/tools/): the toolchain runner with a cloned Wine prefix per job and the COFF object reader (c28tc.py), the map (c28map.py), disassembly through a patched placeholder object (c28disasm.py), per-function matching by linking each source at its real addresses (c28match.py), the whole-image gate in one link (c28gate.py), the .ebss order solver (c28ebss.py), the runtime-library fingerprint (c28libfp.py) and batch search over a function's variants (c28batch.py). What matching taught is in compiler.md, linker.md and match-brief.md.
  • profiles/borland-tc101/: Turbo C++ 1.01, TASM 2.0 and TLINK 3.01. The command lines and name rules the tools use (profile.toml), and what matching taught about each tool (compiler.md, assembler.md, linker.md). profiles/README.md lists the profiles that would come next.
  • skills/: Claude Code skills, one per stage: fingerprint-toolchain, map-binary, match-file, match-asm, verify-and-merge, link-and-diff, modding-build (changes of any size, and the layout audit), readability-pass (headers, constants, fields, names and comments, every byte kept), orchestrate (running several agents), carry-forward (a matched tree carried to the vendor's next release or to a sibling board), port-prep, port-and-verify (from the modding build to a native port verified against DOS), firmware-verify (for firmware: differential tests, the system emulator and the open-toolchain build), ci-bundle (CI with private data on free public runners). Each says where the armcc5 profile differs; fingerprint-toolchain, match-file, link-and-diff, orchestrate and readability-pass also have notes for ti-c2000.
  • docs/: the method end to end (method.md), the configuration (config.md), the link stage, the modding build and layout audits (link.md), the readability pass (readability.md), the native port and its verification (port.md), CI from an encrypted bundle (ci-bundle.md), the third-party code a port uses and its licences (third-party.md), and the UW2 case study with its numbers.
  • examples/uw2/: UW2's configuration, its link stage as a reference implementation, a script that proves Exhume reproduces UW2Decomp's results, the readability pass's plan, struct specs and scripts, and UW2's port: its bindings for the runtime, its fuzz targets and recompiler spec, its CI values, and a script that proves the port tools and runtime on snapshots of Underworld Exhumed's uw2/ (prove-port.sh).

Requirements

  • Python 3.11 or later, with iced-x86, and Unicorn for the port's fuzzing and driver checks (python3 -m venv .venv && .venv/bin/pip install -r tools/requirements.txt).
  • Node 20 or later, and npm install in this directory, which fetches dos-mcp (a headless js-dos). js-dos runs the program itself (rungame.mjs), and runs the DOS toolchain when nothing faster is installed.
  • Optional, for speed: git, make and a C compiler, with which sh tools/setup-emu2.sh builds emu2 into tools/emu2; the toolchain then runs about twenty times faster than in js-dos (on UW2 the whole gate takes 8 seconds instead of 74). Or DOSBox-X (brew install dosbox-x), nearly as fast. docs/method.md, "Choosing the DOS", has the choice and the measurements.
  • For the armcc5 profile: Keil MDK's installer (MDK-Lite needs no licence; profiles/armcc5/setup-mdk.sh unpacks it), and Docker or Wine to run it (on a Mac, Wine Devel with EXHUME_ARMCC_RUNNER=wine); the exact armcc 5.06 update that built the firmware, which may be an older MDK's (skills/fingerprint-toolchain); Unicorn for the emulator, and clang with arm-none-eabi binutils for the open-toolchain build. The firmware image, supplied by you, as the vendor's Intel HEX update file or a raw dump.
  • For the ti-c2000 profile: TI's C2000 code generation tools, the version that built the firmware (TI's Windows installers, from TI's public download server, run under Wine; Wine Devel on a Mac), and a native CGT of 18.12 or later for the object tools. The firmware image, supplied by you, as a binary of 16-bit words.
  • The toolchain, supplied by you: for the Borland profile, the Turbo C++ 1.01 and Turbo Assembler 2.0 disk images, which profiles/borland-tc101/setup-tc.sh and setup-tasm.sh unpack (with mtools and 7z) and check by MD5. Exhume never contains compilers or other Borland software.
  • The program, supplied by you. Exhume never contains game data, and nothing a link takes from your copy of the game is ever written outside the build directory.
  • An IDA listing of the executable for the map stage, and optionally a sibling build with symbols.
  • For a port: clang, pkg-config and SDL3 (brew install sdl3 mt32emu pkgconf, or sh tools/setup-libs.sh on Linux), Nuked OPL3 fetched by sh tools/setup-sound.sh DIR for FM sound, and DOSBox-X for fast DOS replays (docs/port.md).

Quick start

npm install
python3 -m venv .venv && .venv/bin/pip install -r tools/requirements.txt
sh tools/setup-emu2.sh                                  # optional: the fast DOS for the toolchain
node tools/dosbackend.mjs                               # prints the DOS the toolchain will use
profiles/borland-tc101/setup-tc.sh   ~/disks/tc101      # your Disk01.img..Disk04.img
profiles/borland-tc101/setup-tasm.sh ~/disks/tasm20     # your TASM Disk01.img

.venv/bin/python tools/fingerprint.py ~/games/GAME/GAME.EXE

To make the skills available to Claude Code, link them into a skills directory, for example ln -s ~/Exhume/skills/* ~/.claude/skills/ (or a project's .claude/skills/). The skills refer to tools by paths relative to this checkout.

Then make a project directory with an exhume.toml (start from examples/uw2/exhume.toml), and follow docs/method.md, using the skills for each stage. In a project, the loop for one file is:

export EXHUME_CONFIG=~/myproject/exhume.toml
python3 ~/Exhume/tools/match.py src/FILE.C --dis some_function
python3 ~/Exhume/tools/verify.py src/FILE.C
python3 ~/Exhume/tools/merge.py src/FILE.C

Once the whole program matches and links, the gate proves the tree in one command, and is what every later change must pass: copy tools/templates/Makefile beside the project's exhume.toml, fill in its [gate] section, and run make check (or python3 ~/Exhume/tools/gate.py check). A project includes Exhume best as a git submodule (docs/port.md, "Where it is"); the template Makefile finds it at ../exhume, then .exhume, then ~/Exhume, unless $EXHUME names it.

For firmware, name the armcc5 profile in exhume.toml ([toolchain] profile = "armcc5") and point [binary] image at the HEX file; the same match, merge and gate commands then dispatch to the profile's tools. The profile's own tests run without the toolchain (those that need it skip): .venv/bin/python -m unittest discover -s profiles/armcc5/tests -t ..

For a C2000 image, name the ti-c2000 profile ([toolchain] profile = "ti-c2000", home the compiler's directory), point [binary] image at the binary and fill in [c2000] (docs/config.md); tools/match.py and tools/gate.py check then dispatch to its tools. Its tests: .venv/bin/python -m unittest discover -s profiles/ti-c2000/tests -t . (the integration tests need a compiler named in the environment; profiles/ti-c2000/tests/synth.py says how).

To check the toolkit against UW2 (needs an Underworld Exhumed checkout, its build, the toolchain and your UW2.EXE): examples/uw2/prove.sh --map, and python3 tools/gate.py check with EXHUME_CONFIG=examples/uw2/exhume.toml; for the port, sh examples/uw2/prove-port.sh --dos.

A port adds [port] and [replay] to exhume.toml, writes its bindings for runtime/ and builds against the runtime where it is, never copying it (runtime/README.md, skills/port-and-verify): then make port, make verify and make test from tools/templates/Makefile, and sh tools/install-hooks.sh . "make test".

Status

  • Three toolchain profiles: Borland Turbo C++ 1.01 for DOS (profiles/borland-tc101), Arm Compiler 5 for Cortex-M firmware (profiles/armcc5), and TI's C2000 tools for C28x firmware (profiles/ti-c2000). The C2000 profile was proven on a TMS320F2806x firmware in a private project: every word of the image rebuilt from source by one link, with no blobs, every function in C except a few that were assembly in the original. The armcc profile was proven on a commercial Cortex-M3 firmware image of about 500 functions, in a private project: every function written in C (no assembly stand-ins), the whole image rebuilt byte-identical to the vendor's update file, then made readable (folders, names, headers, comments) with every byte kept, and a modded build adding a feature beside the original code. The DOS bullets below describe the Borland profile.

  • Proven on two games with one toolchain profile: UW2, then UW1 (Underworld Exhumed's uw1/, with its own exhume.toml and link stage). For each, every code segment is matched, the exact link is byte-identical to the shipped EXE (UW2.EXE, UW.EXE), and the native port replays ten sessions recorded in DOS identically. Both games come from the same studio and the same Borland tools, so neither has yet tested another compiler or engine. For UW2, examples/uw2/prove.sh rebuilds all 154 UW2 sources, matches and verifies them with the same results as UW2Decomp's own tools, and links an EXE byte-identical to UW2Decomp's and to UW2.EXE itself (docs/case-study-uw2.md has the table). The gate requires that: no known differing bytes. Its modding build gives the same EXE with nothing changed, and with sources grown in an overlay and a resident file it boots into the 3D view.

  • The C and assembly stages (fingerprint, map, match, verify, merge) are general, given a profile. The verify rules assume a Borland medium-model program with VROOMM overlays; other layouts will need work there.

  • The link stage is a reference implementation per game (UW2's in examples/uw2/, UW1's in Underworld Exhumed's uw1/tools; docs/link.md): most of it encodes facts measured from the EXE. The modding build's general parts (tools/modding.py) and the layout scan (tools/addrscan.py) are tools; the scan assumes a Borland medium-model program with a VROOMM segment table.

  • The readability pass is proven on UW2: Exhume's gate passes on UW2Decomp's final tree, and its header generator, given UW2's plan, reproduces UW2Decomp's shared-headers commit token for token. The C parser behind the readability tools reads the top level of period C, not all of C, and the struct layout rules are Borland's.

  • The port tools and runtime are proven on UW2 (examples/uw2/prove-port.sh, docs/case-study-uw2.md): on a snapshot of UW2Decomp, Exhume's tools build a byte-identical port, its replay DOS build is the one the goldens were made with, all eight recorded sessions verify against the committed goldens, the fuzzing is clean, the recompiler reproduces every translated file, and UW2Decomp builds against Exhume's runtime in place, with no copy of it: its gate passes with the runtime's portable.h, and every session verifies, in the port and, for the replay harness, in DOSBox-X. They assume a 16-bit Borland program with a VGA mode X screen and, for the sound library, Miles AIL 2; what is UW2's own (the far data loader, seg021's start-up and input, the renderer's hand-written C and glue) stays in Underworld Exhumed's uw2/, listed in runtime/README.md.

License

MIT; see LICENSE. The toolkit contains no game data and no Borland, Keil or TI binaries: each project supplies its own executable and toolchain.

About

Toolkit and Claude Code skills for byte-matching decompilation of old DOS games, from the Ultima Underworld II decompilation

Resources

Stars

12 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages