Phase 0 + Phase 1: Infrastructure & Preprocessor (zxbpp) - #1
Merged
Conversation
Create WIP implementation plan tracking Phase 0 (infrastructure) and Phase 1 (preprocessor) of the Python-to-C port. Lists all tasks, decisions, and will be updated throughout implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the shared utility library for the C port: - Arena allocator (arena.h/c): Block-based allocator that frees everything at once. 64KiB default blocks, 8-byte aligned. This is the primary memory management strategy per plan.md. - Dynamic string buffer (strbuf.h/c): Growable byte buffer for building strings incrementally. Supports append, printf-style formatting, and detach. Used extensively for output generation. - Type-safe dynamic array (vec.h): Macro-based growable vector with push/pop/foreach. Header-only, no .c file needed. - Hash map (hashmap.h/c): String-keyed hash table using FNV-1a and open addressing with linear probing. Used for symbol tables, macro definition tables, and include-once tracking. All utilities are pure C11 with no external dependencies. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Top-level CMakeLists.txt with C11, flex/bison support, and warnings - zxbpp CMakeLists.txt wired for flex lexer + bison parser generation - Test harness shell script (run_zxbpp_tests.sh) that runs zxbpp on each .bi file and diffs output against expected .out files - Test CMakeLists.txt registers the harness with CTest The build won't succeed yet — flex/bison sources and main.c are still needed. This commit establishes the build skeleton. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Mark Phase 0 tasks complete (arena, strbuf, vec, hashmap, cmake, tests) - Document the remote misconfiguration incident: origin was pointing to the Python repo, now correctly points to zxbasic-c - Note bison 2.3 constraint on macOS (must use old-style grammar syntax) - Log Python preprocessor source study as complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
First working version of the C preprocessor. Uses a recursive-descent approach rather than flex/bison — this is more natural for a preprocessor and easier to match the Python original's exact output. Supports: - #define (object-like and function-like macros) - #undef - #ifdef / #ifndef / #else / #endif - #include with path resolution - #line directive emission - #pragma pass-through - #require / #init pass-through - #error / #warning - Builtin macros: __FILE__, __LINE__, __BASE_FILE__, __ABS_FILE__ - Recursive macro expansion with cycle detection - Command-line: -o, -d, -e, -D, -I, --arch, --expect-warnings Challenge: Switched from flex/bison to hand-written recursive descent. The preprocessor is line-oriented and doesn't need a formal grammar — a pure C approach is simpler and gives more control over output format. Not yet passing tests — discovered that the Python preprocessor strips REM/apostrophe comments to blank lines. Need to implement BASIC comment handling in the next commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Studied the Python PLY grammar (zxbpp.py) and base_pplex.py to understand the exact output model: - Initial #line 1 "file"\n at start - #define: blank line (first production) or #line N+1 (subsequent) - #ifdef (true): blank line; (false): no output - #else (false→true): #line N+1; (true→false): no output - #endif: #line N+1 when parent was enabled - Comments/empty: blank line; disabled: no output Replaced the broken need_line_sync/last_output_line tracking with a has_output flag that correctly distinguishes first vs subsequent defines. 29/91 tests now passing (up from ~2), including prepro02, prepro04, prepro05, prepro09, prepro10, and many more. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
strbuf_detach returns the buffer and resets the strbuf. The code was trying to compute the original pointer from strbuf_cstr after detach, which returned invalid data. Save original pointer before advancing past whitespace. Also fix memory leak in comma-separated arg case. 45/91 tests now passing (up from 29). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Handle the case where a #define has a BASIC comment (') followed by
a continuation backslash (\). The Python lexer treats this as: comment
content is skipped, but the \ at end continues the define body on the
next line. Implemented by replacing \ with \n during line joining and
teaching handle_define to skip comments until \n while preserving the
\n as a line break in the macro body.
Also handle multi-line macro expansion output: when expanded text
contains \n, append \n#line N to resync line numbers.
51/91 tests now passing (up from 45).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Strip trailing comments (' and REM in BASIC, ; in ASM) from content
lines before macro expansion
- Track asm/end asm blocks to switch comment character
- In ASM mode, ' is a valid token (e.g. af') and ; starts comments
- Fixes: prepro33, prepro34, prepro55, prepro58 (54→58 tests passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand macros in function-like macro arguments before substitution
- When object-like macro expands to a function-like macro name followed
by '(' in the remaining text, rescan to handle the call
- Fixes: prepro17, prepro19, prepro62 (58→61 tests passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- # operator converts macro parameter to quoted string literal - ## operator concatenates adjacent tokens in macro body - Stringizing doubles internal quotes per BASIC convention - Token paste strips surrounding whitespace before joining - Fixes: prepro63, stringizing0-2, token-paste0-1 (61→67 tests passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle multi-line block comments /' ... '/ — each line emits blank - Support underscore _ at end of line as BASIC line continuation - _ continuation only when standalone (not part of identifiers like __LINE__) - Fixes: prepro36, prepro37, prepro60 (67→70 tests passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Recursive-descent parser for #if expressions - Supports: ==, !=, <>, <, <=, >, >=, &&, ||, parentheses - Expands macros (object-like and function-like) in expressions - String comparisons include quotes (matching Python behavior) - Numeric values compared as integers for <, <=, >, >= - Fixes: iflogic, prepro44, prepro46, prepro82, prepro90 (70→74 passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- #init directive: trim leading whitespace from rest argument - ASM mode: suppress #line directive output (consumed silently like Python) - Include: remove extra closing #line for included file (Python only emits parent's #line on return) - Include: normalize paths by stripping ./ prefix - ifdef/ifndef/if: reset has_output inside enabled conditional blocks, matching Python's grammar scoping where each ifdef body is a new program - Fixes: init_dot, line_asm, once_base, prepro27, prepro72, prepro75, prepro80, and several others (74→80 tests passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…aths - Convert backslash to forward slash in include filenames (Windows compat) - Strip ./ prefix from include filenames for clean #line directives - Support macro expansion in #include (e.g. #define a <file.bas>, #include a) - Test harness: auto-detect project root, pass -I for stdlib include paths - Test harness: normalize paths in output for cross-platform comparison - Fixes: prepro70, prepro72 and others (80→82 tests passing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Trim leading whitespace from #pragma, #require, #init arguments - Normalize include paths using realpath() to make them relative to CWD - Fixes: prepro00, prepro01, prepro30, prepro64, prepro71, prepro73, prepro74, prepro85 (82→90 tests passing, only 1 failure remaining) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Parse [arch:XXX] modifier in #include directives - Support both orderings: #include once [arch:X] and #include [arch:X] once - Substitute arch in include search paths for arch-specific resolution - All 91 tests now passing (5 skipped = error-only tests with no .out) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ash error Match Python behavior: - Suppress stdout when errors occurred (if not global_.has_errors) - Check macro argument count and error on mismatch - Handle #define foo() as 1 epsilon param (matching Python's PLY grammar) - Report illegal '\' character in ASM content lines - Exit code 1 on errors (matching Python's sys.exit(entry_point())) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add compare_python_c.sh: runs both Python and C preprocessors on all test files and diffs output for drop-in replacement verification - Update run_zxbpp_tests.sh to handle .err files (error tests) in addition to .out files — now covers all 96 test files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 96 test files now have expected output: 91 .out files for normal tests, 5 .err files for error tests. Zero skips. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add detailed testing section covering: - Brew Python 3.12 as ground truth reference - How to run both test harnesses (standalone + Python comparison) - Test file types (.bi, .out, .err) - Current status: 96/96 passing, 91/91 Python-identical Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Testing strategy applies to all components (zxbpp, zxbasm, zxbc), not just the preprocessor. Each gets its own test harness pair. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each component has different input/output conventions: - zxbpp: .bi → .out/.err - zxbasm: .asm → .bin - zxbc: .bas → varies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oadmap - Phase 1 (zxbpp) complete: 96/96 tests, Python-identical output - How to build, test, and use the C preprocessor today - Python ground-truth comparison instructions - Visual roadmap from here to single-binary NextPi toolchain - Updated status table reflecting actual progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Build on Linux x86_64, macOS ARM64, macOS x86_64 - Run 96/96 test suite on all platforms - Python ground-truth comparison on Ubuntu - Release workflow: tag v* to create GitHub Release with binaries - Fix compare_python_c.sh to auto-detect Python (not hardcode brew path) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Show install commands for macOS, Ubuntu, and Fedora instead of just brew. Remove hardcoded /opt/homebrew paths from CLAUDE.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The original project declares python = "^3.11". No reason to demand 3.12. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- C Build badge (live from GitHub Actions — green/red) - zxbpp test count badge (96/96 passing) - Simplified workflow name to 'C Build' for clean badge URLs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
README badges, test counts, CI workflow, plan.md, and test harnesses must all stay in sync with the code. Don't let them lie. Also fixed stale Python 3.12 reference → 3.11+. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Linux x86_64 and macOS ARM64 cover the useful cases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Point python-upstream remote to canonical https://github.com/boriel-basic/zxbasic - Add sync-upstream.sh for manual sync of src/ and tests/functional/ - Add weekly CI workflow that auto-opens a PR when upstream changes - Update README with upstream sync section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Version tracks boriel-basic/zxbasic upstream with +cN port suffix: - csrc/VERSION: "1.18.7+c1" (read by CMake, compiled into binaries) - csrc/UPSTREAM: records upstream repo, version, commit, sync date - zxbpp --version now prints "zxbpp 1.18.7+c1 (C port)" - Fixed sync scripts to use upstream's 'main' branch (not 'master') Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
We are read-only consumers of upstream. All work to origin only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep C port docs out of Boriel's docs root. Updated links in CLAUDE.md, README.md, and WIP progress file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Xalior
added a commit
that referenced
this pull request
May 26, 2026
…ializer (zxbparser.py:712-720) Python anchor — p_var_decl_ini's CONST branch (zxbparser.py:712-720) short-circuits when the initializer is non-static AND non-string: `errmsg.syntax_error_not_constant(...); return` is taken BEFORE `declare_const(...)` is called. The identifier is therefore never registered, so a subsequent reference (e.g. an array bound) hits `access_var` fresh, gets auto-declared as default-implicit-float (with the accompanying `[W100] Using default implicit type 'float'` at the reference site), and `check.is_static` returns False on the bound — so the downstream emission is the correct "Array bounds must be constants" (bound.py:43-44). C anchor — `csrc/zxbc/parser.c` ~6885 calls `symboltable_declare(..., CLASS_const)` UNCONDITIONALLY at the top of the single-name DIM/CONST branch. When the CONST init-static check fails further down (line 7016, `err_not_constant; return NULL`), the entry was already inserted as CLASS_const and the class promotion at line 6920-6922 was already applied. Downstream the DIM upper-bound's `check_is_static` saw CLASS_const and short-circuited to true, `eval_to_num` then failed (no actual value), and the WRONG message "Unknown upper bound for array dimension" surfaced. The :25 W100 reference warning also went missing because the entry was treated as a (broken) const rather than a fresh implicit-float var. Fix — strategy #2 (rollback, surgical). Snapshot the entry's pre-promotion (`class_`, `declared`) state before the unconditional promotion, and on the err_not_constant path do the inverse of Python's "never reached declare_const": - if the entry was freshly created (no prior forward `@name`): drop it from the current scope hashmap, the per-scope ordered list, and the cs->sym_entries_ordered tail (new `symboltable_undeclare`). - if the entry pre-existed as CLASS_unknown (forward `@name`): restore its prior class and declared flags so it stays in the half-resolved state Python leaves it in. Strategy #1 (defer the class+declared assignment until after the static check passes) was rejected because the class promotion happens above the duplicate-decl check and other downstream uses that depend on `class_ != CLASS_unknown` — moving it is a larger structural change with risk of regressions on the CONST happy path (const4, const5, const9, arrconst). Strategy #2 leaves the happy path untouched. Verify: - probe `err_failed_const_decl_must_not_register` RED -> PROBE-EQUAL. - All 9 probe categories: 108 PROBE-EQUAL / 0 PROBE-DIFF (baseline 1 STDERR diff in warnings preserved). - Parse meter: PASS 1030 -> 1031; STDERR_MISMATCH 3 -> 2; dim_const_crash cleared. - Full corpus codegen meter: FULL-EQUAL 888 / FULL-DIFF 0 unchanged. - zxbpp: 96/96 unchanged. - PY-vs-C diff on const4, const5, const9, arrconst, dim_const_crash: all MATCH. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete implementation of Phase 0 (Infrastructure) and Phase 1 (Preprocessor) from plan.md.
Phase 0 — Infrastructure 🏗️
csrc/VERSION1.18.7+c1(upstream version + C port build number)Phase 1 — Preprocessor (zxbpp) ✅
Hand-written recursive-descent C port of the Python preprocessor (~1,600 lines).
#define,#include,#ifdef/#if, macro expansion, token pasting, stringizing, block comments, ASM mode, line continuation,#pragma/#require/#init/#error/#warning, architecture-specific includeszxbppzxbpp --version→zxbpp 1.18.7+c1 (C port)What's new in the repo
Nothing in
src/ortests/was modified — the C port is fully additive.Test plan
run_zxbpp_tests.sh)compare_python_c.sh)🤖 Generated with Claude Code