Introduces advanced natural language pattern matching

.build_meta.json

@@ -1,4 +1,5 @@
 {
-  "year": 2025,
-  "build": 57
+  "year": 25,
+  "month": 8,
+  "build": 3
 }

.claude/settings.local.json

@@ -17,7 +17,15 @@
       "Bash(mv:*)",
       "Bash(cargo fmt:*)",
       "Bash(cargo run:*)",
-      "Bash(cargo clippy:*)"
+      "Bash(cargo clippy:*)",
+      "Bash(cargo test)",
+      "Bash(cargo test:*)",
+      "Bash(cargo update:*)",
+      "Bash(\"C:\\logbie\\wfl\\target\\release\\wfl.exe\" wfl_combiner.wfl)",
+      "Bash(mkdir:*)",
+      "Bash(target/release/wfl.exe:*)",
+      "Bash(VM_DEBUG=1 cargo test test_positive_lookahead -- --nocapture)",
+      "Bash(cargo check:*)"
     ],
     "deny": []
   }

.gitignore

@@ -8,3 +8,5 @@
 #/target
 
 combined/
+
+*.log

CLAUDE.md

@@ -309,6 +309,11 @@ After making changes:
 4. Write tests in the module's test section
 5. Document in function catalog
 
+## Debug and code quality
+
+MUST ALWAYS run cargo clippy --all-targets --all-features -- -D warnings and fix any reported errors
+MUST ALWAYS run cargo fmt --all to fix formatting issues
+
 ## Key Files to Understand
 
 - `src/main.rs` - CLI entry point and command handling
@@ -351,7 +356,7 @@ wfl/
 3. **Error Messages**: Improving clarity and helpfulness
 4. **Documentation**: Keeping all docs up-to-date
 5. **Stability**: Ensuring backward compatibility
-6. **Version**: Currently at v2025.50.0
+6. **Version**: Currently at v25.8.3
 
 ## Debugging 
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "wfl"
-version = "2025.50.0"
+version = "25.8.3"
 edition = "2024"
 description = "WFL (WebFirst Language) is a programming language designed to be readable and intuitive using natural language constructs."
 license = "Apache-2.0"
@@ -10,7 +10,7 @@ authors = ["Logbie LLC <info@logbie.com>"]
 name = "WFL"
 identifier = "com.logbie.wfl"
 icon = ["icons/wfl.png"]
-version = "0.1.0"
+version = "25.8.3"
 copyright = "© 2025 Logbie LLC"
 category = "Developer Tool"
 short_description = "WebFirst Language Compiler and Runtime"

Dev diary/2025-08-05_backreference_implementation.md

@@ -0,0 +1,90 @@
+# Dev Diary: Backreference Implementation
+**Date**: August 5, 2025
+**Author**: Claude
+**Task**: Implement pattern backreferences for Phase 3 advanced features
+
+## Summary
+Successfully implemented backreference support in the WFL pattern matching system, allowing patterns to reference previously captured groups using the syntax `same as captured "name"`.
+
+## Implementation Details
+
+### 1. AST Extension
+Added `Backreference(String)` variant to `PatternExpression` enum in `src/parser/ast.rs`:
+```rust
+pub enum PatternExpression {
+    // ... existing variants ...
+    Backreference(String), // References a named capture group
+}
+```
+
+### 2. Bytecode Instruction
+Added `Backreference(usize)` instruction to `src/pattern/instruction.rs`:
+```rust
+pub enum Instruction {
+    // ... existing instructions ...
+    /// Match a backreference to a previously captured group
+    Backreference(usize), // capture group index
+}
+```
+
+### 3. Parser Updates
+- Added `KeywordSame` and `KeywordCaptured` tokens to the lexer
+- Updated `parse_pattern_element` to recognize `same as captured "name"` syntax
+- Fixed pattern concatenation to properly handle space-separated pattern elements
+- Removed need for "followed by" connectors - patterns now use simple space separation
+
+### 4. Compiler Updates
+Added `compile_backreference` method to resolve capture names to indices:
+```rust
+fn compile_backreference(&mut self, name: &str) -> Result<(), PatternError> {
+    if let Some(&capture_index) = self.capture_map.get(name) {
+        self.program.push(Instruction::Backreference(capture_index));
+        Ok(())
+    } else {
+        Err(PatternError::CompileError(
+            format!("Backreference to undefined capture group: '{}'", name)
+        ))
+    }
+}
+```
+
+### 5. VM Implementation
+Enhanced the VM to handle backreferences by:
+- Storing captured text during execution
+- Matching backreference against previously captured content
+- Properly handling capture state in `VMState`
+- Fixed `StepResult::Match` to include state for capture extraction
+
+### 6. Test Coverage
+Created comprehensive test program `TestPrograms/pattern_backreference_test.wfl` covering:
+- Simple backreference matching (e.g., "aa" matches `capture {any letter} as word same as captured "word"`)
+- Word repetition detection
+- HTML/XML tag matching
+- Multiple captures with backreferences
+- Backreferences in quantified patterns
+
+## Challenges and Solutions
+
+### Pattern Syntax
+**Challenge**: Initial attempt to use "followed by" as a connector caused parsing errors.
+**Solution**: Simplified to space-separated pattern elements, consistent with existing pattern syntax.
+
+### Capture API
+**Challenge**: VM wasn't returning capture information with matches.
+**Solution**: Updated `StepResult::Match` to include the final VM state, enabling capture extraction.
+
+## Results
+All backreference tests pass successfully:
+- ✓ Simple character repetition
+- ✓ Word repetition detection
+- ✓ HTML tag matching with backreferences
+- ✓ Multiple captures and backreferences
+- ✓ Quoted string matching with backreferences
+
+## Backward Compatibility
+Verified that existing pattern tests continue to work correctly. The new feature integrates seamlessly with the existing pattern system without breaking changes.
+
+## Next Steps
+- Implement lookarounds (positive/negative lookaheads and lookbehinds)
+- Add Unicode support for pattern matching
+- Update documentation with new pattern syntax

Dev diary/2025-08-05_lookaround_implementation.md

@@ -0,0 +1,101 @@
+# Dev Diary - August 5, 2025
+
+## Lookaround Implementation for WFL Pattern System
+
+### Overview
+Today I implemented lookaround support for the WFL pattern matching system. This includes positive/negative lookaheads and lookbehinds, allowing patterns to assert conditions about surrounding text without consuming characters.
+
+### Syntax Design
+The natural language syntax for lookarounds:
+- Positive lookahead: `check ahead for {pattern}`
+- Negative lookahead: `check not ahead for {pattern}`
+- Positive lookbehind: `check behind for {pattern}`
+- Negative lookbehind: `check not behind for {pattern}`
+
+Example:
+```wfl
+create pattern digit_before_letter:
+    digit check ahead for {letter}
+end pattern
+```
+
+### Implementation Details
+
+#### 1. AST Extensions (src/parser/ast.rs)
+Added four new variants to `PatternExpression`:
+- `Lookahead(Box<PatternExpression>)`
+- `NegativeLookahead(Box<PatternExpression>)`
+- `Lookbehind(Box<PatternExpression>)`
+- `NegativeLookbehind(Box<PatternExpression>)`
+
+#### 2. Lexer Updates (src/lexer/token.rs)
+Added new keywords:
+- `KeywordAhead`
+- `KeywordBehind`
+
+#### 3. Parser Updates (src/parser/mod.rs)
+Added parsing logic in `parse_pattern_element` to handle:
+- `check [not] ahead for {pattern}`
+- `check [not] behind for {pattern}`
+
+The parser correctly handles nested patterns within braces and tracks whether the lookaround is negative.
+
+#### 4. Bytecode Instructions (src/pattern/instruction.rs)
+Added new instructions:
+- `BeginLookahead` / `EndLookahead`
+- `BeginNegativeLookahead` / `EndNegativeLookahead`
+- `CheckLookbehind(usize)` / `CheckNegativeLookbehind(usize)`
+
+Lookbehinds are simplified to require fixed-length patterns (stored as usize).
+
+#### 5. Compiler Updates (src/pattern/compiler.rs)
+Implemented compilation methods:
+- `compile_lookahead`: Wraps pattern with Begin/End instructions
+- `compile_negative_lookahead`: Similar but for negative assertions
+- `compile_lookbehind`: Validates fixed length and generates CheckLookbehind
+- `compile_negative_lookbehind`: Similar for negative lookbehinds
+- `calculate_pattern_length`: Helper to determine if a pattern has fixed length
+
+#### 6. VM Implementation (src/pattern/vm.rs)
+The VM handles lookarounds by:
+- **Lookaheads**: Save position, execute nested pattern, restore position on success
+- **Negative lookaheads**: Save position, ensure pattern fails, restore position
+- **Lookbehinds**: Currently simplified - only check if enough characters exist behind
+
+The implementation uses a state-based approach with proper handling of nested lookarounds through depth tracking.
+
+### Challenges Encountered
+
+1. **Ownership in VM**: The `step` function takes ownership of VMState, requiring careful management of cloned states for lookaround evaluation.
+
+2. **Nested Pattern Execution**: Lookarounds contain nested patterns that must be executed without affecting the main match position.
+
+3. **WFL Syntax Issues**: The test programs revealed that WFL's property access syntax and function call syntax need clarification. Pattern functions aren't properly exposed in the standard library.
+
+### Testing
+Created test programs to verify lookaround functionality:
+- `pattern_lookaround_expr_test.wfl`: Tests basic lookahead with pattern matching expressions
+- Results show positive lookahead working correctly ("5a" matches `digit check ahead for {letter}`)
+
+### Current Status
+- ✅ AST nodes for all lookaround types
+- ✅ Parser support for natural language syntax
+- ✅ Bytecode instructions defined
+- ✅ Compiler generates correct bytecode
+- ✅ VM executes positive lookaheads correctly
+- ⚠️  Negative lookahead may have issues (test showing incorrect behavior)
+- ⚠️  Lookbehinds are simplified placeholders
+- ⚠️  Integration with WFL standard library needs work
+
+### Next Steps
+1. Debug negative lookahead implementation in VM
+2. Implement full lookbehind support with sub-pattern execution
+3. Fix standard library integration for pattern functions
+4. Add more comprehensive tests
+5. Update documentation with lookaround examples
+
+### Code Quality
+- All code compiles without errors
+- Minor warnings about unused variables addressed
+- Follows existing code patterns and conventions
+- Maintains backward compatibility with existing pattern tests

Dev diary/2025-08-05_lookbehind_implementation.md

@@ -0,0 +1,66 @@
+# Dev Diary - August 5, 2025
+
+## Full Lookbehind Implementation
+
+### Summary
+Successfully implemented full lookbehind support with sub-pattern execution for WFL pattern matching. This completes the lookbehind portion of Phase 3 advanced pattern matching features.
+
+### Changes Made
+
+#### 1. Enhanced Instruction Enum
+Modified `src/pattern/instruction.rs` to change lookbehind instructions from simple length-based checks to full sub-program execution:
+```rust
+// Before:
+CheckLookbehind(usize), // length only
+CheckNegativeLookbehind(usize), // length only
+
+// After:
+CheckLookbehind(Box<Program>), // full sub-program
+CheckNegativeLookbehind(Box<Program>), // full sub-program
+```
+
+#### 2. Updated Compiler
+Modified `src/pattern/compiler.rs` to compile lookbehind patterns into sub-programs:
+- Removed the fixed-length requirement
+- Each lookbehind pattern is compiled into a complete Program
+- The sub-program is embedded in the instruction
+
+#### 3. VM Implementation
+Completely rewrote lookbehind execution in `src/pattern/vm.rs`:
+- Uses sub-VM approach similar to lookaheads
+- Tries matching at different positions before current position
+- Supports variable-length lookbehinds (up to 1000 characters)
+- Ensures the pattern matches ending exactly at current position
+
+### Technical Details
+
+The implementation works by:
+1. Creating a sub-VM for the lookbehind pattern
+2. Trying different starting positions before the current position
+3. For each position, extracting a substring and checking if the pattern matches the entire substring
+4. Success if any position results in a complete match ending at current position
+
+### Test Results
+Created comprehensive test program `TestPrograms/pattern_lookbehind_test.wfl`:
+- ✅ Positive lookbehind for literal patterns
+- ✅ Negative lookbehind for word boundaries
+- ✅ Complex lookbehinds with lookaheads
+- ✅ Variable-length lookbehinds
+- ✅ Lookbehinds at string boundaries
+
+### Known Behavior
+The pattern `check not behind for {"the "}` when applied to "the cat" matches "t" at position 0, not "cat". This is correct behavior because:
+- "t" is not preceded by "the " (nothing precedes it)
+- "h" is preceded by "t", not "the "
+- "e" is preceded by "th", not "the "
+- "c" is preceded by "the ", so it doesn't match
+
+### Performance Considerations
+- Limited lookback distance to 1000 characters to prevent excessive computation
+- Each lookbehind requires trying multiple starting positions
+- Could be optimized for fixed-length patterns in the future
+
+### Next Steps
+- Implement Unicode support for character classes
+- Update documentation with lookbehind syntax and examples
+- Consider optimizations for common lookbehind patterns