From 5b22ab24b1c4ca33fa07d57a9d82617e7ee53b23 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 02:02:19 +0900 Subject: [PATCH 01/24] test: specify the ToDo3463 todo update reader before implementing it Adds the spec (Issue #408), the CLI contract, the fixed production pins and synthetic Git/file/receipt acceptance tests. verifyUpdate is a stub, so every verification test fails for the intended reason. --- docs/todo-update-reader-3190.md | 124 +++ .../cli_windows_test.go | 112 +++ tools/todo-update-reader-3190/go.mod | 3 + tools/todo-update-reader-3190/main_windows.go | 75 ++ .../profile_windows.go | 131 +++ .../reader_windows_test.go | 913 ++++++++++++++++++ .../todo-update-reader-3190/verify_windows.go | 120 +++ 7 files changed, 1478 insertions(+) create mode 100644 docs/todo-update-reader-3190.md create mode 100644 tools/todo-update-reader-3190/cli_windows_test.go create mode 100644 tools/todo-update-reader-3190/go.mod create mode 100644 tools/todo-update-reader-3190/main_windows.go create mode 100644 tools/todo-update-reader-3190/profile_windows.go create mode 100644 tools/todo-update-reader-3190/reader_windows_test.go create mode 100644 tools/todo-update-reader-3190/verify_windows.go diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md new file mode 100644 index 00000000..4281ccf2 --- /dev/null +++ b/docs/todo-update-reader-3190.md @@ -0,0 +1,124 @@ +# 方向記録版todo.exe 1本の更新資格を独立readerで照合する（ToDo3463・親3190） + +対応Issue: https://github.com/AutoDevJapan/cc-pc-tools-rs/issues/408 。親Issueは #325、生成元はPR #376 の通常merge。 +この工程は読取専用の資格確認だけを扱う。`todo.exe` の置換・配布、ToDo 3050/3084/3110/3190 の状態変更、 +本人回答の記録と質問の終了は親3190が担当する。共有guard（`repair-guard`）の source・profile・配布物は変更しない。 + +## 入口と出力 + +```text +todo-update-reader-3190.exe --verify-todo-runtime-update D:/CC/Tools/todo/todo.exe +``` + +`-verify-todo-runtime-update` と `=` 付きの形も受け付ける。引数はこの1組だけで、パスは絶対パスに限る。 +callerからmanifest、hash、source、root、ToDo番号、許可ディレクトリを受け取らない。 +不足・重複・追加の引数と相対パスは、照合を始めずに標準出力へ `{"qualified":false}`、終了コード1を返す。 +引数なしは使い方を標準エラーへ出して終了コード2、`-h` と `--help` は終了コード0で、標準出力は空。 + +資格を得た場合だけ、終了コード0で次の固定項目を返す。 + +| 項目 | 値 | +| --- | --- | +| `qualified` | `true` | +| `domain` / `todo_id` / `verifier_todo_id` | `todo-runtime-update-3190-v1` / 3190 / 3463 | +| `target_kind` / `target` | `todo` / `D:/CC/Tools/todo/todo.exe` | +| `current_sha256` | 照合した現在の旧SHA（固定値と一致した場合だけ） | +| `new_sha256` / `new_bytes` | 新候補の固定SHAとサイズ | +| `source_repository` / `source_root` / `source_head` / `source_tree` | 固定source | +| `build_receipt_sha256` / `build_job` / `build_command` / `build_log_sha256` / `dep_info_sha256` | 原生成記録 | +| `inputs_verified` / `dep_info_entries` / `local_crates` | 照合した件数 | +| `baseline_qualified_by` / `baseline_source_receipt_sha256` | 旧版の資格を与えた既存用途とその証拠SHA | +| `data_access_granted` / `original_registry_recovered` | 常に `false` | + +拒否は標準出力に `{"qualified":false}` だけを出し、終了コード1。標準エラーには固定の理由コード +（`target_path`、`target_leaf`、`baseline_qualification`、`baseline_mismatch`、`source_root`、 +`source_git`、`receipt`、`log`、`dep_info`、`input`、`closure`、`candidate`、 +`guard_classification`、`recheck`）だけを出す。未知の現在SHA、パスの中身、receiptの本文は出さない。 +返却時にhandleは閉じるので、この結果は将来の置換を予約するものではない。置換の直前の再照合と +有限CASは親3190の配布工程が行う。置換後の状態はこのreaderの対象外で、置換後に同じ照合を求めない。 + +## 固定値 + +| 対象 | 固定値 | +| --- | --- | +| 対象leaf | `D:/CC/Tools/todo/todo.exe` | +| 旧版 | SHA256 `f8710b516d3ecdd37564d7a1ab978c8689ab1bdd2ffaec4150d0e4bed4c0b983` | +| 旧版の資格 | 配布済み `D:/CC/Tools/client-records/scrub391.exe --verify-capacity-runtime-update` の結果が domain `capacity-runtime-update-3085`、todo_id 3085、target_kind `todo`、old `4b351db6…364f`、new `f8710b51…b983`、source receipt `a3bc15ae…6e86`、data access false と完全に一致 | +| source | `D:/CC/Work/unattended-operations-20261013-wt/todo-3190-3110`、origin `AutoDevJapan/cc-pc-tools-rs`、HEAD `0e16e81798ff4463c6ae6bcf6c4c57a695d9d13c`、tree `31d45a745d27271420e764f6dff493e3d58b3731`、`origin/main` の祖先 | +| build receipt | `todo3190-build-0e16e817.private.json`、SHA256 `20345f443566c16553dbd959d5cd07b53b8aa0e1579a55bff1a614ab0f75087c` | +| build log | `todo3190-release-0e16e817.private.log`、SHA256 `1fe224b19ba4d5265711e0119e554a035aa15b7cc7f4b0da327478bafb9fe1ef` | +| dep-info | `target/release/todo.d`、SHA256 `da3ce6ae3fcebe72fb5cecb701fc23efa70571086a5cd73a7d7642aca1e96941` | +| 新候補 | `todo3190-todo-0e16e817.private.exe`、SHA256 `d4392e6326f1868e2ffddc31a80e83d3170f93d639b1a04656066be387180718`、8,017,920 bytes | +| 件数 | 入力126件（dep-info 116件と manifest/lock 10件）、ローカルcrate 8件 | + +旧版の資格はToDo3085の旧配布の証拠であり、このreaderはその結果を「現在の配布物が f871 である」 +前提の確認にだけ使う。容量用途の資格を今回の更新資格へ名前だけ変えて流用しない。 + +## 検証の順序 + +1. 指定パスが固定leafと字句的に一致することを確かめる。親ディレクトリ、兄弟、ADS（`:`）、末尾の点、 + `\\?\`・UNC・短い別名は一致しないので拒否する。 +2. 対象leafと全祖先が reparse point でない実ディレクトリ・通常ファイルであることを確かめ、 + 書込・削除・改名を共有しない読取handleで開く。この時点では中身を読まない。 + handleの実パス、リンク数1、reparse属性なしを確認する。 +3. 同じhandleを保持したまま、配布済みguardの既存の容量用途を子プロセスで呼ぶ。結果の項目と値が + 上の表と完全に一致し、終了コード0の場合だけ次へ進む。元client一致などで容量用途が拒否すれば止める。 +4. その後でだけ同じhandleから対象を読み、固定の旧SHAと一致することを確かめる。違えば拒否し、値を出さない。 +5. source rootの祖先と自身が reparse point でないこと、実パス、guardの `--guard-path` が clear であることを確かめる。 + Gitは `GIT_*` 環境を除き、窓を出さず、期限付きで呼ぶ。worktreeの最上位、origin、HEAD、tree、 + 追跡ファイルの差分なし（`core.autocrlf=true`）、`origin/main` の祖先であることを確かめる。 +6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 + 読んだbytesそのものをhashして解析する。 +7. receiptの全項目を固定値と照合する。未知の項目と `data_access_granted` が false でない記録は拒否する。 +8. 各入力は、固定treeの通常ファイル（mode 100644/100755）のblobとreceiptのblobが一致し、 + working fileのSHA256がreceiptと一致することを確かめる。 +9. 新候補のサイズとSHA256を固定値とreceiptの両方に照合する。 +10. 最後に保持した全handleの同一性（volume・file ID・サイズ・更新時刻・リンク数・属性）と実パス、 + source rootの実パス、HEAD、追跡ファイルの差分なしを再確認する。照合中の変更は拒否する。 + +## compiler closure の確認 + +dep-info はコンパイラが読んだローカルファイルの一覧である。次の条件をすべて満たす場合だけ、 +126件の入力がそのビルドのrepo内の入力として過不足ないとみなす。 + +- dep-info の対象が `/target/release/todo.exe` で、全項目がsource root内の入力に含まれる。 +- 入力からdep-infoの項目を除いた残りが、`Cargo.lock`、workspaceの `Cargo.toml`、 + dep-infoの各項目を含むcrateの `Cargo.toml` とちょうど一致する。どのcrateにも属さない項目は拒否する。 +- logの最終行が release の `Finished` で、ほかは `Compiling` 行だけ。source root内のcrateとして + コンパイルされた集合が、dep-infoから求めたcrateの集合と一致する。source root外のローカルpathは拒否する。 +- 固定treeの各crateとrootに `build.rs`、`.cargo/config(.toml)` がなく、rootに `rust-toolchain(.toml)` がない。 + 各 `Cargo.toml` に `build =` の指定がない。 + +repo外の要素は入力に含まれない。具体的には toolchain の版、利用者・上位ディレクトリのCargo設定と +環境変数（`RUSTFLAGS` やsccacheなど）、registry crate の展開済みsource（`--locked` と `Cargo.lock` の +checksumで版を固定）である。このreaderは再ビルドによる再現性の証明ではなく、候補・原記録・固定sourceの +相互の一致を確かめる。 + +## 受入条件とテスト + +テストは `tools/todo-update-reader-3190` の `go test`。合成したGit repo、ファイル、receiptと、 +guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 +理由コードも照合し、別の理由による偶然の拒否を合格にしない。 + +| 条件 | テスト | +| --- | --- | +| 正しい組だけが資格を得る | `TestQualifiesFixedUpdate` | +| 旧版の資格の後でだけ旧targetを読む | `TestBaselineIsReadOnlyAfterPurposeQualification` | +| 旧/新SHAの相違 | `TestRejectsBaselineMismatch`、`TestRejectsCandidateMismatch` | +| source/receipt/log/dep-info の改変と欠落 | `TestRejectsTamperedOrMissingEvidence`、`TestRejectsSourceChanges` | +| 別origin、HEAD/treeの移動、未merge | `TestRejectsSourceChanges` | +| 親・兄弟・ADS・別名、任意入力 | `TestRejectsOtherPaths`、`TestCLIAcceptsOnlyThePurposeFlag` | +| hardlink、junction | `TestRejectsLinks` | +| 照合中の置換・変更 | `TestPinnedFilesResistReplacement`、`TestRejectsChangesBeforeRecheck` | +| 元client一致 | `TestRejectsProtectedResources` | +| data access の拡大 | `TestRejectsDataAccessExpansion` | +| closure の過不足 | `TestRejectsClosureGaps` | +| 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | + +symlinkの作成は権限が必要なため、合成テストは junction と hardlink で reparse 属性とリンク数の拒否を確かめる。 +symlink も同じ reparse 属性で拒否される。 + +## 実測と未完了 + +実装前Red、実装後Green、CI、別モデルのレビュー、merge、merge後のreaderによる実物の照合は、 +それぞれ完了した時点でこの節に追記する。 diff --git a/tools/todo-update-reader-3190/cli_windows_test.go b/tools/todo-update-reader-3190/cli_windows_test.go new file mode 100644 index 00000000..a2658a12 --- /dev/null +++ b/tools/todo-update-reader-3190/cli_windows_test.go @@ -0,0 +1,112 @@ +package main + +import ( + "encoding/json" + "sort" + "strings" + "testing" +) + +const cliTarget = `D:\CC\Tools\todo\todo.exe` + +func TestCLIAcceptsOnlyThePurposeFlag(t *testing.T) { + for _, args := range [][]string{ + {"--verify-todo-runtime-update", cliTarget}, + {"-verify-todo-runtime-update", cliTarget}, + {"--verify-todo-runtime-update=" + cliTarget}, + {"-verify-todo-runtime-update=" + cliTarget}, + } { + called := "" + var out, errOut strings.Builder + code := runCLI(args, &out, &errOut, func(path string) (qualification, string) { + called = path + return qualification{}, "baseline_mismatch" + }) + if called != cliTarget || code != 1 || out.String() != "{\"qualified\":false}\n" { + t.Errorf("%q: called %q code %d stdout %q", args, called, code, out.String()) + } + } + for _, args := range [][]string{ + {"--verify-todo-runtime-update"}, + {"--verify-todo-runtime-update", "todo.exe"}, + {"--verify-todo-runtime-update", `tools\todo\todo.exe`}, + {"--verify-todo-runtime-update", cliTarget, "extra"}, + {"--verify-todo-runtime-update", cliTarget, "--verify-todo-runtime-update", cliTarget}, + {"--manifest", "m.json", "--verify-todo-runtime-update", cliTarget}, + {"--verify-todo-runtime-update", cliTarget, "--sha256", strings.Repeat("0", 64)}, + {"--guard-path", cliTarget}, + {"--verify-capacity-runtime-update", cliTarget}, + {"--verify-model-runtime-update", cliTarget}, + {"--verify-todo-runtime-update="}, + {"--verify-todo-runtime-update=todo.exe"}, + {"verify-todo-runtime-update", cliTarget}, + {"---verify-todo-runtime-update", cliTarget}, + {"--Verify-Todo-Runtime-Update", cliTarget}, + {"--help", "--verify-todo-runtime-update", cliTarget}, + } { + called := false + var out, errOut strings.Builder + code := runCLI(args, &out, &errOut, func(string) (qualification, string) { + called = true + return qualification{Qualified: true}, "" + }) + if called || code != 1 || out.String() != "{\"qualified\":false}\n" || !strings.Contains(errOut.String(), "argument") { + t.Errorf("%q: called %v code %d stdout %q stderr %q", args, called, code, out.String(), errOut.String()) + } + } +} + +func TestCLIHelpAndUsage(t *testing.T) { + for _, args := range [][]string{{"-h"}, {"--help"}, {"-help"}} { + var out, errOut strings.Builder + code := runCLI(args, &out, &errOut, func(string) (qualification, string) { + t.Fatal("help must not verify") + return qualification{}, "" + }) + if code != 0 || out.Len() != 0 || !strings.Contains(errOut.String(), "-verify-todo-runtime-update") { + t.Errorf("%q: code %d stdout %q stderr %q", args, code, out.String(), errOut.String()) + } + } + var out, errOut strings.Builder + if code := runCLI(nil, &out, &errOut, nil); code != 2 || out.Len() != 0 || !strings.Contains(errOut.String(), "Usage") { + t.Errorf("no arguments: code %d stdout %q stderr %q", code, out.String(), errOut.String()) + } +} + +func TestCLIPrintsOnlyTheFixedQualification(t *testing.T) { + var out, errOut strings.Builder + code := runCLI([]string{"--verify-todo-runtime-update", cliTarget}, &out, &errOut, func(string) (qualification, string) { + return qualification{Qualified: true, Domain: "todo-runtime-update-3190-v1", TodoID: 3190, VerifierTodoID: 3463, TargetKind: "todo", BuildCommand: []string{"cargo"}}, "" + }) + if code != 0 || errOut.Len() != 0 { + t.Fatalf("code %d stderr %q", code, errOut.String()) + } + var got map[string]any + if err := json.Unmarshal([]byte(out.String()), &got); err != nil { + t.Fatal(err) + } + var keys []string + for k := range got { + keys = append(keys, k) + } + sort.Strings(keys) + want := "baseline_qualified_by baseline_source_receipt_sha256 build_command build_job build_log_sha256 build_receipt_sha256 current_sha256 data_access_granted dep_info_entries dep_info_sha256 domain inputs_verified local_crates new_bytes new_sha256 original_registry_recovered qualified source_head source_repository source_root source_tree target target_kind todo_id verifier_todo_id" + if strings.Join(keys, " ") != want { + t.Fatalf("keys %s\nwant %s", strings.Join(keys, " "), want) + } + if got["data_access_granted"] != false || got["qualified"] != true { + t.Fatalf("output %v", got) + } +} + +func TestCLIHidesRejectionDetails(t *testing.T) { + var out, errOut strings.Builder + leak := strings.Repeat("d", 64) + code := runCLI([]string{"--verify-todo-runtime-update", cliTarget}, &out, &errOut, func(string) (qualification, string) { + return qualification{CurrentSHA256: leak, SourceRoot: "D:/private"}, "baseline_mismatch" + }) + if code != 1 || out.String() != "{\"qualified\":false}\n" || !strings.Contains(errOut.String(), "baseline_mismatch") || + strings.Contains(out.String()+errOut.String(), leak) || strings.Contains(out.String()+errOut.String(), "private") { + t.Fatalf("code %d stdout %q stderr %q", code, out.String(), errOut.String()) + } +} diff --git a/tools/todo-update-reader-3190/go.mod b/tools/todo-update-reader-3190/go.mod new file mode 100644 index 00000000..2edf3771 --- /dev/null +++ b/tools/todo-update-reader-3190/go.mod @@ -0,0 +1,3 @@ +module cc/todo-update-reader-3190 + +go 1.26 diff --git a/tools/todo-update-reader-3190/main_windows.go b/tools/todo-update-reader-3190/main_windows.go new file mode 100644 index 00000000..9341fc6f --- /dev/null +++ b/tools/todo-update-reader-3190/main_windows.go @@ -0,0 +1,75 @@ +// Command todo-update-reader-3190 qualifies exactly one software update of +// D:/CC/Tools/todo/todo.exe for ToDo 3190. It is a non-distributed, +// read-only reader: it replaces nothing, reads only software and build +// evidence, and never grants data access. See docs/todo-update-reader-3190.md. +package main + +import ( + "encoding/json" + "fmt" + "io" + "os" + "path/filepath" + "strings" +) + +const purposeFlag = "verify-todo-runtime-update" + +func usage(w io.Writer) { + fmt.Fprintln(w, "Usage of todo-update-reader-3190:") + fmt.Fprintln(w, " -verify-todo-runtime-update string") + fmt.Fprintln(w, " \tqualify the one fixed ToDo3190 update of D:/CC/Tools/todo/todo.exe;") + fmt.Fprintln(w, " \treads software and build evidence only, replaces nothing, grants no data access") +} + +// runCLI accepts exactly one purpose flag with one absolute path. Any other +// shape is refused before verification starts, so no caller input can add a +// manifest, digest, source, root or grant. +func runCLI(args []string, stdout, stderr io.Writer, verify func(string) (qualification, string)) int { + if len(args) == 0 { + usage(stderr) + return 2 + } + if len(args) == 1 { + switch args[0] { + case "-h", "-help", "--help": + usage(stderr) + return 0 + } + } + result, reason := qualification{}, "argument" + if path, ok := parsePurpose(args); ok { + result, reason = verify(path) + } + if !result.Qualified { + // Only a fixed reason code; never a digest, path content or receipt text. + fmt.Fprintf(stderr, "todo-update-reader-3190: not qualified: %s\n", reason) + _ = json.NewEncoder(stdout).Encode(map[string]bool{"qualified": false}) + return 1 + } + if err := json.NewEncoder(stdout).Encode(result); err != nil { + return 1 + } + return 0 +} + +func parsePurpose(args []string) (string, bool) { + path := "" + switch { + case len(args) == 2 && (args[0] == "--"+purposeFlag || args[0] == "-"+purposeFlag): + path = args[1] + case len(args) == 1: + for _, prefix := range []string{"--" + purposeFlag + "=", "-" + purposeFlag + "="} { + if strings.HasPrefix(args[0], prefix) { + path = strings.TrimPrefix(args[0], prefix) + } + } + } + return path, path != "" && filepath.IsAbs(path) +} + +func main() { + os.Exit(runCLI(os.Args[1:], os.Stdout, os.Stderr, func(path string) (qualification, string) { + return verifyUpdate(path, productionProfile, productionEnvironment()) + })) +} diff --git a/tools/todo-update-reader-3190/profile_windows.go b/tools/todo-update-reader-3190/profile_windows.go new file mode 100644 index 00000000..c672844d --- /dev/null +++ b/tools/todo-update-reader-3190/profile_windows.go @@ -0,0 +1,131 @@ +package main + +import "path/filepath" + +// The qualification this reader issues. Field names are the public contract +// returned to the ToDo3190 installer; failures print only {"qualified":false}. +type qualification struct { + Qualified bool `json:"qualified"` + Domain string `json:"domain"` + TodoID int `json:"todo_id"` + VerifierTodoID int `json:"verifier_todo_id"` + TargetKind string `json:"target_kind"` + Target string `json:"target"` + CurrentSHA256 string `json:"current_sha256"` + NewSHA256 string `json:"new_sha256"` + NewBytes int64 `json:"new_bytes"` + SourceRepository string `json:"source_repository"` + SourceRoot string `json:"source_root"` + SourceHead string `json:"source_head"` + SourceTree string `json:"source_tree"` + BuildReceiptSHA256 string `json:"build_receipt_sha256"` + BuildJob string `json:"build_job"` + BuildCommand []string `json:"build_command"` + BuildLogSHA256 string `json:"build_log_sha256"` + DepInfoSHA256 string `json:"dep_info_sha256"` + InputsVerified int `json:"inputs_verified"` + DepInfoEntries int `json:"dep_info_entries"` + LocalCrates int `json:"local_crates"` + BaselineQualifiedBy string `json:"baseline_qualified_by"` + BaselineSourceReceiptSHA256 string `json:"baseline_source_receipt_sha256"` + DataAccessGranted bool `json:"data_access_granted"` + OriginalRegistryRecovered bool `json:"original_registry_recovered"` +} + +const ( + qualificationDomain = "todo-runtime-update-3190-v1" + qualificationTodo = 3190 + verifierTodo = 3463 + qualificationKind = "todo" +) + +// The installed baseline must first be qualified by the deployed guard's +// existing ToDo3085 capacity purpose. That old qualification only proves the +// current software; it is not renamed into this update's qualification. +type capacityExpectation struct { + domain string + todoID int64 + targetKind string + oldSHA string + receiptSHA string +} + +// The independent build receipt written by ToDo3190's normal Cargo run. +type receiptExpectation struct { + domain string + todoID int64 + job string + repository string + command []string + baselineQualifiedBy string +} + +// profile holds every compiled pin. Only tests construct other profiles; the +// CLI always uses productionProfile and accepts no caller-supplied values. +type profile struct { + target string + root string + head string + tree string + origins []string + mainRef string + receiptName string + receiptSHA string + logName string + logSHA string + depInfoName string + depInfoSHA string + candidateName string + candidateSHA string + candidateBytes int64 + baselineSHA string + capacity capacityExpectation + receipt receiptExpectation + inputs int + depEntries int + localCrates int +} + +var repositoryOrigins = []string{ + "https://github.com/autodevjapan/cc-pc-tools-rs.git", + "https://github.com/autodevjapan/cc-pc-tools-rs", + "git@github.com:autodevjapan/cc-pc-tools-rs.git", +} + +// One leaf, one baseline, one candidate, one source, one receipt. +var productionProfile = profile{ + target: filepath.Clean("D:/CC/Tools/todo/todo.exe"), + root: filepath.Clean("D:/CC/Work/unattended-operations-20261013-wt/todo-3190-3110"), + head: "0e16e81798ff4463c6ae6bcf6c4c57a695d9d13c", + tree: "31d45a745d27271420e764f6dff493e3d58b3731", + origins: repositoryOrigins, + mainRef: "refs/remotes/origin/main", + receiptName: "todo3190-build-0e16e817.private.json", + receiptSHA: "20345f443566c16553dbd959d5cd07b53b8aa0e1579a55bff1a614ab0f75087c", + logName: "todo3190-release-0e16e817.private.log", + logSHA: "1fe224b19ba4d5265711e0119e554a035aa15b7cc7f4b0da327478bafb9fe1ef", + depInfoName: "target/release/todo.d", + depInfoSHA: "da3ce6ae3fcebe72fb5cecb701fc23efa70571086a5cd73a7d7642aca1e96941", + candidateName: "todo3190-todo-0e16e817.private.exe", + candidateSHA: "d4392e6326f1868e2ffddc31a80e83d3170f93d639b1a04656066be387180718", + candidateBytes: 8017920, + baselineSHA: "f8710b516d3ecdd37564d7a1ab978c8689ab1bdd2ffaec4150d0e4bed4c0b983", + capacity: capacityExpectation{ + domain: "capacity-runtime-update-3085", + todoID: 3085, + targetKind: "todo", + oldSHA: "4b351db686ae315b884d75be952453c20204d25ab81eb1643df106e335b3364f", + receiptSHA: "a3bc15ae1a7b8eb9cc1f06dbf379b587eecea9f379e1eab48768372c91e46786", + }, + receipt: receiptExpectation{ + domain: "todo3190-production-build-v1", + todoID: 3190, + job: "20261024-001451-todo-3190", + repository: "AutoDevJapan/cc-pc-tools-rs", + command: []string{"cargo", "build", "--release", "--locked", "-p", "cc-todo"}, + baselineQualifiedBy: "scrub391 --verify-capacity-runtime-update D:/CC/Tools/todo/todo.exe", + }, + inputs: 126, + depEntries: 116, + localCrates: 8, +} diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go new file mode 100644 index 00000000..4cf4b01a --- /dev/null +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -0,0 +1,913 @@ +package main + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "sync" + "syscall" + "testing" + "unsafe" +) + +const ( + fixtureBaseline = "installed todo.exe from the earlier qualified distribution\n" + fixtureCandidate = "new todo.exe candidate built once from the fixed merged source\n" + fixtureOldSHA = "1111111111111111111111111111111111111111111111111111111111111111" + fixtureCapacityReceiptSHA = "2222222222222222222222222222222222222222222222222222222222222222" + fixtureReceiptName = "todo-build.private.json" + fixtureLogName = "todo-release.private.log" + fixtureCandidateName = "todo-candidate.private.exe" +) + +// A small workspace with the same shape as the real one: a virtual workspace +// manifest, two local crates, an include_str! input and tracked non-inputs. +var fixtureTracked = map[string]string{ + ".gitignore": "*.private.*\n/target/\n", + "Cargo.toml": "[workspace]\nresolver = \"3\"\nmembers = [\"app\", \"lib-a\"]\n", + "Cargo.lock": "# This file is automatically @generated by Cargo.\nversion = 4\n", + "README.md": "A tracked file that is not a compiler input.\n", + "app/Cargo.toml": "[package]\nname = \"app\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", + "app/src/main.rs": "fn main() { println!(\"{}\", include_str!(\"preamble.txt\")); }\n", + "app/src/preamble.txt": "preamble\n", + "app/src/unused.rs": "// tracked but not compiled\n", + "lib-a/Cargo.toml": "[package]\nname = \"lib-a\"\nversion = \"0.1.0\"\nedition = \"2024\"\n", + "lib-a/src/lib.rs": "pub fn a() {}\n", +} + +var ( + fixtureDeps = []string{"app/src/main.rs", "app/src/preamble.txt", "lib-a/src/lib.rs"} + fixtureManifests = []string{"Cargo.lock", "Cargo.toml", "app/Cargo.toml", "lib-a/Cargo.toml"} +) + +type fixtureInput struct { + Path string `json:"path"` + Blob string `json:"blob"` + SHA256 string `json:"sha256"` +} + +type fixture struct { + t *testing.T + base string + root string + target string + receipt map[string]any + inputs []fixtureInput + p profile + mu sync.Mutex + events []string + protected map[string]string + capacity string + capacityExit int + onCapacity func() + onRecheck func() +} + +func sha(data []byte) string { + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} + +func crlf(content string) string { return strings.ReplaceAll(content, "\n", "\r\n") } + +func newFixture(t *testing.T, edit func(map[string]string)) *fixture { + t.Helper() + tracked := map[string]string{} + for name, content := range fixtureTracked { + tracked[name] = content + } + if edit != nil { + edit(tracked) + } + base := testLongPath(t, t.TempDir()) + f := &fixture{ + t: t, base: base, root: filepath.Join(base, "source"), + target: filepath.Join(base, "tools", "todo", "todo.exe"), + protected: map[string]string{}, + } + if err := os.MkdirAll(f.root, 0o755); err != nil { + t.Fatal(err) + } + f.git("init", "-q", "-b", "main") + f.git("config", "core.autocrlf", "true") + f.git("remote", "add", "origin", "https://github.com/AutoDevJapan/cc-pc-tools-rs.git") + for name, content := range tracked { + // Mirror the Windows checkout: CRLF working files, LF blobs. + f.write(name, crlf(content)) + } + f.git("add", "-A") + f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "-m", "fixture") + f.git("update-ref", "refs/remotes/origin/main", "HEAD") + head, tree := f.gitOut("rev-parse", "HEAD"), f.gitOut("rev-parse", "HEAD^{tree}") + + writeFile(t, f.target, fixtureBaseline) + f.write(fixtureCandidateName, fixtureCandidate) + var deps []string + for _, dep := range fixtureDeps { + deps = append(deps, f.path(dep)) + } + depInfo := strings.ToLower(filepath.Join(f.root, "target", "release", "todo.exe")) + ": " + strings.Join(deps, " ") + "\n" + f.write("target/release/todo.d", depInfo) + log := " Compiling serde v1.0.229\n" + + " Compiling lib-a v0.1.0 (" + f.path("lib-a") + ")\n" + + " Compiling app v0.1.0 (" + f.path("app") + ")\n" + + " Finished `release` profile [optimized] target(s) in 1.25s\n" + f.write(fixtureLogName, log) + + names := append(append([]string{}, fixtureManifests...), fixtureDeps...) + sort.Strings(names) + for _, name := range names { + f.inputs = append(f.inputs, f.input(name)) + } + f.p = profile{ + target: f.target, root: f.root, head: head, tree: tree, + origins: repositoryOrigins, mainRef: "refs/remotes/origin/main", + receiptName: fixtureReceiptName, + logName: fixtureLogName, logSHA: sha([]byte(log)), + depInfoName: "target/release/todo.d", depInfoSHA: sha([]byte(depInfo)), + candidateName: fixtureCandidateName, candidateSHA: sha([]byte(fixtureCandidate)), candidateBytes: int64(len(fixtureCandidate)), + baselineSHA: sha([]byte(fixtureBaseline)), + capacity: capacityExpectation{ + domain: "capacity-runtime-update-3085", todoID: 3085, targetKind: "todo", + oldSHA: fixtureOldSHA, receiptSHA: fixtureCapacityReceiptSHA, + }, + receipt: productionProfile.receipt, + inputs: len(names), + depEntries: len(fixtureDeps), + localCrates: 2, + } + f.receipt = map[string]any{ + "domain": f.p.receipt.domain, + "todo_id": f.p.receipt.todoID, + "job": f.p.receipt.job, + "at": "2026-10-24T00:44:51.4600710+09:00", + "repository": f.p.receipt.repository, + "source_root": f.root, + "source_head": head, + "source_tree": tree, + "command": f.p.receipt.command, + "compiler_exit_code": 0, + "log_file": fixtureLogName, + "log_sha256": f.p.logSHA, + "dep_info_file": "target/release/todo.d", + "dep_info_sha256": f.p.depInfoSHA, + "artifact": fixtureCandidateName, + "artifact_sha256": f.p.candidateSHA, + "artifact_bytes": f.p.candidateBytes, + "baseline_qualified_by": f.p.receipt.baselineQualifiedBy, + "baseline_sha256": f.p.baselineSHA, + "functional_test_status": "pending Go integration on merged source", + "data_access_granted": false, + } + f.writeReceipt() + f.capacity = f.capacityReport(nil) + return f +} + +func (f *fixture) path(rel string) string { return filepath.Join(f.root, filepath.FromSlash(rel)) } + +func (f *fixture) write(rel, content string) { + f.t.Helper() + writeFile(f.t, f.path(rel), content) +} + +func writeFile(t *testing.T, path, content string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} + +func (f *fixture) input(rel string) fixtureInput { + f.t.Helper() + data, err := os.ReadFile(f.path(rel)) + if err != nil { + f.t.Fatal(err) + } + return fixtureInput{Path: rel, Blob: f.gitOut("rev-parse", "HEAD:"+rel), SHA256: sha(data)} +} + +func testGit(t *testing.T, root string, args ...string) (string, error) { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", root}, args...)...) + cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} + cmd.Env = childEnvironment(os.Environ()) + out, err := cmd.CombinedOutput() + return strings.TrimSpace(string(out)), err +} + +func (f *fixture) git(args ...string) { + f.t.Helper() + if out, err := testGit(f.t, f.root, args...); err != nil { + f.t.Fatalf("git %v: %v\n%s", args, err, out) + } +} + +func (f *fixture) gitOut(args ...string) string { + f.t.Helper() + out, err := testGit(f.t, f.root, args...) + if err != nil { + f.t.Fatalf("git %v: %v\n%s", args, err, out) + } + return out +} + +// writeReceipt stores the receipt and pins its new digest, as the parent did. +func (f *fixture) writeReceipt() { + f.t.Helper() + f.receipt["inputs"] = f.inputs + data, err := json.MarshalIndent(f.receipt, "", " ") + if err != nil { + f.t.Fatal(err) + } + f.write(fixtureReceiptName, string(data)) + f.p.receiptSHA = sha(data) + f.p.inputs = len(f.inputs) +} + +func (f *fixture) setLog(content string) { + f.write(fixtureLogName, content) + f.p.logSHA = sha([]byte(content)) + f.receipt["log_sha256"] = f.p.logSHA + f.writeReceipt() +} + +func (f *fixture) setDepInfo(content string, entries int) { + f.write("target/release/todo.d", content) + f.p.depInfoSHA = sha([]byte(content)) + f.p.depEntries = entries + f.receipt["dep_info_sha256"] = f.p.depInfoSHA + f.writeReceipt() +} + +func (f *fixture) capacityReport(edit func(map[string]any)) string { + report := map[string]any{ + "data_access_granted": false, "domain": "capacity-runtime-update-3085", + "new_sha256": f.p.baselineSHA, "old_sha256": fixtureOldSHA, + "original_registry_recovered": false, "qualified": true, + "source_receipt_sha256": fixtureCapacityReceiptSHA, "target_kind": "todo", "todo_id": 3085, + } + if edit != nil { + edit(report) + } + data, err := json.Marshal(report) + if err != nil { + f.t.Fatal(err) + } + return string(data) +} + +func (f *fixture) record(event string) { + f.mu.Lock() + defer f.mu.Unlock() + f.events = append(f.events, event) +} + +func key(path string) string { return strings.ToLower(filepath.Clean(path)) } + +const clearVerdict = `{"access_allowed":true,"ci_retirement_qualified":false,"classification_unverified":false,"client_resource":false,"ok":true,"original_registry_recovered":false,"protection_categories":[],"protection_status":"clear","scope":"fixture"}` +const clientVerdict = `{"access_allowed":false,"ci_retirement_qualified":false,"classification_unverified":false,"client_resource":true,"ok":true,"original_registry_recovered":false,"protection_categories":["client_project"],"protection_status":"protected","scope":"fixture"}` +const unverifiedVerdict = `{"access_allowed":false,"ci_retirement_qualified":false,"classification_unverified":true,"client_resource":true,"ok":true,"original_registry_recovered":false,"protection_categories":[],"protection_status":"unverified","scope":"fixture"}` + +// guard imitates the two existing guard CLI entries the reader may call. +func (f *fixture) guard(args ...string) ([]byte, int, bool) { + switch { + case len(args) == 2 && args[0] == "--guard-path": + f.record("guard-path:" + key(args[1])) + f.mu.Lock() + verdict, ok := f.protected[key(args[1])] + f.mu.Unlock() + if !ok { + verdict = clearVerdict + } + if verdict == "exit-2" { + return []byte(clearVerdict + "\n"), 2, true + } + return []byte(verdict + "\n"), 0, true + case len(args) == 2 && args[0] == "--verify-capacity-runtime-update": + f.record("capacity:" + key(args[1])) + if f.onCapacity != nil { + f.onCapacity() + } + return []byte(f.capacity + "\n"), f.capacityExit, true + } + f.record(fmt.Sprintf("unexpected:%v", args)) + return nil, 2, true +} + +func (f *fixture) env() environment { + return environment{ + guard: f.guard, + git: runGit, + beforeRead: func(path string) { f.record("read:" + key(path)) }, + beforeRecheck: func() { + if f.onRecheck != nil { + f.onRecheck() + } + }, + } +} + +func (f *fixture) verify() (qualification, string) { + return verifyUpdate(f.target, f.p, f.env()) +} + +func (f *fixture) index(event string) int { + f.mu.Lock() + defer f.mu.Unlock() + for i, e := range f.events { + if e == event { + return i + } + } + return -1 +} + +func expectReason(t *testing.T, f *fixture, want string) { + t.Helper() + got, reason := f.verify() + if got.Qualified || reason != want { + t.Fatalf("verify = qualified %v reason %q, want rejection %q", got.Qualified, reason, want) + } + if got.CurrentSHA256 != "" || got.NewSHA256 != "" || got.SourceHead != "" { + t.Fatalf("rejection leaked a partial qualification: %+v", got) + } +} + +var procFinalPath = syscall.NewLazyDLL("kernel32.dll").NewProc("GetFinalPathNameByHandleW") + +// testLongPath resolves 8.3 names and links in the temporary directory itself, +// so fixtures start from the same physical spelling the reader compares. +func testLongPath(t *testing.T, dir string) string { + t.Helper() + name, err := syscall.UTF16PtrFromString(dir) + if err != nil { + t.Fatal(err) + } + h, err := syscall.CreateFile(name, 0x80, syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE|syscall.FILE_SHARE_DELETE, nil, syscall.OPEN_EXISTING, 0x02000000, 0) + if err != nil { + t.Fatal(err) + } + defer syscall.CloseHandle(h) + buf := make([]uint16, 32768) + n, _, _ := procFinalPath.Call(uintptr(h), uintptr(unsafe.Pointer(&buf[0])), uintptr(len(buf)), 0) + if n == 0 || n >= uintptr(len(buf)) { + t.Fatalf("final path of %s", dir) + } + return strings.TrimPrefix(syscall.UTF16ToString(buf[:n]), "\\\\?\\") +} + +func junction(t *testing.T, link, target string) { + t.Helper() + cmd := exec.Command("cmd", "/c", "mklink", "/J", link, target) + cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("mklink /J: %v\n%s", err, out) + } +} + +func TestQualifiesFixedUpdate(t *testing.T) { + f := newFixture(t, nil) + got, reason := f.verify() + if !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q, want the fixed update qualified", got.Qualified, reason) + } + want := qualification{ + Qualified: true, Domain: "todo-runtime-update-3190-v1", TodoID: 3190, VerifierTodoID: 3463, TargetKind: "todo", + Target: filepath.ToSlash(f.target), CurrentSHA256: f.p.baselineSHA, NewSHA256: f.p.candidateSHA, NewBytes: f.p.candidateBytes, + SourceRepository: "AutoDevJapan/cc-pc-tools-rs", SourceRoot: filepath.ToSlash(f.root), SourceHead: f.p.head, SourceTree: f.p.tree, + BuildReceiptSHA256: f.p.receiptSHA, BuildJob: "20261024-001451-todo-3190", BuildCommand: productionProfile.receipt.command, + BuildLogSHA256: f.p.logSHA, DepInfoSHA256: f.p.depInfoSHA, InputsVerified: 7, DepInfoEntries: 3, LocalCrates: 2, + BaselineQualifiedBy: "capacity-runtime-update-3085", BaselineSourceReceiptSHA256: fixtureCapacityReceiptSHA, + } + gotJSON, _ := json.Marshal(got) + wantJSON, _ := json.Marshal(want) + if string(gotJSON) != string(wantJSON) { + t.Fatalf("qualification\n got %s\nwant %s", gotJSON, wantJSON) + } + if !strings.Contains(string(gotJSON), `"data_access_granted":false`) { + t.Fatalf("data access must be explicitly false: %s", gotJSON) + } + // Every file the reader reads is classified by the official guard first; + // the installed leaf is qualified only by the existing purpose entry. + paths := []string{f.root, f.path(fixtureReceiptName), f.path(fixtureLogName), f.path("target/release/todo.d"), f.path(fixtureCandidateName)} + for _, in := range f.inputs { + paths = append(paths, f.path(in.Path)) + } + for _, path := range paths { + classified := f.index("guard-path:" + key(path)) + read := f.index("read:" + key(path)) + if classified < 0 { + t.Errorf("%s was not classified by the guard", path) + } + if read >= 0 && read < classified { + t.Errorf("%s was read before its guard classification", path) + } + } + if f.index("guard-path:"+key(f.target)) >= 0 { + t.Errorf("the general classifier was used for the installed leaf") + } +} + +func TestBaselineIsReadOnlyAfterPurposeQualification(t *testing.T) { + f := newFixture(t, nil) + f.capacity, f.capacityExit = `{"qualified":false}`, 1 + expectReason(t, f, "baseline_qualification") + if f.index("read:"+key(f.target)) >= 0 { + t.Fatalf("the installed leaf was read without its purpose qualification: %v", f.events) + } + + f = newFixture(t, nil) + if got, reason := f.verify(); !got.Qualified { + t.Fatalf("verify rejected %q", reason) + } + capacity, read := f.index("capacity:"+key(f.target)), f.index("read:"+key(f.target)) + if capacity < 0 || read < 0 || read < capacity { + t.Fatalf("installed leaf read at %d before purpose qualification at %d: %v", read, capacity, f.events) + } +} + +func TestRejectsBaselineMismatch(t *testing.T) { + f := newFixture(t, nil) + writeFile(t, f.target, "an unknown executable that the guard purpose accepted\n") + expectReason(t, f, "baseline_mismatch") + + // The CLI never reveals the unknown digest. + var out, errOut strings.Builder + code := runCLI([]string{"--verify-todo-runtime-update", f.target}, &out, &errOut, func(path string) (qualification, string) { + return verifyUpdate(path, f.p, f.env()) + }) + unknown := sha([]byte("an unknown executable that the guard purpose accepted\n")) + if code != 1 || out.String() != "{\"qualified\":false}\n" || strings.Contains(out.String()+errOut.String(), unknown) { + t.Fatalf("code %d stdout %q stderr %q", code, out.String(), errOut.String()) + } +} + +func TestRejectsCandidateMismatch(t *testing.T) { + f := newFixture(t, nil) + f.write(fixtureCandidateName, "a different candidate\n") + expectReason(t, f, "candidate") + + f = newFixture(t, nil) + f.p.candidateSHA = sha([]byte("another pinned candidate")) + f.receipt["artifact_sha256"] = f.p.candidateSHA + f.writeReceipt() + expectReason(t, f, "candidate") +} + +func TestRejectsTamperedOrMissingEvidence(t *testing.T) { + cases := []struct { + name, reason string + edit func(*fixture) + }{ + {"receipt bytes changed", "receipt", func(f *fixture) { + data, _ := os.ReadFile(f.path(fixtureReceiptName)) + f.write(fixtureReceiptName, string(data)+" ") + }}, + {"receipt missing", "receipt", func(f *fixture) { os.Remove(f.path(fixtureReceiptName)) }}, + {"receipt names another head", "receipt", func(f *fixture) { + f.receipt["source_head"] = strings.Repeat("a", 40) + f.writeReceipt() + }}, + {"receipt names another source root", "receipt", func(f *fixture) { + f.receipt["source_root"] = filepath.Join(f.base, "elsewhere") + f.writeReceipt() + }}, + {"receipt has an unknown field", "receipt", func(f *fixture) { + f.receipt["allowed_directories"] = []string{f.base} + f.writeReceipt() + }}, + {"receipt duplicates a key", "receipt", func(f *fixture) { + data, _ := os.ReadFile(f.path(fixtureReceiptName)) + edited := strings.Replace(string(data), "{", "{\n \"artifact_sha256\": \""+strings.Repeat("0", 64)+"\",", 1) + f.write(fixtureReceiptName, edited) + f.p.receiptSHA = sha([]byte(edited)) + }}, + {"receipt reports a failed compiler", "receipt", func(f *fixture) { + f.receipt["compiler_exit_code"] = 101 + f.writeReceipt() + }}, + {"receipt omits the compiler result", "receipt", func(f *fixture) { + delete(f.receipt, "compiler_exit_code") + f.writeReceipt() + }}, + {"receipt uses another command", "receipt", func(f *fixture) { + f.receipt["command"] = []string{"cargo", "build", "--release", "-p", "cc-todo"} + f.writeReceipt() + }}, + {"receipt names another baseline", "receipt", func(f *fixture) { + f.receipt["baseline_sha256"] = fixtureOldSHA + f.writeReceipt() + }}, + {"receipt input count differs from the pin", "receipt", func(f *fixture) { f.p.inputs++ }}, + {"input sha256 differs", "input", func(f *fixture) { + f.inputs[0].SHA256 = strings.Repeat("3", 64) + f.writeReceipt() + }}, + {"input blob differs", "input", func(f *fixture) { + f.inputs[0].Blob = strings.Repeat("4", 40) + f.writeReceipt() + }}, + {"input is not in the fixed tree", "input", func(f *fixture) { + f.write("app/src/untracked.rs", "fn x() {}\n") + f.inputs = append(f.inputs, fixtureInput{Path: "app/src/untracked.rs", Blob: strings.Repeat("5", 40), SHA256: sha([]byte("fn x() {}\n"))}) + f.writeReceipt() + }}, + {"input path escapes the source", "receipt", func(f *fixture) { + f.inputs[0].Path = "../outside.rs" + f.writeReceipt() + }}, + {"input path names an alternate stream", "receipt", func(f *fixture) { + f.inputs[0].Path = "Cargo.lock:stream" + f.writeReceipt() + }}, + {"input path repeated with other case", "receipt", func(f *fixture) { + f.inputs = append(f.inputs, f.inputs[0]) + f.inputs[len(f.inputs)-1].Path = strings.ToUpper(f.inputs[0].Path) + f.writeReceipt() + }}, + {"log bytes changed", "log", func(f *fixture) { + data, _ := os.ReadFile(f.path(fixtureLogName)) + f.write(fixtureLogName, string(data)+"\n") + }}, + {"log missing", "log", func(f *fixture) { os.Remove(f.path(fixtureLogName)) }}, + {"log reports an error", "log", func(f *fixture) { + f.setLog(" Compiling lib-a v0.1.0 (" + f.path("lib-a") + ")\nerror: could not compile `app`\n Finished `release` profile [optimized] target(s) in 1.25s\n") + }}, + {"log does not finish the release build", "log", func(f *fixture) { + f.setLog(" Compiling lib-a v0.1.0 (" + f.path("lib-a") + ")\n Compiling app v0.1.0 (" + f.path("app") + ")\n") + }}, + {"log compiles a local path outside the source", "log", func(f *fixture) { + f.setLog(" Compiling lib-a v0.1.0 (" + f.path("lib-a") + ")\n Compiling app v0.1.0 (" + f.path("app") + ")\n Compiling helper v0.1.0 (" + filepath.Join(f.base, "helper") + ")\n Finished `release` profile [optimized] target(s) in 1.25s\n") + }}, + {"dep-info bytes changed", "dep_info", func(f *fixture) { + data, _ := os.ReadFile(f.path("target/release/todo.d")) + f.write("target/release/todo.d", strings.TrimSuffix(string(data), "\n")+" \n") + }}, + {"dep-info missing", "dep_info", func(f *fixture) { os.Remove(f.path("target/release/todo.d")) }}, + {"dep-info describes another artifact", "dep_info", func(f *fixture) { + data, _ := os.ReadFile(f.path("target/release/todo.d")) + f.setDepInfo(strings.Replace(string(data), "todo.exe:", "other.exe:", 1), 3) + }}, + {"dep-info reads a file outside the source", "dep_info", func(f *fixture) { + data, _ := os.ReadFile(f.path("target/release/todo.d")) + f.setDepInfo(strings.TrimSuffix(string(data), "\n")+" "+filepath.Join(f.base, "outside.rs")+"\n", 4) + }}, + {"candidate missing", "candidate", func(f *fixture) { os.Remove(f.path(fixtureCandidateName)) }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t, nil) + tc.edit(f) + expectReason(t, f, tc.reason) + }) + } +} + +func TestRejectsSourceChanges(t *testing.T) { + cases := []struct { + name string + edit func(*fixture) + }{ + {"another origin", func(f *fixture) { + f.git("remote", "set-url", "origin", "https://github.com/someone-else/cc-pc-tools-rs.git") + }}, + {"no origin", func(f *fixture) { f.git("remote", "remove", "origin") }}, + {"HEAD moved after the build", func(f *fixture) { + f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "later") + }}, + {"tracked input edited", func(f *fixture) { f.write("lib-a/src/lib.rs", crlf("pub fn a() { panic!() }\n")) }}, + {"tracked input deleted", func(f *fixture) { os.Remove(f.path("lib-a/src/lib.rs")) }}, + {"another tree is pinned", func(f *fixture) { f.p.tree = strings.Repeat("6", 40) }}, + {"head not merged into origin/main", func(f *fixture) { f.git("update-ref", "-d", "refs/remotes/origin/main") }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t, nil) + tc.edit(f) + expectReason(t, f, "source_git") + }) + } + t.Run("source root is a subdirectory of the worktree", func(t *testing.T) { + f := newFixture(t, nil) + f.p.root = filepath.Join(f.root, "app") + expectReason(t, f, "source_git") + }) +} + +func TestRejectsOtherPaths(t *testing.T) { + f := newFixture(t, nil) + writeFile(t, filepath.Join(filepath.Dir(f.target), "todo.json"), "{}") + writeFile(t, filepath.Join(filepath.Dir(f.target), "todo-other.exe"), fixtureBaseline) + for _, path := range []string{ + filepath.Dir(f.target), + filepath.Join(filepath.Dir(f.target), "todo.json"), + filepath.Join(filepath.Dir(f.target), "todo-other.exe"), + f.target + ":stream", + f.target + "::$DATA", + f.target + ".", + f.target + " ", + "\\\\?\\" + f.target, + "\\\\.\\" + f.target, + "todo.exe", + f.path(fixtureCandidateName), + } { + got, reason := verifyUpdate(path, f.p, f.env()) + if got.Qualified || reason != "target_path" { + t.Errorf("verify(%q) = qualified %v reason %q, want target_path", path, got.Qualified, reason) + } + } + if f.index("capacity:"+key(f.target)) >= 0 { + t.Fatalf("a refused path reached the purpose entry: %v", f.events) + } + for _, spelling := range []string{filepath.ToSlash(f.target), strings.ToUpper(f.target)} { + if got, reason := verifyUpdate(spelling, f.p, f.env()); !got.Qualified { + t.Errorf("verify(%q) rejected %q; the same leaf spelled with other separators or case must match", spelling, reason) + } + } +} + +func TestRejectsLinks(t *testing.T) { + t.Run("installed leaf has a second hard link", func(t *testing.T) { + f := newFixture(t, nil) + if err := os.Link(f.target, filepath.Join(f.base, "alias.exe")); err != nil { + t.Fatal(err) + } + expectReason(t, f, "target_leaf") + }) + t.Run("candidate has a second hard link", func(t *testing.T) { + f := newFixture(t, nil) + if err := os.Link(f.path(fixtureCandidateName), filepath.Join(f.base, "candidate-alias.exe")); err != nil { + t.Fatal(err) + } + expectReason(t, f, "candidate") + }) + t.Run("receipt has a second hard link", func(t *testing.T) { + f := newFixture(t, nil) + if err := os.Link(f.path(fixtureReceiptName), filepath.Join(f.base, "receipt-alias.json")); err != nil { + t.Fatal(err) + } + expectReason(t, f, "receipt") + }) + t.Run("input has a second hard link", func(t *testing.T) { + f := newFixture(t, nil) + if err := os.Link(f.path("lib-a/src/lib.rs"), filepath.Join(f.base, "lib-alias.rs")); err != nil { + t.Fatal(err) + } + expectReason(t, f, "input") + }) + t.Run("installed leaf directory is a junction", func(t *testing.T) { + f := newFixture(t, nil) + dir := filepath.Dir(f.target) + if err := os.Rename(dir, dir+"-real"); err != nil { + t.Fatal(err) + } + junction(t, dir, dir+"-real") + expectReason(t, f, "target_leaf") + }) + t.Run("installed leaf ancestor is a junction", func(t *testing.T) { + f := newFixture(t, nil) + tools := filepath.Join(f.base, "tools") + if err := os.Rename(tools, tools+"-real"); err != nil { + t.Fatal(err) + } + junction(t, tools, tools+"-real") + expectReason(t, f, "target_leaf") + }) + t.Run("source root is reached through a junction", func(t *testing.T) { + f := newFixture(t, nil) + linked := filepath.Join(f.base, "linked-source") + junction(t, linked, f.root) + f.p.root = linked + f.receipt["source_root"] = linked + f.writeReceipt() + expectReason(t, f, "source_root") + }) +} + +func TestPinnedFilesResistReplacement(t *testing.T) { + f := newFixture(t, nil) + var refused []string + attempt := func(name string, op func() error) { + if err := op(); err != nil { + refused = append(refused, name) + } + } + f.onCapacity = func() { + attempt("write installed leaf", func() error { return os.WriteFile(f.target, []byte("replaced"), 0o644) }) + attempt("rename installed leaf", func() error { return os.Rename(f.target, f.target+".moved") }) + attempt("delete installed leaf", func() error { return os.Remove(f.target) }) + } + f.onRecheck = func() { + attempt("write candidate", func() error { return os.WriteFile(f.path(fixtureCandidateName), []byte("replaced"), 0o644) }) + attempt("rename receipt", func() error { return os.Rename(f.path(fixtureReceiptName), f.path("moved.json")) }) + attempt("write input", func() error { return os.WriteFile(f.path("lib-a/src/lib.rs"), []byte("replaced"), 0o644) }) + attempt("rename source root", func() error { return os.Rename(f.root, f.root+"-moved") }) + } + got, reason := f.verify() + if !got.Qualified { + t.Fatalf("verify rejected %q after refused replacements %v", reason, refused) + } + want := []string{"write installed leaf", "rename installed leaf", "delete installed leaf", "write candidate", "rename receipt", "write input", "rename source root"} + if strings.Join(refused, ",") != strings.Join(want, ",") { + t.Fatalf("refused %v, want every replacement of pinned evidence refused %v", refused, want) + } +} + +func TestRejectsChangesBeforeRecheck(t *testing.T) { + cases := map[string]func(*fixture){ + "tracked non-input edited": func(f *fixture) { f.write("README.md", "changed during verification\n") }, + "new commit": func(f *fixture) { + f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "racing") + }, + "origin changed": func(f *fixture) { + f.git("remote", "set-url", "origin", "https://github.com/someone-else/cc-pc-tools-rs.git") + }, + } + for name, change := range cases { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + f.onRecheck = func() { change(f) } + expectReason(t, f, "recheck") + }) + } +} + +func TestRejectsProtectedResources(t *testing.T) { + t.Run("purpose entry refuses the installed leaf", func(t *testing.T) { + f := newFixture(t, nil) + f.capacity, f.capacityExit = `{"qualified":false}`, 1 + expectReason(t, f, "baseline_qualification") + }) + t.Run("purpose entry fails after printing a qualification", func(t *testing.T) { + f := newFixture(t, nil) + f.capacityExit = 1 + expectReason(t, f, "baseline_qualification") + }) + t.Run("source root is a client resource", func(t *testing.T) { + f := newFixture(t, nil) + f.protected[key(f.root)] = clientVerdict + expectReason(t, f, "source_root") + }) + for _, rel := range []string{fixtureReceiptName, fixtureLogName, "target/release/todo.d", "lib-a/src/lib.rs", "Cargo.lock", fixtureCandidateName} { + for name, verdict := range map[string]string{"client": clientVerdict, "unverified": unverifiedVerdict, "guard failure": "exit-2"} { + t.Run(rel+" "+name, func(t *testing.T) { + f := newFixture(t, nil) + f.protected[key(f.path(rel))] = verdict + expectReason(t, f, "guard_classification") + }) + } + } +} + +func TestRejectsDataAccessExpansion(t *testing.T) { + for name, edit := range map[string]func(map[string]any){ + "purpose grants data access": func(m map[string]any) { m["data_access_granted"] = true }, + "purpose adds a scope": func(m map[string]any) { m["data_access_scope"] = "D:/CC/Tools/todo" }, + "purpose omits data access": func(m map[string]any) { delete(m, "data_access_granted") }, + "purpose for another kind": func(m map[string]any) { m["target_kind"] = "capacity" }, + "purpose for another domain": func(m map[string]any) { m["domain"] = "model-runtime-update-3081-v1" }, + "purpose for another update": func(m map[string]any) { m["new_sha256"] = fixtureOldSHA }, + "purpose recovered a registry": func(m map[string]any) { m["original_registry_recovered"] = true }, + "purpose todo id as text": func(m map[string]any) { m["todo_id"] = "3085" }, + } { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + f.capacity = f.capacityReport(edit) + expectReason(t, f, "baseline_qualification") + }) + } + t.Run("receipt grants data access", func(t *testing.T) { + f := newFixture(t, nil) + f.receipt["data_access_granted"] = true + f.writeReceipt() + expectReason(t, f, "receipt") + }) + t.Run("data files beside the leaf are never qualified", func(t *testing.T) { + f := newFixture(t, nil) + for _, name := range []string{"todo.json", "todo.json.lock", "history.jsonl", "specs"} { + if got, reason := verifyUpdate(filepath.Join(filepath.Dir(f.target), name), f.p, f.env()); got.Qualified || reason != "target_path" { + t.Errorf("%s: qualified %v reason %q", name, got.Qualified, reason) + } + } + }) +} + +func TestRejectsClosureGaps(t *testing.T) { + t.Run("dep-info names a file missing from the inputs", func(t *testing.T) { + f := newFixture(t, nil) + data, _ := os.ReadFile(f.path("target/release/todo.d")) + f.setDepInfo(strings.TrimSuffix(string(data), "\n")+" "+f.path("app/src/unused.rs")+"\n", 4) + expectReason(t, f, "closure") + }) + t.Run("inputs include a file the compiler did not read", func(t *testing.T) { + f := newFixture(t, nil) + f.inputs = append(f.inputs, f.input("app/src/unused.rs")) + f.writeReceipt() + expectReason(t, f, "closure") + }) + t.Run("a crate manifest is missing", func(t *testing.T) { + f := newFixture(t, nil) + var kept []fixtureInput + for _, in := range f.inputs { + if in.Path != "lib-a/Cargo.toml" { + kept = append(kept, in) + } + } + f.inputs = kept + f.writeReceipt() + expectReason(t, f, "closure") + }) + t.Run("the lock file is missing", func(t *testing.T) { + f := newFixture(t, nil) + var kept []fixtureInput + for _, in := range f.inputs { + if in.Path != "Cargo.lock" { + kept = append(kept, in) + } + } + f.inputs = kept + f.writeReceipt() + expectReason(t, f, "closure") + }) + t.Run("the log compiled another set of local crates", func(t *testing.T) { + f := newFixture(t, nil) + f.setLog(" Compiling app v0.1.0 (" + f.path("app") + ")\n Finished `release` profile [optimized] target(s) in 1.25s\n") + expectReason(t, f, "closure") + }) + t.Run("dep entry count differs from the pin", func(t *testing.T) { + f := newFixture(t, nil) + f.p.depEntries++ + expectReason(t, f, "closure") + }) + t.Run("local crate count differs from the pin", func(t *testing.T) { + f := newFixture(t, nil) + f.p.localCrates++ + expectReason(t, f, "closure") + }) + for name, edit := range map[string]func(map[string]string){ + "build script": func(m map[string]string) { m["lib-a/build.rs"] = "fn main() {}\n" }, + "cargo config": func(m map[string]string) { m[".cargo/config.toml"] = "[build]\nrustflags = []\n" }, + "crate cargo config": func(m map[string]string) { m["app/.cargo/config"] = "[build]\n" }, + "toolchain file": func(m map[string]string) { m["rust-toolchain.toml"] = "[toolchain]\nchannel = \"stable\"\n" }, + "declared build key": func(m map[string]string) { + m["lib-a/Cargo.toml"] = "[package]\nname = \"lib-a\"\nversion = \"0.1.0\"\nbuild = \"src/gen.rs\"\n" + }, + } { + t.Run("tree has an uncovered "+name, func(t *testing.T) { + f := newFixture(t, edit) + expectReason(t, f, "closure") + }) + } +} + +func TestProductionProfileBindsOneFixedUpdate(t *testing.T) { + p := productionProfile + checks := map[string][2]string{ + "target": {p.target, `D:\CC\Tools\todo\todo.exe`}, + "root": {p.root, `D:\CC\Work\unattended-operations-20261013-wt\todo-3190-3110`}, + "head": {p.head, "0e16e81798ff4463c6ae6bcf6c4c57a695d9d13c"}, + "tree": {p.tree, "31d45a745d27271420e764f6dff493e3d58b3731"}, + "mainRef": {p.mainRef, "refs/remotes/origin/main"}, + "receipt": {p.receiptName + " " + p.receiptSHA, "todo3190-build-0e16e817.private.json 20345f443566c16553dbd959d5cd07b53b8aa0e1579a55bff1a614ab0f75087c"}, + "log": {p.logName + " " + p.logSHA, "todo3190-release-0e16e817.private.log 1fe224b19ba4d5265711e0119e554a035aa15b7cc7f4b0da327478bafb9fe1ef"}, + "dep-info": {p.depInfoName + " " + p.depInfoSHA, "target/release/todo.d da3ce6ae3fcebe72fb5cecb701fc23efa70571086a5cd73a7d7642aca1e96941"}, + "candidate": {fmt.Sprintf("%s %s %d", p.candidateName, p.candidateSHA, p.candidateBytes), "todo3190-todo-0e16e817.private.exe d4392e6326f1868e2ffddc31a80e83d3170f93d639b1a04656066be387180718 8017920"}, + "baseline": {p.baselineSHA, "f8710b516d3ecdd37564d7a1ab978c8689ab1bdd2ffaec4150d0e4bed4c0b983"}, + "capacity": {fmt.Sprintf("%s %d %s %s %s", p.capacity.domain, p.capacity.todoID, p.capacity.targetKind, p.capacity.oldSHA, p.capacity.receiptSHA), "capacity-runtime-update-3085 3085 todo 4b351db686ae315b884d75be952453c20204d25ab81eb1643df106e335b3364f a3bc15ae1a7b8eb9cc1f06dbf379b587eecea9f379e1eab48768372c91e46786"}, + "build receipt": {fmt.Sprintf("%s %d %s %s %v", p.receipt.domain, p.receipt.todoID, p.receipt.job, p.receipt.repository, p.receipt.command), "todo3190-production-build-v1 3190 20261024-001451-todo-3190 AutoDevJapan/cc-pc-tools-rs [cargo build --release --locked -p cc-todo]"}, + "baseline by": {p.receipt.baselineQualifiedBy, "scrub391 --verify-capacity-runtime-update D:/CC/Tools/todo/todo.exe"}, + "counts": {fmt.Sprintf("%d %d %d", p.inputs, p.depEntries, p.localCrates), "126 116 8"}, + "guard": {guardExecutable, `D:\CC\Tools\client-records\scrub391.exe`}, + "origins": {strings.Join(p.origins, " "), "https://github.com/autodevjapan/cc-pc-tools-rs.git https://github.com/autodevjapan/cc-pc-tools-rs git@github.com:autodevjapan/cc-pc-tools-rs.git"}, + } + for name, pair := range checks { + if pair[0] != pair[1] { + t.Errorf("%s = %q, want %q", name, pair[0], pair[1]) + } + } + // The capacity qualification proves only the installed baseline. The new + // candidate is never the old capacity pin under another name. + if p.candidateSHA == p.baselineSHA || p.candidateSHA == p.capacity.oldSHA { + t.Fatal("the candidate reuses an old capacity pin") + } + for _, path := range []string{"D:/CC/Tools/todo/todo.exe", `d:\cc\tools\todo\todo.exe`} { + if !exactLeaf(path, p.target) { + t.Errorf("exactLeaf(%q) refused the fixed leaf", path) + } + } + for _, path := range []string{"D:/CC/Tools/todo", "D:/CC/Tools/todo/todo.json", "D:/CC/Tools/todo/todo.exe:x", "D:/CC/Tools/todo/TODO~1.EXE", "D:/CC/Tools/todo/todo.exe.", "\\\\?\\D:\\CC\\Tools\\todo\\todo.exe"} { + if exactLeaf(path, p.target) { + t.Errorf("exactLeaf(%q) accepted another resource", path) + } + } +} diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go new file mode 100644 index 00000000..93f79526 --- /dev/null +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -0,0 +1,120 @@ +package main + +import ( + "bytes" + "context" + "errors" + "io" + "os" + "os/exec" + "strings" + "syscall" + "time" +) + +// environment is the reader's only connection to the outside: the deployed +// guard's CLI and Git. Tests replace the guard with a recorded fake. +type environment struct { + guard func(args ...string) (stdout []byte, exitCode int, ran bool) + git func(root string, args ...string) ([]byte, error) + beforeRead func(path string) + beforeRecheck func() +} + +const guardExecutable = `D:\CC\Tools\client-records\scrub391.exe` + +const createNoWindow = 0x08000000 + +func productionEnvironment() environment { + return environment{guard: runGuard, git: runGit} +} + +// verifyUpdate is not implemented yet (ToDo3463 Red). +func verifyUpdate(path string, p profile, env environment) (qualification, string) { + return qualification{}, "unimplemented" +} + +// exactLeaf is not implemented yet (ToDo3463 Red). +func exactLeaf(path, target string) bool { + return false +} + +type limitedBuffer struct { + bytes.Buffer + limit int + overflow bool +} + +func (b *limitedBuffer) Write(p []byte) (int, error) { + if b.Len()+len(p) > b.limit { + b.overflow = true + return 0, errors.New("output limit exceeded") + } + return b.Buffer.Write(p) +} + +// childEnvironment removes inherited GIT_* overrides so another repository, +// index or object store cannot stand in for the declared source. +func childEnvironment(parent []string) []string { + out := make([]string, 0, len(parent)+3) + for _, entry := range parent { + key, _, _ := strings.Cut(entry, "=") + if strings.HasPrefix(strings.ToUpper(key), "GIT_") { + continue + } + out = append(out, entry) + } + return append(out, "GIT_NO_LAZY_FETCH=1", "GIT_OPTIONAL_LOCKS=0", "GIT_TERMINAL_PROMPT=0") +} + +func hiddenProcess(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} + cmd.WaitDelay = time.Second + cmd.Env = childEnvironment(os.Environ()) + cmd.Stderr = io.Discard +} + +func runGit(root string, args ...string) ([]byte, error) { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + argv := append([]string{"-C", root, "--no-replace-objects", "-c", "core.fsmonitor=false"}, args...) + cmd := exec.CommandContext(ctx, "git", argv...) + hiddenProcess(cmd) + out := &limitedBuffer{limit: 32 << 20} + cmd.Stdout = out + err := cmd.Run() + if out.overflow { + return nil, errors.New("git output limit exceeded") + } + return out.Bytes(), err +} + +// runGuard executes the deployed guard by its fixed path. The executable must +// be a plain file; its classification is trusted as the PC's protection authority. +func runGuard(args ...string) ([]byte, int, bool) { + info, err := os.Lstat(guardExecutable) + if err != nil || !info.Mode().IsRegular() { + return nil, -1, false + } + if data, ok := info.Sys().(*syscall.Win32FileAttributeData); !ok || data.FileAttributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT != 0 { + return nil, -1, false + } + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, guardExecutable, args...) + hiddenProcess(cmd) + out := &limitedBuffer{limit: 1 << 20} + cmd.Stdout = out + err = cmd.Run() + if out.overflow { + return nil, -1, false + } + if err != nil { + var exit *exec.ExitError + if errors.As(err, &exit) { + return out.Bytes(), exit.ExitCode(), true + } + return nil, -1, false + } + return out.Bytes(), 0, true +} From 76fdc92a5dacdbb550bd62eba0331780c78f7eca Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 02:16:03 +0900 Subject: [PATCH 02/24] feat: qualify the one ToDo3190 todo.exe update with an independent reader The reader pins the installed leaf, asks the deployed guard's existing capacity purpose to qualify the f871 baseline, and only then hashes it. It binds the d439 candidate to the fixed merged source 0e16e817, its tree, all 126 compiler inputs, the original build receipt, log and dep-info, checks the compiler closure, classifies every read file with the deployed guard, and rechecks every pinned handle and the worktree before returning. It replaces nothing and always reports data_access_granted false. The shared guard is unchanged. Refs #408 --- docs/todo-update-reader-3190.md | 35 +- .../evidence_windows.go | 360 ++++++++++++++++++ tools/todo-update-reader-3190/pin_windows.go | 234 ++++++++++++ .../todo-update-reader-3190/verify_windows.go | 301 ++++++++++++++- 4 files changed, 912 insertions(+), 18 deletions(-) create mode 100644 tools/todo-update-reader-3190/evidence_windows.go create mode 100644 tools/todo-update-reader-3190/pin_windows.go diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index 4281ccf2..d44b805b 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -120,5 +120,36 @@ symlink も同じ reparse 属性で拒否される。 ## 実測と未完了 -実装前Red、実装後Green、CI、別モデルのレビュー、merge、merge後のreaderによる実物の照合は、 -それぞれ完了した時点でこの節に追記する。 +### Red（実装前） + +commit `5b22ab24` は仕様、CLI、固定値、受入テストだけを含み、`verifyUpdate` は常に `unimplemented` を返す。 +`tools/todo-update-reader-3190` で `go test -count=1 ./...` は終了コード1。検証系の14テストが +subtest込み102行すべて `unimplemented` か `exactLeaf` の未実装で失敗し、fixture作成の失敗は0。 +CLIの4テストはこの時点で実装済みの契約として合格した。コンパイル失敗をRedに数えていない。 + +### Green（実装後） + +`go test -race -count=1 ./...` は終了コード0。上位18件、subtest込み106件が合格し、失敗0・skip 0・ +data race 0、136秒。`go vet ./...` とgofmtも差分なし。Cargo、GPU、Civ6は使っていない。 + +照合の要所を壊した8つの版を `go test -overlay` で当て、7つは対応するテストが失敗した +（容量用途の未知の項目、読取の順序、hardlink、最後の再照合、receiptの重複key、build script、入力のguard分類）。 +祖先のreparse確認を外した版は生き残った。handleの実パス照合が祖先のjunctionを独立に拒否するためで、 +junctionの拒否自体はテストで確認している。 + +### 実物への事前の観測（merge前） + +実装commitから生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 +`qualified:true`、current `f8710b51…b983`、new `d4392e63…0718`（8,017,920 bytes）、 +source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false` を返した。これはmerge前のsourceによる観測で、最終の結果はmerge後のsourceから +生成したreaderで取り直す。対象の置換、guardの変更、ToDoの状態変更はしていない。 + +補足として、親の作業場所に残るCargoの出力 `target/release/todo.exe` は、配布用コピーと同じ +SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/deps/todo.exe` とhardlinkされている。 +リンクを持つためreaderの照合対象ではなく、コピーと原出力が同じbytesであることの観測として記録する。 + +### 未完了 + +正式CI、別モデルのレビューとMustの解消、通常merge、merge後のsourceから生成したreaderによる実物の照合、 +親3190への結果の返却が残る。完了した時点でこの節に追記する。 diff --git a/tools/todo-update-reader-3190/evidence_windows.go b/tools/todo-update-reader-3190/evidence_windows.go new file mode 100644 index 00000000..1c5711bb --- /dev/null +++ b/tools/todo-update-reader-3190/evidence_windows.go @@ -0,0 +1,360 @@ +package main + +import ( + "bytes" + "encoding/json" + "io" + "path" + "path/filepath" + "regexp" + "slices" + "strings" + "time" + "unicode/utf8" +) + +type receiptInput struct { + Path string `json:"path"` + Blob string `json:"blob"` + SHA256 string `json:"sha256"` +} + +// buildReceipt is the complete schema of ToDo3190's original build record. +// Unknown fields are refused, so a receipt cannot carry an extra grant. +type buildReceipt struct { + Domain string `json:"domain"` + TodoID int64 `json:"todo_id"` + Job string `json:"job"` + At string `json:"at"` + Repository string `json:"repository"` + SourceRoot string `json:"source_root"` + SourceHead string `json:"source_head"` + SourceTree string `json:"source_tree"` + Command []string `json:"command"` + CompilerExitCode *int64 `json:"compiler_exit_code"` + LogFile string `json:"log_file"` + LogSHA256 string `json:"log_sha256"` + DepInfoFile string `json:"dep_info_file"` + DepInfoSHA256 string `json:"dep_info_sha256"` + Artifact string `json:"artifact"` + ArtifactSHA256 string `json:"artifact_sha256"` + ArtifactBytes int64 `json:"artifact_bytes"` + Inputs []receiptInput `json:"inputs"` + BaselineQualifiedBy string `json:"baseline_qualified_by"` + BaselineSHA256 string `json:"baseline_sha256"` + FunctionalTestStatus string `json:"functional_test_status"` + DataAccessGranted *bool `json:"data_access_granted"` +} + +type treeEntry struct{ mode, kind, object string } + +var ( + relPathPattern = regexp.MustCompile(`^[A-Za-z0-9_.-]+(?:/[A-Za-z0-9_.-]+)*$`) + lowerHex = regexp.MustCompile(`^[0-9a-f]+$`) + compilingLine = regexp.MustCompile(`^ +Compiling ([A-Za-z0-9_-]+) v([0-9][0-9A-Za-z.+-]*)(?: \(([^()]+)\))?$`) + finishedLine = regexp.MustCompile("^ +Finished `release` profile \\[optimized\\] target\\(s\\) in (?:[0-9]+(?:\\.[0-9]+)?s|[0-9]+m [0-9]+s)$") +) + +func hexDigest(value string, length int) bool { + return len(value) == length && lowerHex.MatchString(value) +} + +// validRelPath accepts one plain repository path: no absolute or parent +// parts, no stream or drive separators, no trailing dot alias, no device name. +func validRelPath(rel string) bool { + if !relPathPattern.MatchString(rel) { + return false + } + for _, part := range strings.Split(rel, "/") { + if part == "." || part == ".." || strings.HasSuffix(part, ".") || reservedDeviceName(part) { + return false + } + } + return true +} + +func reservedDeviceName(part string) bool { + base, _, _ := strings.Cut(strings.ToUpper(part), ".") + switch base { + case "CON", "PRN", "AUX", "NUL", "CONIN$", "CONOUT$": + return true + } + if len(base) == 4 && (strings.HasPrefix(base, "COM") || strings.HasPrefix(base, "LPT")) && base[3] >= '1' && base[3] <= '9' { + return true + } + return false +} + +// relativeTo converts a native absolute path strictly inside root. +func relativeTo(root, native string) (string, bool) { + if !filepath.IsAbs(native) || filepath.Clean(native) != native || len(native) <= len(root)+1 || + !strings.EqualFold(native[:len(root)], root) || native[len(root)] != '\\' { + return "", false + } + return filepath.ToSlash(native[len(root)+1:]), true +} + +// noDuplicateKeys refuses JSON in which any object repeats a key; the +// standard decoder would silently keep only the last value. +func noDuplicateKeys(data []byte) bool { + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + var value func(depth int) bool + value = func(depth int) bool { + if depth > 32 { + return false + } + token, err := dec.Token() + if err != nil { + return false + } + switch token { + case json.Delim('{'): + seen := map[string]bool{} + for dec.More() { + name, err := dec.Token() + key, ok := name.(string) + if err != nil || !ok || seen[key] { + return false + } + seen[key] = true + if !value(depth + 1) { + return false + } + } + end, err := dec.Token() + return err == nil && end == json.Delim('}') + case json.Delim('['): + for dec.More() { + if !value(depth + 1) { + return false + } + } + end, err := dec.Token() + return err == nil && end == json.Delim(']') + } + return true + } + if !value(0) { + return false + } + _, err := dec.Token() + return err == io.EOF +} + +// decodeSingle decodes exactly one JSON value followed only by whitespace. +func decodeSingle(data []byte, v any, strict bool) bool { + if !utf8.Valid(data) || !noDuplicateKeys(data) { + return false + } + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + if strict { + dec.DisallowUnknownFields() + } + if dec.Decode(v) != nil { + return false + } + _, err := dec.Token() + return err == io.EOF +} + +func receiptMatches(r buildReceipt, p profile) bool { + at, err := time.Parse(time.RFC3339Nano, r.At) + return err == nil && !at.IsZero() && + r.Domain == p.receipt.domain && r.TodoID == p.receipt.todoID && r.Job == p.receipt.job && + r.Repository == p.receipt.repository && filepath.IsAbs(r.SourceRoot) && + strings.EqualFold(filepath.Clean(r.SourceRoot), p.root) && + r.SourceHead == p.head && r.SourceTree == p.tree && slices.Equal(r.Command, p.receipt.command) && + r.CompilerExitCode != nil && *r.CompilerExitCode == 0 && + r.LogFile == p.logName && r.LogSHA256 == p.logSHA && + r.DepInfoFile == p.depInfoName && r.DepInfoSHA256 == p.depInfoSHA && + r.Artifact == p.candidateName && r.ArtifactSHA256 == p.candidateSHA && r.ArtifactBytes == p.candidateBytes && + r.BaselineQualifiedBy == p.receipt.baselineQualifiedBy && r.BaselineSHA256 == p.baselineSHA && + r.DataAccessGranted != nil && !*r.DataAccessGranted && + len(r.Inputs) == p.inputs && validInputs(r.Inputs) +} + +func validInputs(inputs []receiptInput) bool { + seen := map[string]bool{} + for _, in := range inputs { + folded := strings.ToLower(in.Path) + if !validRelPath(in.Path) || seen[folded] || !hexDigest(in.SHA256, 64) || !hexDigest(in.Blob, 40) { + return false + } + seen[folded] = true + } + return true +} + +// parseTree reads `git ls-tree -r -z --full-tree` output. +func parseTree(out []byte) (map[string]treeEntry, bool) { + tree := map[string]treeEntry{} + for _, record := range strings.Split(string(out), "\x00") { + if record == "" { + continue + } + meta, name, ok := strings.Cut(record, "\t") + fields := strings.Fields(meta) + if !ok || len(fields) != 3 || name == "" { + return nil, false + } + if _, dup := tree[name]; dup { + return nil, false + } + tree[name] = treeEntry{mode: fields[0], kind: fields[1], object: fields[2]} + } + return tree, len(tree) > 0 +} + +// parseDepInfo reads Cargo's one-rule dep-info for artifact. Every compiler +// input must be a plain path inside root; escapes and variables are refused. +func parseDepInfo(data []byte, root, artifact string) ([]string, bool) { + text := string(data) + if !utf8.ValidString(text) || !strings.HasSuffix(text, "\n") { + return nil, false + } + line := strings.TrimSuffix(strings.TrimSuffix(text, "\n"), "\r") + if line == "" || strings.ContainsAny(line, "\n\r\t$#") || strings.Contains(line, `\ `) { + return nil, false + } + target, list, ok := strings.Cut(line, ": ") + if !ok || !strings.EqualFold(target, artifact) { + return nil, false + } + seen := map[string]bool{} + var deps []string + for _, dep := range strings.Split(list, " ") { + rel, ok := relativeTo(root, dep) + if !ok || !validRelPath(rel) || seen[strings.ToLower(rel)] { + return nil, false + } + seen[strings.ToLower(rel)] = true + deps = append(deps, rel) + } + return deps, len(deps) > 0 +} + +// parseBuildLog accepts only Cargo's Compiling lines and a final release +// Finished line. It returns the source-relative directories of local crates. +func parseBuildLog(data []byte, root string) ([]string, bool) { + text := string(data) + if !utf8.ValidString(text) { + return nil, false + } + lines := strings.Split(strings.ReplaceAll(text, "\r\n", "\n"), "\n") + for len(lines) > 0 && lines[len(lines)-1] == "" { + lines = lines[:len(lines)-1] + } + if len(lines) < 2 || !finishedLine.MatchString(lines[len(lines)-1]) { + return nil, false + } + seen := map[string]bool{} + var crates []string + for _, line := range lines[:len(lines)-1] { + match := compilingLine.FindStringSubmatch(line) + if match == nil { + return nil, false + } + if match[3] == "" { + continue // a registry crate pinned by Cargo.lock + } + rel, ok := relativeTo(root, match[3]) + if !ok || !validRelPath(rel) || seen[rel] { + return nil, false + } + seen[rel] = true + crates = append(crates, rel) + } + return crates, len(crates) > 0 +} + +func crateOf(dep string, inputs map[string]bool) (string, bool) { + for dir := path.Dir(dep); dir != "." && dir != "/"; dir = path.Dir(dir) { + if inputs[dir+"/Cargo.toml"] { + return dir, true + } + } + return "", false +} + +func declaresBuildScript(manifest []byte) bool { + for _, line := range strings.Split(string(manifest), "\n") { + trimmed := strings.TrimSpace(line) + for _, key := range []string{"build", `"build"`, "'build'"} { + if rest, ok := strings.CutPrefix(trimmed, key); ok && strings.HasPrefix(strings.TrimSpace(rest), "=") { + return true + } + } + } + return false +} + +// closureComplete checks that the receipt's inputs are exactly the compiler's +// local inputs plus the manifests and lock file that define the build, and +// that the fixed tree has no build script, Cargo configuration or toolchain +// file that the receipt would not cover. It returns the local crate count. +func closureComplete(inputs []receiptInput, deps, logCrates []string, tree map[string]treeEntry, manifests map[string][]byte, p profile) (int, bool) { + inputSet := map[string]bool{} + for _, in := range inputs { + inputSet[in.Path] = true + } + depSet := map[string]bool{} + crates := map[string]bool{} + for _, dep := range deps { + crate, ok := crateOf(dep, inputSet) + if !inputSet[dep] || !ok { + return 0, false + } + depSet[dep] = true + crates[crate] = true + } + expected := map[string]bool{"Cargo.lock": true, "Cargo.toml": true} + for crate := range crates { + expected[crate+"/Cargo.toml"] = true + } + for name := range inputSet { + if depSet[name] { + continue + } + if !expected[name] { + return 0, false + } + delete(expected, name) + } + if len(expected) != 0 || len(logCrates) != len(crates) { + return 0, false + } + for _, crate := range logCrates { + if !crates[crate] { + return 0, false + } + } + folded := map[string]bool{} + for name := range tree { + folded[strings.ToLower(name)] = true + } + dirs := []string{""} + for crate := range crates { + dirs = append(dirs, crate) + } + for _, dir := range dirs { + for _, name := range []string{"build.rs", ".cargo/config", ".cargo/config.toml"} { + if folded[strings.ToLower(path.Join(dir, name))] { + return 0, false + } + } + } + if folded["rust-toolchain"] || folded["rust-toolchain.toml"] { + return 0, false + } + for _, manifest := range manifests { + if declaresBuildScript(manifest) { + return 0, false + } + } + if len(inputSet) != p.inputs || len(deps) != p.depEntries || len(crates) != p.localCrates { + return 0, false + } + return len(crates), true +} diff --git a/tools/todo-update-reader-3190/pin_windows.go b/tools/todo-update-reader-3190/pin_windows.go new file mode 100644 index 00000000..c2620eb9 --- /dev/null +++ b/tools/todo-update-reader-3190/pin_windows.go @@ -0,0 +1,234 @@ +package main + +import ( + "crypto/sha256" + "encoding/hex" + "io" + "os" + "path/filepath" + "strings" + "syscall" + "unsafe" +) + +const ( + fileFlagOpenReparsePoint = 0x00200000 + fileFlagBackupSemantics = 0x02000000 + fileFlagSequentialScan = 0x08000000 + fileListDirectory = 0x00000001 + fileReadAttributes = 0x00000080 +) + +var procGetFinalPathNameByHandleW = syscall.NewLazyDLL("kernel32.dll").NewProc("GetFinalPathNameByHandleW") + +// pinnedFile is one evidence file held open for reading with writes, deletes +// and renames denied to everyone else until the session closes. +type pinnedFile struct { + file *os.File + path string + info syscall.ByHandleFileInformation + size int64 +} + +// pinnedDir keeps the identity of the source root. Others may still create +// files in it, so only its identity and physical path are rechecked. +type pinnedDir struct { + handle syscall.Handle + path string + info syscall.ByHandleFileInformation +} + +type session struct { + env environment + files []*pinnedFile + dirs []pinnedDir + dirCache map[string]bool +} + +func newSession(env environment) *session { + return &session{env: env, dirCache: map[string]bool{}} +} + +func (s *session) close() { + for _, f := range s.files { + f.file.Close() + } + for _, d := range s.dirs { + syscall.CloseHandle(d.handle) + } +} + +// finalPath returns the handle's physical DOS path. Links and junctions in +// any ancestor, and 8.3 aliases, make it differ from the requested spelling. +func finalPath(h syscall.Handle) (string, bool) { + buf := make([]uint16, 32768) + n, _, _ := procGetFinalPathNameByHandleW.Call(uintptr(h), uintptr(unsafe.Pointer(&buf[0])), uintptr(len(buf)), 0) + if n == 0 || n >= uintptr(len(buf)) { + return "", false + } + path := syscall.UTF16ToString(buf[:n]) + if strings.HasPrefix(path, `\\?\UNC\`) || !strings.HasPrefix(path, `\\?\`) { + return "", false + } + path = path[4:] + if len(path) < 3 || path[1] != ':' || path[2] != '\\' { + return "", false + } + return path, true +} + +func reparse(attributes uint32) bool { + return attributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT != 0 +} + +// lstatPlain reads the Windows attributes directly: recent Go releases do not +// report every reparse point (such as a junction) as a symlink. +func lstatPlain(path string) (os.FileInfo, bool) { + info, err := os.Lstat(path) + if err != nil { + return nil, false + } + data, ok := info.Sys().(*syscall.Win32FileAttributeData) + return info, ok && !reparse(data.FileAttributes) +} + +func (s *session) plainDir(dir string) bool { + key := strings.ToLower(dir) + if ok, cached := s.dirCache[key]; cached { + return ok + } + info, ok := lstatPlain(dir) + ok = ok && info.IsDir() + s.dirCache[key] = ok + return ok +} + +// plainAncestors checks every directory between the volume root and path. +func (s *session) plainAncestors(path string) bool { + for dir := filepath.Dir(path); filepath.Dir(dir) != dir; dir = filepath.Dir(dir) { + if !s.plainDir(dir) { + return false + } + } + return true +} + +func plainFile(info syscall.ByHandleFileInformation) bool { + return !reparse(info.FileAttributes) && info.FileAttributes&syscall.FILE_ATTRIBUTE_DIRECTORY == 0 && info.NumberOfLinks == 1 +} + +// pin opens one regular, single-link file without following a reparse point +// and without reading it. The handle shares only reads. +func (s *session) pin(path string, limit int64) (*pinnedFile, bool) { + if !s.plainAncestors(path) { + return nil, false + } + if info, ok := lstatPlain(path); !ok || !info.Mode().IsRegular() { + return nil, false + } + name, err := syscall.UTF16PtrFromString(path) + if err != nil { + return nil, false + } + h, err := syscall.CreateFile(name, syscall.GENERIC_READ, syscall.FILE_SHARE_READ, nil, syscall.OPEN_EXISTING, fileFlagOpenReparsePoint|fileFlagSequentialScan, 0) + if err != nil { + return nil, false + } + f := &pinnedFile{file: os.NewFile(uintptr(h), path), path: path} + if syscall.GetFileInformationByHandle(h, &f.info) != nil || !plainFile(f.info) { + f.file.Close() + return nil, false + } + f.size = int64(f.info.FileSizeHigh)<<32 | int64(f.info.FileSizeLow) + if actual, ok := finalPath(h); !ok || !strings.EqualFold(actual, path) || f.size < 1 || f.size > limit { + f.file.Close() + return nil, false + } + s.files = append(s.files, f) + return f, true +} + +// read returns exactly the bytes behind the pinned handle and their digest. +// Callers parse these same bytes; nothing is reopened by name. +func (s *session) read(f *pinnedFile) ([]byte, string, bool) { + if s.env.beforeRead != nil { + s.env.beforeRead(f.path) + } + if _, err := f.file.Seek(0, io.SeekStart); err != nil { + return nil, "", false + } + data, err := io.ReadAll(io.LimitReader(f.file, f.size+1)) + if err != nil || int64(len(data)) != f.size || !s.unchanged(f) { + return nil, "", false + } + sum := sha256.Sum256(data) + return data, hex.EncodeToString(sum[:]), true +} + +// unchanged compares the handle's identity, size, write time, link count and +// attributes with the values at pin time; only the access time may differ. +func (s *session) unchanged(f *pinnedFile) bool { + h := syscall.Handle(f.file.Fd()) + var now syscall.ByHandleFileInformation + if syscall.GetFileInformationByHandle(h, &now) != nil || !plainFile(now) { + return false + } + before := f.info + before.LastAccessTime = now.LastAccessTime + if before != now { + return false + } + actual, ok := finalPath(h) + info, plain := lstatPlain(f.path) + return ok && strings.EqualFold(actual, f.path) && plain && info.Mode().IsRegular() +} + +func (s *session) pinDir(path string) bool { + if !s.plainAncestors(path) || !s.plainDir(path) { + return false + } + name, err := syscall.UTF16PtrFromString(path) + if err != nil { + return false + } + h, err := syscall.CreateFile(name, fileListDirectory|fileReadAttributes, syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE|syscall.FILE_SHARE_DELETE, nil, syscall.OPEN_EXISTING, fileFlagBackupSemantics|fileFlagOpenReparsePoint, 0) + if err != nil { + return false + } + d := pinnedDir{handle: h, path: path} + actual, ok := finalPath(h) + if syscall.GetFileInformationByHandle(h, &d.info) != nil || reparse(d.info.FileAttributes) || + d.info.FileAttributes&syscall.FILE_ATTRIBUTE_DIRECTORY == 0 || !ok || !strings.EqualFold(actual, path) { + syscall.CloseHandle(h) + return false + } + s.dirs = append(s.dirs, d) + return true +} + +func dirUnchanged(d pinnedDir) bool { + var now syscall.ByHandleFileInformation + if syscall.GetFileInformationByHandle(d.handle, &now) != nil { + return false + } + actual, ok := finalPath(d.handle) + return ok && strings.EqualFold(actual, d.path) && now.VolumeSerialNumber == d.info.VolumeSerialNumber && + now.FileIndexHigh == d.info.FileIndexHigh && now.FileIndexLow == d.info.FileIndexLow && + !reparse(now.FileAttributes) && now.FileAttributes&syscall.FILE_ATTRIBUTE_DIRECTORY != 0 +} + +// recheck repeats every physical check with a fresh ancestor cache. +func (s *session) recheck() bool { + s.dirCache = map[string]bool{} + for _, f := range s.files { + if !s.plainAncestors(f.path) || !s.unchanged(f) { + return false + } + } + for _, d := range s.dirs { + if !s.plainAncestors(d.path) || !s.plainDir(d.path) || !dirUnchanged(d) { + return false + } + } + return true +} diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 93f79526..6de4bf66 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -3,11 +3,18 @@ package main import ( "bytes" "context" + "encoding/json" "errors" "io" "os" "os/exec" + "path" + "path/filepath" + "slices" + "strconv" "strings" + "sync" + "sync/atomic" "syscall" "time" ) @@ -23,20 +30,286 @@ type environment struct { const guardExecutable = `D:\CC\Tools\client-records\scrub391.exe` -const createNoWindow = 0x08000000 +const ( + createNoWindow = 0x08000000 + maxEvidenceBytes = 1 << 20 + maxInputBytes = 16 << 20 + maxArtifactBytes = 64 << 20 + guardWorkers = 4 +) func productionEnvironment() environment { return environment{guard: runGuard, git: runGit} } -// verifyUpdate is not implemented yet (ToDo3463 Red). -func verifyUpdate(path string, p profile, env environment) (qualification, string) { - return qualification{}, "unimplemented" +// exactLeaf accepts only the compiled leaf, spelled with either separator or +// any case. Parents, siblings, streams, trailing dots or spaces, device and +// extended-length prefixes and short names never compare equal. +func exactLeaf(path, target string) bool { + if len(path) < 3 || !filepath.IsAbs(path) || strings.HasPrefix(path, `\\`) || strings.HasPrefix(path, "//") || + strings.ContainsAny(path[2:], `:*?"<>|`) { + return false + } + return strings.EqualFold(filepath.Clean(path), target) } -// exactLeaf is not implemented yet (ToDo3463 Red). -func exactLeaf(path, target string) bool { - return false +// verifyUpdate qualifies the one fixed update or returns a fixed reason code. +// It never returns a partially filled qualification. +func verifyUpdate(requested string, p profile, env environment) (qualification, string) { + if !exactLeaf(requested, p.target) { + return qualification{}, "target_path" + } + s := newSession(env) + defer s.close() + + // Hold the installed leaf, unread, so the purpose entry and the digest + // below observe one file object that nobody can write or replace. + target, ok := s.pin(p.target, maxArtifactBytes) + if !ok { + return qualification{}, "target_leaf" + } + if !capacityQualified(env, p) { + return qualification{}, "baseline_qualification" + } + _, current, ok := s.read(target) + if !ok { + return qualification{}, "target_leaf" + } + if current != p.baselineSHA { + return qualification{}, "baseline_mismatch" + } + + if !s.pinDir(p.root) || !guardClear(env, p.root) { + return qualification{}, "source_root" + } + tree, ok := checkSource(p, env) + if !ok { + return qualification{}, "source_git" + } + + data, reason := s.evidence(filepath.Join(p.root, p.receiptName), p.receiptSHA, maxEvidenceBytes, "receipt") + if reason != "" { + return qualification{}, reason + } + var receipt buildReceipt + if !decodeSingle(data, &receipt, true) || !receiptMatches(receipt, p) { + return qualification{}, "receipt" + } + + data, reason = s.evidence(filepath.Join(p.root, p.logName), p.logSHA, maxEvidenceBytes, "log") + if reason != "" { + return qualification{}, reason + } + logCrates, ok := parseBuildLog(data, p.root) + if !ok { + return qualification{}, "log" + } + + data, reason = s.evidence(filepath.Join(p.root, filepath.FromSlash(p.depInfoName)), p.depInfoSHA, maxEvidenceBytes, "dep_info") + if reason != "" { + return qualification{}, reason + } + artifact := filepath.Join(p.root, filepath.FromSlash(strings.TrimSuffix(p.depInfoName, path.Ext(p.depInfoName))+".exe")) + deps, ok := parseDepInfo(data, p.root, artifact) + if !ok { + return qualification{}, "dep_info" + } + + manifests, reason := s.inputs(receipt.Inputs, tree, p) + if reason != "" { + return qualification{}, reason + } + crates, ok := closureComplete(receipt.Inputs, deps, logCrates, tree, manifests, p) + if !ok { + return qualification{}, "closure" + } + + candidate, ok := s.pin(filepath.Join(p.root, p.candidateName), maxArtifactBytes) + if !ok { + return qualification{}, "candidate" + } + if !guardClear(env, candidate.path) { + return qualification{}, "guard_classification" + } + if _, sum, ok := s.read(candidate); !ok || candidate.size != p.candidateBytes || sum != p.candidateSHA { + return qualification{}, "candidate" + } + + if env.beforeRecheck != nil { + env.beforeRecheck() + } + if !s.recheck() || !sourceStillFixed(p, env) { + return qualification{}, "recheck" + } + return qualification{ + Qualified: true, Domain: qualificationDomain, TodoID: qualificationTodo, VerifierTodoID: verifierTodo, + TargetKind: qualificationKind, Target: filepath.ToSlash(p.target), + CurrentSHA256: current, NewSHA256: p.candidateSHA, NewBytes: p.candidateBytes, + SourceRepository: receipt.Repository, SourceRoot: filepath.ToSlash(p.root), SourceHead: p.head, SourceTree: p.tree, + BuildReceiptSHA256: p.receiptSHA, BuildJob: receipt.Job, BuildCommand: slices.Clone(receipt.Command), + BuildLogSHA256: p.logSHA, DepInfoSHA256: p.depInfoSHA, + InputsVerified: len(receipt.Inputs), DepInfoEntries: len(deps), LocalCrates: crates, + BaselineQualifiedBy: p.capacity.domain, BaselineSourceReceiptSHA256: p.capacity.receiptSHA, + }, "" +} + +// evidence pins one build record, has the guard classify it, and returns the +// exact bytes whose digest matched the compiled pin. +func (s *session) evidence(name, pin string, limit int64, reason string) ([]byte, string) { + f, ok := s.pin(name, limit) + if !ok { + return nil, reason + } + if !guardClear(s.env, name) { + return nil, "guard_classification" + } + data, sum, ok := s.read(f) + if !ok || sum != pin { + return nil, reason + } + return data, "" +} + +// inputs pins every receipt input, has the guard classify all of them, then +// hashes each pinned file. Manifest bytes are kept for the closure check. +func (s *session) inputs(inputs []receiptInput, tree map[string]treeEntry, p profile) (map[string][]byte, string) { + files := make([]*pinnedFile, len(inputs)) + for i, in := range inputs { + entry, ok := tree[in.Path] + if !ok || entry.kind != "blob" || (entry.mode != "100644" && entry.mode != "100755") || entry.object != in.Blob { + return nil, "input" + } + if files[i], ok = s.pin(filepath.Join(p.root, filepath.FromSlash(in.Path)), maxInputBytes); !ok { + return nil, "input" + } + } + if !s.classifyAll(files) { + return nil, "guard_classification" + } + manifests := map[string][]byte{} + for i, in := range inputs { + data, sum, ok := s.read(files[i]) + if !ok || sum != in.SHA256 { + return nil, "input" + } + if path.Base(in.Path) == "Cargo.toml" { + manifests[in.Path] = data + } + } + return manifests, "" +} + +func (s *session) classifyAll(files []*pinnedFile) bool { + var failed atomic.Bool + var wg sync.WaitGroup + next := make(chan string) + for range guardWorkers { + wg.Add(1) + go func() { + defer wg.Done() + for name := range next { + if !failed.Load() && !guardClear(s.env, name) { + failed.Store(true) + } + } + }() + } + for _, f := range files { + next <- f.path + } + close(next) + wg.Wait() + return !failed.Load() +} + +type guardVerdict struct { + OK *bool `json:"ok"` + AccessAllowed *bool `json:"access_allowed"` + ClientResource *bool `json:"client_resource"` + ClassificationUnverified *bool `json:"classification_unverified"` + ProtectionStatus *string `json:"protection_status"` + ProtectionCategories *[]string `json:"protection_categories"` +} + +// guardClear asks the deployed guard's general classifier about one path and +// accepts only an explicit clear verdict with no protection category. +func guardClear(env environment, name string) bool { + out, code, ran := env.guard("--guard-path", name) + var v guardVerdict + if !ran || code != 0 || !decodeSingle(out, &v, false) { + return false + } + return v.OK != nil && *v.OK && v.AccessAllowed != nil && *v.AccessAllowed && + v.ClientResource != nil && !*v.ClientResource && + v.ClassificationUnverified != nil && !*v.ClassificationUnverified && + v.ProtectionStatus != nil && *v.ProtectionStatus == "clear" && + v.ProtectionCategories != nil && len(*v.ProtectionCategories) == 0 +} + +// capacityQualified requires the existing ToDo3085 purpose entry to return +// exactly its known qualification for the installed baseline, nothing more. +func capacityQualified(env environment, p profile) bool { + out, code, ran := env.guard("--verify-capacity-runtime-update", p.target) + var report map[string]any + if !ran || code != 0 || !decodeSingle(out, &report, false) { + return false + } + want := map[string]any{ + "qualified": true, "domain": p.capacity.domain, + "todo_id": json.Number(strconv.FormatInt(p.capacity.todoID, 10)), "target_kind": p.capacity.targetKind, + "old_sha256": p.capacity.oldSHA, "new_sha256": p.baselineSHA, "source_receipt_sha256": p.capacity.receiptSHA, + "data_access_granted": false, "original_registry_recovered": false, + } + if len(report) != len(want) { + return false + } + for name, value := range want { + if report[name] != value { + return false + } + } + return true +} + +func gitText(env environment, root string, args ...string) (string, bool) { + out, err := env.git(root, args...) + return strings.TrimSpace(string(out)), err == nil +} + +// checkSource verifies the fixed worktree and returns the fixed commit's tree. +func checkSource(p profile, env environment) (map[string]treeEntry, bool) { + if !sourceStillFixed(p, env) { + return nil, false + } + if tree, ok := gitText(env, p.root, "rev-parse", "--verify", "--quiet", p.head+"^{tree}"); !ok || tree != p.tree { + return nil, false + } + if _, err := env.git(p.root, "merge-base", "--is-ancestor", p.head, p.mainRef); err != nil { + return nil, false + } + out, err := env.git(p.root, "ls-tree", "-r", "-z", "--full-tree", p.head) + if err != nil { + return nil, false + } + return parseTree(out) +} + +// sourceStillFixed is checked before reading and again after every file is +// pinned: the same worktree top, origin, HEAD and clean tracked content. +func sourceStillFixed(p profile, env environment) bool { + top, ok := gitText(env, p.root, "rev-parse", "--show-toplevel") + if !ok || !strings.EqualFold(filepath.Clean(filepath.FromSlash(top)), p.root) { + return false + } + origin, ok := gitText(env, p.root, "remote", "get-url", "origin") + if !ok || !slices.Contains(p.origins, strings.ToLower(origin)) { + return false + } + if head, ok := gitText(env, p.root, "rev-parse", "--verify", "HEAD"); !ok || head != p.head { + return false + } + _, err := env.git(p.root, "-c", "core.autocrlf=true", "diff", "--quiet", "--no-ext-diff", "--no-textconv", "HEAD", "--") + return err == nil } type limitedBuffer struct { @@ -58,8 +331,8 @@ func (b *limitedBuffer) Write(p []byte) (int, error) { func childEnvironment(parent []string) []string { out := make([]string, 0, len(parent)+3) for _, entry := range parent { - key, _, _ := strings.Cut(entry, "=") - if strings.HasPrefix(strings.ToUpper(key), "GIT_") { + name, _, _ := strings.Cut(entry, "=") + if strings.HasPrefix(strings.ToUpper(name), "GIT_") { continue } out = append(out, entry) @@ -90,13 +363,9 @@ func runGit(root string, args ...string) ([]byte, error) { } // runGuard executes the deployed guard by its fixed path. The executable must -// be a plain file; its classification is trusted as the PC's protection authority. +// be a plain file; its classification is the PC's protection authority. func runGuard(args ...string) ([]byte, int, bool) { - info, err := os.Lstat(guardExecutable) - if err != nil || !info.Mode().IsRegular() { - return nil, -1, false - } - if data, ok := info.Sys().(*syscall.Win32FileAttributeData); !ok || data.FileAttributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT != 0 { + if info, ok := lstatPlain(guardExecutable); !ok || !info.Mode().IsRegular() { return nil, -1, false } ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) @@ -105,7 +374,7 @@ func runGuard(args ...string) ([]byte, int, bool) { hiddenProcess(cmd) out := &limitedBuffer{limit: 1 << 20} cmd.Stdout = out - err = cmd.Run() + err := cmd.Run() if out.overflow { return nil, -1, false } From d9edfd4d71e0e8447bf08656efd37fb65224835b Mon Sep 17 00:00:00 2001 From: Makhmeto Date: Sat, 24 Oct 2026 02:21:01 +0900 Subject: [PATCH 03/24] ci: test the ToDo3190 todo update reader on the Windows runner Runs the module's race tests, vet and build on the self-hosted Windows runner, which has the Git, hard link and junction support the synthetic fixtures need. Refs #408 --- .github/workflows/todo-update-reader-3190.yml | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 .github/workflows/todo-update-reader-3190.yml diff --git a/.github/workflows/todo-update-reader-3190.yml b/.github/workflows/todo-update-reader-3190.yml new file mode 100644 index 00000000..100dc0eb --- /dev/null +++ b/.github/workflows/todo-update-reader-3190.yml @@ -0,0 +1,44 @@ +name: ToDo3190 todo update reader +on: + push: + branches: [main] + paths: ['tools/todo-update-reader-3190/**', '.github/workflows/todo-update-reader-3190.yml'] + pull_request: + paths: ['tools/todo-update-reader-3190/**', '.github/workflows/todo-update-reader-3190.yml'] +permissions: + contents: read +# Superseded PR heads need no second verification; preserve every main run. +concurrency: + group: ${{ github.workflow }}-pr-head-v2-${{ github.event_name == 'pull_request' && github.run_attempt == 1 && github.event.pull_request.number || github.run_id }} + cancel-in-progress: ${{ github.event_name == 'pull_request' && github.run_attempt == 1 }} +jobs: + windows: + if: github.repository == 'AutoDevJapan/cc-pc-tools-rs' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + # The reader uses Windows handle, link and final-path semantics, and its + # synthetic Git repositories, hard links and junctions need the PC runner. + runs-on: [self-hosted, Windows, X64, cc-pc-tools-windows] + timeout-minutes: 20 + defaults: + run: + working-directory: tools/todo-update-reader-3190 + steps: + - name: Admit current Windows source before checkout + env: + CI_JOB_TOKEN: ${{ github.token }} + working-directory: ${{ runner.temp }} + shell: pwsh + run: | + & $env:PC_CI_ADMISSION admit + exit $LASTEXITCODE + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1 + - uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 + with: + go-version: '1.27.x' + cache: false + - run: go test -race -timeout 900s ./... + - run: go vet ./... + - run: go build -trimpath . From dd4b0273ccf214139552b2a38fe29591ac7fa71a Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 03:12:49 +0900 Subject: [PATCH 04/24] test: require the ToDo3190 reader to keep Git off working files Review ToDo3482 (Must): git diff HEAD read every tracked file before the guard classified it. These tests record every Git call, hold a protected non-input README unreadable, and require the reader itself to tie pinned input bytes to the fixed blobs. checkedOut and gitBlobID are stubs here (Red). --- .../checkout_windows.go | 8 + .../reader_windows_test.go | 39 +++-- .../source_scope_windows_test.go | 165 ++++++++++++++++++ 3 files changed, 199 insertions(+), 13 deletions(-) create mode 100644 tools/todo-update-reader-3190/checkout_windows.go create mode 100644 tools/todo-update-reader-3190/source_scope_windows_test.go diff --git a/tools/todo-update-reader-3190/checkout_windows.go b/tools/todo-update-reader-3190/checkout_windows.go new file mode 100644 index 00000000..0c1d306f --- /dev/null +++ b/tools/todo-update-reader-3190/checkout_windows.go @@ -0,0 +1,8 @@ +package main + +// gitBlobID is the SHA-1 object name Git gives data stored as a blob. +func gitBlobID(data []byte) string { return "" } + +// checkedOut reports whether working bytes are exactly a fixed blob as Git +// writes it out on this PC. +func checkedOut(data []byte, blob string) bool { return false } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index 4cf4b01a..abaa9b2c 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -306,8 +306,11 @@ func (f *fixture) guard(args ...string) ([]byte, int, bool) { func (f *fixture) env() environment { return environment{ - guard: f.guard, - git: runGit, + guard: f.guard, + git: func(root string, args ...string) ([]byte, error) { + f.record("git:" + strings.Join(args, " ")) + return runGit(root, args...) + }, beforeRead: func(path string) { f.record("read:" + key(path)) }, beforeRecheck: func() { if f.onRecheck != nil { @@ -416,6 +419,10 @@ func TestQualifiesFixedUpdate(t *testing.T) { if f.index("guard-path:"+key(f.target)) >= 0 { t.Errorf("the general classifier was used for the installed leaf") } + // Git reads only refs, configuration and objects, never a working file. + if expectGitObjectOnly(t, f) == 0 { + t.Errorf("the source was verified without Git") + } } func TestBaselineIsReadOnlyAfterPurposeQualification(t *testing.T) { @@ -575,26 +582,27 @@ func TestRejectsTamperedOrMissingEvidence(t *testing.T) { func TestRejectsSourceChanges(t *testing.T) { cases := []struct { - name string - edit func(*fixture) + name, want string + edit func(*fixture) }{ - {"another origin", func(f *fixture) { + {"another origin", "source_git", func(f *fixture) { f.git("remote", "set-url", "origin", "https://github.com/someone-else/cc-pc-tools-rs.git") }}, - {"no origin", func(f *fixture) { f.git("remote", "remove", "origin") }}, - {"HEAD moved after the build", func(f *fixture) { + {"no origin", "source_git", func(f *fixture) { f.git("remote", "remove", "origin") }}, + {"HEAD moved after the build", "source_git", func(f *fixture) { f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "later") }}, - {"tracked input edited", func(f *fixture) { f.write("lib-a/src/lib.rs", crlf("pub fn a() { panic!() }\n")) }}, - {"tracked input deleted", func(f *fixture) { os.Remove(f.path("lib-a/src/lib.rs")) }}, - {"another tree is pinned", func(f *fixture) { f.p.tree = strings.Repeat("6", 40) }}, - {"head not merged into origin/main", func(f *fixture) { f.git("update-ref", "-d", "refs/remotes/origin/main") }}, + // The reader compares inputs itself; Git never reads them. + {"tracked input edited", "input", func(f *fixture) { f.write("lib-a/src/lib.rs", crlf("pub fn a() { panic!() }\n")) }}, + {"tracked input deleted", "input", func(f *fixture) { os.Remove(f.path("lib-a/src/lib.rs")) }}, + {"another tree is pinned", "source_git", func(f *fixture) { f.p.tree = strings.Repeat("6", 40) }}, + {"head not merged into origin/main", "source_git", func(f *fixture) { f.git("update-ref", "-d", "refs/remotes/origin/main") }}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { f := newFixture(t, nil) tc.edit(f) - expectReason(t, f, "source_git") + expectReason(t, f, tc.want) }) } t.Run("source root is a subdirectory of the worktree", func(t *testing.T) { @@ -724,8 +732,9 @@ func TestPinnedFilesResistReplacement(t *testing.T) { } func TestRejectsChangesBeforeRecheck(t *testing.T) { + // Inputs stay pinned against writes and tracked files outside the inputs are + // never read (TestGitNeverReadsWorkingFiles), so the races left are Git's. cases := map[string]func(*fixture){ - "tracked non-input edited": func(f *fixture) { f.write("README.md", "changed during verification\n") }, "new commit": func(f *fixture) { f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "racing") }, @@ -764,6 +773,10 @@ func TestRejectsProtectedResources(t *testing.T) { f := newFixture(t, nil) f.protected[key(f.path(rel))] = verdict expectReason(t, f, "guard_classification") + if f.index("read:"+key(f.path(rel))) >= 0 { + t.Fatalf("%s was read although the guard refused it", rel) + } + expectGitObjectOnly(t, f) }) } } diff --git a/tools/todo-update-reader-3190/source_scope_windows_test.go b/tools/todo-update-reader-3190/source_scope_windows_test.go new file mode 100644 index 00000000..c7c5c358 --- /dev/null +++ b/tools/todo-update-reader-3190/source_scope_windows_test.go @@ -0,0 +1,165 @@ +package main + +import ( + "os" + "strings" + "syscall" + "testing" + "time" +) + +// objectOnlyGit lists the Git commands that read only refs, configuration and +// objects. Others, such as diff, status, ls-files, hash-object or add, can open +// working files that the guard has not classified one by one. +var objectOnlyGit = map[string]bool{"rev-parse": true, "remote": true, "merge-base": true, "ls-tree": true} + +func gitSubcommand(args string) string { + fields := strings.Fields(args) + for i := 0; i < len(fields); i++ { + if fields[i] == "-c" { + i++ + continue + } + return fields[i] + } + return "" +} + +// expectGitObjectOnly fails on any recorded Git call that can read working +// files and returns how many Git calls the reader made. +func expectGitObjectOnly(t *testing.T, f *fixture) int { + t.Helper() + f.mu.Lock() + defer f.mu.Unlock() + calls := 0 + for _, event := range f.events { + if args, ok := strings.CutPrefix(event, "git:"); ok { + calls++ + if !objectOnlyGit[gitSubcommand(args)] { + t.Errorf("git %s can read working files the guard has not classified", args) + } + } + } + return calls +} + +// denyReads holds name open without sharing, so every other open fails until +// the returned function closes it. +func denyReads(t *testing.T, name string) func() { + t.Helper() + p, err := syscall.UTF16PtrFromString(name) + if err != nil { + t.Fatal(err) + } + h, err := syscall.CreateFile(p, syscall.GENERIC_READ, 0, nil, syscall.OPEN_EXISTING, syscall.FILE_ATTRIBUTE_NORMAL, 0) + if err != nil { + t.Fatal(err) + } + if _, err := os.ReadFile(name); err == nil { + syscall.CloseHandle(h) + t.Fatalf("%s could still be read while held", name) + } + return func() { syscall.CloseHandle(h) } +} + +// rebuildReceipt records the working inputs as they are now, as a receipt +// written after the change would; the blobs stay those of the fixed tree. +func (f *fixture) rebuildReceipt() { + f.t.Helper() + for i, in := range f.inputs { + f.inputs[i] = f.input(in.Path) + } + f.writeReceipt() +} + +// The guard classifies files one by one, so Git may read only refs, +// configuration and objects. A protected tracked file outside the inputs is +// never opened: it cannot decide the result even when it changes and nobody +// may read it. +func TestGitNeverReadsWorkingFiles(t *testing.T) { + cases := map[string]func(t *testing.T, name string){ + "timestamp changed": func(t *testing.T, name string) { + later := time.Now().Add(time.Hour) + if err := os.Chtimes(name, later, later); err != nil { + t.Fatal(err) + } + }, + "content changed": func(t *testing.T, name string) { writeFile(t, name, "private change\r\n") }, + } + for caseName, change := range cases { + t.Run(caseName, func(t *testing.T) { + f := newFixture(t, nil) + protected := f.path("README.md") + f.protected[key(protected)] = clientVerdict + change(t, protected) + release := denyReads(t, protected) + defer release() + got, reason := f.verify() + if !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q: a protected file outside the inputs decided the result", got.Qualified, reason) + } + if f.index("read:"+key(protected)) >= 0 || f.index("guard-path:"+key(protected)) >= 0 { + t.Fatalf("a file outside the inputs was classified or read: %v", f.events) + } + if expectGitObjectOnly(t, f) == 0 { + t.Fatalf("the source was verified without Git") + } + }) + } +} + +// With Git kept off the working files, the reader itself ties each pinned +// input to the fixed tree's blob, in the form Git writes it out on this PC. +func TestInputsMustBeTheCommittedBytes(t *testing.T) { + t.Run("LF working file is the blob itself", func(t *testing.T) { + f := newFixture(t, nil) + f.write("lib-a/src/lib.rs", fixtureTracked["lib-a/src/lib.rs"]) + f.rebuildReceipt() + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q for the blob written without conversion", got.Qualified, reason) + } + }) + cases := map[string]func(*fixture){ + "edited input with a receipt that agrees": func(f *fixture) { + f.write("lib-a/src/lib.rs", crlf("pub fn a() { panic!() }\n")) + }, + "mixed line endings": func(f *fixture) { + f.write("Cargo.toml", strings.Replace(crlf(fixtureTracked["Cargo.toml"]), "\r\n", "\n", 1)) + }, + } + for name, edit := range cases { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + edit(f) + f.rebuildReceipt() + expectReason(t, f, "input") + }) + } +} + +func TestCheckedOutForms(t *testing.T) { + // Object names printed by git hash-object. + if got := gitBlobID(nil); got != "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391" { + t.Fatalf("empty blob = %q", got) + } + if got := gitBlobID([]byte("hello\n")); got != "ce013625030ba8dba906f756967f9e9ca394464a" { + t.Fatalf("hello blob = %q", got) + } + cases := []struct { + name, working, blob string + want bool + }{ + {"the blob itself", "a\nb\n", "a\nb\n", true}, + {"every LF written as CRLF", "a\r\nb\r\n", "a\nb\n", true}, + {"a blob that already has CRLF", "a\r\nb\r\n", "a\r\nb\r\n", true}, + {"one LF left as LF", "a\r\nb\n", "a\nb\n", false}, + {"an extra CR", "a\r\r\nb\r\n", "a\r\nb\n", false}, + {"binary is never converted", "a\x00\r\nb\r\n", "a\x00\nb\n", false}, + {"other bytes", "a\r\nc\r\n", "a\nb\n", false}, + } + for _, tc := range cases { + if got := checkedOut([]byte(tc.working), gitBlobID([]byte(tc.blob))); got != tc.want { + t.Errorf("%s: checkedOut = %v, want %v", tc.name, got, tc.want) + } + } +} From 74c87e3042355bcc58931a04b73d75650d38ce47 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 03:16:20 +0900 Subject: [PATCH 05/24] fix: keep Git off working files in the ToDo3190 reader Git now reads only refs, configuration and objects (rev-parse, remote get-url, merge-base, ls-tree). The whole-tree git diff HEAD is gone; the reader hashes each guard-classified, pinned input itself and requires its bytes to be the fixed blob, either as is or with every LF written as CRLF. Tracked files outside the inputs are never classified or read. The spec now lists the argument reason and states exactly what the closure check covers. --- docs/todo-update-reader-3190.md | 30 ++++++++++------ .../checkout_windows.go | 34 ++++++++++++++++--- .../todo-update-reader-3190/verify_windows.go | 17 +++++----- 3 files changed, 59 insertions(+), 22 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index d44b805b..61a94118 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -31,9 +31,10 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト | `data_access_granted` / `original_registry_recovered` | 常に `false` | 拒否は標準出力に `{"qualified":false}` だけを出し、終了コード1。標準エラーには固定の理由コード -（`target_path`、`target_leaf`、`baseline_qualification`、`baseline_mismatch`、`source_root`、 +（`argument`、`target_path`、`target_leaf`、`baseline_qualification`、`baseline_mismatch`、`source_root`、 `source_git`、`receipt`、`log`、`dep_info`、`input`、`closure`、`candidate`、 -`guard_classification`、`recheck`）だけを出す。未知の現在SHA、パスの中身、receiptの本文は出さない。 +`guard_classification`、`recheck`）だけを出す。`argument` は引数の不足・重複・追加と相対パスで、照合を始めずに返す。 +未知の現在SHA、パスの中身、receiptの本文は出さない。 返却時にhandleは閉じるので、この結果は将来の置換を予約するものではない。置換の直前の再照合と 有限CASは親3190の配布工程が行う。置換後の状態はこのreaderの対象外で、置換後に同じ照合を求めない。 @@ -66,20 +67,26 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト 4. その後でだけ同じhandleから対象を読み、固定の旧SHAと一致することを確かめる。違えば拒否し、値を出さない。 5. source rootの祖先と自身が reparse point でないこと、実パス、guardの `--guard-path` が clear であることを確かめる。 Gitは `GIT_*` 環境を除き、窓を出さず、期限付きで呼ぶ。worktreeの最上位、origin、HEAD、tree、 - 追跡ファイルの差分なし（`core.autocrlf=true`）、`origin/main` の祖先であることを確かめる。 + `origin/main` の祖先であることを確かめる。Gitに読ませるのは参照・設定・オブジェクトだけで + （`rev-parse`、`remote get-url`、`merge-base`、`ls-tree`）、作業ファイルは開かせない。 + guardは1件ずつ分類するので、rootがclearでも、分類していない追跡ファイルをGitに読ませない。 6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 読んだbytesそのものをhashして解析する。 7. receiptの全項目を固定値と照合する。未知の項目と `data_access_granted` が false でない記録は拒否する。 8. 各入力は、固定treeの通常ファイル（mode 100644/100755）のblobとreceiptのblobが一致し、 - working fileのSHA256がreceiptと一致することを確かめる。 + working fileのSHA256がreceiptと一致することを確かめる。さらに、保持したhandleから読んだbytesの + Git blob値をreader自身が計算し、固定treeのblobと照合する。一致とみなすのは、bytesがblobそのものか、 + `core.autocrlf=true` でLFをすべてCRLFにした形の場合だけ。NUL、単独のCR、CRLFとLFの混在は変換として認めない。 + 入力以外の追跡ファイルは分類も読取もしないので、その変更は結果を左右しない。 9. 新候補のサイズとSHA256を固定値とreceiptの両方に照合する。 10. 最後に保持した全handleの同一性（volume・file ID・サイズ・更新時刻・リンク数・属性）と実パス、 - source rootの実パス、HEAD、追跡ファイルの差分なしを再確認する。照合中の変更は拒否する。 + source rootの実パス、worktreeの最上位・origin・HEADを再確認する。照合中の変更は拒否する。 + 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。 ## compiler closure の確認 dep-info はコンパイラが読んだローカルファイルの一覧である。次の条件をすべて満たす場合だけ、 -126件の入力がそのビルドのrepo内の入力として過不足ないとみなす。 +receiptの126件の入力が、原dep-infoの116件と manifest/lock の10件にちょうど対応するとみなす。 - dep-info の対象が `/target/release/todo.exe` で、全項目がsource root内の入力に含まれる。 - 入力からdep-infoの項目を除いた残りが、`Cargo.lock`、workspaceの `Cargo.toml`、 @@ -89,10 +96,11 @@ dep-info はコンパイラが読んだローカルファイルの一覧であ - 固定treeの各crateとrootに `build.rs`、`.cargo/config(.toml)` がなく、rootに `rust-toolchain(.toml)` がない。 各 `Cargo.toml` に `build =` の指定がない。 -repo外の要素は入力に含まれない。具体的には toolchain の版、利用者・上位ディレクトリのCargo設定と -環境変数（`RUSTFLAGS` やsccacheなど）、registry crate の展開済みsource（`--locked` と `Cargo.lock` の -checksumで版を固定）である。このreaderは再ビルドによる再現性の証明ではなく、候補・原記録・固定sourceの -相互の一致を確かめる。 +次のものは入力に含まれず、このreaderは照合しない。toolchain の版、利用者・上位ディレクトリのCargo設定と +環境変数（`RUSTFLAGS` やsccacheなど）、worktree内の未追跡ファイル（未追跡の `build.rs` や +`.cargo/config.toml` など。固定treeに無いことだけを確かめる）、registry crate の展開済みsourceの実bytes +（`--locked` と `Cargo.lock` のchecksumで版を固定）。このreaderは再ビルドによる再現性の証明ではなく、 +候補・原記録・固定sourceの相互の一致を確かめる。 ## 受入条件とテスト @@ -113,6 +121,8 @@ guardのCLI応答を模した関数で、実物の配布先・guard・親の作 | 元client一致 | `TestRejectsProtectedResources` | | data access の拡大 | `TestRejectsDataAccessExpansion` | | closure の過不足 | `TestRejectsClosureGaps` | +| Gitに作業ファイルを読ませない。入力以外の保護された追跡ファイルは結果を左右しない | `TestGitNeverReadsWorkingFiles`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | +| 入力のbytesが固定treeのblob（そのままか、LFをすべてCRLFにした形）である | `TestInputsMustBeTheCommittedBytes`、`TestCheckedOutForms` | | 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | symlinkの作成は権限が必要なため、合成テストは junction と hardlink で reparse 属性とリンク数の拒否を確かめる。 diff --git a/tools/todo-update-reader-3190/checkout_windows.go b/tools/todo-update-reader-3190/checkout_windows.go index 0c1d306f..256a6a01 100644 --- a/tools/todo-update-reader-3190/checkout_windows.go +++ b/tools/todo-update-reader-3190/checkout_windows.go @@ -1,8 +1,34 @@ package main -// gitBlobID is the SHA-1 object name Git gives data stored as a blob. -func gitBlobID(data []byte) string { return "" } +import ( + "bytes" + "crypto/sha1" + "encoding/hex" + "strconv" +) + +// gitBlobID is the SHA-1 object name Git gives data stored as a blob. It ties +// bytes to an entry of the fixed tree; SHA-256 pins the same bytes separately. +func gitBlobID(data []byte) string { + h := sha1.New() + h.Write([]byte("blob " + strconv.Itoa(len(data)) + "\x00")) + h.Write(data) + return hex.EncodeToString(h.Sum(nil)) +} // checkedOut reports whether working bytes are exactly a fixed blob as Git -// writes it out on this PC. -func checkedOut(data []byte, blob string) bool { return false } +// writes it out on this PC: the blob itself or, under core.autocrlf=true, the +// blob with every LF written as CRLF. Git converts only text, so bytes with a +// NUL, a lone CR or a lone LF are never accepted as a conversion. Only the +// reader's pinned bytes are hashed; Git never opens the working file. +func checkedOut(data []byte, blob string) bool { + if gitBlobID(data) == blob { + return true + } + pairs := bytes.Count(data, []byte("\r\n")) + if pairs == 0 || bytes.IndexByte(data, 0) >= 0 || + bytes.Count(data, []byte("\r")) != pairs || bytes.Count(data, []byte("\n")) != pairs { + return false + } + return gitBlobID(bytes.ReplaceAll(data, []byte("\r\n"), []byte("\n"))) == blob +} diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 6de4bf66..05c57183 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -171,7 +171,8 @@ func (s *session) evidence(name, pin string, limit int64, reason string) ([]byte } // inputs pins every receipt input, has the guard classify all of them, then -// hashes each pinned file. Manifest bytes are kept for the closure check. +// hashes each pinned file and ties its bytes to the fixed tree's blob. Git +// never reads these files. Manifest bytes are kept for the closure check. func (s *session) inputs(inputs []receiptInput, tree map[string]treeEntry, p profile) (map[string][]byte, string) { files := make([]*pinnedFile, len(inputs)) for i, in := range inputs { @@ -189,7 +190,7 @@ func (s *session) inputs(inputs []receiptInput, tree map[string]treeEntry, p pro manifests := map[string][]byte{} for i, in := range inputs { data, sum, ok := s.read(files[i]) - if !ok || sum != in.SHA256 { + if !ok || sum != in.SHA256 || !checkedOut(data, in.Blob) { return nil, "input" } if path.Base(in.Path) == "Cargo.toml" { @@ -295,7 +296,10 @@ func checkSource(p profile, env environment) (map[string]treeEntry, bool) { } // sourceStillFixed is checked before reading and again after every file is -// pinned: the same worktree top, origin, HEAD and clean tracked content. +// pinned: the same worktree top, origin and HEAD. Git reads only refs, +// configuration and objects. It never opens a working file: the guard +// classifies files one by one, and only the pinned inputs are compared, by +// the reader itself. func sourceStillFixed(p profile, env environment) bool { top, ok := gitText(env, p.root, "rev-parse", "--show-toplevel") if !ok || !strings.EqualFold(filepath.Clean(filepath.FromSlash(top)), p.root) { @@ -305,11 +309,8 @@ func sourceStillFixed(p profile, env environment) bool { if !ok || !slices.Contains(p.origins, strings.ToLower(origin)) { return false } - if head, ok := gitText(env, p.root, "rev-parse", "--verify", "HEAD"); !ok || head != p.head { - return false - } - _, err := env.git(p.root, "-c", "core.autocrlf=true", "diff", "--quiet", "--no-ext-diff", "--no-textconv", "HEAD", "--") - return err == nil + head, ok := gitText(env, p.root, "rev-parse", "--verify", "HEAD") + return ok && head == p.head } type limitedBuffer struct { From 308cb24d5090e440803f0448c8ccddc74525c98d Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 03:17:45 +0900 Subject: [PATCH 06/24] docs: record the ToDo3482 review and the measured Must fix Adds the review result (Must 1, Suggestion 2, Nit 1), the Red/Green/mutation measurements of the fix and the pre-merge observation of the reader built from 74c87e30, and separates official CI, re-review, merge and the post-merge check that remain. --- docs/todo-update-reader-3190.md | 48 ++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index 61a94118..e9912af0 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -130,14 +130,14 @@ symlink も同じ reparse 属性で拒否される。 ## 実測と未完了 -### Red（実装前） +### 最初の実装のRed commit `5b22ab24` は仕様、CLI、固定値、受入テストだけを含み、`verifyUpdate` は常に `unimplemented` を返す。 `tools/todo-update-reader-3190` で `go test -count=1 ./...` は終了コード1。検証系の14テストが subtest込み102行すべて `unimplemented` か `exactLeaf` の未実装で失敗し、fixture作成の失敗は0。 CLIの4テストはこの時点で実装済みの契約として合格した。コンパイル失敗をRedに数えていない。 -### Green（実装後） +### 最初の実装のGreen `go test -race -count=1 ./...` は終了コード0。上位18件、subtest込み106件が合格し、失敗0・skip 0・ data race 0、136秒。`go vet ./...` とgofmtも差分なし。Cargo、GPU、Civ6は使っていない。 @@ -147,13 +147,46 @@ data race 0、136秒。`go vet ./...` とgofmtも差分なし。Cargo、GPU、Ci 祖先のreparse確認を外した版は生き残った。handleの実パス照合が祖先のjunctionを独立に拒否するためで、 junctionの拒否自体はテストで確認している。 +### 別モデルのレビュー（ToDo3482） + +gpt-6.1-sol・推論max（job `20261024-022652-todo-3482`、6分34秒、終了コード0）が、base `dbb1c79c`・head `d9edfd4d` の +変更10ファイルを全文読んだ。レビュー担当はテスト・ビルド・CIを実行していない。結果は Must 1件、Suggestion 2件、Nit 1件。 + +- Must: 追跡ファイル全体への `git diff HEAD` が、1件ずつの保護判定より先にGitへ作業ファイルを読ませていた。 + 126件の入力以外にも読取が広がり、テストはreader自身の読取だけを見ていたので検出できなかった。 +- Suggestion: closureの節の「repo内の入力として過不足ない」が広く読める。PR本文でも、実測、正式CI、レビュー、 + merge、merge後の照合を分けて書く。 +- Nit: 理由コードの一覧に `argument` がない。 + +### Mustの修正 + +Gitに読ませるのを参照・設定・オブジェクト（`rev-parse`、`remote get-url`、`merge-base`、`ls-tree`）だけにし、 +作業ツリー全体の差分確認を外した。代わりに、guardの分類と固定の後で読んだ各入力のbytesから reader 自身が +Git blob値を計算し、固定treeのblobと照合する（検証の順序の8）。入力以外の追跡ファイルは分類も読取もしない。 +Suggestion と Nit はこの文書へ反映した。 + +- Red（commit `dd4b0273`、照合関数はstub）: `go test -count=1 ./...` は終了コード1、96秒。失敗はテスト6件と + subtest 24件、合格83、skip 0、ビルド失敗0で、すべて意図した理由による。正例と保護対象の18 subtestで、 + Gitの呼び出しに `git diff` が記録された。保護したREADMEの日時か中身を変えると `source_git` で拒否された。 + CRLFとLFが混在した入力は資格を得ていた。 +- Green（commit `74c87e30`）: `go test -race -count=1 ./...` は終了コード0、98.5秒。上位21件、subtest込み113件が合格し、 + 失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。 +- 新しい照合の要所を壊した3つの版（blob照合を呼ばない、NULの確認を外す、CRとLFの対の確認を外す）は、 + すべて対応するテストが失敗した。 + ### 実物への事前の観測（merge前） -実装commitから生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 +最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 `qualified:true`、current `f8710b51…b983`、new `d4392e63…0718`（8,017,920 bytes）、 source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力126件・dep-info 116件・crate 8件、 -`data_access_granted:false` を返した。これはmerge前のsourceによる観測で、最終の結果はmerge後のsourceから -生成したreaderで取り直す。対象の置換、guardの変更、ToDoの状態変更はしていない。 +`data_access_granted:false` を返した。 + +修正後のcommit `74c87e30` から `go build -trimpath` で生成したreader（SHA256 `c4fb0ad3…c11b`、vcs.modified=false） +でも実物を1回照合した。終了コード0、8.7秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。実物の126件はすべて、固定treeのblobそのものか、LFをすべてCRLFにした形だった。 + +どちらもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 +対象の置換、guardの変更、ToDoの状態変更はしていない。 補足として、親の作業場所に残るCargoの出力 `target/release/todo.exe` は、配布用コピーと同じ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/deps/todo.exe` とhardlinkされている。 @@ -161,5 +194,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d ### 未完了 -正式CI、別モデルのレビューとMustの解消、通常merge、merge後のsourceから生成したreaderによる実物の照合、 -親3190への結果の返却が残る。完了した時点でこの節に追記する。 +修正前のhead `d9edfd4d` では、専用workflow（windows、run 39994455364）とCodeQL（run 39994455228）が成功した。 +修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる実物の照合、 +親3190への結果の返却が残る。 From 6bda9f63d986a7a852a70864e862111c39208315 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 03:55:46 +0900 Subject: [PATCH 07/24] test: require Git to read only objects in the ToDo3190 reader Review ToDo3491 (Must): Git still read configuration, and so any include, before the guard classified it. The fixture is now a linked worktree like the real source. These tests demand exactly three object reads with exact arguments, ignore a global include of a protected unreadable file, refuse repository includes, includeIf and URL rewrites before Git runs, and read the worktree link, HEAD, refs, configuration and shallow boundary through the guard. isolatedEnvironment is a stub here (Red). --- .../isolated_git_windows.go | 4 + .../reader_windows_test.go | 51 ++-- .../source_scope_windows_test.go | 225 +++++++++++++++--- 3 files changed, 231 insertions(+), 49 deletions(-) create mode 100644 tools/todo-update-reader-3190/isolated_git_windows.go diff --git a/tools/todo-update-reader-3190/isolated_git_windows.go b/tools/todo-update-reader-3190/isolated_git_windows.go new file mode 100644 index 00000000..0887589d --- /dev/null +++ b/tools/todo-update-reader-3190/isolated_git_windows.go @@ -0,0 +1,4 @@ +package main + +// isolatedEnvironment is the environment of the scratch Git process. +func isolatedEnvironment(parent []string, gitDir, objects string) []string { return nil } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index abaa9b2c..66d4d5fc 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -56,6 +56,10 @@ type fixture struct { t *testing.T base string root string + repo string + common string + gitDir string + main string target string receipt map[string]any inputs []fixtureInput @@ -87,24 +91,29 @@ func newFixture(t *testing.T, edit func(map[string]string)) *fixture { } base := testLongPath(t, t.TempDir()) f := &fixture{ - t: t, base: base, root: filepath.Join(base, "source"), + t: t, base: base, root: filepath.Join(base, "source"), repo: filepath.Join(base, "repo"), target: filepath.Join(base, "tools", "todo", "todo.exe"), protected: map[string]string{}, } - if err := os.MkdirAll(f.root, 0o755); err != nil { + if err := os.MkdirAll(f.repo, 0o755); err != nil { t.Fatal(err) } - f.git("init", "-q", "-b", "main") - f.git("config", "core.autocrlf", "true") - f.git("remote", "add", "origin", "https://github.com/AutoDevJapan/cc-pc-tools-rs.git") + // Mirror the real source: a linked worktree of a repository elsewhere, + // checked out on Windows with CRLF working files and LF blobs. + f.gitIn(f.repo, "init", "-q", "-b", "main") + f.gitIn(f.repo, "config", "core.autocrlf", "true") + f.gitIn(f.repo, "remote", "add", "origin", "https://github.com/AutoDevJapan/cc-pc-tools-rs.git") for name, content := range tracked { - // Mirror the Windows checkout: CRLF working files, LF blobs. - f.write(name, crlf(content)) - } - f.git("add", "-A") - f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "-m", "fixture") - f.git("update-ref", "refs/remotes/origin/main", "HEAD") + writeFile(t, filepath.Join(f.repo, filepath.FromSlash(name)), crlf(content)) + } + f.gitIn(f.repo, "add", "-A") + f.gitIn(f.repo, "-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "-m", "fixture") + f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", "HEAD") + f.gitIn(f.repo, "worktree", "add", "-q", "-b", "work", f.root) + f.common = filepath.Join(f.repo, ".git") + f.gitDir = filepath.Join(f.common, "worktrees", "source") head, tree := f.gitOut("rev-parse", "HEAD"), f.gitOut("rev-parse", "HEAD^{tree}") + f.main = head writeFile(t, f.target, fixtureBaseline) f.write(fixtureCandidateName, fixtureCandidate) @@ -207,7 +216,12 @@ func testGit(t *testing.T, root string, args ...string) (string, error) { func (f *fixture) git(args ...string) { f.t.Helper() - if out, err := testGit(f.t, f.root, args...); err != nil { + f.gitIn(f.root, args...) +} + +func (f *fixture) gitIn(dir string, args ...string) { + f.t.Helper() + if out, err := testGit(f.t, dir, args...); err != nil { f.t.Fatalf("git %v: %v\n%s", args, err, out) } } @@ -402,7 +416,10 @@ func TestQualifiesFixedUpdate(t *testing.T) { } // Every file the reader reads is classified by the official guard first; // the installed leaf is qualified only by the existing purpose entry. - paths := []string{f.root, f.path(fixtureReceiptName), f.path(fixtureLogName), f.path("target/release/todo.d"), f.path(fixtureCandidateName)} + paths := []string{f.root, f.path(fixtureReceiptName), f.path(fixtureLogName), f.path("target/release/todo.d"), f.path(fixtureCandidateName), + f.path(".git"), filepath.Join(f.gitDir, "gitdir"), filepath.Join(f.gitDir, "commondir"), filepath.Join(f.gitDir, "HEAD"), + filepath.Join(f.common, "config"), filepath.Join(f.common, "refs", "heads", "work"), + filepath.Join(f.common, "refs", "remotes", "origin", "main"), filepath.Join(f.common, "objects")} for _, in := range f.inputs { paths = append(paths, f.path(in.Path)) } @@ -419,10 +436,8 @@ func TestQualifiesFixedUpdate(t *testing.T) { if f.index("guard-path:"+key(f.target)) >= 0 { t.Errorf("the general classifier was used for the installed leaf") } - // Git reads only refs, configuration and objects, never a working file. - if expectGitObjectOnly(t, f) == 0 { - t.Errorf("the source was verified without Git") - } + // Git ran only the three object reads, isolated from every other file. + expectObjectReads(t, f) } func TestBaselineIsReadOnlyAfterPurposeQualification(t *testing.T) { @@ -776,7 +791,7 @@ func TestRejectsProtectedResources(t *testing.T) { if f.index("read:"+key(f.path(rel))) >= 0 { t.Fatalf("%s was read although the guard refused it", rel) } - expectGitObjectOnly(t, f) + expectOnlyObjectReads(t, f) }) } } diff --git a/tools/todo-update-reader-3190/source_scope_windows_test.go b/tools/todo-update-reader-3190/source_scope_windows_test.go index c7c5c358..8a669045 100644 --- a/tools/todo-update-reader-3190/source_scope_windows_test.go +++ b/tools/todo-update-reader-3190/source_scope_windows_test.go @@ -2,47 +2,62 @@ package main import ( "os" + "path/filepath" + "slices" "strings" "syscall" "testing" "time" ) -// objectOnlyGit lists the Git commands that read only refs, configuration and -// objects. Others, such as diff, status, ls-files, hash-object or add, can open -// working files that the guard has not classified one by one. -var objectOnlyGit = map[string]bool{"rev-parse": true, "remote": true, "merge-base": true, "ls-tree": true} - -func gitSubcommand(args string) string { - fields := strings.Fields(args) - for i := 0; i < len(fields); i++ { - if fields[i] == "-c" { - i++ - continue - } - return fields[i] +// objectReads are the only Git commands the reader may run, with exactly these +// arguments. Each reads commits or trees from the fixed object store. +func objectReads(f *fixture) []string { + return []string{ + "rev-parse --verify --quiet " + f.p.head + "^{tree}", + "merge-base --is-ancestor " + f.p.head + " " + f.main, + "ls-tree -r -z --full-tree " + f.p.head, } - return "" } -// expectGitObjectOnly fails on any recorded Git call that can read working -// files and returns how many Git calls the reader made. -func expectGitObjectOnly(t *testing.T, f *fixture) int { - t.Helper() +func gitCalls(f *fixture) []string { f.mu.Lock() defer f.mu.Unlock() - calls := 0 + var calls []string for _, event := range f.events { if args, ok := strings.CutPrefix(event, "git:"); ok { - calls++ - if !objectOnlyGit[gitSubcommand(args)] { - t.Errorf("git %s can read working files the guard has not classified", args) - } + calls = append(calls, args) } } return calls } +// expectOnlyObjectReads fails on any Git call other than the object reads. +func expectOnlyObjectReads(t *testing.T, f *fixture) { + t.Helper() + allowed := objectReads(f) + for _, call := range gitCalls(f) { + if !slices.Contains(allowed, call) { + t.Errorf("git %s is not one of the object reads", call) + } + } +} + +// expectObjectReads requires exactly the object reads, each once, in order. +func expectObjectReads(t *testing.T, f *fixture) { + t.Helper() + if got, want := gitCalls(f), objectReads(f); !slices.Equal(got, want) { + t.Errorf("git calls\n got %q\nwant %q", got, want) + } +} + +func expectUntouched(t *testing.T, f *fixture, name string) { + t.Helper() + if f.index("read:"+key(name)) >= 0 || f.index("guard-path:"+key(name)) >= 0 { + t.Fatalf("%s was classified or read: %v", name, f.events) + } +} + // denyReads holds name open without sharing, so every other open fails until // the returned function closes it. func denyReads(t *testing.T, name string) func() { @@ -62,6 +77,18 @@ func denyReads(t *testing.T, name string) func() { return func() { syscall.CloseHandle(h) } } +func appendFile(t *testing.T, name, content string) { + t.Helper() + file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY, 0) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteString(content); err != nil { + t.Fatal(err) + } +} + // rebuildReceipt records the working inputs as they are now, as a receipt // written after the change would; the blobs stay those of the fixed tree. func (f *fixture) rebuildReceipt() { @@ -72,10 +99,9 @@ func (f *fixture) rebuildReceipt() { f.writeReceipt() } -// The guard classifies files one by one, so Git may read only refs, -// configuration and objects. A protected tracked file outside the inputs is -// never opened: it cannot decide the result even when it changes and nobody -// may read it. +// The guard classifies files one by one, so Git never opens a working file. +// A protected tracked file outside the inputs cannot decide the result even +// when it changes and nobody may read it. func TestGitNeverReadsWorkingFiles(t *testing.T) { cases := map[string]func(t *testing.T, name string){ "timestamp changed": func(t *testing.T, name string) { @@ -98,16 +124,153 @@ func TestGitNeverReadsWorkingFiles(t *testing.T) { if !got.Qualified || reason != "" { t.Fatalf("verify = qualified %v reason %q: a protected file outside the inputs decided the result", got.Qualified, reason) } - if f.index("read:"+key(protected)) >= 0 || f.index("guard-path:"+key(protected)) >= 0 { - t.Fatalf("a file outside the inputs was classified or read: %v", f.events) + expectUntouched(t, f, protected) + expectObjectReads(t, f) + }) + } +} + +// Git reads no configuration file. An include that names a protected, +// unreadable file never reaches Git, and the reader refuses a repository +// configuration that Git would have to follow or that rewrites the remote. +func TestGitReadsNoConfiguration(t *testing.T) { + t.Run("global include", func(t *testing.T) { + f := newFixture(t, nil) + protected := f.path("README.md") + f.protected[key(protected)] = clientVerdict + home := filepath.Join(f.base, "home") + writeFile(t, filepath.Join(home, ".gitconfig"), "[include]\n\tpath = "+filepath.ToSlash(protected)+"\n") + t.Setenv("HOME", home) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, "xdg")) + release := denyReads(t, protected) + defer release() + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q: the global configuration reached Git", got.Qualified, reason) + } + expectUntouched(t, f, protected) + expectObjectReads(t, f) + }) + for name, section := range map[string]string{ + "include": "[include]\n\tpath = PROTECTED\n", + "conditional include": "[includeIf \"gitdir:C:/\"]\n\tpath = PROTECTED\n", + "URL rewrite": "[url \"https://example.invalid/\"]\n\tinsteadOf = https://github.com/\n", + } { + t.Run("repository "+name, func(t *testing.T) { + f := newFixture(t, nil) + protected := f.path("README.md") + f.protected[key(protected)] = clientVerdict + appendFile(t, filepath.Join(f.common, "config"), strings.ReplaceAll(section, "PROTECTED", filepath.ToSlash(protected))) + release := denyReads(t, protected) + defer release() + expectReason(t, f, "source_git") + if calls := gitCalls(f); len(calls) != 0 { + t.Fatalf("Git ran %q before the configuration was refused", calls) + } + expectUntouched(t, f, protected) + }) + } +} + +// The reader reads the worktree link, HEAD, refs, configuration and shallow +// boundary itself, each classified by the guard first, and refuses what it +// cannot tie to the fixed source. +func TestReadsGitMetadataItself(t *testing.T) { + t.Run("branch only in packed-refs", func(t *testing.T) { + f := newFixture(t, nil) + f.gitIn(f.repo, "pack-refs", "--all") + if _, err := os.Lstat(filepath.Join(f.common, "refs", "heads", "work")); !os.IsNotExist(err) { + t.Fatalf("pack-refs kept the loose branch: %v", err) + } + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q with packed refs", got.Qualified, reason) + } + expectObjectReads(t, f) + }) + t.Run("shallow boundary", func(t *testing.T) { + f := newFixture(t, nil) + writeFile(t, filepath.Join(f.common, "shallow"), f.p.head+"\n") + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q with a shallow boundary", got.Qualified, reason) + } + expectObjectReads(t, f) + }) + rejections := map[string]func(f *fixture){ + "worktree file names another worktree": func(f *fixture) { + f.gitIn(f.repo, "worktree", "add", "-q", "-b", "other", filepath.Join(f.base, "other")) + writeFile(f.t, f.path(".git"), "gitdir: "+filepath.ToSlash(filepath.Join(f.common, "worktrees", "other"))+"\n") + }, + "worktree registration names another checkout": func(f *fixture) { + writeFile(f.t, filepath.Join(f.gitDir, "gitdir"), filepath.ToSlash(filepath.Join(f.base, "elsewhere", ".git"))+"\n") + }, + "common directory elsewhere": func(f *fixture) { + writeFile(f.t, filepath.Join(f.gitDir, "commondir"), "../../..\n") + }, + "HEAD outside refs": func(f *fixture) { + writeFile(f.t, filepath.Join(f.gitDir, "HEAD"), "ref: refs/heads/../../config\n") + }, + "symbolic remote-tracking ref": func(f *fixture) { + writeFile(f.t, filepath.Join(f.common, "refs", "remotes", "origin", "main"), "ref: refs/heads/main\n") + }, + "per-worktree configuration": func(f *fixture) { + f.gitIn(f.repo, "config", "extensions.worktreeConfig", "true") + writeFile(f.t, filepath.Join(f.gitDir, "config.worktree"), "[core]\n\tbare = false\n") + }, + "alternate object store": func(f *fixture) { + writeFile(f.t, filepath.Join(f.common, "objects", "info", "alternates"), filepath.ToSlash(filepath.Join(f.base, "elsewhere"))+"\n") + }, + "reftable refs": func(f *fixture) { + appendFile(f.t, filepath.Join(f.common, "config"), "[extensions]\n\trefStorage = reftable\n") + }, + "second origin URL": func(f *fixture) { + appendFile(f.t, filepath.Join(f.common, "config"), "[remote \"origin\"]\n\turl = https://github.com/AutoDevJapan/cc-pc-tools-rs.git\n") + }, + "configuration line not understood": func(f *fixture) { + appendFile(f.t, filepath.Join(f.common, "config"), "[core]\n\tbroken = \"unterminated\n") + }, + "malformed shallow boundary": func(f *fixture) { + writeFile(f.t, filepath.Join(f.common, "shallow"), "not a commit\n") + }, + } + for name, edit := range rejections { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + edit(f) + expectReason(t, f, "source_git") + expectOnlyObjectReads(t, f) + }) + } + for name, path := range map[string]func(*fixture) string{ + "worktree file": func(f *fixture) string { return f.path(".git") }, + "worktree HEAD": func(f *fixture) string { return filepath.Join(f.gitDir, "HEAD") }, + "repository configuration": func(f *fixture) string { return filepath.Join(f.common, "config") }, + "remote-tracking ref": func(f *fixture) string { return filepath.Join(f.common, "refs", "remotes", "origin", "main") }, + "object store": func(f *fixture) string { return filepath.Join(f.common, "objects") }, + } { + t.Run("protected "+name, func(t *testing.T) { + f := newFixture(t, nil) + refused := path(f) + f.protected[key(refused)] = clientVerdict + expectReason(t, f, "guard_classification") + if f.index("read:"+key(refused)) >= 0 { + t.Fatalf("%s was read although the guard refused it", refused) } - if expectGitObjectOnly(t, f) == 0 { - t.Fatalf("the source was verified without Git") + if calls := gitCalls(f); len(calls) != 0 { + t.Fatalf("Git ran %q although the source was refused", calls) } }) } } +func TestIsolatedGitEnvironment(t *testing.T) { + parent := []string{`PATH=C:\bin`, `HOME=C:\home`, `GIT_DIR=C:\evil`, `Git_Config_Global=C:\evil.cfg`, "GIT_CONFIG_COUNT=1", "GIT_CONFIG_KEY_0=include.path"} + got := isolatedEnvironment(parent, `C:\tmp\g`, `C:\repo\.git\objects`) + want := []string{`PATH=C:\bin`, `HOME=C:\home`, `GIT_DIR=C:\tmp\g`, `GIT_OBJECT_DIRECTORY=C:\repo\.git\objects`, + "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_NO_LAZY_FETCH=1", "GIT_OPTIONAL_LOCKS=0", "GIT_TERMINAL_PROMPT=0"} + if !slices.Equal(got, want) { + t.Fatalf("environment\n got %q\nwant %q", got, want) + } +} + // With Git kept off the working files, the reader itself ties each pinned // input to the fixed tree's blob, in the form Git writes it out on this PC. func TestInputsMustBeTheCommittedBytes(t *testing.T) { From 2f804f13947e22572223eee7b45e6588b05da6ef Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 04:51:14 +0900 Subject: [PATCH 08/24] fix: let Git read only the object store in the ToDo3190 reader Review ToDo3491 (Must): `git remote get-url` and the other source checks let Git read the system, global and repository configuration, and so any include.path or includeIf target, before the guard classified it. The reader now reads the worktree link, its registration (commondir, gitdir), HEAD, the branch and origin/main refs (loose, else packed-refs), the repository configuration and the shallow boundary itself. The guard classifies each file first; the reader reads it once from a plain single-link handle and closes it so the other worktrees can keep updating the shared repository. A strict parser reads the configuration as Git does and refuses include, includeIf, url, core keys other than those of a normal non-bare repository, extensions other than worktreeConfig, and syntax it cannot read the same way (continuations, text after a header, the old [a.b] form, unknown escapes, open quotes, control characters). config.worktree, info/grafts and objects/info/alternates must be absent. Git then runs in an empty scratch repository whose only outside data is the classified object store and a copy of the shallow boundary, with no GIT_* inheritance and no system or global configuration. It runs exactly rev-parse ^{tree}, merge-base --is-ancestor and ls-tree. The recheck reopens every released metadata file and compares identity and SHA-256, and requires every absent file to stay absent. Tests added with the fix: a shallow clone whose parent object is missing, a bare or core.worktree configuration, refs packed or alternates added or a ref rewritten in place before the recheck, Git's own reading of each accepted configuration (git config --list) and the packed-refs format. go test -race: 153 passed, 0 failed. Eleven mutants of the new checks are each killed by the intended test. --- .../gitmeta_windows.go | 510 ++++++++++++++++++ .../gitmeta_windows_test.go | 112 ++++ .../isolated_git_windows.go | 62 ++- tools/todo-update-reader-3190/pin_windows.go | 60 ++- .../reader_windows_test.go | 31 +- .../source_scope_windows_test.go | 21 +- .../todo-update-reader-3190/verify_windows.go | 83 ++- 7 files changed, 817 insertions(+), 62 deletions(-) create mode 100644 tools/todo-update-reader-3190/gitmeta_windows.go create mode 100644 tools/todo-update-reader-3190/gitmeta_windows_test.go diff --git a/tools/todo-update-reader-3190/gitmeta_windows.go b/tools/todo-update-reader-3190/gitmeta_windows.go new file mode 100644 index 00000000..ed0c1563 --- /dev/null +++ b/tools/todo-update-reader-3190/gitmeta_windows.go @@ -0,0 +1,510 @@ +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" +) + +// maxMetadataBytes bounds each Git metadata file the reader reads itself. +const maxMetadataBytes = 4 << 20 + +// gitSource is all that Git sees of the fixed source: the repository's object +// store and, for a shallow clone, its boundary commits. Git reads no +// configuration, reference or working file of the source; the reader reads +// those itself, each after the guard has classified it. +type gitSource struct { + objects string + shallow []byte +} + +// readSource ties the source root to its repository from bytes the reader +// read itself, the way Git's files backend would: the worktree link and its +// registration, HEAD and the branch it names, the origin in the repository +// configuration and the main ref. It refuses what Git would read in another +// way: an include or URL rewrite, per-worktree configuration, grafts, +// alternates or another ref storage. It returns the object store Git may read +// and the commit of the main ref. +func (s *session) readSource(p profile) (gitSource, string, string) { + fail := func(reason string) (gitSource, string, string) { return gitSource{}, "", reason } + + link, reason := s.required(filepath.Join(p.root, ".git")) + if reason != "" { + return fail(reason) + } + gitDir, ok := gitdirLink(link) + if !ok || !strings.EqualFold(filepath.Base(filepath.Dir(gitDir)), "worktrees") { + return fail("source_git") + } + common := filepath.Dir(filepath.Dir(gitDir)) + commondir, reason := s.required(filepath.Join(gitDir, "commondir")) + if reason != "" { + return fail(reason) + } + backlink, reason := s.required(filepath.Join(gitDir, "gitdir")) + if reason != "" { + return fail(reason) + } + back, ok := strings.CutSuffix(string(backlink), "\n") + if string(commondir) != "../..\n" || !ok || !strings.EqualFold(filepath.FromSlash(back), filepath.Join(p.root, ".git")) { + return fail("source_git") + } + + // Nothing may redirect what Git reads for this worktree. + for _, path := range []string{ + filepath.Join(gitDir, "config.worktree"), + filepath.Join(common, "info", "grafts"), + filepath.Join(common, "objects", "info", "alternates"), + } { + if !s.absentFile(path) { + return fail("source_git") + } + } + + config, reason := s.required(filepath.Join(common, "config")) + if reason != "" { + return fail(reason) + } + if entries, ok := parseConfig(config); !ok || !sourceConfig(entries, p.origins) { + return fail("source_git") + } + + headFile, reason := s.required(filepath.Join(gitDir, "HEAD")) + if reason != "" { + return fail(reason) + } + branch, head, ok := headTarget(headFile) + if !ok { + return fail("source_git") + } + refs := &refReader{s: s, common: common} + if branch != "" { + if head, reason = refs.resolve(branch); reason != "" { + return fail(reason) + } + } + if head != p.head { + return fail("source_git") + } + mainID, reason := refs.resolve(p.mainRef) + if reason != "" { + return fail(reason) + } + + shallow, present, reason := s.metadata(filepath.Join(common, "shallow"), false) + if reason != "" { + return fail(reason) + } + if present && !shallowBoundary(shallow) { + return fail("source_git") + } + + objects := filepath.Join(common, "objects") + if _, err := os.Lstat(objects); err != nil { + return fail("source_git") + } + if !guardClear(s.env, objects) { + return fail("guard_classification") + } + if !s.pinDir(objects) { + return fail("source_git") + } + return gitSource{objects: objects, shallow: shallow}, mainID, "" +} + +// metadata reads one Git metadata file. The guard classifies it first; then it +// is read once from a plain single-link handle and closed again, so the +// worktrees sharing the repository can keep updating it. The recheck compares +// its identity and digest. A missing optional file is recorded and must stay +// missing; a missing required file refuses the source. +func (s *session) metadata(path string, required bool) ([]byte, bool, string) { + if _, err := os.Lstat(path); err != nil { + if !required && os.IsNotExist(err) { + s.absent = append(s.absent, path) + return nil, false, "" + } + return nil, false, "source_git" + } + if !guardClear(s.env, path) { + return nil, false, "guard_classification" + } + f, ok := s.open(path, maxMetadataBytes) + if !ok { + return nil, false, "source_git" + } + data, sum, ok := s.read(f) + if !ok { + f.file.Close() + return nil, false, "source_git" + } + s.release(f, sum) + return data, true, "" +} + +func (s *session) required(path string) ([]byte, string) { + data, _, reason := s.metadata(path, true) + return data, reason +} + +// refReader resolves a ref as Git's files backend does: a loose file wins, +// otherwise packed-refs. A missing loose file must stay missing. +type refReader struct { + s *session + common string + packed map[string]string + loaded bool +} + +func (r *refReader) resolve(name string) (string, string) { + if !refName(name) { + return "", "source_git" + } + data, present, reason := r.s.metadata(filepath.Join(r.common, filepath.FromSlash(name)), false) + if reason != "" { + return "", reason + } + if present { + id, ok := strings.CutSuffix(string(data), "\n") + if !ok || !hexDigest(id, 40) { + return "", "source_git" + } + return id, "" + } + if !r.loaded { + data, present, reason := r.s.metadata(filepath.Join(r.common, "packed-refs"), false) + if reason != "" { + return "", reason + } + r.loaded = true + if present { + refs, ok := packedRefs(data) + if !ok { + return "", "source_git" + } + r.packed = refs + } + } + id, ok := r.packed[name] + if !ok { + return "", "source_git" + } + return id, "" +} + +// gitdirLink reads a worktree's .git file: "gitdir: " and the absolute path +// of its registration, as Git writes it. +func gitdirLink(data []byte) (string, bool) { + line, ok := strings.CutSuffix(string(data), "\n") + value, found := strings.CutPrefix(line, "gitdir: ") + if !ok || !found { + return "", false + } + dir := filepath.FromSlash(value) + if len(dir) < 4 || !filepath.IsAbs(dir) || strings.HasPrefix(dir, `\\`) || filepath.Clean(dir) != dir || + strings.ContainsFunc(dir[2:], func(r rune) bool { return r < ' ' || strings.ContainsRune(`:*?"<>|`, r) }) { + return "", false + } + return dir, true +} + +// headTarget reads a worktree HEAD: a branch under refs/heads or, detached, +// a commit. +func headTarget(data []byte) (branch, commit string, ok bool) { + line, ok := strings.CutSuffix(string(data), "\n") + if !ok { + return "", "", false + } + if name, found := strings.CutPrefix(line, "ref: "); found { + if !strings.HasPrefix(name, "refs/heads/") || !refName(name) { + return "", "", false + } + return name, "", true + } + if !hexDigest(line, 40) { + return "", "", false + } + return "", line, true +} + +// refName accepts a ref that the files backend stores as a plain path: parts +// of letters, digits, '-', '_' and '.', none starting with a dot or ending in +// a dot or ".lock", and no "..". +func refName(name string) bool { + if !strings.HasPrefix(name, "refs/") || !validRelPath(name) || strings.Contains(name, "..") { + return false + } + for _, part := range strings.Split(name, "/") { + if strings.HasPrefix(part, ".") || strings.HasSuffix(part, ".lock") { + return false + } + } + return true +} + +// packedRefs reads the packed-refs file Git writes: an optional header line, +// " " lines and "^" lines that peel the tag before them. It +// refuses a repeated name and anything else. +func packedRefs(data []byte) (map[string]string, bool) { + text, ok := strings.CutSuffix(string(data), "\n") + if !ok { + return nil, false + } + refs := map[string]string{} + peelable := false + for i, line := range strings.Split(text, "\n") { + if i == 0 && strings.HasPrefix(line, "# pack-refs with:") { + continue + } + if id, found := strings.CutPrefix(line, "^"); found { + if !peelable || !hexDigest(id, 40) { + return nil, false + } + peelable = false + continue + } + id, name, found := strings.Cut(line, " ") + if !found || !hexDigest(id, 40) || !strings.HasPrefix(name, "refs/") || + strings.ContainsFunc(name, func(r rune) bool { return r <= ' ' || r == 0x7f }) { + return nil, false + } + if _, dup := refs[name]; dup { + return nil, false + } + refs[name] = id + peelable = true + } + return refs, true +} + +// shallowBoundary accepts the shallow file Git writes: one commit per line. +func shallowBoundary(data []byte) bool { + text, ok := strings.CutSuffix(string(data), "\n") + if !ok { + return false + } + for _, line := range strings.Split(text, "\n") { + if !hexDigest(line, 40) { + return false + } + } + return true +} + +// configEntry is one line of a Git configuration file: a section header +// (empty key) or a variable. Section and key names are lower case, the +// subsection keeps its case, and the value is the one Git reads. +type configEntry struct { + section, subsection, key, value string + hasValue bool +} + +// parseConfig reads Git's configuration syntax. It accepts a subset of what +// Git accepts and reads every accepted line as Git does. It refuses a +// continuation line, text after a section header, the old [section.sub] form, +// an empty subsection, a variable outside a section, an unknown escape, an +// open quote and any control character other than a tab; refusing only +// narrows what qualifies. +func parseConfig(data []byte) ([]configEntry, bool) { + text := strings.ReplaceAll(strings.TrimPrefix(string(data), "\ufeff"), "\r\n", "\n") + var entries []configEntry + section, subsection, inSection := "", "", false + for _, line := range strings.Split(text, "\n") { + if strings.ContainsFunc(line, func(r rune) bool { return r < ' ' && r != '\t' || r == 0x7f }) { + return nil, false + } + line = strings.TrimLeft(line, " \t") + switch { + case line == "" || line[0] == '#' || line[0] == ';': + case line[0] == '[': + name, sub, rest, ok := sectionHeader(line[1:]) + if !ok || !blankOrComment(rest) { + return nil, false + } + section, subsection, inSection = name, sub, true + entries = append(entries, configEntry{section: section, subsection: subsection}) + default: + key, value, hasValue, ok := configVariable(line) + if !ok || !inSection { + return nil, false + } + entries = append(entries, configEntry{section, subsection, key, value, hasValue}) + } + } + return entries, true +} + +func keyChar(c byte) bool { + return c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' +} + +func blankOrComment(rest string) bool { + rest = strings.TrimLeft(rest, " \t") + return rest == "" || rest[0] == '#' || rest[0] == ';' +} + +// sectionHeader parses what follows "[": name] or name "subsection"]. In the +// subsection only the escapes \" and \\ are accepted. +func sectionHeader(s string) (name, sub, rest string, ok bool) { + i := 0 + for i < len(s) && keyChar(s[i]) { + i++ + } + if i == 0 || i == len(s) { + return "", "", "", false + } + name = strings.ToLower(s[:i]) + if s[i] == ']' { + return name, "", s[i+1:], true + } + if s[i] != ' ' && s[i] != '\t' { + return "", "", "", false + } + for i < len(s) && (s[i] == ' ' || s[i] == '\t') { + i++ + } + if i == len(s) || s[i] != '"' { + return "", "", "", false + } + var b strings.Builder + for i++; i < len(s); i++ { + switch s[i] { + case '"': + if b.Len() == 0 || i+1 == len(s) || s[i+1] != ']' { + return "", "", "", false + } + return name, b.String(), s[i+2:], true + case '\\': + if i++; i == len(s) || s[i] != '"' && s[i] != '\\' { + return "", "", "", false + } + } + b.WriteByte(s[i]) + } + return "", "", "", false +} + +// configVariable parses "name", "name = value" or "name=value". +func configVariable(line string) (key, value string, hasValue, ok bool) { + c := line[0] + if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z') { + return "", "", false, false + } + i := 0 + for i < len(line) && keyChar(line[i]) { + i++ + } + key = strings.ToLower(line[:i]) + for i < len(line) && (line[i] == ' ' || line[i] == '\t') { + i++ + } + if i == len(line) { + return key, "", false, true + } + if line[i] != '=' { + return "", "", false, false + } + value, ok = configValue(line[i+1:]) + return key, value, true, ok +} + +// configValue reads a value as Git does: leading blanks are skipped, inner +// blanks kept, trailing blanks and a comment outside quotes dropped, quotes +// removed and the escapes \" \\ \n \t \b decoded. +func configValue(s string) (string, bool) { + var b strings.Builder + quote := false + trim := -1 + for i := 0; i < len(s); i++ { + c := s[i] + if (c == ' ' || c == '\t') && !quote { + if b.Len() > 0 { + if trim < 0 { + trim = b.Len() + } + b.WriteByte(c) + } + continue + } + if !quote && (c == '#' || c == ';') { + break + } + trim = -1 + switch c { + case '"': + quote = !quote + continue + case '\\': + if i++; i == len(s) { + return "", false + } + switch s[i] { + case 't': + c = '\t' + case 'b': + c = '\b' + case 'n': + c = '\n' + case '\\', '"': + c = s[i] + default: + return "", false + } + } + b.WriteByte(c) + } + if quote { + return "", false + } + value := b.String() + if trim >= 0 { + value = value[:trim] + } + return value, true +} + +// sourceConfig checks what the configuration means for this repository: the +// origin has exactly one URL and it is a fixed origin; the core keys are those +// of a normal non-bare repository; the only extension is worktreeConfig; and +// no include, conditional include or URL rewrite changes what Git reads. +func sourceConfig(entries []configEntry, origins []string) bool { + urls := 0 + for _, e := range entries { + switch e.section { + case "include", "includeif", "url": + return false + case "core": + if e.subsection != "" || !coreSetting(e) { + return false + } + case "extensions": + if e.subsection != "" || e.key != "" && (e.key != "worktreeconfig" || !configBool(e)) { + return false + } + case "remote": + if e.subsection == "origin" && e.key == "url" { + if !e.hasValue || !slices.Contains(origins, strings.ToLower(e.value)) { + return false + } + urls++ + } + } + } + return urls == 1 +} + +func coreSetting(e configEntry) bool { + switch e.key { + case "", "filemode", "logallrefupdates", "ignorecase", "symlinks", "autocrlf", "longpaths": + return true + case "repositoryformatversion": + return e.hasValue && (e.value == "0" || e.value == "1") + case "bare": + return e.hasValue && slices.Contains([]string{"false", "no", "off", "0"}, strings.ToLower(e.value)) + } + return false +} + +func configBool(e configEntry) bool { + return !e.hasValue || slices.Contains([]string{"true", "false", "yes", "no", "on", "off", "1", "0"}, strings.ToLower(e.value)) +} diff --git a/tools/todo-update-reader-3190/gitmeta_windows_test.go b/tools/todo-update-reader-3190/gitmeta_windows_test.go new file mode 100644 index 00000000..347a8539 --- /dev/null +++ b/tools/todo-update-reader-3190/gitmeta_windows_test.go @@ -0,0 +1,112 @@ +package main + +import ( + "os" + "os/exec" + "path/filepath" + "slices" + "strings" + "syscall" + "testing" +) + +// gitConfigList is what Git itself reads from a configuration file: one +// "name" or "name\nvalue" per variable, in file order. +func gitConfigList(t *testing.T, config string) []string { + t.Helper() + name := filepath.Join(t.TempDir(), "config") + if err := os.WriteFile(name, []byte(config), 0o644); err != nil { + t.Fatal(err) + } + cmd := exec.Command("git", "config", "--file", name, "--null", "--list") + cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} + cmd.Env = append(childEnvironment(os.Environ()), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null") + out, err := cmd.Output() + if err != nil { + t.Fatalf("git config --list: %v", err) + } + return strings.Split(strings.TrimSuffix(string(out), "\x00"), "\x00") +} + +// Every configuration the reader accepts means to it what it means to Git. +func TestConfigParserAgreesWithGit(t *testing.T) { + origin := "https://github.com/AutoDevJapan/cc-pc-tools-rs.git" + for name, config := range map[string]string{ + "written by Git": "[core]\n\trepositoryformatversion = 0\n\tbare = false\n[remote \"origin\"]\n\turl = " + origin + "\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n", + "quotes and comments": "# top\n[remote \"origin\"] ; header\n\turl = \"" + origin + "\" # note\n\tpushurl = \"a#b;c\"\n", + "case and spacing": "[REMOTE \"origin\"]\n URL=" + origin + "#x\n", + "byte order mark and CRLF": "\ufeff[remote \"origin\"]\r\n\turl = " + origin + "\r\n", + "escapes and blanks": "[a \"b\\\"c\\\\d\"]\n\tk = \"q\\\\ \\t\\n\\b\" mid tail \t\n\tflag\n\tempty =\n", + "no final newline": "[core]\n\tbare = false", + "origin twice": "[remote \"origin\"]\n\turl = a\n[remote \"origin\"]\n\turl = b\n", + } { + t.Run(name, func(t *testing.T) { + entries, ok := parseConfig([]byte(config)) + if !ok { + t.Fatalf("parseConfig refused %q", config) + } + var got []string + for _, e := range entries { + if e.key == "" { + continue + } + name := e.section + "." + if e.subsection != "" { + name += e.subsection + "." + } + name += e.key + if e.hasValue { + name += "\n" + e.value + } + got = append(got, name) + } + if want := gitConfigList(t, config); !slices.Equal(got, want) { + t.Fatalf("reader read\n%q\nGit read\n%q", got, want) + } + }) + } +} + +// Lines the reader might not read as Git does are refused, which only narrows +// what qualifies. +func TestConfigParserRefusesWhatItDoesNotRead(t *testing.T) { + for name, config := range map[string]string{ + "continuation line": "[core]\n\tbare = fal\\\nse\n", + "variable after a header": "[core] bare = false\n", + "old subsection form": "[remote.origin]\n\turl = x\n", + "empty subsection": "[core \"\"]\n\tbare = true\n", + "other escape in a subsection": "[remote \"or\\igin\"]\n\turl = x\n", + "open quote": "[core]\n\tbare = \"false\n", + "unknown escape": "[core]\n\tbare = \\q\n", + "variable outside a section": "bare = false\n", + "comment after a name": "[core]\n\tbare # true\n", + "control character": "[core]\n\tbare = fa\x01lse\n", + "lone carriage return": "[core]\n\tbare = false\r\tx = y\n", + "open header": "[core\n\tbare = false\n", + } { + if _, ok := parseConfig([]byte(config)); ok { + t.Errorf("%s: parseConfig accepted %q", name, config) + } + } +} + +func TestPackedRefs(t *testing.T) { + a, b := strings.Repeat("a", 40), strings.Repeat("b", 40) + refs, ok := packedRefs([]byte("# pack-refs with: peeled fully-peeled sorted \n" + a + " refs/heads/work\n" + b + " refs/tags/v1\n^" + a + "\n")) + if !ok || len(refs) != 2 || refs["refs/heads/work"] != a || refs["refs/tags/v1"] != b { + t.Fatalf("packedRefs = %v, %v", refs, ok) + } + for name, data := range map[string]string{ + "repeated name": a + " refs/heads/work\n" + b + " refs/heads/work\n", + "peel without a ref": "^" + a + "\n", + "second peel": b + " refs/tags/v1\n^" + a + "\n^" + a + "\n", + "header not first": a + " refs/heads/work\n# pack-refs with: peeled\n", + "short object name": "abc refs/heads/work\n", + "name outside refs": a + " HEAD\n", + "no final newline": a + " refs/heads/work", + } { + if _, ok := packedRefs([]byte(data)); ok { + t.Errorf("%s: packedRefs accepted %q", name, data) + } + } +} diff --git a/tools/todo-update-reader-3190/isolated_git_windows.go b/tools/todo-update-reader-3190/isolated_git_windows.go index 0887589d..fec48aec 100644 --- a/tools/todo-update-reader-3190/isolated_git_windows.go +++ b/tools/todo-update-reader-3190/isolated_git_windows.go @@ -1,4 +1,62 @@ package main -// isolatedEnvironment is the environment of the scratch Git process. -func isolatedEnvironment(parent []string, gitDir, objects string) []string { return nil } +import ( + "context" + "errors" + "os" + "os/exec" + "path/filepath" + "strings" + "time" +) + +// isolatedEnvironment gives Git a scratch repository whose only outside data +// is the fixed object store. It drops every inherited GIT_* setting and turns +// off the system and global configuration, so Git reads no configuration +// file and follows no include. +func isolatedEnvironment(parent []string, gitDir, objects string) []string { + out := make([]string, 0, len(parent)+7) + for _, entry := range parent { + name, _, _ := strings.Cut(entry, "=") + if !strings.HasPrefix(strings.ToUpper(name), "GIT_") { + out = append(out, entry) + } + } + return append(out, "GIT_DIR="+gitDir, "GIT_OBJECT_DIRECTORY="+objects, "GIT_CONFIG_NOSYSTEM=1", + "GIT_CONFIG_GLOBAL=/dev/null", "GIT_NO_LAZY_FETCH=1", "GIT_OPTIONAL_LOCKS=0", "GIT_TERMINAL_PROMPT=0") +} + +// runGit runs one object read in a new scratch repository that holds only +// HEAD, an empty refs directory and, for a shallow source, its boundary. The +// repository has no configuration file; the objects are the source's own. +func runGit(src gitSource, args ...string) ([]byte, error) { + dir, err := os.MkdirTemp("", "todo-update-reader-3190-") + if err != nil { + return nil, err + } + defer os.RemoveAll(dir) + if err := os.Mkdir(filepath.Join(dir, "refs"), 0o700); err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(dir, "HEAD"), []byte("ref: refs/heads/main\n"), 0o600); err != nil { + return nil, err + } + if len(src.shallow) > 0 { + if err := os.WriteFile(filepath.Join(dir, "shallow"), src.shallow, 0o600); err != nil { + return nil, err + } + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "git", append([]string{"--no-replace-objects"}, args...)...) + hiddenProcess(cmd) + cmd.Dir = dir + cmd.Env = isolatedEnvironment(os.Environ(), dir, src.objects) + out := &limitedBuffer{limit: 32 << 20} + cmd.Stdout = out + err = cmd.Run() + if out.overflow { + return nil, errors.New("git output limit exceeded") + } + return out.Bytes(), err +} diff --git a/tools/todo-update-reader-3190/pin_windows.go b/tools/todo-update-reader-3190/pin_windows.go index c2620eb9..d835de3a 100644 --- a/tools/todo-update-reader-3190/pin_windows.go +++ b/tools/todo-update-reader-3190/pin_windows.go @@ -38,10 +38,21 @@ type pinnedDir struct { info syscall.ByHandleFileInformation } +// releasedFile is Git metadata that was classified, read and closed again, so +// the worktrees sharing the repository can keep updating it. The recheck +// reopens it and compares its identity, size, write time and digest. +type releasedFile struct { + path string + info syscall.ByHandleFileInformation + sum string +} + type session struct { env environment files []*pinnedFile dirs []pinnedDir + released []releasedFile + absent []string dirCache map[string]bool } @@ -117,9 +128,9 @@ func plainFile(info syscall.ByHandleFileInformation) bool { return !reparse(info.FileAttributes) && info.FileAttributes&syscall.FILE_ATTRIBUTE_DIRECTORY == 0 && info.NumberOfLinks == 1 } -// pin opens one regular, single-link file without following a reparse point +// open opens one regular, single-link file without following a reparse point // and without reading it. The handle shares only reads. -func (s *session) pin(path string, limit int64) (*pinnedFile, bool) { +func (s *session) open(path string, limit int64) (*pinnedFile, bool) { if !s.plainAncestors(path) { return nil, false } @@ -144,10 +155,34 @@ func (s *session) pin(path string, limit int64) (*pinnedFile, bool) { f.file.Close() return nil, false } - s.files = append(s.files, f) return f, true } +// pin opens a file and keeps it open until the session closes. +func (s *session) pin(path string, limit int64) (*pinnedFile, bool) { + f, ok := s.open(path, limit) + if ok { + s.files = append(s.files, f) + } + return f, ok +} + +// release closes a file that was read once and keeps its identity and the +// digest of the bytes read for the recheck. +func (s *session) release(f *pinnedFile, sum string) { + s.released = append(s.released, releasedFile{path: f.path, info: f.info, sum: sum}) + f.file.Close() +} + +// absentFile records a path that must not exist now or at the recheck. +func (s *session) absentFile(path string) bool { + if _, err := os.Lstat(path); !os.IsNotExist(err) { + return false + } + s.absent = append(s.absent, path) + return true +} + // read returns exactly the bytes behind the pinned handle and their digest. // Callers parse these same bytes; nothing is reopened by name. func (s *session) read(f *pinnedFile) ([]byte, string, bool) { @@ -230,5 +265,24 @@ func (s *session) recheck() bool { return false } } + for _, r := range s.released { + f, ok := s.open(r.path, maxMetadataBytes) + if !ok { + return false + } + _, sum, read := s.read(f) + now := f.info + f.file.Close() + before := r.info + before.LastAccessTime = now.LastAccessTime + if !read || sum != r.sum || before != now { + return false + } + } + for _, path := range s.absent { + if _, err := os.Lstat(path); !os.IsNotExist(err) { + return false + } + } return true } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index 66d4d5fc..f10bdbec 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -321,9 +321,9 @@ func (f *fixture) guard(args ...string) ([]byte, int, bool) { func (f *fixture) env() environment { return environment{ guard: f.guard, - git: func(root string, args ...string) ([]byte, error) { + git: func(src gitSource, args ...string) ([]byte, error) { f.record("git:" + strings.Join(args, " ")) - return runGit(root, args...) + return runGit(src, args...) }, beforeRead: func(path string) { f.record("read:" + key(path)) }, beforeRecheck: func() { @@ -748,7 +748,8 @@ func TestPinnedFilesResistReplacement(t *testing.T) { func TestRejectsChangesBeforeRecheck(t *testing.T) { // Inputs stay pinned against writes and tracked files outside the inputs are - // never read (TestGitNeverReadsWorkingFiles), so the races left are Git's. + // never read (TestGitNeverReadsWorkingFiles), so the races left are in the + // Git metadata, which is read once and closed for the other worktrees. cases := map[string]func(*fixture){ "new commit": func(f *fixture) { f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "racing") @@ -756,6 +757,30 @@ func TestRejectsChangesBeforeRecheck(t *testing.T) { "origin changed": func(f *fixture) { f.git("remote", "set-url", "origin", "https://github.com/someone-else/cc-pc-tools-rs.git") }, + "branch packed": func(f *fixture) { f.gitIn(f.repo, "pack-refs", "--all") }, + "alternate object store added": func(f *fixture) { + writeFile(f.t, filepath.Join(f.common, "objects", "info", "alternates"), filepath.ToSlash(filepath.Join(f.base, "elsewhere"))+"\n") + }, + // Same file, same size, write time restored: only the digest differs. + "main ref rewritten in place": func(f *fixture) { + name := filepath.Join(f.common, "refs", "remotes", "origin", "main") + info, err := os.Stat(name) + if err != nil { + f.t.Fatal(err) + } + file, err := os.OpenFile(name, os.O_WRONLY, 0) + if err != nil { + f.t.Fatal(err) + } + _, err = file.WriteAt([]byte(strings.Repeat("0", 40)), 0) + file.Close() + if err != nil { + f.t.Fatal(err) + } + if err := os.Chtimes(name, info.ModTime(), info.ModTime()); err != nil { + f.t.Fatal(err) + } + }, } for name, change := range cases { t.Run(name, func(t *testing.T) { diff --git a/tools/todo-update-reader-3190/source_scope_windows_test.go b/tools/todo-update-reader-3190/source_scope_windows_test.go index 8a669045..d8341710 100644 --- a/tools/todo-update-reader-3190/source_scope_windows_test.go +++ b/tools/todo-update-reader-3190/source_scope_windows_test.go @@ -187,8 +187,21 @@ func TestReadsGitMetadataItself(t *testing.T) { expectObjectReads(t, f) }) t.Run("shallow boundary", func(t *testing.T) { + // As in the real shallow clone, HEAD's parent is not in the object + // store. Only the boundary given to Git keeps the ancestry check from + // looking for it. f := newFixture(t, nil) - writeFile(t, filepath.Join(f.common, "shallow"), f.p.head+"\n") + parent := f.p.head + f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "boundary") + head := f.gitOut("rev-parse", "HEAD") + f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", head) + f.p.head, f.main = head, head + f.receipt["source_head"] = head + f.writeReceipt() + if err := os.Remove(filepath.Join(f.common, "objects", parent[:2], parent[2:])); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(f.common, "shallow"), head+"\n") if got, reason := f.verify(); !got.Qualified || reason != "" { t.Fatalf("verify = qualified %v reason %q with a shallow boundary", got.Qualified, reason) } @@ -221,6 +234,12 @@ func TestReadsGitMetadataItself(t *testing.T) { "reftable refs": func(f *fixture) { appendFile(f.t, filepath.Join(f.common, "config"), "[extensions]\n\trefStorage = reftable\n") }, + "bare repository": func(f *fixture) { + appendFile(f.t, filepath.Join(f.common, "config"), "[core]\n\tbare = true\n") + }, + "work tree set by configuration": func(f *fixture) { + appendFile(f.t, filepath.Join(f.common, "config"), "[core]\n\tworktree = "+filepath.ToSlash(filepath.Join(f.base, "elsewhere"))+"\n") + }, "second origin URL": func(f *fixture) { appendFile(f.t, filepath.Join(f.common, "config"), "[remote \"origin\"]\n\turl = https://github.com/AutoDevJapan/cc-pc-tools-rs.git\n") }, diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 05c57183..6ea64a75 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -20,10 +20,11 @@ import ( ) // environment is the reader's only connection to the outside: the deployed -// guard's CLI and Git. Tests replace the guard with a recorded fake. +// guard's CLI and Git, which sees only the fixed object store. Tests replace +// the guard with a recorded fake. type environment struct { guard func(args ...string) (stdout []byte, exitCode int, ran bool) - git func(root string, args ...string) ([]byte, error) + git func(src gitSource, args ...string) ([]byte, error) beforeRead func(path string) beforeRecheck func() } @@ -82,9 +83,9 @@ func verifyUpdate(requested string, p profile, env environment) (qualification, if !s.pinDir(p.root) || !guardClear(env, p.root) { return qualification{}, "source_root" } - tree, ok := checkSource(p, env) - if !ok { - return qualification{}, "source_git" + tree, reason := s.checkSource(p) + if reason != "" { + return qualification{}, reason } data, reason := s.evidence(filepath.Join(p.root, p.receiptName), p.receiptSHA, maxEvidenceBytes, "receipt") @@ -138,7 +139,7 @@ func verifyUpdate(requested string, p profile, env environment) (qualification, if env.beforeRecheck != nil { env.beforeRecheck() } - if !s.recheck() || !sourceStillFixed(p, env) { + if !s.recheck() { return qualification{}, "recheck" } return qualification{ @@ -272,45 +273,36 @@ func capacityQualified(env environment, p profile) bool { return true } -func gitText(env environment, root string, args ...string) (string, bool) { - out, err := env.git(root, args...) +func gitText(env environment, src gitSource, args ...string) (string, bool) { + out, err := env.git(src, args...) return strings.TrimSpace(string(out)), err == nil } -// checkSource verifies the fixed worktree and returns the fixed commit's tree. -func checkSource(p profile, env environment) (map[string]treeEntry, bool) { - if !sourceStillFixed(p, env) { - return nil, false +// checkSource reads the fixed worktree's Git metadata itself (readSource), +// then lets Git read only the object store: the fixed commit's tree, its +// ancestry to the main commit and the tree listing. It never lets Git open a +// configuration, reference or working file; the guard classifies files one by +// one, and only the pinned inputs are compared, by the reader itself. +func (s *session) checkSource(p profile) (map[string]treeEntry, string) { + src, mainID, reason := s.readSource(p) + if reason != "" { + return nil, reason } - if tree, ok := gitText(env, p.root, "rev-parse", "--verify", "--quiet", p.head+"^{tree}"); !ok || tree != p.tree { - return nil, false + if tree, ok := gitText(s.env, src, "rev-parse", "--verify", "--quiet", p.head+"^{tree}"); !ok || tree != p.tree { + return nil, "source_git" } - if _, err := env.git(p.root, "merge-base", "--is-ancestor", p.head, p.mainRef); err != nil { - return nil, false + if _, err := s.env.git(src, "merge-base", "--is-ancestor", p.head, mainID); err != nil { + return nil, "source_git" } - out, err := env.git(p.root, "ls-tree", "-r", "-z", "--full-tree", p.head) + out, err := s.env.git(src, "ls-tree", "-r", "-z", "--full-tree", p.head) if err != nil { - return nil, false + return nil, "source_git" } - return parseTree(out) -} - -// sourceStillFixed is checked before reading and again after every file is -// pinned: the same worktree top, origin and HEAD. Git reads only refs, -// configuration and objects. It never opens a working file: the guard -// classifies files one by one, and only the pinned inputs are compared, by -// the reader itself. -func sourceStillFixed(p profile, env environment) bool { - top, ok := gitText(env, p.root, "rev-parse", "--show-toplevel") - if !ok || !strings.EqualFold(filepath.Clean(filepath.FromSlash(top)), p.root) { - return false - } - origin, ok := gitText(env, p.root, "remote", "get-url", "origin") - if !ok || !slices.Contains(p.origins, strings.ToLower(origin)) { - return false + tree, ok := parseTree(out) + if !ok { + return nil, "source_git" } - head, ok := gitText(env, p.root, "rev-parse", "--verify", "HEAD") - return ok && head == p.head + return tree, "" } type limitedBuffer struct { @@ -327,8 +319,8 @@ func (b *limitedBuffer) Write(p []byte) (int, error) { return b.Buffer.Write(p) } -// childEnvironment removes inherited GIT_* overrides so another repository, -// index or object store cannot stand in for the declared source. +// childEnvironment removes inherited GIT_* overrides from the guard's +// environment. Git itself runs in isolatedEnvironment. func childEnvironment(parent []string) []string { out := make([]string, 0, len(parent)+3) for _, entry := range parent { @@ -348,21 +340,6 @@ func hiddenProcess(cmd *exec.Cmd) { cmd.Stderr = io.Discard } -func runGit(root string, args ...string) ([]byte, error) { - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - argv := append([]string{"-C", root, "--no-replace-objects", "-c", "core.fsmonitor=false"}, args...) - cmd := exec.CommandContext(ctx, "git", argv...) - hiddenProcess(cmd) - out := &limitedBuffer{limit: 32 << 20} - cmd.Stdout = out - err := cmd.Run() - if out.overflow { - return nil, errors.New("git output limit exceeded") - } - return out.Bytes(), err -} - // runGuard executes the deployed guard by its fixed path. The executable must // be a plain file; its classification is the PC's protection authority. func runGuard(args ...string) ([]byte, int, bool) { From 8b4f34b9e405b02ee93442fc295e8dfdead36f5a Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 04:53:49 +0900 Subject: [PATCH 09/24] docs: record the ToDo3491 review and the isolated Git fix Steps 5 and 10 now describe the metadata the reader reads itself, the scratch repository Git runs in with only the object store, and the recheck of released and absent files. Adds the ToDo3491 review, the second Must fix with its Red, Green and mutant results, and the pre-merge observation of the reader built from 2f804f13. --- docs/todo-update-reader-3190.md | 88 ++++++++++++++++++++++++++++----- 1 file changed, 75 insertions(+), 13 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index e9912af0..a631ccc0 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -66,10 +66,27 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト 上の表と完全に一致し、終了コード0の場合だけ次へ進む。元client一致などで容量用途が拒否すれば止める。 4. その後でだけ同じhandleから対象を読み、固定の旧SHAと一致することを確かめる。違えば拒否し、値を出さない。 5. source rootの祖先と自身が reparse point でないこと、実パス、guardの `--guard-path` が clear であることを確かめる。 - Gitは `GIT_*` 環境を除き、窓を出さず、期限付きで呼ぶ。worktreeの最上位、origin、HEAD、tree、 - `origin/main` の祖先であることを確かめる。Gitに読ませるのは参照・設定・オブジェクトだけで - （`rev-parse`、`remote get-url`、`merge-base`、`ls-tree`）、作業ファイルは開かせない。 - guardは1件ずつ分類するので、rootがclearでも、分類していない追跡ファイルをGitに読ませない。 + 続けて、Gitの設定と参照をreader自身が読む。対象は、worktreeの `.git`、その登録先の `commondir`・`gitdir`・`HEAD`、 + 共有repoの `config`、HEADの指すbranchと `refs/remotes/origin/main`（looseが無ければ `packed-refs`）、`shallow` である。 + 各ファイルはguardの clear を確かめてから、reparse pointでないリンク数1のhandleで1回だけ読み、すぐ閉じる。 + 共有repoはほかの多くのworktreeも使うので、読んだ後まで開いたままにしない。読んだbytesで次を確かめる。 + - `.git` は `gitdir: <絶対パス>` の1行で、登録先の `commondir` は `../..`、`gitdir` はこのworktreeの `.git` を指す。 + - `config` を厳密に解析する。`include`・`includeIf`・`url` の節を拒否する。core は通常の非bare repoのキーだけ、 + 拡張は `worktreeConfig` だけを認める。origin のURLはちょうど1つで、固定のoriginと一致する。Gitと同じ意味に + 読めない書き方（継続行、見出しの後の値、旧形式の見出し、未知のescape、閉じない引用符、制御文字）も拒否する。 + - HEADの指すbranchが固定HEADのcommitを指し、`refs/remotes/origin/main` のcommitを求められる。 + - `config.worktree`・`info/grafts`・`objects/info/alternates` が無い。`shallow` は40桁のcommitの行だけ。 + + その後でだけ、guardの clear を確かめたobjectsディレクトリをGitに渡す。Gitは `%TEMP%` に作る空のrepo + （HEAD、空のrefs、必要なら `shallow` の写し）で動き、objectsだけが共有repoのものになる。`GIT_*` の環境は引き継がず、 + system・globalの設定も読まない（`GIT_CONFIG_NOSYSTEM=1`、`GIT_CONFIG_GLOBAL=/dev/null`）。窓を出さず、期限付きで呼ぶ。 + Gitに許すのは次の3回だけで、引数まで固定する。 + - `rev-parse --verify --quiet ^{tree}`（結果が固定treeと一致する） + - `merge-base --is-ancestor ` + - `ls-tree -r -z --full-tree ` + + Gitは設定・参照・作業ファイルを1つも開かない。guardは1件ずつ分類するので、rootがclearでも、 + 分類していないファイルをGitに読ませない。 6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 読んだbytesそのものをhashして解析する。 7. receiptの全項目を固定値と照合する。未知の項目と `data_access_granted` が false でない記録は拒否する。 @@ -79,9 +96,12 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト `core.autocrlf=true` でLFをすべてCRLFにした形の場合だけ。NUL、単独のCR、CRLFとLFの混在は変換として認めない。 入力以外の追跡ファイルは分類も読取もしないので、その変更は結果を左右しない。 9. 新候補のサイズとSHA256を固定値とreceiptの両方に照合する。 -10. 最後に保持した全handleの同一性（volume・file ID・サイズ・更新時刻・リンク数・属性）と実パス、 - source rootの実パス、worktreeの最上位・origin・HEADを再確認する。照合中の変更は拒否する。 - 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。 +10. 最後に、保持した全handleの同一性（volume・file ID・サイズ・更新時刻・リンク数・属性）と実パス、 + source rootとobjectsディレクトリの実パスを再確認する。読んで閉じたGitの設定と参照は開き直し、同一性と + SHA256が最初に読んだときと同じことを確かめる。無かったファイル（`config.worktree`、`info/grafts`、 + `objects/info/alternates`、looseで無かったref、`shallow` など）が今も無いことも確かめる。照合中の変更は拒否する。 + 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。共有repoのconfigやrefは + ほかのworktreeが書き換えることがあり、照合中に書き換わると `recheck` で拒否される。その場合は再実行する。 ## compiler closure の確認 @@ -104,8 +124,8 @@ receiptの126件の入力が、原dep-infoの116件と manifest/lock の10件に ## 受入条件とテスト -テストは `tools/todo-update-reader-3190` の `go test`。合成したGit repo、ファイル、receiptと、 -guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 +テストは `tools/todo-update-reader-3190` の `go test`。合成したGit repo（実物と同じく、別の場所のrepoに属する +linked worktree）、ファイル、receiptと、guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 理由コードも照合し、別の理由による偶然の拒否を合格にしない。 | 条件 | テスト | @@ -121,7 +141,8 @@ guardのCLI応答を模した関数で、実物の配布先・guard・親の作 | 元client一致 | `TestRejectsProtectedResources` | | data access の拡大 | `TestRejectsDataAccessExpansion` | | closure の過不足 | `TestRejectsClosureGaps` | -| Gitに作業ファイルを読ませない。入力以外の保護された追跡ファイルは結果を左右しない | `TestGitNeverReadsWorkingFiles`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | +| Gitに設定・参照・作業ファイルを読ませない。Gitの呼び出しは引数まで固定の3回だけ。入力以外の保護された追跡ファイルは結果を左右しない | `TestGitNeverReadsWorkingFiles`、`TestGitReadsNoConfiguration`、`TestIsolatedGitEnvironment`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | +| Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs` | | 入力のbytesが固定treeのblob（そのままか、LFをすべてCRLFにした形）である | `TestInputsMustBeTheCommittedBytes`、`TestCheckedOutForms` | | 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | @@ -174,6 +195,41 @@ Suggestion と Nit はこの文書へ反映した。 - 新しい照合の要所を壊した3つの版（blob照合を呼ばない、NULの確認を外す、CRとLFの対の確認を外す）は、 すべて対応するテストが失敗した。 +### 別モデルの再レビュー（ToDo3491） + +gpt-6.1-sol・推論max（job `20261024-033144-todo-3491`、4分17秒、終了コード0）が、base `d9edfd4d`・head `308cb24d` の +変更5ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のMustは「一部解消」、 +前回のSuggestion 2件とNitは解消と判定された。新たな指摘は Must 1件、Suggestion 1件。 + +- Must: `remote get-url` などは既存のGit設定を読む。system・global・repo・worktreeの設定にある `include.path` と + `includeIf.*.path` を隔離していないので、設定が保護されたファイルをincludeすれば、guardの1件ずつの分類より先に + Gitがその中身を読み得る。修正案は、既存の設定を読ませないGitの実行環境にすること、必要な設定はguardが分類して + 固定したbytesからincludeを追わずに取ること、include先を保護した回帰テストを足すこと。実環境にそうした設定が + あるという指摘ではなく、Gitの仕様とコードからの判断である。 +- Suggestion: テストでGitの呼び出しを許す判定を、サブコマンド名ではなく引数列の全体で照合する。 + +### 2回目のMustの修正 + +Gitに渡すのを固定objectsの読取だけにした（検証の順序の5）。worktreeの登録・HEAD・ref・config・shallowは、 +guardの分類の後でreader自身が読む。Gitは設定を1つも読まない空のrepoで動き、呼び出しは引数まで固定の3回だけ。 +最後の再照合では、読んで閉じたファイルの同一性とSHA256、無かったファイルが今も無いことを確かめる（検証の順序の10）。 +Suggestionは、テストの許可判定を引数列全体の一致にして反映した。 + +- Red（commit `6bda9f63`、隔離した環境はstub）: fixtureを実物と同じlinked worktreeにした。`go test -count=1 ./...` は + 終了コード1、131秒。合格90、失敗48、skip 0、ビルド失敗0。global設定から保護ファイルをincludeした例は `source_git` で + 失敗し、Gitが実際にinclude先を開こうとしたことを示した。 +- Green（commit `2f804f13`）: `go test -race -count=1 ./...` は終了コード0、144秒。上位27件、subtest込み153件が合格し、 + 失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。Redのテストだけでも、同じ実装で138件が合格した。 + 修正と同時に足したテストは、親commitのobjectが無いshallow clone、bareや `core.worktree` の設定、再照合の前の + refのpack・alternatesの追加・同じ大きさと日時でのrefの書換え、受け付けたconfigをGit自身（`git config --list`）も + 同じ値に読むこと、packed-refsの書式。 +- 新しい照合の要所を壊した11の版を `go test -overlay` で当て、すべて対応するテストが失敗した。壊したのは、 + include・includeIf・urlの拒否、登録先の逆リンク、`config.worktree` の不在、global設定の無効化、shallowの受け渡し、 + 読んで閉じたファイルの再照合、不在の再照合、SHA256の比較、HEADの照合、originの照合、alternatesの不在。 +- 実物の親の作業場所は、別の場所の共有repoに属するlinked worktreeで、branchは `packed-refs` にだけあり、 + 共有repoはshallow cloneである。configには800件を超えるbranchの節と core・extensions・remote があり、includeは無い。 + 読んだメタデータはguardの分類で clear だった。configの中身はログや文書へ出していない。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -185,7 +241,12 @@ source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力12 でも実物を1回照合した。終了コード0、8.7秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 `data_access_granted:false`）を返した。実物の126件はすべて、固定treeのblobそのものか、LFをすべてCRLFにした形だった。 -どちらもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 +2回目の修正のcommit `2f804f13` から `go build -trimpath` で生成したreader（SHA256 `2528d5cc…1825`、vcs.modified=false） +でも実物を1回照合した。終了コード0、10.0秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。reader自身が実物のconfig・HEAD・ref（branchは `packed-refs`、`origin/main` はloose）と +`shallow` を読んで受け付け、Gitは空のrepoで固定objectsとshallowの写しだけを読んだ。 + +いずれもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 対象の置換、guardの変更、ToDoの状態変更はしていない。 補足として、親の作業場所に残るCargoの出力 `target/release/todo.exe` は、配布用コピーと同じ @@ -195,5 +256,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d ### 未完了 修正前のhead `d9edfd4d` では、専用workflow（windows、run 39994455364）とCodeQL（run 39994455228）が成功した。 -修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる実物の照合、 -親3190への結果の返却が残る。 +1回目の修正のhead `308cb24d` でも、専用workflow（windows、run 40002129622）とCodeQL（run 40002129620）が成功した。 +2回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +実物の照合、親3190への結果の返却が残る。 From 3b9b98d30743906f468d428ed286edf5bb993bb2 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 05:51:44 +0900 Subject: [PATCH 10/24] test: require the ToDo3190 reader to read objects and packed-refs as Git does Review ToDo3505 found two Musts. Git reads objects/info/alternates of the shared store, so an alternates file added while the reader runs and removed before the recheck lets Git open an unclassified store. The packed-refs parser skipped the header, so it accepted a header Git refuses and records out of the declared sorted order, where Git binary search cannot find the main ref. Against the current reader these tests fail: an object only in a transient alternate store still qualifies, and both packed-refs files qualify although Git itself cannot resolve refs/remotes/origin/main from them. --- .../gitmeta_windows_test.go | 32 +++-- .../objects_windows_test.go | 122 ++++++++++++++++++ .../reader_windows_test.go | 8 +- 3 files changed, 151 insertions(+), 11 deletions(-) create mode 100644 tools/todo-update-reader-3190/objects_windows_test.go diff --git a/tools/todo-update-reader-3190/gitmeta_windows_test.go b/tools/todo-update-reader-3190/gitmeta_windows_test.go index 347a8539..2f63c5c8 100644 --- a/tools/todo-update-reader-3190/gitmeta_windows_test.go +++ b/tools/todo-update-reader-3190/gitmeta_windows_test.go @@ -92,18 +92,30 @@ func TestConfigParserRefusesWhatItDoesNotRead(t *testing.T) { func TestPackedRefs(t *testing.T) { a, b := strings.Repeat("a", 40), strings.Repeat("b", 40) - refs, ok := packedRefs([]byte("# pack-refs with: peeled fully-peeled sorted \n" + a + " refs/heads/work\n" + b + " refs/tags/v1\n^" + a + "\n")) - if !ok || len(refs) != 2 || refs["refs/heads/work"] != a || refs["refs/tags/v1"] != b { - t.Fatalf("packedRefs = %v, %v", refs, ok) + header := "# pack-refs with: peeled fully-peeled sorted \n" + for name, data := range map[string]string{ + "written by Git": header + a + " refs/heads/work\n" + b + " refs/tags/v1\n^" + a + "\n", + "no header in any order": b + " refs/tags/v1\n" + a + " refs/heads/work\n", + "not declared sorted": "# pack-refs with: peeled \n" + b + " refs/tags/v1\n" + a + " refs/heads/work\n", + } { + refs, ok := packedRefs([]byte(data)) + if !ok || len(refs) != 2 || refs["refs/heads/work"] != a || refs["refs/tags/v1"] != b { + t.Errorf("%s: packedRefs = %v, %v", name, refs, ok) + } } for name, data := range map[string]string{ - "repeated name": a + " refs/heads/work\n" + b + " refs/heads/work\n", - "peel without a ref": "^" + a + "\n", - "second peel": b + " refs/tags/v1\n^" + a + "\n^" + a + "\n", - "header not first": a + " refs/heads/work\n# pack-refs with: peeled\n", - "short object name": "abc refs/heads/work\n", - "name outside refs": a + " HEAD\n", - "no final newline": a + " refs/heads/work", + "repeated name": a + " refs/heads/work\n" + b + " refs/heads/work\n", + "peel without a ref": "^" + a + "\n", + "second peel": b + " refs/tags/v1\n^" + a + "\n^" + a + "\n", + "header not first": a + " refs/heads/work\n# pack-refs with: peeled\n", + "short object name": "abc refs/heads/work\n", + "name outside refs": a + " HEAD\n", + "no final newline": a + " refs/heads/work", + "declared sorted, out of order": header + b + " refs/tags/v1\n" + a + " refs/heads/work\n", + "declared sorted, longer first": header + a + " refs/heads/work/x\n" + b + " refs/heads/work\n", + "no space after the colon": "# pack-refs with:peeled fully-peeled sorted \n" + a + " refs/heads/work\n", + "another comment first": "# written by hand\n" + a + " refs/heads/work\n", + "unknown trait": "# pack-refs with: peeled future \n" + a + " refs/heads/work\n", } { if _, ok := packedRefs([]byte(data)); ok { t.Errorf("%s: packedRefs accepted %q", name, data) diff --git a/tools/todo-update-reader-3190/objects_windows_test.go b/tools/todo-update-reader-3190/objects_windows_test.go new file mode 100644 index 00000000..890c16c4 --- /dev/null +++ b/tools/todo-update-reader-3190/objects_windows_test.go @@ -0,0 +1,122 @@ +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +// moveObject moves the loose file of id from one object store to another. +func moveObject(t *testing.T, from, to, id string) { + t.Helper() + src := filepath.Join(from, id[:2], id[2:]) + data, err := os.ReadFile(src) + if err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(to, id[:2], id[2:]), string(data)) + if err := os.Chmod(src, 0o644); err != nil { + t.Fatal(err) + } + if err := os.Remove(src); err != nil { + t.Fatal(err) + } +} + +// alternateDuringReads names dir in the alternates file of the source's store +// right after the reader has found none and has first read the main ref, and +// removes the file again before the recheck, which therefore cannot see it. +func (f *fixture) alternateDuringReads(dir string) { + alternates := filepath.Join(f.common, "objects", "info", "alternates") + mainRef := key(filepath.Join(f.common, "refs", "remotes", "origin", "main")) + written := false + f.onRead = func(path string) { + if path == mainRef && !written { + written = true + writeFile(f.t, alternates, filepath.ToSlash(dir)+"\n") + } + } + f.onRecheck = func() { + if err := os.Remove(alternates); err != nil { + f.t.Fatal(err) + } + } +} + +// expectUntouchedBelow fails if anything at or below dir was classified or read. +func expectUntouchedBelow(t *testing.T, f *fixture, dir string) { + t.Helper() + prefix := key(dir) + f.mu.Lock() + defer f.mu.Unlock() + for _, event := range f.events { + for _, kind := range []string{"read:", "guard-path:"} { + if path, ok := strings.CutPrefix(event, kind); ok && (path == prefix || strings.HasPrefix(path, prefix+`\`)) { + t.Errorf("%s was classified or read: %s", dir, event) + } + } + } +} + +// The reader takes objects from the source's own store only. An alternate +// store named only while the reads run is never read: an object that only it +// holds is missing, and a protected store it names is never classified. +func TestObjectsIgnoreAlternates(t *testing.T) { + t.Run("object only in the alternate store", func(t *testing.T) { + f := newFixture(t, nil) + elsewhere := filepath.Join(f.base, "elsewhere", "objects") + moveObject(t, filepath.Join(f.common, "objects"), elsewhere, f.p.head) + f.alternateDuringReads(elsewhere) + expectReason(t, f, "source_git") + expectUntouchedBelow(t, f, filepath.Join(f.base, "elsewhere")) + }) + t.Run("protected alternate store", func(t *testing.T) { + f := newFixture(t, nil) + elsewhere := filepath.Join(f.base, "elsewhere", "objects") + if err := os.MkdirAll(elsewhere, 0o755); err != nil { + t.Fatal(err) + } + f.protected[key(elsewhere)] = clientVerdict + f.alternateDuringReads(elsewhere) + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q with every object in the source's store", got.Qualified, reason) + } + expectUntouchedBelow(t, f, filepath.Join(f.base, "elsewhere")) + }) +} + +// Git trusts the "sorted" trait of packed-refs and finds a name by binary +// search, and it refuses a header without the space after the colon. The +// reader refuses both files instead of reading a ref that Git would not. +func TestPackedRefsAsGitReadsThem(t *testing.T) { + for name, rewrite := range map[string]func(header string, records []string) string{ + "records out of the declared order": func(header string, records []string) string { + slices.Reverse(records) + return header + "\n" + strings.Join(records, "\n") + "\n" + }, + "header without the space after the colon": func(header string, records []string) string { + return strings.Replace(header, "with: ", "with:", 1) + "\n" + strings.Join(records, "\n") + "\n" + }, + } { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + f.gitIn(f.repo, "pack-refs", "--all") + packed := filepath.Join(f.common, "packed-refs") + data, err := os.ReadFile(packed) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(strings.TrimSuffix(string(data), "\n"), "\n") + if len(lines) < 3 || lines[0] != "# pack-refs with: peeled fully-peeled sorted " { + t.Fatalf("pack-refs wrote %q", data) + } + writeFile(t, packed, rewrite(lines[0], lines[1:])) + if out, err := testGit(t, f.root, "rev-parse", "--verify", "--quiet", f.p.mainRef); err == nil { + t.Fatalf("Git still resolved %s to %s", f.p.mainRef, out) + } + expectReason(t, f, "source_git") + }) + } +} diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index f10bdbec..59f61eb0 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -71,6 +71,7 @@ type fixture struct { capacityExit int onCapacity func() onRecheck func() + onRead func(path string) } func sha(data []byte) string { @@ -325,7 +326,12 @@ func (f *fixture) env() environment { f.record("git:" + strings.Join(args, " ")) return runGit(src, args...) }, - beforeRead: func(path string) { f.record("read:" + key(path)) }, + beforeRead: func(path string) { + f.record("read:" + key(path)) + if f.onRead != nil { + f.onRead(key(path)) + } + }, beforeRecheck: func() { if f.onRecheck != nil { f.onRecheck() From fe1d5941e910f430eedebe97521a8e4ff962c675 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 06:10:58 +0900 Subject: [PATCH 11/24] fix: read objects without Git and packed-refs as Git does in the ToDo3190 reader Review ToDo3505 found that Git still followed objects/info/alternates of the shared store, so a file placed while the reader ran and removed before the recheck could make Git read an unclassified store, and that packed-refs was parsed without its header and declared order. The reader no longer runs Git. It reads loose objects and version 2 pack indexes with their packs from the source store only, applies offset and named-base deltas within a pack, and uses an object only when its SHA-1 name matches. The guard classifies the pack directory, every index, pack and loose file first. Alternates, commit-graph, multi-pack-index and replace refs are never read. packed-refs accepts only the header and traits Git writes and requires strict order when the file declares sorted. Tests compare every object, commit, tree listing and ancestry with Git for loose objects, offset deltas, named-base deltas and a mixed store. --- docs/todo-update-reader-3190.md | 47 +- .../checkout_windows.go | 11 +- .../evidence_windows.go | 20 - .../gitmeta_windows.go | 60 +- .../isolated_git_windows.go | 62 -- .../objects_windows.go | 764 ++++++++++++++++++ .../objects_windows_test.go | 413 +++++++++- tools/todo-update-reader-3190/pin_windows.go | 13 +- .../reader_windows_test.go | 29 +- .../source_scope_windows_test.go | 138 ++-- .../todo-update-reader-3190/verify_windows.go | 62 +- 11 files changed, 1361 insertions(+), 258 deletions(-) delete mode 100644 tools/todo-update-reader-3190/isolated_git_windows.go create mode 100644 tools/todo-update-reader-3190/objects_windows.go diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index a631ccc0..0fcba00b 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -75,18 +75,27 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト 拡張は `worktreeConfig` だけを認める。origin のURLはちょうど1つで、固定のoriginと一致する。Gitと同じ意味に 読めない書き方（継続行、見出しの後の値、旧形式の見出し、未知のescape、閉じない引用符、制御文字）も拒否する。 - HEADの指すbranchが固定HEADのcommitを指し、`refs/remotes/origin/main` のcommitを求められる。 + - `packed-refs` はGitと同じに読む。先頭の行が `#` で始まれば、`# pack-refs with: ` と既知の特性 + （`peeled`・`fully-peeled`・`sorted`）だけを認める。Gitは `sorted` を宣言したファイルを二分探索するので、 + 宣言があれば名前がbyte順に厳密な昇順であることを求める。宣言が無ければ、Gitと同じく並びを問わない。 - `config.worktree`・`info/grafts`・`objects/info/alternates` が無い。`shallow` は40桁のcommitの行だけ。 - その後でだけ、guardの clear を確かめたobjectsディレクトリをGitに渡す。Gitは `%TEMP%` に作る空のrepo - （HEAD、空のrefs、必要なら `shallow` の写し）で動き、objectsだけが共有repoのものになる。`GIT_*` の環境は引き継がず、 - system・globalの設定も読まない（`GIT_CONFIG_NOSYSTEM=1`、`GIT_CONFIG_GLOBAL=/dev/null`）。窓を出さず、期限付きで呼ぶ。 - Gitに許すのは次の3回だけで、引数まで固定する。 - - `rev-parse --verify --quiet ^{tree}`（結果が固定treeと一致する） - - `merge-base --is-ancestor ` - - `ls-tree -r -z --full-tree ` - - Gitは設定・参照・作業ファイルを1つも開かない。guardは1件ずつ分類するので、rootがclearでも、 - 分類していないファイルをGitに読ませない。 + Gitは実行しない。objectsもreader自身が読む。読むのは共有repoのobjectsディレクトリにあるloose objectと、 + `objects/pack` の `pack-<40桁>.idx`（version 2）と対になる `.pack`（version 2か3）だけである。 + alternates・commit-graph・multi-pack-index・replace refは読まない。guardが `objects` と `objects/pack` の + ディレクトリ、各index・pack・loose objectを1件ずつ clear と分類してから開く。indexは末尾のchecksumを確かめ、 + packは先頭の件数と末尾のchecksumがindexの記録と一致する場合だけ使う。packの各entryは、そのままの形か、 + offset delta・名前を指すdeltaとして同じpackの中から組み立てる。どのobjectも、型と中身から計算したSHA-1の + 名前が求めた名前と一致する場合だけ使う。そのため、照合中に共有repoのobjectが増えたり、別のstoreを指す + ファイルが一時的に置かれたりしても、readerが読む中身は変わらない。objectsで確かめるのは次の3つ。 + - 固定HEADのcommitのtreeが、固定treeと一致する。 + - `refs/remotes/origin/main` のcommitから親をたどると、固定HEADに着く。`shallow` にあるcommitは、 + Gitのshallow cloneと同じく親を持たないものとして扱う。読めない親があれば拒否する。 + - 固定treeを再帰的にたどった一覧。`git ls-tree -r --full-tree` と同じく、通常ファイル・link・submoduleの + entryを並べ、directoryは並べない。 + + Gitが設定・別のstore・作業ファイルを読むことはない。guardは1件ずつ分類するので、rootがclearでも、 + 分類していないファイルは読まない。 6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 読んだbytesそのものをhashして解析する。 7. receiptの全項目を固定値と照合する。未知の項目と `data_access_granted` が false でない記録は拒否する。 @@ -97,9 +106,12 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト 入力以外の追跡ファイルは分類も読取もしないので、その変更は結果を左右しない。 9. 新候補のサイズとSHA256を固定値とreceiptの両方に照合する。 10. 最後に、保持した全handleの同一性（volume・file ID・サイズ・更新時刻・リンク数・属性）と実パス、 - source rootとobjectsディレクトリの実パスを再確認する。読んで閉じたGitの設定と参照は開き直し、同一性と + source rootと `objects`・`objects/pack` ディレクトリの実パスを再確認する。読んで閉じたGitの設定と参照は開き直し、同一性と SHA256が最初に読んだときと同じことを確かめる。無かったファイル（`config.worktree`、`info/grafts`、 - `objects/info/alternates`、looseで無かったref、`shallow` など）が今も無いことも確かめる。照合中の変更は拒否する。 + `objects/info/alternates`、looseで無かったref、`shallow` など）が今も無いことも確かめる。 + この再確認が拒否できるのは、再確認の時点で残っている変更である。途中で作られて再確認の前に消えたファイルは + 検出できない。そのためreaderは、alternates・grafts・worktreeごとの設定を読む処理を持たず、Gitも実行しない。 + objectは名前で中身を確かめるので、再確認の対象にしない。 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。共有repoのconfigやrefは ほかのworktreeが書き換えることがあり、照合中に書き換わると `recheck` で拒否される。その場合は再実行する。 @@ -126,7 +138,8 @@ receiptの126件の入力が、原dep-infoの116件と manifest/lock の10件に テストは `tools/todo-update-reader-3190` の `go test`。合成したGit repo（実物と同じく、別の場所のrepoに属する linked worktree）、ファイル、receiptと、guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 -理由コードも照合し、別の理由による偶然の拒否を合格にしない。 +理由コードも照合し、別の理由による偶然の拒否を合格にしない。objectの読取は、Git自身の出力 +（`git cat-file --batch-all-objects`、`git ls-tree`、`git rev-list`）と比べる。 | 条件 | テスト | | --- | --- | @@ -137,12 +150,14 @@ linked worktree）、ファイル、receiptと、guardのCLI応答を模した | 別origin、HEAD/treeの移動、未merge | `TestRejectsSourceChanges` | | 親・兄弟・ADS・別名、任意入力 | `TestRejectsOtherPaths`、`TestCLIAcceptsOnlyThePurposeFlag` | | hardlink、junction | `TestRejectsLinks` | -| 照合中の置換・変更 | `TestPinnedFilesResistReplacement`、`TestRejectsChangesBeforeRecheck` | +| 照合中の置換・変更 | `TestPinnedFilesResistReplacement`、`TestRejectsChangesBeforeRecheck`、`TestObjectsIgnoreAlternates` | | 元client一致 | `TestRejectsProtectedResources` | | data access の拡大 | `TestRejectsDataAccessExpansion` | | closure の過不足 | `TestRejectsClosureGaps` | -| Gitに設定・参照・作業ファイルを読ませない。Gitの呼び出しは引数まで固定の3回だけ。入力以外の保護された追跡ファイルは結果を左右しない | `TestGitNeverReadsWorkingFiles`、`TestGitReadsNoConfiguration`、`TestIsolatedGitEnvironment`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | -| Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs` | +| Gitを実行せず、設定・別のstore・作業ファイルを読まない。読む全ファイルをguardが先に分類する。入力以外の保護された追跡ファイルは結果を左右しない | `TestNeverReadsWorkingFiles`、`TestReadsNoGitConfiguration`、`TestObjectsIgnoreAlternates`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | +| Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs`、`TestPackedRefsAsGitReadsThem` | +| objectをGitと同じに読む（loose、pack、offset delta、名前を指すdelta、commit・tree・祖先） | `TestObjectStoreAgreesWithGit`、`TestQualifiesFromPacks` | +| 壊れたobject・別のobject・保護されたobjectのファイルを拒否する | `TestRejectsDamagedObjects`、`TestApplyDeltaRefusesMalformedDeltas` | | 入力のbytesが固定treeのblob（そのままか、LFをすべてCRLFにした形）である | `TestInputsMustBeTheCommittedBytes`、`TestCheckedOutForms` | | 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | diff --git a/tools/todo-update-reader-3190/checkout_windows.go b/tools/todo-update-reader-3190/checkout_windows.go index 256a6a01..76720fa0 100644 --- a/tools/todo-update-reader-3190/checkout_windows.go +++ b/tools/todo-update-reader-3190/checkout_windows.go @@ -7,15 +7,18 @@ import ( "strconv" ) -// gitBlobID is the SHA-1 object name Git gives data stored as a blob. It ties -// bytes to an entry of the fixed tree; SHA-256 pins the same bytes separately. -func gitBlobID(data []byte) string { +// objectName is the SHA-1 name Git gives an object of kind with data. +func objectName(kind string, data []byte) string { h := sha1.New() - h.Write([]byte("blob " + strconv.Itoa(len(data)) + "\x00")) + h.Write([]byte(kind + " " + strconv.Itoa(len(data)) + "\x00")) h.Write(data) return hex.EncodeToString(h.Sum(nil)) } +// gitBlobID is the name Git gives data stored as a blob. It ties bytes to an +// entry of the fixed tree; SHA-256 pins the same bytes separately. +func gitBlobID(data []byte) string { return objectName("blob", data) } + // checkedOut reports whether working bytes are exactly a fixed blob as Git // writes it out on this PC: the blob itself or, under core.autocrlf=true, the // blob with every LF written as CRLF. Git converts only text, so bytes with a diff --git a/tools/todo-update-reader-3190/evidence_windows.go b/tools/todo-update-reader-3190/evidence_windows.go index 1c5711bb..a58ab97d 100644 --- a/tools/todo-update-reader-3190/evidence_windows.go +++ b/tools/todo-update-reader-3190/evidence_windows.go @@ -187,26 +187,6 @@ func validInputs(inputs []receiptInput) bool { return true } -// parseTree reads `git ls-tree -r -z --full-tree` output. -func parseTree(out []byte) (map[string]treeEntry, bool) { - tree := map[string]treeEntry{} - for _, record := range strings.Split(string(out), "\x00") { - if record == "" { - continue - } - meta, name, ok := strings.Cut(record, "\t") - fields := strings.Fields(meta) - if !ok || len(fields) != 3 || name == "" { - return nil, false - } - if _, dup := tree[name]; dup { - return nil, false - } - tree[name] = treeEntry{mode: fields[0], kind: fields[1], object: fields[2]} - } - return tree, len(tree) > 0 -} - // parseDepInfo reads Cargo's one-rule dep-info for artifact. Every compiler // input must be a plain path inside root; escapes and variables are refused. func parseDepInfo(data []byte, root, artifact string) ([]string, bool) { diff --git a/tools/todo-update-reader-3190/gitmeta_windows.go b/tools/todo-update-reader-3190/gitmeta_windows.go index ed0c1563..1b02c5e7 100644 --- a/tools/todo-update-reader-3190/gitmeta_windows.go +++ b/tools/todo-update-reader-3190/gitmeta_windows.go @@ -10,24 +10,15 @@ import ( // maxMetadataBytes bounds each Git metadata file the reader reads itself. const maxMetadataBytes = 4 << 20 -// gitSource is all that Git sees of the fixed source: the repository's object -// store and, for a shallow clone, its boundary commits. Git reads no -// configuration, reference or working file of the source; the reader reads -// those itself, each after the guard has classified it. -type gitSource struct { - objects string - shallow []byte -} - // readSource ties the source root to its repository from bytes the reader // read itself, the way Git's files backend would: the worktree link and its // registration, HEAD and the branch it names, the origin in the repository // configuration and the main ref. It refuses what Git would read in another // way: an include or URL rewrite, per-worktree configuration, grafts, -// alternates or another ref storage. It returns the object store Git may read -// and the commit of the main ref. -func (s *session) readSource(p profile) (gitSource, string, string) { - fail := func(reason string) (gitSource, string, string) { return gitSource{}, "", reason } +// alternates or another ref storage. It returns the object store, which the +// reader then reads itself, and the commit of the main ref. +func (s *session) readSource(p profile) (sourceObjects, string, string) { + fail := func(reason string) (sourceObjects, string, string) { return sourceObjects{}, "", reason } link, reason := s.required(filepath.Join(p.root, ".git")) if reason != "" { @@ -51,7 +42,8 @@ func (s *session) readSource(p profile) (gitSource, string, string) { return fail("source_git") } - // Nothing may redirect what Git reads for this worktree. + // Nothing may change what Git would read for this worktree. The reader never + // reads an alternate store; a store that borrows objects is refused. for _, path := range []string{ filepath.Join(gitDir, "config.worktree"), filepath.Join(common, "info", "grafts"), @@ -110,7 +102,7 @@ func (s *session) readSource(p profile) (gitSource, string, string) { if !s.pinDir(objects) { return fail("source_git") } - return gitSource{objects: objects, shallow: shallow}, mainID, "" + return sourceObjects{dir: objects, shallow: shallow}, mainID, "" } // metadata reads one Git metadata file. The guard classifies it first; then it @@ -242,20 +234,40 @@ func refName(name string) bool { return true } -// packedRefs reads the packed-refs file Git writes: an optional header line, -// " " lines and "^" lines that peel the tag before them. It -// refuses a repeated name and anything else. +// packedRefs reads packed-refs as Git does (refs/packed-backend.c): an +// optional first line "# pack-refs with: " and its traits, " " +// records and "^" lines that peel the record before them. Git trusts the +// "sorted" trait and finds a name by binary search, so a file that declares it +// must have its names in strictly ascending byte order; without the trait Git +// sorts the records itself. The reader also refuses an unknown trait, a +// repeated name and any other line. func packedRefs(data []byte) (map[string]string, bool) { text, ok := strings.CutSuffix(string(data), "\n") if !ok { return nil, false } + lines := strings.Split(text, "\n") + sorted := false + if strings.HasPrefix(lines[0], "#") { + traits, found := strings.CutPrefix(lines[0], "# pack-refs with: ") + if !found { + return nil, false + } + for _, trait := range strings.Split(traits, " ") { + switch trait { + case "", "peeled", "fully-peeled": + case "sorted": + sorted = true + default: + return nil, false + } + } + lines = lines[1:] + } refs := map[string]string{} + previous := "" peelable := false - for i, line := range strings.Split(text, "\n") { - if i == 0 && strings.HasPrefix(line, "# pack-refs with:") { - continue - } + for _, line := range lines { if id, found := strings.CutPrefix(line, "^"); found { if !peelable || !hexDigest(id, 40) { return nil, false @@ -271,7 +283,11 @@ func packedRefs(data []byte) (map[string]string, bool) { if _, dup := refs[name]; dup { return nil, false } + if sorted && previous != "" && name <= previous { + return nil, false + } refs[name] = id + previous = name peelable = true } return refs, true diff --git a/tools/todo-update-reader-3190/isolated_git_windows.go b/tools/todo-update-reader-3190/isolated_git_windows.go deleted file mode 100644 index fec48aec..00000000 --- a/tools/todo-update-reader-3190/isolated_git_windows.go +++ /dev/null @@ -1,62 +0,0 @@ -package main - -import ( - "context" - "errors" - "os" - "os/exec" - "path/filepath" - "strings" - "time" -) - -// isolatedEnvironment gives Git a scratch repository whose only outside data -// is the fixed object store. It drops every inherited GIT_* setting and turns -// off the system and global configuration, so Git reads no configuration -// file and follows no include. -func isolatedEnvironment(parent []string, gitDir, objects string) []string { - out := make([]string, 0, len(parent)+7) - for _, entry := range parent { - name, _, _ := strings.Cut(entry, "=") - if !strings.HasPrefix(strings.ToUpper(name), "GIT_") { - out = append(out, entry) - } - } - return append(out, "GIT_DIR="+gitDir, "GIT_OBJECT_DIRECTORY="+objects, "GIT_CONFIG_NOSYSTEM=1", - "GIT_CONFIG_GLOBAL=/dev/null", "GIT_NO_LAZY_FETCH=1", "GIT_OPTIONAL_LOCKS=0", "GIT_TERMINAL_PROMPT=0") -} - -// runGit runs one object read in a new scratch repository that holds only -// HEAD, an empty refs directory and, for a shallow source, its boundary. The -// repository has no configuration file; the objects are the source's own. -func runGit(src gitSource, args ...string) ([]byte, error) { - dir, err := os.MkdirTemp("", "todo-update-reader-3190-") - if err != nil { - return nil, err - } - defer os.RemoveAll(dir) - if err := os.Mkdir(filepath.Join(dir, "refs"), 0o700); err != nil { - return nil, err - } - if err := os.WriteFile(filepath.Join(dir, "HEAD"), []byte("ref: refs/heads/main\n"), 0o600); err != nil { - return nil, err - } - if len(src.shallow) > 0 { - if err := os.WriteFile(filepath.Join(dir, "shallow"), src.shallow, 0o600); err != nil { - return nil, err - } - } - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - cmd := exec.CommandContext(ctx, "git", append([]string{"--no-replace-objects"}, args...)...) - hiddenProcess(cmd) - cmd.Dir = dir - cmd.Env = isolatedEnvironment(os.Environ(), dir, src.objects) - out := &limitedBuffer{limit: 32 << 20} - cmd.Stdout = out - err = cmd.Run() - if out.overflow { - return nil, errors.New("git output limit exceeded") - } - return out.Bytes(), err -} diff --git a/tools/todo-update-reader-3190/objects_windows.go b/tools/todo-update-reader-3190/objects_windows.go new file mode 100644 index 00000000..a71dab74 --- /dev/null +++ b/tools/todo-update-reader-3190/objects_windows.go @@ -0,0 +1,764 @@ +package main + +import ( + "bytes" + "compress/zlib" + "container/heap" + "crypto/sha1" + "encoding/binary" + "encoding/hex" + "io" + "os" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" +) + +// Bounds of the object reads. The commits and trees read here, and the delta +// bases on the way to them, are far smaller than maxObjectBytes. +const ( + maxObjectBytes = 16 << 20 + maxLooseBytes = 16 << 20 + maxIndexBytes = 256 << 20 + maxPackBytes = 1 << 40 + maxDeltaChain = 4096 + maxCommits = 1 << 20 + maxTreeDepth = 128 + maxTreeEntries = 1 << 20 + maxCachedBytes = 64 << 20 +) + +// Pack entry types, as Git's pack format numbers them. +const ( + objCommit = 1 + objTree = 2 + objBlob = 3 + objTag = 4 + objOfsDelta = 6 + objRefDelta = 7 +) + +var objectKinds = map[int]string{objCommit: "commit", objTree: "tree", objBlob: "blob", objTag: "tag"} + +var packIndexName = regexp.MustCompile(`^pack-[0-9a-f]{40}\.idx$`) + +// treeModes maps each mode Git writes in a tree to the type and the mode that +// ls-tree prints for the entry. +var treeModes = map[string]treeEntry{ + "100644": {mode: "100644", kind: "blob"}, + "100755": {mode: "100755", kind: "blob"}, + "120000": {mode: "120000", kind: "blob"}, + "160000": {mode: "160000", kind: "commit"}, + "40000": {mode: "040000", kind: "tree"}, +} + +// sourceObjects is the fixed source's own object directory and the shallow +// file that the reader has already read itself. +type sourceObjects struct { + dir string + shallow []byte +} + +// objectStore reads objects the way Git's loose and pack backends store them, +// from the source's own object directory only, and never runs Git. It reads +// no alternates, commit-graph, multi-pack index or replacement ref, so a file +// added to the shared repository while it reads cannot bring in another +// store. The guard classifies every file before it is opened, each file is +// read from a plain single-link handle that denies writes, and every object +// is checked against its name before it is used. The result therefore rests +// on the objects' names, and the store's files are not rechecked. +type objectStore struct { + s *session + dir string + packs []*pack + shallow map[string]bool + reason string + cache map[packedAt]packedObject + cached int +} + +type packedAt struct { + p *pack + off int64 +} + +type packedObject struct { + kind int + data []byte +} + +// pack is one pack file and its version 2 index. +type pack struct { + path string + count int + fanout [256]uint32 + names []byte + offsets []byte + large []byte + checksum []byte + file *pinnedFile + failed bool +} + +// entryHeader is the start of a pack entry: its type, the size of its +// inflated data, where that data starts and, for a delta, where its base is. +type entryHeader struct { + kind int + size int64 + data int64 + base int64 + baseID []byte +} + +type namedEntry struct { + name string + entry treeEntry +} + +type queuedCommit struct { + id string + parents []string + when int64 + order int +} + +// commitQueue holds queued commits newest first, then in the order queued. +type commitQueue []queuedCommit + +func (q commitQueue) Len() int { return len(q) } + +func (q commitQueue) Less(i, j int) bool { + if q[i].when != q[j].when { + return q[i].when > q[j].when + } + return q[i].order < q[j].order +} + +func (q commitQueue) Swap(i, j int) { q[i], q[j] = q[j], q[i] } + +func (q *commitQueue) Push(x any) { *q = append(*q, x.(queuedCommit)) } + +func (q *commitQueue) Pop() any { + old := *q + last := old[len(old)-1] + *q = old[:len(old)-1] + return last +} + +// openObjects lists the store's packs and reads their indexes. The guard +// classifies the pack directory, then every index and pack, before any is +// opened. An index without its pack is skipped, as Git skips it, and no other +// file of the directory is opened. +func (s *session) openObjects(src sourceObjects) (*objectStore, string) { + o := &objectStore{s: s, dir: src.dir, shallow: map[string]bool{}, cache: map[packedAt]packedObject{}} + if text, ok := strings.CutSuffix(string(src.shallow), "\n"); ok { + for _, id := range strings.Split(text, "\n") { + o.shallow[id] = true + } + } + dir := filepath.Join(src.dir, "pack") + if _, err := os.Lstat(dir); os.IsNotExist(err) { + return o, "" + } else if err != nil { + return nil, "source_git" + } + if !guardClear(s.env, dir) { + return nil, "guard_classification" + } + if !s.pinDir(dir) { + return nil, "source_git" + } + entries, err := os.ReadDir(dir) + if err != nil { + return nil, "source_git" + } + present := map[string]bool{} + for _, e := range entries { + present[e.Name()] = true + } + var paths []string + for _, e := range entries { + packName := strings.TrimSuffix(e.Name(), ".idx") + ".pack" + if packIndexName.MatchString(e.Name()) && present[packName] { + paths = append(paths, filepath.Join(dir, e.Name()), filepath.Join(dir, packName)) + } + } + if !s.classifyPaths(paths) { + return nil, "guard_classification" + } + for i := 0; i < len(paths); i += 2 { + f, ok := s.openObject(paths[i], maxIndexBytes) + if !ok { + return nil, "source_git" + } + data, _, ok := s.read(f) + f.file.Close() + p := &pack{path: paths[i+1]} + if !ok || !p.parseIndex(data) { + return nil, "source_git" + } + o.packs = append(o.packs, p) + } + return o, "" +} + +func (o *objectStore) close() { + for _, p := range o.packs { + if p.file != nil { + p.file.file.Close() + p.file = nil + } + } +} + +// refuse keeps the first reason why an object could not be read. +func (o *objectStore) refuse(reason string) { + if o.reason == "" { + o.reason = reason + } +} + +// failure is the reason code of the first refusal: source_git unless the +// guard refused a file of the store. +func (o *objectStore) failure() string { + if o.reason == "" { + return "source_git" + } + return o.reason +} + +// parseIndex reads a version 2 pack index: the fanout table, the sorted +// object names, their CRCs and offsets, the large offsets and the checksum of +// the pack. The index's own trailing checksum must match its bytes. +func (p *pack) parseIndex(data []byte) bool { + const head = 8 + 256*4 + if len(data) < head+40 || !bytes.Equal(data[:8], []byte{0xff, 't', 'O', 'c', 0, 0, 0, 2}) { + return false + } + if sum := sha1.Sum(data[:len(data)-20]); !bytes.Equal(sum[:], data[len(data)-20:]) { + return false + } + for i := range p.fanout { + p.fanout[i] = binary.BigEndian.Uint32(data[8+4*i:]) + if i > 0 && p.fanout[i] < p.fanout[i-1] { + return false + } + } + body, count := len(data)-head-40, int(p.fanout[255]) + if count > body/28 || (body-28*count)%8 != 0 { + return false + } + p.count = count + p.names = data[head : head+20*count] + p.offsets = data[head+24*count : head+28*count] + p.large = data[head+28*count : len(data)-40] + p.checksum = data[len(data)-40 : len(data)-20] + for i := range count { + name := p.names[20*i : 20*i+20] + lo := uint32(0) + if name[0] > 0 { + lo = p.fanout[name[0]-1] + } + if uint32(i) < lo || uint32(i) >= p.fanout[name[0]] || i > 0 && bytes.Compare(p.names[20*i-20:20*i], name) >= 0 { + return false + } + } + return true +} + +// find returns the index position of the object named id. +func (p *pack) find(id []byte) (int, bool) { + lo, hi := 0, int(p.fanout[id[0]]) + if id[0] > 0 { + lo = int(p.fanout[id[0]-1]) + } + i := lo + sort.Search(hi-lo, func(k int) bool { return bytes.Compare(p.names[20*(lo+k):20*(lo+k)+20], id) >= 0 }) + return i, i < hi && bytes.Equal(p.names[20*i:20*i+20], id) +} + +// offset returns where the entry at index position i starts in the pack. +func (p *pack) offset(i int) (int64, bool) { + v := binary.BigEndian.Uint32(p.offsets[4*i:]) + if v&0x80000000 == 0 { + return int64(v), true + } + j := int(v & 0x7fffffff) + if j >= len(p.large)/8 { + return 0, false + } + off := binary.BigEndian.Uint64(p.large[8*j:]) + return int64(off), off < 1<<62 +} + +// readAt reads exactly len(buf) bytes at off from a held file. +func readAt(f *pinnedFile, buf []byte, off int64) bool { + n, err := f.file.ReadAt(buf, off) + return n == len(buf) && (err == nil || err == io.EOF) +} + +// openPack opens a pack the first time an object is found in its index. The +// pack's header must count the index's objects, and its trailer must be the +// pack checksum the index recorded. +func (o *objectStore) openPack(p *pack) bool { + if p.file != nil || p.failed { + return p.file != nil + } + p.failed = true + f, ok := o.s.openObject(p.path, maxPackBytes) + if !ok { + return false + } + if o.s.env.beforeRead != nil { + o.s.env.beforeRead(p.path) + } + head, tail := make([]byte, 12), make([]byte, 20) + ok = f.size >= 32 && readAt(f, head, 0) && readAt(f, tail, f.size-20) + version := binary.BigEndian.Uint32(head[4:8]) + if !ok || !bytes.Equal(head[:4], []byte("PACK")) || (version != 2 && version != 3) || + binary.BigEndian.Uint32(head[8:12]) != uint32(p.count) || !bytes.Equal(tail, p.checksum) { + f.file.Close() + return false + } + p.file, p.failed = f, false + return true +} + +// header reads the entry at off as Git's unpack_object_header and +// get_delta_base do. +func (p *pack) header(off int64) (entryHeader, bool) { + end := p.file.size - 20 + if off < 12 || off >= end { + return entryHeader{}, false + } + buf := make([]byte, min(32, end-off)) + if !readAt(p.file, buf, off) { + return entryHeader{}, false + } + c := buf[0] + h := entryHeader{kind: int(c >> 4 & 7), size: int64(c & 15)} + i := 1 + for shift := 4; c&0x80 != 0; shift += 7 { + if i == len(buf) || shift > 56 { + return entryHeader{}, false + } + c = buf[i] + i++ + h.size |= int64(c&0x7f) << shift + } + switch h.kind { + case objCommit, objTree, objBlob, objTag: + case objOfsDelta: + if i == len(buf) { + return entryHeader{}, false + } + c = buf[i] + i++ + rel := int64(c & 0x7f) + for c&0x80 != 0 { + if i == len(buf) || rel >= 1<<48 { + return entryHeader{}, false + } + c = buf[i] + i++ + rel = (rel+1)<<7 | int64(c&0x7f) + } + if h.base = off - rel; rel <= 0 || h.base <= 0 { + return entryHeader{}, false + } + case objRefDelta: + if i+20 > len(buf) { + return entryHeader{}, false + } + h.baseID = buf[i : i+20] + i += 20 + default: + return entryHeader{}, false + } + h.data = off + int64(i) + return h, h.size <= maxObjectBytes +} + +// inflate decompresses the zlib stream at at. It must hold exactly size bytes +// and end there with a matching checksum. +func (p *pack) inflate(at, size int64) ([]byte, bool) { + z, err := zlib.NewReader(io.NewSectionReader(p.file.file, at, p.file.size-20-at)) + if err != nil { + return nil, false + } + defer z.Close() + return inflateExact(z, size) +} + +func inflateExact(z io.Reader, size int64) ([]byte, bool) { + data := make([]byte, size) + if _, err := io.ReadFull(z, data); err != nil { + return nil, false + } + n, err := z.Read(make([]byte, 1)) + return data, n == 0 && err == io.EOF +} + +// entry inflates the entry at off and applies its deltas against bases in the +// same pack, as Git's unpack_entry does for a pack stored on disk. +func (o *objectStore) entry(p *pack, off int64, depth int) (int, []byte, bool) { + if cached, ok := o.cache[packedAt{p, off}]; ok { + return cached.kind, cached.data, true + } + h, ok := p.header(off) + if !ok || depth > maxDeltaChain { + return 0, nil, false + } + data, ok := p.inflate(h.data, h.size) + if !ok { + return 0, nil, false + } + kind := h.kind + if kind == objOfsDelta || kind == objRefDelta { + base := h.base + if kind == objRefDelta { + i, in := p.find(h.baseID) + if !in { + return 0, nil, false + } + if base, ok = p.offset(i); !ok { + return 0, nil, false + } + } + baseKind, baseData, ok := o.entry(p, base, depth+1) + if !ok { + return 0, nil, false + } + if data, ok = applyDelta(baseData, data); !ok { + return 0, nil, false + } + kind = baseKind + } + if o.cached+len(data) <= maxCachedBytes { + o.cache[packedAt{p, off}] = packedObject{kind, data} + o.cached += len(data) + } + return kind, data, true +} + +// applyDelta rebuilds an object from its base and a delta, as Git's +// patch_delta does: the delta starts with the base's size and the result's +// size, then copies ranges of the base or inserts literal bytes. +func applyDelta(base, delta []byte) ([]byte, bool) { + if len(delta) < 4 { + return nil, false + } + from, delta, ok := deltaSize(delta) + if !ok || from != uint64(len(base)) { + return nil, false + } + size, delta, ok := deltaSize(delta) + if !ok || size > maxObjectBytes { + return nil, false + } + out := make([]byte, 0, size) + for len(delta) > 0 { + cmd := delta[0] + delta = delta[1:] + switch { + case cmd&0x80 != 0: + var off, n uint64 + for bit := range 7 { + if cmd&(1< uint64(len(base)) || n > size-uint64(len(out)) { + return nil, false + } + out = append(out, base[off:off+n]...) + case cmd != 0: + n := int(cmd) + if n > len(delta) || uint64(n) > size-uint64(len(out)) { + return nil, false + } + out = append(out, delta[:n]...) + delta = delta[n:] + default: + return nil, false + } + } + return out, uint64(len(out)) == size +} + +// deltaSize reads one of a delta's two leading sizes. +func deltaSize(d []byte) (uint64, []byte, bool) { + var size uint64 + for shift := 0; shift < 64; shift += 7 { + if len(d) == 0 { + return 0, nil, false + } + c := d[0] + d = d[1:] + size |= uint64(c&0x7f) << shift + if c&0x80 == 0 { + return size, d, true + } + } + return 0, nil, false +} + +// object returns the type and bytes of the object named id, taken from a pack +// or a loose file of the store and checked against id. +func (o *objectStore) object(id string) (string, []byte, bool) { + raw, err := hex.DecodeString(id) + if err != nil || !hexDigest(id, 40) { + o.refuse("source_git") + return "", nil, false + } + for _, p := range o.packs { + if i, in := p.find(raw); in { + kind, data, ok := o.packed(p, i) + return o.checked(id, kind, data, ok) + } + } + kind, data, ok := o.loose(id) + return o.checked(id, kind, data, ok) +} + +// checked accepts an object only if its name is id. +func (o *objectStore) checked(id, kind string, data []byte, ok bool) (string, []byte, bool) { + if !ok || kind == "" || objectName(kind, data) != id { + o.refuse("source_git") + return "", nil, false + } + return kind, data, true +} + +// packed reads the object at index position i of p. +func (o *objectStore) packed(p *pack, i int) (string, []byte, bool) { + off, ok := p.offset(i) + if !ok || !o.openPack(p) { + return "", nil, false + } + kind, data, ok := o.entry(p, off, 0) + return objectKinds[kind], data, ok +} + +// loose reads the loose file of id. A missing file means that the store does +// not hold the object. +func (o *objectStore) loose(id string) (string, []byte, bool) { + path := filepath.Join(o.dir, id[:2], id[2:]) + if _, err := os.Lstat(path); err != nil { + return "", nil, false + } + if !guardClear(o.s.env, path) { + o.refuse("guard_classification") + return "", nil, false + } + f, ok := o.s.openObject(path, maxLooseBytes) + if !ok { + return "", nil, false + } + data, _, ok := o.s.read(f) + f.file.Close() + if !ok { + return "", nil, false + } + return inflateLoose(data) +} + +// inflateLoose reads a loose object file: one zlib stream of +// " \x00" and exactly size bytes. +func inflateLoose(data []byte) (string, []byte, bool) { + z, err := zlib.NewReader(bytes.NewReader(data)) + if err != nil { + return "", nil, false + } + defer z.Close() + var header []byte + b := make([]byte, 1) + for { + if _, err := io.ReadFull(z, b); err != nil || len(header) == 32 { + return "", nil, false + } + if b[0] == 0 { + break + } + header = append(header, b[0]) + } + kind, text, _ := strings.Cut(string(header), " ") + size, err := strconv.ParseInt(text, 10, 64) + if err != nil || size < 0 || size > maxObjectBytes || strconv.FormatInt(size, 10) != text || !knownKind(kind) { + return "", nil, false + } + content, ok := inflateExact(z, size) + return kind, content, ok +} + +func knownKind(kind string) bool { + for _, k := range objectKinds { + if k == kind { + return true + } + } + return false +} + +// commit reads a commit's tree, its parents (none for a shallow boundary, as +// Git's shallow file defines it) and its committer time. +func (o *objectStore) commit(id string) (string, []string, int64, bool) { + kind, data, ok := o.object(id) + if !ok { + return "", nil, 0, false + } + tree, parents, when, parsed := parseCommit(data) + if kind != "commit" || !parsed { + o.refuse("source_git") + return "", nil, 0, false + } + if o.shallow[id] { + parents = nil + } + return tree, parents, when, true +} + +// parseCommit reads the header Git requires of a commit: the tree line first, +// then the parent lines. The committer line's time only orders the walk. +func parseCommit(data []byte) (string, []string, int64, bool) { + header, _, _ := strings.Cut(string(data), "\n\n") + lines := strings.Split(header, "\n") + tree, ok := strings.CutPrefix(lines[0], "tree ") + if !ok || !hexDigest(tree, 40) { + return "", nil, 0, false + } + var parents []string + for lines = lines[1:]; len(lines) > 0 && strings.HasPrefix(lines[0], "parent "); lines = lines[1:] { + id := strings.TrimPrefix(lines[0], "parent ") + if !hexDigest(id, 40) { + return "", nil, 0, false + } + parents = append(parents, id) + } + var when int64 + for _, line := range lines { + if who, ok := strings.CutPrefix(line, "committer "); ok { + if i := strings.LastIndexByte(who, '>'); i >= 0 { + if fields := strings.Fields(who[i+1:]); len(fields) > 0 { + when, _ = strconv.ParseInt(fields[0], 10, 64) + } + } + break + } + } + return tree, parents, when, true +} + +// reaches reports whether target is tip or one of its ancestors. Like Git's +// merge-base walk it reads each commit when it is queued and takes the newest +// first; a shallow boundary has no parents. A commit it cannot read refuses +// the source. +func (o *objectStore) reaches(tip, target string) bool { + q := &commitQueue{} + seen := map[string]bool{} + queue := func(id string) bool { + _, parents, when, ok := o.commit(id) + if ok { + seen[id] = true + heap.Push(q, queuedCommit{id: id, parents: parents, when: when, order: len(seen)}) + } + return ok + } + if !queue(tip) { + return false + } + for q.Len() > 0 && len(seen) <= maxCommits { + c := heap.Pop(q).(queuedCommit) + if c.id == target { + return true + } + for _, parent := range c.parents { + if !seen[parent] && !queue(parent) { + return false + } + } + } + return false +} + +// listTree lists every entry below the tree id as `git ls-tree -r --full-tree` +// does: subtrees are entered and not listed; blobs, links and submodule +// commits are listed under their path. +func (o *objectStore) listTree(id string) (map[string]treeEntry, bool) { + out := map[string]treeEntry{} + var walk func(id, prefix string, depth int) bool + walk = func(id, prefix string, depth int) bool { + kind, data, ok := o.object(id) + if !ok { + return false + } + if kind != "tree" || depth > maxTreeDepth { + o.refuse("source_git") + return false + } + entries, ok := parseTreeObject(data) + if !ok { + o.refuse("source_git") + return false + } + for _, e := range entries { + path := prefix + e.name + if e.entry.kind == "tree" { + if !walk(e.entry.object, path+"/", depth+1) { + return false + } + continue + } + if _, dup := out[path]; dup || len(out) == maxTreeEntries { + o.refuse("source_git") + return false + } + out[path] = e.entry + } + return true + } + if !walk(id, "", 0) { + return nil, false + } + return out, len(out) > 0 +} + +// parseTreeObject reads a tree's entries: " \x00" and the 20-byte +// object name. It accepts only the modes Git writes, and names that are not +// empty, "." or ".." and contain no "/". +func parseTreeObject(data []byte) ([]namedEntry, bool) { + var entries []namedEntry + for len(data) > 0 { + sp := bytes.IndexByte(data, ' ') + if sp < 0 { + return nil, false + } + entry, known := treeModes[string(data[:sp])] + data = data[sp+1:] + nul := bytes.IndexByte(data, 0) + if !known || nul < 0 || len(data) < nul+21 { + return nil, false + } + name := string(data[:nul]) + if name == "" || name == "." || name == ".." || strings.Contains(name, "/") { + return nil, false + } + entry.object = hex.EncodeToString(data[nul+1 : nul+21]) + entries = append(entries, namedEntry{name, entry}) + data = data[nul+21:] + } + return entries, true +} diff --git a/tools/todo-update-reader-3190/objects_windows_test.go b/tools/todo-update-reader-3190/objects_windows_test.go index 890c16c4..c4abf97b 100644 --- a/tools/todo-update-reader-3190/objects_windows_test.go +++ b/tools/todo-update-reader-3190/objects_windows_test.go @@ -1,13 +1,40 @@ package main import ( + "bytes" + "encoding/binary" + "fmt" + "maps" "os" + "os/exec" "path/filepath" "slices" + "strconv" "strings" + "syscall" "testing" ) +func looseObject(f *fixture, id string) string { + return filepath.Join(f.common, "objects", id[:2], id[2:]) +} + +// writable clears the read-only attribute Git gives object files. +func writable(t *testing.T, path string) { + t.Helper() + if err := os.Chmod(path, 0o644); err != nil { + t.Fatal(err) + } +} + +func removeObject(t *testing.T, path string) { + t.Helper() + writable(t, path) + if err := os.Remove(path); err != nil { + t.Fatal(err) + } +} + // moveObject moves the loose file of id from one object store to another. func moveObject(t *testing.T, from, to, id string) { t.Helper() @@ -17,12 +44,28 @@ func moveObject(t *testing.T, from, to, id string) { t.Fatal(err) } writeFile(t, filepath.Join(to, id[:2], id[2:]), string(data)) - if err := os.Chmod(src, 0o644); err != nil { + removeObject(t, src) +} + +// looseCount counts the loose object files of a store. +func looseCount(t *testing.T, objects string) int { + t.Helper() + entries, err := os.ReadDir(objects) + if err != nil { t.Fatal(err) } - if err := os.Remove(src); err != nil { - t.Fatal(err) + n := 0 + for _, e := range entries { + if !e.IsDir() || len(e.Name()) != 2 { + continue + } + files, err := os.ReadDir(filepath.Join(objects, e.Name())) + if err != nil { + t.Fatal(err) + } + n += len(files) } + return n } // alternateDuringReads names dir in the alternates file of the source's store @@ -60,6 +103,23 @@ func expectUntouchedBelow(t *testing.T, f *fixture, dir string) { } } +// expectReadsClassified requires every file the reader opened, other than the +// installed leaf that its purpose entry qualifies, to have been classified by +// the guard before. +func expectReadsClassified(t *testing.T, f *fixture) { + t.Helper() + f.mu.Lock() + defer f.mu.Unlock() + classified := map[string]bool{} + for _, event := range f.events { + if path, ok := strings.CutPrefix(event, "guard-path:"); ok { + classified[path] = true + } else if path, ok := strings.CutPrefix(event, "read:"); ok && !classified[path] && path != key(f.target) { + t.Errorf("%s was read before its guard classification", path) + } + } +} + // The reader takes objects from the source's own store only. An alternate // store named only while the reads run is never read: an object that only it // holds is missing, and a protected store it names is never classified. @@ -120,3 +180,350 @@ func TestPackedRefsAsGitReadsThem(t *testing.T) { }) } } + +// gitRaw returns Git's standard output exactly. +func gitRaw(t *testing.T, dir string, args ...string) []byte { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} + cmd.Env = childEnvironment(os.Environ()) + out, err := cmd.Output() + if err != nil { + t.Fatalf("git %v: %v", args, err) + } + return out +} + +type gitObject struct { + kind string + data []byte +} + +// gitObjects returns every object of the repository at dir as Git reads it. +func gitObjects(t *testing.T, dir string) map[string]gitObject { + t.Helper() + out := gitRaw(t, dir, "cat-file", "--batch-all-objects", "--batch") + objects := map[string]gitObject{} + for len(out) > 0 { + line, rest, _ := bytes.Cut(out, []byte("\n")) + fields := strings.Fields(string(line)) + if len(fields) != 3 { + t.Fatalf("cat-file printed %q", line) + } + size, err := strconv.Atoi(fields[2]) + if err != nil || len(rest) < size+1 || rest[size] != '\n' { + t.Fatalf("cat-file printed %q and %d more bytes", line, len(rest)) + } + objects[fields[0]] = gitObject{kind: fields[1], data: rest[:size]} + out = rest[size+1:] + } + return objects +} + +// parseLsTree reads `git ls-tree -r -z --full-tree` output. +func parseLsTree(t *testing.T, out []byte) map[string]treeEntry { + t.Helper() + tree := map[string]treeEntry{} + for _, record := range strings.Split(strings.TrimSuffix(string(out), "\x00"), "\x00") { + meta, name, ok := strings.Cut(record, "\t") + fields := strings.Fields(meta) + if !ok || len(fields) != 3 { + t.Fatalf("ls-tree printed %q", record) + } + tree[name] = treeEntry{mode: fields[0], kind: fields[1], object: fields[2]} + } + return tree +} + +// historyRepo is a repository whose history gives Git much to store as deltas: +// a long file edited a line at a time and a directory edited a file at a time, +// then an executable, a symbolic link and a submodule entry. +func historyRepo(t *testing.T) (string, func(args ...string) string) { + t.Helper() + dir := filepath.Join(testLongPath(t, t.TempDir()), "history") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + git := func(args ...string) string { + t.Helper() + out, err := testGit(t, dir, args...) + if err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } + return out + } + commit := func(message string) { + git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", message) + } + git("init", "-q", "-b", "main") + git("config", "core.autocrlf", "false") + lines := make([]string, 300) + for i := range lines { + lines[i] = fmt.Sprintf("line %d of a file long enough for Git to store its versions as deltas", i) + } + for round := range 8 { + lines[round*37] = fmt.Sprintf("line edited in round %d", round) + writeFile(t, filepath.Join(dir, "long.txt"), strings.Join(lines, "\n")+"\n") + for i := range 40 { + content := fmt.Sprintf("file %d\n", i) + if i == round { + content += fmt.Sprintf("edited in round %d\n", round) + } + writeFile(t, filepath.Join(dir, "many", fmt.Sprintf("file-%02d.txt", i)), content) + } + git("add", "-A") + commit(fmt.Sprintf("round %d", round)) + } + git("update-index", "--chmod=+x", "many/file-00.txt") + git("update-index", "--add", "--cacheinfo", "120000,"+git("rev-parse", "HEAD:long.txt")+",link") + git("update-index", "--add", "--cacheinfo", "160000,"+git("rev-parse", "HEAD")+",module") + commit("modes") + return dir, git +} + +func clearGuard(args ...string) ([]byte, int, bool) { + return []byte(clearVerdict + "\n"), 0, true +} + +// The reader reads every object as Git does, loose, whole in a pack or as a +// delta against an offset or a named base, and reads commits, trees and +// ancestry as Git does. +func TestObjectStoreAgreesWithGit(t *testing.T) { + repack := []string{"repack", "-a", "-d", "-f", "-q", "--depth=50", "--window=50"} + cases := map[string]struct { + prepare func(git func(args ...string) string) + ofs, ref, loose bool + }{ + "loose objects": {prepare: func(func(args ...string) string) {}, loose: true}, + "offset deltas": {prepare: func(git func(args ...string) string) { git(repack...) }, ofs: true}, + "named-base deltas": {prepare: func(git func(args ...string) string) { + git(append([]string{"-c", "repack.useDeltaBaseOffset=false"}, repack...)...) + }, ref: true}, + "pack and loose objects": {prepare: func(git func(args ...string) string) { + git(repack...) + git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "after the repack") + }, ofs: true, loose: true}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + dir, git := historyRepo(t) + tc.prepare(git) + objects := filepath.Join(dir, ".git", "objects") + s := newSession(environment{guard: clearGuard}) + defer s.close() + store, reason := s.openObjects(sourceObjects{dir: objects}) + if reason != "" { + t.Fatalf("openObjects refused the store: %s", reason) + } + defer store.close() + + kinds := map[int]int{} + for _, p := range store.packs { + if !store.openPack(p) { + t.Fatalf("%s did not open", p.path) + } + for i := range p.count { + off, ok := p.offset(i) + h, read := p.header(off) + if !ok || !read { + t.Fatalf("entry %d of %s", i, p.path) + } + kinds[h.kind]++ + } + } + if kinds[objOfsDelta] > 0 != tc.ofs || kinds[objRefDelta] > 0 != tc.ref { + t.Fatalf("pack entry types %v", kinds) + } + if loose := looseCount(t, objects); loose > 0 != tc.loose { + t.Fatalf("%d loose objects", loose) + } + + for id, want := range gitObjects(t, dir) { + kind, data, ok := store.object(id) + if !ok || kind != want.kind || !bytes.Equal(data, want.data) { + t.Fatalf("object %s: reader read %q of %d bytes (%v), Git %q of %d bytes", id, kind, len(data), ok, want.kind, len(want.data)) + } + } + + head := git("rev-parse", "HEAD") + tree, parents, _, ok := store.commit(head) + wantParents := strings.Fields(git("rev-list", "--parents", "-n", "1", "HEAD"))[1:] + if !ok || tree != git("rev-parse", "HEAD^{tree}") || !slices.Equal(parents, wantParents) { + t.Fatalf("commit %s: tree %s parents %v (%v)", head, tree, parents, ok) + } + listed, ok := store.listTree(tree) + if want := parseLsTree(t, gitRaw(t, dir, "ls-tree", "-r", "-z", "--full-tree", "HEAD")); !ok || !maps.Equal(listed, want) { + t.Fatalf("tree listing\n got %v\nwant %v", listed, want) + } + if root := git("rev-list", "--max-parents=0", "HEAD"); !store.reaches(head, root) || store.reaches(root, head) { + t.Fatal("the reader's ancestry differs from Git's") + } + }) + } +} + +// A store whose objects are all in a pack qualifies as a loose one does, and +// every index and pack is classified before it is read. +func TestQualifiesFromPacks(t *testing.T) { + f := newFixture(t, nil) + f.gitIn(f.repo, "repack", "-a", "-d", "-q") + if loose := looseCount(t, filepath.Join(f.common, "objects")); loose != 0 { + t.Fatalf("%d loose objects are left after the repack", loose) + } + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q from a pack", got.Qualified, reason) + } + expectReadsClassified(t, f) + for _, ext := range []string{"idx", "pack"} { + matches, err := filepath.Glob(filepath.Join(f.common, "objects", "pack", "pack-*."+ext)) + if err != nil || len(matches) != 1 || f.index("read:"+key(matches[0])) < 0 { + t.Fatalf("the %s file %v was not read (%v)", ext, matches, err) + } + } +} + +// The reader refuses objects that it cannot read as Git wrote them, and files +// of the store that the guard does not clear, before reading them. +func TestRejectsDamagedObjects(t *testing.T) { + repack := func(f *fixture) { f.gitIn(f.repo, "repack", "-a", "-d", "-q") } + packFile := func(f *fixture, ext string) string { + matches, err := filepath.Glob(filepath.Join(f.common, "objects", "pack", "pack-*."+ext)) + if err != nil || len(matches) != 1 { + f.t.Fatalf("pack files %v: %v", matches, err) + } + return matches[0] + } + flip := func(t *testing.T, path string, at int64) { + t.Helper() + writable(t, path) + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if at < 0 { + at += int64(len(data)) + } + data[at] ^= 0xff + if err := os.WriteFile(path, data, 0o644); err != nil { + t.Fatal(err) + } + } + // The middle of an entry's compressed data, from git verify-pack -v: + // " ". + middleOf := func(f *fixture, id string) int64 { + for _, line := range strings.Split(f.gitOut("verify-pack", "-v", packFile(f, "idx")), "\n") { + if fields := strings.Fields(line); len(fields) >= 5 && fields[0] == id { + size, err1 := strconv.ParseInt(fields[3], 10, 64) + off, err2 := strconv.ParseInt(fields[4], 10, 64) + if err1 != nil || err2 != nil { + f.t.Fatalf("verify-pack printed %q", line) + } + return off + size/2 + } + } + f.t.Fatalf("%s is not in the pack", id) + return 0 + } + cases := map[string]func(f *fixture){ + "loose file holds another object": func(f *fixture) { + data, err := os.ReadFile(looseObject(f, f.p.tree)) + if err != nil { + f.t.Fatal(err) + } + writable(f.t, looseObject(f, f.p.head)) + writeFile(f.t, looseObject(f, f.p.head), string(data)) + }, + // A commit of the same tree with no parent would pass every other check: + // only the object's name refuses it. + "loose file holds another commit of the same tree": func(f *fixture) { + other := f.gitOut("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit-tree", f.p.tree, "-m", "another commit") + data, err := os.ReadFile(looseObject(f, other)) + if err != nil { + f.t.Fatal(err) + } + writable(f.t, looseObject(f, f.p.head)) + writeFile(f.t, looseObject(f, f.p.head), string(data)) + }, + "loose file is not a zlib stream": func(f *fixture) { + writable(f.t, looseObject(f, f.p.head)) + writeFile(f.t, looseObject(f, f.p.head), "not an object") + }, + "loose file has a second hard link": func(f *fixture) { + if err := os.Link(looseObject(f, f.p.head), filepath.Join(f.base, "object-alias")); err != nil { + f.t.Fatal(err) + } + }, + "pack entry damaged": func(f *fixture) { + repack(f) + flip(f.t, packFile(f, "pack"), middleOf(f, f.p.head)) + }, + // The first object's CRC, which nothing but the index's own checksum covers. + "pack index damaged": func(f *fixture) { + repack(f) + idx := packFile(f, "idx") + data, err := os.ReadFile(idx) + if err != nil { + f.t.Fatal(err) + } + flip(f.t, idx, int64(8+256*4+20*binary.BigEndian.Uint32(data[8+255*4:]))) + }, + "pack is not the one indexed": func(f *fixture) { + repack(f) + flip(f.t, packFile(f, "pack"), -1) + }, + "pack directory is a junction": func(f *fixture) { + repack(f) + dir := filepath.Join(f.common, "objects", "pack") + if err := os.Rename(dir, dir+"-real"); err != nil { + f.t.Fatal(err) + } + junction(f.t, dir, dir+"-real") + }, + } + for name, damage := range cases { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + damage(f) + expectReason(t, f, "source_git") + }) + } + for name, path := range map[string]func(*fixture) string{ + "loose object": func(f *fixture) string { return looseObject(f, f.p.head) }, + "pack index": func(f *fixture) string { repack(f); return packFile(f, "idx") }, + "pack": func(f *fixture) string { repack(f); return packFile(f, "pack") }, + "pack directory": func(f *fixture) string { return filepath.Join(f.common, "objects", "pack") }, + } { + t.Run("protected "+name, func(t *testing.T) { + f := newFixture(t, nil) + refused := path(f) + f.protected[key(refused)] = clientVerdict + expectReason(t, f, "guard_classification") + if f.index("read:"+key(refused)) >= 0 { + t.Fatalf("%s was read although the guard refused it", refused) + } + }) + } +} + +// Deltas the reader must refuse rather than misapply. +func TestApplyDeltaRefusesMalformedDeltas(t *testing.T) { + base := []byte("0123456789") + // A delta from the 10-byte base to "345XY": copy 3 bytes at 3, insert "XY". + good := []byte{10, 5, 0x80 | 0x01 | 0x10, 3, 3, 2, 'X', 'Y'} + if out, ok := applyDelta(base, good); !ok || string(out) != "345XY" { + t.Fatalf("applyDelta = %q, %v", out, ok) + } + for name, delta := range map[string][]byte{ + "base size differs": {9, 5, 0x80 | 0x01 | 0x10, 3, 3, 2, 'X', 'Y'}, + "result longer than said": {10, 4, 0x80 | 0x01 | 0x10, 3, 3, 2, 'X', 'Y'}, + "result shorter than said": {10, 6, 0x80 | 0x01 | 0x10, 3, 3, 2, 'X', 'Y'}, + "copy beyond the base": {10, 5, 0x80 | 0x01 | 0x10, 9, 3, 2, 'X', 'Y'}, + "reserved opcode": {10, 5, 0, 0x80 | 0x01 | 0x10, 3, 3, 2, 'X', 'Y'}, + "insert past the end": {10, 5, 0x80 | 0x01 | 0x10, 3, 3, 4, 'X', 'Y'}, + "copy argument missing": {10, 5, 0x80 | 0x01 | 0x10, 3}, + } { + if out, ok := applyDelta(base, delta); ok { + t.Errorf("%s: applyDelta accepted %v as %q", name, delta, out) + } + } +} diff --git a/tools/todo-update-reader-3190/pin_windows.go b/tools/todo-update-reader-3190/pin_windows.go index d835de3a..2995e776 100644 --- a/tools/todo-update-reader-3190/pin_windows.go +++ b/tools/todo-update-reader-3190/pin_windows.go @@ -131,6 +131,17 @@ func plainFile(info syscall.ByHandleFileInformation) bool { // open opens one regular, single-link file without following a reparse point // and without reading it. The handle shares only reads. func (s *session) open(path string, limit int64) (*pinnedFile, bool) { + return s.openShared(path, limit, syscall.FILE_SHARE_READ) +} + +// openObject opens a file of the object store the same way, but lets the +// worktrees sharing the repository delete or rename it meanwhile, as Git does +// when it repacks. Writes stay denied, and objects are checked by their names. +func (s *session) openObject(path string, limit int64) (*pinnedFile, bool) { + return s.openShared(path, limit, syscall.FILE_SHARE_READ|syscall.FILE_SHARE_DELETE) +} + +func (s *session) openShared(path string, limit int64, share uint32) (*pinnedFile, bool) { if !s.plainAncestors(path) { return nil, false } @@ -141,7 +152,7 @@ func (s *session) open(path string, limit int64) (*pinnedFile, bool) { if err != nil { return nil, false } - h, err := syscall.CreateFile(name, syscall.GENERIC_READ, syscall.FILE_SHARE_READ, nil, syscall.OPEN_EXISTING, fileFlagOpenReparsePoint|fileFlagSequentialScan, 0) + h, err := syscall.CreateFile(name, syscall.GENERIC_READ, share, nil, syscall.OPEN_EXISTING, fileFlagOpenReparsePoint|fileFlagSequentialScan, 0) if err != nil { return nil, false } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index 59f61eb0..0a6a9c45 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -322,10 +322,6 @@ func (f *fixture) guard(args ...string) ([]byte, int, bool) { func (f *fixture) env() environment { return environment{ guard: f.guard, - git: func(src gitSource, args ...string) ([]byte, error) { - f.record("git:" + strings.Join(args, " ")) - return runGit(src, args...) - }, beforeRead: func(path string) { f.record("read:" + key(path)) if f.onRead != nil { @@ -425,7 +421,8 @@ func TestQualifiesFixedUpdate(t *testing.T) { paths := []string{f.root, f.path(fixtureReceiptName), f.path(fixtureLogName), f.path("target/release/todo.d"), f.path(fixtureCandidateName), f.path(".git"), filepath.Join(f.gitDir, "gitdir"), filepath.Join(f.gitDir, "commondir"), filepath.Join(f.gitDir, "HEAD"), filepath.Join(f.common, "config"), filepath.Join(f.common, "refs", "heads", "work"), - filepath.Join(f.common, "refs", "remotes", "origin", "main"), filepath.Join(f.common, "objects")} + filepath.Join(f.common, "refs", "remotes", "origin", "main"), filepath.Join(f.common, "objects"), + filepath.Join(f.common, "objects", "pack"), looseObject(f, f.p.head), looseObject(f, f.p.tree)} for _, in := range f.inputs { paths = append(paths, f.path(in.Path)) } @@ -442,8 +439,8 @@ func TestQualifiesFixedUpdate(t *testing.T) { if f.index("guard-path:"+key(f.target)) >= 0 { t.Errorf("the general classifier was used for the installed leaf") } - // Git ran only the three object reads, isolated from every other file. - expectObjectReads(t, f) + // No Git runs; every file the reader opened was classified first. + expectReadsClassified(t, f) } func TestBaselineIsReadOnlyAfterPurposeQualification(t *testing.T) { @@ -616,8 +613,20 @@ func TestRejectsSourceChanges(t *testing.T) { // The reader compares inputs itself; Git never reads them. {"tracked input edited", "input", func(f *fixture) { f.write("lib-a/src/lib.rs", crlf("pub fn a() { panic!() }\n")) }}, {"tracked input deleted", "input", func(f *fixture) { os.Remove(f.path("lib-a/src/lib.rs")) }}, - {"another tree is pinned", "source_git", func(f *fixture) { f.p.tree = strings.Repeat("6", 40) }}, + // A tree in the store that differs from HEAD's only outside the inputs, + // pinned and recorded in the receipt: only HEAD's own tree refuses it. + {"another tree is pinned", "source_git", func(f *fixture) { + f.write("README.md", crlf("another tracked file that is not a compiler input\n")) + f.git("add", "README.md") + f.p.tree = f.gitOut("write-tree") + f.receipt["source_tree"] = f.p.tree + f.writeReceipt() + }}, {"head not merged into origin/main", "source_git", func(f *fixture) { f.git("update-ref", "-d", "refs/remotes/origin/main") }}, + {"origin/main does not contain HEAD", "source_git", func(f *fixture) { + other := f.gitOut("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit-tree", f.p.tree, "-m", "unrelated") + f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", other) + }}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -754,7 +763,7 @@ func TestPinnedFilesResistReplacement(t *testing.T) { func TestRejectsChangesBeforeRecheck(t *testing.T) { // Inputs stay pinned against writes and tracked files outside the inputs are - // never read (TestGitNeverReadsWorkingFiles), so the races left are in the + // never read (TestNeverReadsWorkingFiles), so the races left are in the // Git metadata, which is read once and closed for the other worktrees. cases := map[string]func(*fixture){ "new commit": func(f *fixture) { @@ -822,7 +831,7 @@ func TestRejectsProtectedResources(t *testing.T) { if f.index("read:"+key(f.path(rel))) >= 0 { t.Fatalf("%s was read although the guard refused it", rel) } - expectOnlyObjectReads(t, f) + expectReadsClassified(t, f) }) } } diff --git a/tools/todo-update-reader-3190/source_scope_windows_test.go b/tools/todo-update-reader-3190/source_scope_windows_test.go index d8341710..e51df5fc 100644 --- a/tools/todo-update-reader-3190/source_scope_windows_test.go +++ b/tools/todo-update-reader-3190/source_scope_windows_test.go @@ -3,54 +3,12 @@ package main import ( "os" "path/filepath" - "slices" "strings" "syscall" "testing" "time" ) -// objectReads are the only Git commands the reader may run, with exactly these -// arguments. Each reads commits or trees from the fixed object store. -func objectReads(f *fixture) []string { - return []string{ - "rev-parse --verify --quiet " + f.p.head + "^{tree}", - "merge-base --is-ancestor " + f.p.head + " " + f.main, - "ls-tree -r -z --full-tree " + f.p.head, - } -} - -func gitCalls(f *fixture) []string { - f.mu.Lock() - defer f.mu.Unlock() - var calls []string - for _, event := range f.events { - if args, ok := strings.CutPrefix(event, "git:"); ok { - calls = append(calls, args) - } - } - return calls -} - -// expectOnlyObjectReads fails on any Git call other than the object reads. -func expectOnlyObjectReads(t *testing.T, f *fixture) { - t.Helper() - allowed := objectReads(f) - for _, call := range gitCalls(f) { - if !slices.Contains(allowed, call) { - t.Errorf("git %s is not one of the object reads", call) - } - } -} - -// expectObjectReads requires exactly the object reads, each once, in order. -func expectObjectReads(t *testing.T, f *fixture) { - t.Helper() - if got, want := gitCalls(f), objectReads(f); !slices.Equal(got, want) { - t.Errorf("git calls\n got %q\nwant %q", got, want) - } -} - func expectUntouched(t *testing.T, f *fixture, name string) { t.Helper() if f.index("read:"+key(name)) >= 0 || f.index("guard-path:"+key(name)) >= 0 { @@ -99,10 +57,10 @@ func (f *fixture) rebuildReceipt() { f.writeReceipt() } -// The guard classifies files one by one, so Git never opens a working file. -// A protected tracked file outside the inputs cannot decide the result even -// when it changes and nobody may read it. -func TestGitNeverReadsWorkingFiles(t *testing.T) { +// The guard classifies files one by one, and nothing reads a working file on +// the reader's behalf. A protected tracked file outside the inputs cannot +// decide the result even when it changes and nobody may read it. +func TestNeverReadsWorkingFiles(t *testing.T) { cases := map[string]func(t *testing.T, name string){ "timestamp changed": func(t *testing.T, name string) { later := time.Now().Add(time.Hour) @@ -125,15 +83,16 @@ func TestGitNeverReadsWorkingFiles(t *testing.T) { t.Fatalf("verify = qualified %v reason %q: a protected file outside the inputs decided the result", got.Qualified, reason) } expectUntouched(t, f, protected) - expectObjectReads(t, f) + expectReadsClassified(t, f) }) } } -// Git reads no configuration file. An include that names a protected, -// unreadable file never reaches Git, and the reader refuses a repository -// configuration that Git would have to follow or that rewrites the remote. -func TestGitReadsNoConfiguration(t *testing.T) { +// No Git configuration is read. A global include that names a protected, +// unreadable file changes nothing, and the reader refuses a repository +// configuration that Git would have to follow or that rewrites the remote +// before it reads any object. +func TestReadsNoGitConfiguration(t *testing.T) { t.Run("global include", func(t *testing.T) { f := newFixture(t, nil) protected := f.path("README.md") @@ -145,10 +104,10 @@ func TestGitReadsNoConfiguration(t *testing.T) { release := denyReads(t, protected) defer release() if got, reason := f.verify(); !got.Qualified || reason != "" { - t.Fatalf("verify = qualified %v reason %q: the global configuration reached Git", got.Qualified, reason) + t.Fatalf("verify = qualified %v reason %q: the global configuration was read", got.Qualified, reason) } expectUntouched(t, f, protected) - expectObjectReads(t, f) + expectReadsClassified(t, f) }) for name, section := range map[string]string{ "include": "[include]\n\tpath = PROTECTED\n", @@ -163,8 +122,8 @@ func TestGitReadsNoConfiguration(t *testing.T) { release := denyReads(t, protected) defer release() expectReason(t, f, "source_git") - if calls := gitCalls(f); len(calls) != 0 { - t.Fatalf("Git ran %q before the configuration was refused", calls) + if f.index("read:"+key(looseObject(f, f.p.head))) >= 0 { + t.Fatal("objects were read before the configuration was refused") } expectUntouched(t, f, protected) }) @@ -184,29 +143,34 @@ func TestReadsGitMetadataItself(t *testing.T) { if got, reason := f.verify(); !got.Qualified || reason != "" { t.Fatalf("verify = qualified %v reason %q with packed refs", got.Qualified, reason) } - expectObjectReads(t, f) - }) - t.Run("shallow boundary", func(t *testing.T) { - // As in the real shallow clone, HEAD's parent is not in the object - // store. Only the boundary given to Git keeps the ancestry check from - // looking for it. - f := newFixture(t, nil) - parent := f.p.head - f.git("-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit", "-q", "--allow-empty", "-m", "boundary") - head := f.gitOut("rev-parse", "HEAD") - f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", head) - f.p.head, f.main = head, head - f.receipt["source_head"] = head - f.writeReceipt() - if err := os.Remove(filepath.Join(f.common, "objects", parent[:2], parent[2:])); err != nil { - t.Fatal(err) - } - writeFile(t, filepath.Join(f.common, "shallow"), head+"\n") - if got, reason := f.verify(); !got.Qualified || reason != "" { - t.Fatalf("verify = qualified %v reason %q with a shallow boundary", got.Qualified, reason) - } - expectObjectReads(t, f) + expectReadsClassified(t, f) }) + // As in the real shallow clone, a boundary commit's parent is not in the + // store. The main commit merges a boundary, which the walk reaches first, + // and the fixed HEAD: with the shallow file the walk stops at the + // boundary, and without it the missing parent refuses the store. + for name, shallow := range map[string]bool{"shallow boundary on another path": true, "missing parent without a shallow boundary": false} { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + commit := func(args ...string) string { + return f.gitOut(append([]string{"-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit-tree", f.p.tree}, args...)...) + } + missing := commit("-m", "outside the clone") + boundary := commit("-p", missing, "-m", "boundary") + main := commit("-p", boundary, "-p", f.p.head, "-m", "merge") + f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", main) + removeObject(t, looseObject(f, missing)) + if !shallow { + expectReason(t, f, "source_git") + return + } + writeFile(t, filepath.Join(f.common, "shallow"), boundary+"\n") + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q with a shallow boundary", got.Qualified, reason) + } + expectReadsClassified(t, f) + }) + } rejections := map[string]func(f *fixture){ "worktree file names another worktree": func(f *fixture) { f.gitIn(f.repo, "worktree", "add", "-q", "-b", "other", filepath.Join(f.base, "other")) @@ -255,7 +219,7 @@ func TestReadsGitMetadataItself(t *testing.T) { f := newFixture(t, nil) edit(f) expectReason(t, f, "source_git") - expectOnlyObjectReads(t, f) + expectReadsClassified(t, f) }) } for name, path := range map[string]func(*fixture) string{ @@ -273,25 +237,15 @@ func TestReadsGitMetadataItself(t *testing.T) { if f.index("read:"+key(refused)) >= 0 { t.Fatalf("%s was read although the guard refused it", refused) } - if calls := gitCalls(f); len(calls) != 0 { - t.Fatalf("Git ran %q although the source was refused", calls) + if f.index("read:"+key(looseObject(f, f.p.head))) >= 0 { + t.Fatalf("objects were read although %s was refused", refused) } }) } } -func TestIsolatedGitEnvironment(t *testing.T) { - parent := []string{`PATH=C:\bin`, `HOME=C:\home`, `GIT_DIR=C:\evil`, `Git_Config_Global=C:\evil.cfg`, "GIT_CONFIG_COUNT=1", "GIT_CONFIG_KEY_0=include.path"} - got := isolatedEnvironment(parent, `C:\tmp\g`, `C:\repo\.git\objects`) - want := []string{`PATH=C:\bin`, `HOME=C:\home`, `GIT_DIR=C:\tmp\g`, `GIT_OBJECT_DIRECTORY=C:\repo\.git\objects`, - "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_NO_LAZY_FETCH=1", "GIT_OPTIONAL_LOCKS=0", "GIT_TERMINAL_PROMPT=0"} - if !slices.Equal(got, want) { - t.Fatalf("environment\n got %q\nwant %q", got, want) - } -} - -// With Git kept off the working files, the reader itself ties each pinned -// input to the fixed tree's blob, in the form Git writes it out on this PC. +// The reader itself ties each pinned input to the fixed tree's blob, in the +// form Git writes it out on this PC. func TestInputsMustBeTheCommittedBytes(t *testing.T) { t.Run("LF working file is the blob itself", func(t *testing.T) { f := newFixture(t, nil) diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 6ea64a75..0c55f3e6 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -20,11 +20,10 @@ import ( ) // environment is the reader's only connection to the outside: the deployed -// guard's CLI and Git, which sees only the fixed object store. Tests replace -// the guard with a recorded fake. +// guard's CLI. The reader runs no Git; it reads the repository's files itself. +// Tests replace the guard with a recorded fake and observe every read. type environment struct { guard func(args ...string) (stdout []byte, exitCode int, ran bool) - git func(src gitSource, args ...string) ([]byte, error) beforeRead func(path string) beforeRecheck func() } @@ -40,7 +39,7 @@ const ( ) func productionEnvironment() environment { - return environment{guard: runGuard, git: runGit} + return environment{guard: runGuard} } // exactLeaf accepts only the compiled leaf, spelled with either separator or @@ -172,8 +171,8 @@ func (s *session) evidence(name, pin string, limit int64, reason string) ([]byte } // inputs pins every receipt input, has the guard classify all of them, then -// hashes each pinned file and ties its bytes to the fixed tree's blob. Git -// never reads these files. Manifest bytes are kept for the closure check. +// hashes each pinned file and ties its bytes to the fixed tree's blob. Nothing +// else reads these files. Manifest bytes are kept for the closure check. func (s *session) inputs(inputs []receiptInput, tree map[string]treeEntry, p profile) (map[string][]byte, string) { files := make([]*pinnedFile, len(inputs)) for i, in := range inputs { @@ -201,7 +200,17 @@ func (s *session) inputs(inputs []receiptInput, tree map[string]treeEntry, p pro return manifests, "" } +// classifyAll has the guard classify every pinned file. func (s *session) classifyAll(files []*pinnedFile) bool { + paths := make([]string, len(files)) + for i, f := range files { + paths[i] = f.path + } + return s.classifyPaths(paths) +} + +// classifyPaths has the guard classify every path, guardWorkers at a time. +func (s *session) classifyPaths(paths []string) bool { var failed atomic.Bool var wg sync.WaitGroup next := make(chan string) @@ -216,8 +225,8 @@ func (s *session) classifyAll(files []*pinnedFile) bool { } }() } - for _, f := range files { - next <- f.path + for _, path := range paths { + next <- path } close(next) wg.Wait() @@ -273,34 +282,31 @@ func capacityQualified(env environment, p profile) bool { return true } -func gitText(env environment, src gitSource, args ...string) (string, bool) { - out, err := env.git(src, args...) - return strings.TrimSpace(string(out)), err == nil -} - -// checkSource reads the fixed worktree's Git metadata itself (readSource), -// then lets Git read only the object store: the fixed commit's tree, its -// ancestry to the main commit and the tree listing. It never lets Git open a -// configuration, reference or working file; the guard classifies files one by -// one, and only the pinned inputs are compared, by the reader itself. +// checkSource reads the fixed worktree's Git metadata (readSource) and then +// the objects it needs (objectStore), both by the reader itself: the fixed +// commit's tree, its ancestry from the main commit and the tree's listing. No +// Git runs, so no configuration, alternate store or working file is read on +// the reader's behalf. The guard classifies files one by one, and only the +// pinned inputs are compared with the tree, by the reader itself. func (s *session) checkSource(p profile) (map[string]treeEntry, string) { src, mainID, reason := s.readSource(p) if reason != "" { return nil, reason } - if tree, ok := gitText(s.env, src, "rev-parse", "--verify", "--quiet", p.head+"^{tree}"); !ok || tree != p.tree { - return nil, "source_git" + store, reason := s.openObjects(src) + if reason != "" { + return nil, reason } - if _, err := s.env.git(src, "merge-base", "--is-ancestor", p.head, mainID); err != nil { - return nil, "source_git" + defer store.close() + if tree, _, _, ok := store.commit(p.head); !ok || tree != p.tree { + return nil, store.failure() } - out, err := s.env.git(src, "ls-tree", "-r", "-z", "--full-tree", p.head) - if err != nil { - return nil, "source_git" + if !store.reaches(mainID, p.head) { + return nil, store.failure() } - tree, ok := parseTree(out) + tree, ok := store.listTree(p.tree) if !ok { - return nil, "source_git" + return nil, store.failure() } return tree, "" } @@ -320,7 +326,7 @@ func (b *limitedBuffer) Write(p []byte) (int, error) { } // childEnvironment removes inherited GIT_* overrides from the guard's -// environment. Git itself runs in isolatedEnvironment. +// environment. func childEnvironment(parent []string) []string { out := make([]string, 0, len(parent)+3) for _, entry := range parent { From 206181d4e24c4eae09b2e174087dca7fb3347a74 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 06:12:08 +0900 Subject: [PATCH 12/24] docs: record the ToDo3505 review and the reader that runs no Git Record the two Musts of review ToDo3505, the Red and Green of the third fix, the 18 mutants (one survivor fixed by a stronger test), the hard link counts of the real shared store and the pre-merge run of the fe1d5941 reader against the real target, and the CI of 8b4f34b9. --- docs/todo-update-reader-3190.md | 46 ++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index 0fcba00b..b43f6828 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -245,6 +245,44 @@ Suggestionは、テストの許可判定を引数列全体の一致にして反 共有repoはshallow cloneである。configには800件を超えるbranchの節と core・extensions・remote があり、includeは無い。 読んだメタデータはguardの分類で clear だった。configの中身はログや文書へ出していない。 +### 別モデルの再レビュー（ToDo3505） + +gpt-6.1-sol・推論max（job `20261024-051252-todo-3505`、6分28秒、終了コード0）が、base `308cb24d`・head `8b4f34b9` の +変更8ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のMustとSuggestionは +解消と判定された。新たな指摘は Must 2件で、どちらもコードとGitの一次資料からの判断であり、再現テストは伴わない。 + +- Must: `objects/info/alternates` が無いことを確かめた後でも、共有repoのobjectsは変更できる。Gitの起動前に + alternatesが置かれると、Gitはその内容と参照先を分類なしで読み得る。再確認の前に消されれば、最後の不在確認も通る。 + `GIT_OBJECT_DIRECTORY` の指定は、読取先の隔離にはならない。 +- Must: `packed-refs` の見出しを読み飛ばし、`sorted` の宣言と実際の並びを確かめていない。Gitは `sorted` を信じて + 二分探索するので、逆順の記録ではGitが解けないbranchを、readerだけが受け付け得る。コロンの後の空白を欠く見出しも、 + Gitは拒否するがreaderは受け付けていた。 +- 合わせて、この文書の「照合中の変更は拒否する」という断定は、途中で作られて消える変更の検出を保証しないと指摘された。 + +### 3回目のMustの修正 + +Gitの実行をやめ、objectsもreader自身が読むようにした（検証の順序の5）。looseとpack（version 2のindex、 +offset delta、名前を指すdelta）を読み、どのobjectも名前（SHA-1）で中身を確かめる。alternates・commit-graph・ +multi-pack-index・replace refは読まないので、照合中に別のstoreを指すファイルが置かれても、読む中身は変わらない。 +`packed-refs` はGitと同じに見出しと `sorted` を解釈し、宣言と並びが合わないファイルを拒否する。 +再確認の説明は、検出できる範囲に合わせて書き直した（検証の順序の10）。 + +- Red（commit `3b9b98d3`）: `go test -race -count=1 ./...` は終了コード1、163秒。上位26件が合格、3件が失敗し、 + subtest込みで合格153、失敗6、skip 0、ビルド失敗0、data race 0。失敗はすべて意図した理由による。 + alternatesだけにあるobjectで資格を得た。宣言と逆順の `packed-refs` と、空白を欠く見出しの `packed-refs` でも + 資格を得た（どちらもGit自身は `origin/main` を解けない）。`packedRefs` は逆順・見出しの誤り・未知の特性を受け付けた。 +- Green（commit `fe1d5941`）: `go test -race -count=1 ./...` は終了コード0、156秒。上位32件、subtest込み180件が合格し、 + 失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。readerが読んだ全object、commit、tree一覧、祖先は、 + looseだけ、offset delta、名前を指すdelta、packとlooseの混在の4通りの合成repoで、Git自身の出力と一致した。 +- 照合の要所を壊した18の版を `go test -overlay` で当てた。最初は17が検出され、HEADのcommitのtreeと固定treeの照合を + 外した版が生き残った。存在しないtreeを固定したテストでは、後のtree一覧の読取でも拒否されていたためである。 + READMEだけが違う実在のtreeを固定値とreceiptに記録するテストへ替えると、その版は資格を出し、テストが失敗した。 + 壊したのは、`sorted` の並び、見出しの空白、未知の特性、objectの名前の照合、indexとpackのguard分類、 + loose objectのguard分類、shallowの境界、packのchecksum、indexのchecksum、deltaの複写範囲、deltaの結果の大きさ、 + treeの再帰、祖先の判定、HEADのtreeの照合、読んで閉じたファイルの再照合、不在の再照合、includeの拒否、alternatesの不在。 +- 実物の共有repoのobjects（loose 558件、pack 6件とそのindex、multi-pack-index）は、すべてリンク数1だった。 + 固定HEADのcommitはpackの中にある。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -261,6 +299,11 @@ source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力12 `data_access_granted:false`）を返した。reader自身が実物のconfig・HEAD・ref（branchは `packed-refs`、`origin/main` はloose）と `shallow` を読んで受け付け、Gitは空のrepoで固定objectsとshallowの写しだけを読んだ。 +3回目の修正のcommit `fe1d5941` から `go build -trimpath` で生成したreader（SHA256 `fec6e90d…10dd`、vcs.modified=false） +でも実物を1回照合した。終了コード0、11.8秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。Gitは実行せず、reader自身が実物のpackから固定HEADのcommitを読み、 +`origin/main` からの祖先とtreeの一覧を求めた。 + いずれもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 対象の置換、guardの変更、ToDoの状態変更はしていない。 @@ -272,5 +315,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d 修正前のhead `d9edfd4d` では、専用workflow（windows、run 39994455364）とCodeQL（run 39994455228）が成功した。 1回目の修正のhead `308cb24d` でも、専用workflow（windows、run 40002129622）とCodeQL（run 40002129620）が成功した。 -2回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +2回目の修正のhead `8b4f34b9` でも、専用workflow（windows、run 40015024286）とCodeQL（run 40015024382）が成功した。 +3回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる 実物の照合、親3190への結果の返却が残る。 From 9fb6c2b50466ff0c56728c1d9f487c9276b85851 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 07:02:04 +0900 Subject: [PATCH 13/24] test: require the ToDo3190 reader to read commits as Git does and bound its object reads Review ToDo3513 found three Musts. These tests reproduce each of them against the current reader: - A commit whose last line is a "parent " line without its newline has no parent to Git, but the reader read one, so a crafted tip of main made the fixed commit look like its ancestor. TestParseCommitAgreesWithGit compares parseCommit with git log for eleven commit shapes, and TestRefusesParentsGitDoesNotRead checks with git merge-base that Git finds no ancestry where the reader qualified the source. - The pack directory was listed by its path after it was pinned, so a junction put in its place listed names from a directory that the guard never classified. TestListsOnlyThePinnedPackDirectory swaps the directory right before the listing. - A delta was inflated before its base was read, so a chain that returns to itself held one inflated delta per step up to the depth limit: TestDeltaCyclesInflateNothing measures about 250 MB allocated for 16 KiB deltas. Nothing bounded the packs, the index bytes, the bytes read, the commits walked or the tree listed. TestDeltaDepthLimit, TestReadLimit, TestReadLimitCountsCompressedBytes and TestObjectLimits set each bound one below what the source needs. The profile gains the bounds, and the environment a hook that runs before the pack directory is listed, so that the tests compile. The reader does not use either yet. --- .../commit_parse_windows_test.go | 114 ++++++ .../object_bounds_windows_test.go | 329 ++++++++++++++++++ .../objects_windows.go | 9 +- .../objects_windows_test.go | 38 +- .../profile_windows.go | 22 ++ .../reader_windows_test.go | 9 + .../todo-update-reader-3190/verify_windows.go | 6 +- 7 files changed, 522 insertions(+), 5 deletions(-) create mode 100644 tools/todo-update-reader-3190/commit_parse_windows_test.go create mode 100644 tools/todo-update-reader-3190/object_bounds_windows_test.go diff --git a/tools/todo-update-reader-3190/commit_parse_windows_test.go b/tools/todo-update-reader-3190/commit_parse_windows_test.go new file mode 100644 index 00000000..092f76df --- /dev/null +++ b/tools/todo-update-reader-3190/commit_parse_windows_test.go @@ -0,0 +1,114 @@ +package main + +import ( + "bytes" + "compress/zlib" + "errors" + "os" + "os/exec" + "path/filepath" + "slices" + "strconv" + "strings" + "testing" +) + +// writeLooseObject stores body as a loose object of kind, compressed as Git +// stores one, and returns the object's name. +func writeLooseObject(t *testing.T, objects, kind, body string) string { + t.Helper() + id := objectName(kind, []byte(body)) + var buf bytes.Buffer + z := zlib.NewWriter(&buf) + if _, err := z.Write([]byte(kind + " " + strconv.Itoa(len(body)) + "\x00" + body)); err != nil { + t.Fatal(err) + } + if err := z.Close(); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(objects, id[:2], id[2:]), buf.String()) + return id +} + +// gitCommitView returns the tree and the parents that Git reads from the +// commit id, or false if Git refuses to read the commit. +func gitCommitView(t *testing.T, dir, id string) (string, []string, bool) { + t.Helper() + out, err := testGit(t, dir, "log", "--no-walk", "--format=%T %P", id) + if err != nil { + return "", nil, false + } + fields := strings.Fields(out) + if len(fields) == 0 { + t.Fatalf("git log printed %q", out) + } + return fields[0], fields[1:], true +} + +// Git's parse_commit_buffer reads a "tree " line that more bytes follow, then +// takes a line that starts with "parent " as a parent only while more than +// such a line's length follows it, and refuses the commit if that line is not +// "parent \n" with more bytes after it. The reader refuses each commit +// that Git refuses, and also each commit whose "parent " line Git would not +// read, so every commit it accepts has the tree and parents Git reads. +func TestParseCommitAgreesWithGit(t *testing.T) { + const ( + tree = "4b825dc642cb6eb9a060e54bf8d69288fbee4904" + p1 = "1111111111111111111111111111111111111111" + p2 = "2222222222222222222222222222222222222222" + who = "author A 1700000000 +0000\ncommitter C 1700000001 +0000\n" + ) + refused := []string(nil) + cases := []struct { + name string + body string + git, reader []string // the parents read; nil if the commit is refused + }{ + {"root commit", "tree " + tree + "\n" + who + "\nmessage\n", []string{}, []string{}}, + {"two parents", "tree " + tree + "\nparent " + p1 + "\nparent " + p2 + "\n" + who + "\nmessage\n", []string{p1, p2}, []string{p1, p2}}, + {"parent line in the message", "tree " + tree + "\n\nparent " + p1 + "\n", []string{}, []string{}}, + {"tree line only", "tree " + tree + "\n", refused, refused}, + {"tree line without its newline", "tree " + tree, refused, refused}, + {"parent line without its newline", "tree " + tree + "\nparent " + p1, []string{}, refused}, + {"commit ends with a parent line", "tree " + tree + "\nparent " + p1 + "\n", refused, refused}, + {"second parent line without its newline", "tree " + tree + "\nparent " + p1 + "\nparent " + p2, []string{p1}, refused}, + {"short parent", "tree " + tree + "\nparent 1111\n" + who, refused, refused}, + {"long parent", "tree " + tree + "\nparent " + p1 + "1\n" + who, refused, refused}, + {"upper-case tree", "tree " + strings.ToUpper(tree) + "\n" + who, []string{}, refused}, + } + dir := filepath.Join(testLongPath(t, t.TempDir()), "repo") + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if out, err := testGit(t, dir, "init", "-q"); err != nil { + t.Fatalf("git init: %v\n%s", err, out) + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + id := writeLooseObject(t, filepath.Join(dir, ".git", "objects"), "commit", tc.body) + gitTree, gitParents, gitRead := gitCommitView(t, dir, id) + if gitRead != (tc.git != nil) || gitRead && (gitTree != tree || !slices.Equal(gitParents, tc.git)) { + t.Fatalf("Git read tree %q parents %q (%v), want parents %q", gitTree, gitParents, gitRead, tc.git) + } + gotTree, gotParents, _, ok := parseCommit([]byte(tc.body)) + if ok != (tc.reader != nil) || ok && (gotTree != gitTree || !slices.Equal(gotParents, tc.reader)) { + t.Fatalf("parseCommit read tree %q parents %q (%v), want parents %q", gotTree, gotParents, ok, tc.reader) + } + }) + } +} + +// A commit that ends in a "parent " line without its newline has no parent to +// Git. Made the tip of main, it must not let the reader find the fixed commit +// behind it where Git finds no ancestry. +func TestRefusesParentsGitDoesNotRead(t *testing.T) { + f := newFixture(t, nil) + id := writeLooseObject(t, filepath.Join(f.common, "objects"), "commit", "tree "+f.p.tree+"\nparent "+f.p.head) + writeFile(t, filepath.Join(f.common, "refs", "remotes", "origin", "main"), id+"\n") + _, err := testGit(t, f.root, "merge-base", "--is-ancestor", f.p.head, id) + var exit *exec.ExitError + if !errors.As(err, &exit) || exit.ExitCode() != 1 { + t.Fatalf("git merge-base --is-ancestor %s %s: %v, want exit status 1", f.p.head, id, err) + } + expectReason(t, f, "source_git") +} diff --git a/tools/todo-update-reader-3190/object_bounds_windows_test.go b/tools/todo-update-reader-3190/object_bounds_windows_test.go new file mode 100644 index 00000000..047d24e5 --- /dev/null +++ b/tools/todo-update-reader-3190/object_bounds_windows_test.go @@ -0,0 +1,329 @@ +package main + +import ( + "bytes" + "compress/zlib" + "crypto/sha1" + "encoding/binary" + "encoding/hex" + "hash/adler32" + "hash/crc32" + "os" + "path/filepath" + "runtime" + "slices" + "strconv" + "strings" + "testing" +) + +// entryHead encodes a pack entry's type and inflated size as Git does. +func entryHead(kind, size int) []byte { + head := []byte{byte(kind<<4) | byte(size&15)} + for size >>= 4; size > 0; size >>= 7 { + head[len(head)-1] |= 0x80 + head = append(head, byte(size&0x7f)) + } + return head +} + +// ofsBase encodes how far back an offset delta's base starts, as Git does. +func ofsBase(distance int) []byte { + out := []byte{byte(distance & 0x7f)} + for distance >>= 7; distance > 0; distance >>= 7 { + distance-- + out = append([]byte{0x80 | byte(distance&0x7f)}, out...) + } + return out +} + +// deflated is data as one zlib stream. +func deflated(t *testing.T, data []byte) []byte { + t.Helper() + var buf bytes.Buffer + z := zlib.NewWriter(&buf) + if _, err := z.Write(data); err != nil { + t.Fatal(err) + } + if err := z.Close(); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +// rawEntry is one entry of a pack that writeRawPack writes: the name that the +// index gives it, and its bytes from its type and size to the end of its zlib +// stream. +type rawEntry struct { + name []byte + data []byte +} + +// writeRawPack writes a version 2 pack that holds the entries in their order +// from offset 12, and its version 2 index, into dir under Git's names. +func writeRawPack(t *testing.T, dir string, entries []rawEntry) { + t.Helper() + pack := binary.BigEndian.AppendUint32(binary.BigEndian.AppendUint32([]byte("PACK"), 2), uint32(len(entries))) + offsets := make([]uint32, len(entries)) + for i, e := range entries { + offsets[i] = uint32(len(pack)) + pack = append(pack, e.data...) + } + packSum := sha1.Sum(pack) + pack = append(pack, packSum[:]...) + + order := make([]int, len(entries)) + for i := range order { + order[i] = i + } + slices.SortFunc(order, func(a, b int) int { return bytes.Compare(entries[a].name, entries[b].name) }) + idx := []byte{0xff, 't', 'O', 'c', 0, 0, 0, 2} + for first := range 256 { + n := 0 + for _, e := range entries { + if int(e.name[0]) <= first { + n++ + } + } + idx = binary.BigEndian.AppendUint32(idx, uint32(n)) + } + for _, i := range order { + idx = append(idx, entries[i].name...) + } + for _, i := range order { + idx = binary.BigEndian.AppendUint32(idx, crc32.ChecksumIEEE(entries[i].data)) + } + for _, i := range order { + idx = binary.BigEndian.AppendUint32(idx, offsets[i]) + } + idx = append(idx, packSum[:]...) + idxSum := sha1.Sum(idx) + idx = append(idx, idxSum[:]...) + + name := filepath.Join(dir, "pack-"+hex.EncodeToString(packSum[:])) + writeFile(t, name+".pack", string(pack)) + writeFile(t, name+".idx", string(idx)) +} + +// openRawStore opens the object store at objects under limits, as the reader +// opens the source's store. +func openRawStore(t *testing.T, objects string, limits objectLimits) *objectStore { + t.Helper() + s := newSession(environment{guard: clearGuard}) + t.Cleanup(s.close) + store, reason := s.openObjects(sourceObjects{dir: objects}, limits) + if reason != "" { + t.Fatalf("openObjects refused the store: %s", reason) + } + t.Cleanup(store.close) + return store +} + +// A delta chain that returns to an entry is refused once the entries' headers +// show the loop, before anything is inflated, however large the deltas say +// they are. Inflating each delta on the way down would hold one per step until +// the depth limit: tens of gigabytes for 16 MiB deltas. +func TestDeltaCyclesInflateNothing(t *testing.T) { + const size = 16 << 10 + delta := deflated(t, make([]byte, size)) + a, b := sha1.Sum([]byte("a")), sha1.Sum([]byte("b")) + basedOnB := slices.Concat(entryHead(objRefDelta, size), b[:], delta) + cases := map[string][]rawEntry{ + "delta based on itself": {{name: a[:], data: slices.Concat(entryHead(objRefDelta, size), a[:], delta)}}, + "two deltas based on each other": { + {name: a[:], data: basedOnB}, + {name: b[:], data: slices.Concat(entryHead(objOfsDelta, size), ofsBase(len(basedOnB)), delta)}, + }, + } + for name, entries := range cases { + t.Run(name, func(t *testing.T) { + objects := filepath.Join(testLongPath(t, t.TempDir()), "objects") + writeRawPack(t, filepath.Join(objects, "pack"), entries) + store := openRawStore(t, objects, productionProfile.limits) + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + _, _, ok := store.object(hex.EncodeToString(a[:])) + runtime.ReadMemStats(&after) + if ok || store.failure() != "source_git" { + t.Fatalf("object = %v, reason %q; want the cycle refused", ok, store.failure()) + } + if grown := after.TotalAlloc - before.TotalAlloc; grown > 1<<20 { + t.Fatalf("refusing the cycle allocated %d bytes", grown) + } + if store.spent != 0 { + t.Fatalf("refusing the cycle counted %d bytes as read", store.spent) + } + }) + } +} + +// A chain of more deltas than the depth limit is refused before any of its +// entries is inflated, and a chain that meets the limit is read. +func TestDeltaDepthLimit(t *testing.T) { + dir, git := historyRepo(t) + git("repack", "-a", "-d", "-f", "-q", "--depth=50", "--window=50") + objects := filepath.Join(dir, ".git", "objects") + indexes, err := filepath.Glob(filepath.Join(objects, "pack", "pack-*.idx")) + if err != nil || len(indexes) != 1 { + t.Fatalf("pack indexes %v: %v", indexes, err) + } + // git verify-pack -v prints " + // " for each delta. + deepest, depth := "", 0 + for _, line := range strings.Split(git("verify-pack", "-v", indexes[0]), "\n") { + if fields := strings.Fields(line); len(fields) == 7 { + if d, err := strconv.Atoi(fields[5]); err == nil && d > depth { + deepest, depth = fields[0], d + } + } + } + if depth < 2 { + t.Fatalf("the deepest delta chain holds %d deltas", depth) + } + for _, limit := range []int{depth, depth - 1} { + limits := productionProfile.limits + limits.deltaDepth = limit + store := openRawStore(t, objects, limits) + _, _, ok := store.object(deepest) + if ok != (limit == depth) { + t.Fatalf("with %d deltas allowed, reading a chain of %d deltas = %v", limit, depth, ok) + } + if !ok && store.spent != 0 { + t.Fatalf("refusing the chain counted %d bytes as read", store.spent) + } + } +} + +// The compressed bytes read count against the read limit, so a stream padded +// with empty blocks is refused under a limit that its inflated size meets. +func TestReadLimitCountsCompressedBytes(t *testing.T) { + data := []byte("a blob whose stream is padded with empty blocks\n") + id := objectName("blob", data) + name, err := hex.DecodeString(id) + if err != nil { + t.Fatal(err) + } + // A zlib header, 2^18 empty stored blocks, a final stored block of the + // data and the data's Adler-32 checksum. + n := len(data) + stream := slices.Concat([]byte{0x78, 0x01}, bytes.Repeat([]byte{0, 0, 0, 0xff, 0xff}, 1<<18), + []byte{1, byte(n), byte(n >> 8), ^byte(n), ^byte(n >> 8)}, data) + stream = binary.BigEndian.AppendUint32(stream, adler32.Checksum(data)) + objects := filepath.Join(testLongPath(t, t.TempDir()), "objects") + writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{{name: name, data: slices.Concat(entryHead(objBlob, n), stream)}}) + + store := openRawStore(t, objects, productionProfile.limits) + if kind, got, ok := store.object(id); !ok || kind != "blob" || !bytes.Equal(got, data) { + t.Fatalf("object = %q %q %v", kind, got, ok) + } + if store.spent < int64(len(stream)) { + t.Fatalf("reading a stream of %d bytes counted %d bytes", len(stream), store.spent) + } + limits := productionProfile.limits + limits.readBytes = int64(len(stream)) / 2 + if _, _, ok := openRawStore(t, objects, limits).object(id); ok { + t.Fatalf("a stream of %d bytes was read under a limit of %d bytes", len(stream), limits.readBytes) + } +} + +// A source that needs more bytes than the read limit is refused, whether its +// objects are loose or packed. +func TestReadLimit(t *testing.T) { + for name, prepare := range map[string]func(f *fixture){ + "loose objects": func(*fixture) {}, + "packed objects": func(f *fixture) { f.gitIn(f.repo, "repack", "-a", "-d", "-q") }, + } { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + prepare(f) + f.p.limits.readBytes = 64 + expectReason(t, f, "source_git") + }) + } +} + +// Each bound on the object store accepts a source that needs exactly the +// bound, and refuses it under a bound one lower. +func TestObjectLimits(t *testing.T) { + commit := func(f *fixture, args ...string) string { + return f.gitOut(append([]string{"-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit-tree", f.p.tree}, args...)...) + } + repack := func(f *fixture) { f.gitIn(f.repo, "repack", "-a", "-d", "-q") } + packFiles := func(f *fixture, pattern string) []string { + matches, err := filepath.Glob(filepath.Join(f.common, "objects", "pack", pattern)) + if err != nil || len(matches) == 0 { + f.t.Fatalf("pack files %v: %v", matches, err) + } + return matches + } + listed := func(f *fixture, args ...string) []string { + return strings.Split(f.gitOut(append([]string{"ls-tree", "-r", "--full-tree"}, args...)...), "\n") + } + cases := map[string]struct { + set func(l *objectLimits, n int64) + need func(f *fixture) int64 // shapes the store and returns what it needs + }{ + "index and pack pairs": { + func(l *objectLimits, n int64) { l.packs = int(n) }, + func(f *fixture) int64 { + repack(f) + f.gitIn(f.repo, "update-ref", "refs/tags/second-pack", commit(f, "-m", "a commit in a second pack")) + f.gitIn(f.repo, "repack", "-d", "-q") + return int64(len(packFiles(f, "pack-*.idx"))) + }, + }, + "pack directory entries": { + func(l *objectLimits, n int64) { l.packEntries = int(n) }, + func(f *fixture) int64 { repack(f); return int64(len(packFiles(f, "*"))) }, + }, + "index bytes": { + func(l *objectLimits, n int64) { l.indexBytes = n }, + func(f *fixture) int64 { + repack(f) + var total int64 + for _, idx := range packFiles(f, "pack-*.idx") { + info, err := os.Stat(idx) + if err != nil { + f.t.Fatal(err) + } + total += info.Size() + } + return total + }, + }, + "commits walked": { + func(l *objectLimits, n int64) { l.commits = int(n) }, + func(f *fixture) int64 { + ahead := commit(f, "-p", f.p.head, "-m", "one ahead") + f.gitIn(f.repo, "update-ref", "refs/remotes/origin/main", commit(f, "-p", ahead, "-m", "two ahead")) + return 3 + }, + }, + "tree entries": { + func(l *objectLimits, n int64) { l.treeEntries = int(n) }, + func(f *fixture) int64 { return int64(len(listed(f, "-t", "HEAD"))) }, + }, + "path bytes": { + func(l *objectLimits, n int64) { l.pathBytes = n }, + func(f *fixture) int64 { + var total int64 + for _, path := range listed(f, "--name-only", "HEAD") { + total += int64(len(path)) + } + return total + }, + }, + } + for name, b := range cases { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + need := b.need(f) + b.set(&f.p.limits, need) + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("with the bound at the %d needed, verify = qualified %v reason %q", need, got.Qualified, reason) + } + b.set(&f.p.limits, need-1) + expectReason(t, f, "source_git") + }) + } +} diff --git a/tools/todo-update-reader-3190/objects_windows.go b/tools/todo-update-reader-3190/objects_windows.go index a71dab74..b404aa9f 100644 --- a/tools/todo-update-reader-3190/objects_windows.go +++ b/tools/todo-update-reader-3190/objects_windows.go @@ -72,11 +72,13 @@ type sourceObjects struct { type objectStore struct { s *session dir string + limits objectLimits packs []*pack shallow map[string]bool reason string cache map[packedAt]packedObject cached int + spent int64 } type packedAt struct { @@ -151,8 +153,8 @@ func (q *commitQueue) Pop() any { // classifies the pack directory, then every index and pack, before any is // opened. An index without its pack is skipped, as Git skips it, and no other // file of the directory is opened. -func (s *session) openObjects(src sourceObjects) (*objectStore, string) { - o := &objectStore{s: s, dir: src.dir, shallow: map[string]bool{}, cache: map[packedAt]packedObject{}} +func (s *session) openObjects(src sourceObjects, limits objectLimits) (*objectStore, string) { + o := &objectStore{s: s, dir: src.dir, limits: limits, shallow: map[string]bool{}, cache: map[packedAt]packedObject{}} if text, ok := strings.CutSuffix(string(src.shallow), "\n"); ok { for _, id := range strings.Split(text, "\n") { o.shallow[id] = true @@ -170,6 +172,9 @@ func (s *session) openObjects(src sourceObjects) (*objectStore, string) { if !s.pinDir(dir) { return nil, "source_git" } + if s.env.beforeList != nil { + s.env.beforeList(dir) + } entries, err := os.ReadDir(dir) if err != nil { return nil, "source_git" diff --git a/tools/todo-update-reader-3190/objects_windows_test.go b/tools/todo-update-reader-3190/objects_windows_test.go index c4abf97b..ce0b8195 100644 --- a/tools/todo-update-reader-3190/objects_windows_test.go +++ b/tools/todo-update-reader-3190/objects_windows_test.go @@ -311,7 +311,7 @@ func TestObjectStoreAgreesWithGit(t *testing.T) { objects := filepath.Join(dir, ".git", "objects") s := newSession(environment{guard: clearGuard}) defer s.close() - store, reason := s.openObjects(sourceObjects{dir: objects}) + store, reason := s.openObjects(sourceObjects{dir: objects}, productionProfile.limits) if reason != "" { t.Fatalf("openObjects refused the store: %s", reason) } @@ -382,6 +382,42 @@ func TestQualifiesFromPacks(t *testing.T) { } } +// The reader lists the pack directory through the handle that it verified. A +// junction put in the directory's place after the pin leads the reader +// nowhere: no name from the junction's target is listed, classified or read. +func TestListsOnlyThePinnedPackDirectory(t *testing.T) { + f := newFixture(t, nil) + f.gitIn(f.repo, "repack", "-a", "-d", "-q") + dir := filepath.Join(f.common, "objects", "pack") + elsewhere := filepath.Join(f.base, "elsewhere") + unlisted := "pack-" + strings.Repeat("f", 40) + writeFile(t, filepath.Join(elsewhere, unlisted+".idx"), "an index that the guard never classified") + writeFile(t, filepath.Join(elsewhere, unlisted+".pack"), "a pack that the guard never classified") + swapped := false + f.onList = func(listed string) { + if listed != key(dir) || swapped { + return + } + if err := os.Rename(dir, dir+"-pinned"); err != nil { + t.Fatalf("the pinned pack directory could not be moved: %v", err) + } + junction(t, dir, elsewhere) + swapped = true + } + expectReason(t, f, "source_git") + if !swapped { + t.Fatal("the reader never listed the pack directory") + } + f.mu.Lock() + for _, event := range f.events { + if strings.Contains(event, unlisted) { + t.Errorf("a name from the junction's target reached the reader: %s", event) + } + } + f.mu.Unlock() + expectUntouchedBelow(t, f, elsewhere) +} + // The reader refuses objects that it cannot read as Git wrote them, and files // of the store that the guard does not clear, before reading them. func TestRejectsDamagedObjects(t *testing.T) { diff --git a/tools/todo-update-reader-3190/profile_windows.go b/tools/todo-update-reader-3190/profile_windows.go index c672844d..b94393e5 100644 --- a/tools/todo-update-reader-3190/profile_windows.go +++ b/tools/todo-update-reader-3190/profile_windows.go @@ -60,6 +60,20 @@ type receiptExpectation struct { baselineQualifiedBy string } +// objectLimits bounds what the reader takes from the source's object store, +// so that a store shaped to exhaust memory or time refuses the source. Every +// bound is far above what the fixed source needs. +type objectLimits struct { + packs int // index and pack pairs in the pack directory + packEntries int // entries of the pack directory + indexBytes int64 // the indexes' total size + readBytes int64 // compressed bytes read, and bytes inflated or rebuilt from deltas + deltaDepth int // deltas between a pack entry and its whole base + commits int // commits that the ancestry walk reads + treeEntries int // entries below the fixed tree, subtrees included + pathBytes int64 // the listed paths' total length +} + // profile holds every compiled pin. Only tests construct other profiles; the // CLI always uses productionProfile and accepts no caller-supplied values. type profile struct { @@ -84,6 +98,7 @@ type profile struct { inputs int depEntries int localCrates int + limits objectLimits } var repositoryOrigins = []string{ @@ -128,4 +143,11 @@ var productionProfile = profile{ inputs: 126, depEntries: 116, localCrates: 8, + // Git writes at most 4095 deltas in a chain. The shared store holds 6 + // packs with about 530 KB of index; the fixed tree has 1,657 entries, + // subtrees included, and 53,698 bytes of listed paths. + limits: objectLimits{ + packs: 64, packEntries: 4096, indexBytes: 256 << 20, readBytes: 512 << 20, + deltaDepth: 4095, commits: 65536, treeEntries: 65536, pathBytes: 16 << 20, + }, } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index 0a6a9c45..778e21e8 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -72,6 +72,7 @@ type fixture struct { onCapacity func() onRecheck func() onRead func(path string) + onList func(dir string) } func sha(data []byte) string { @@ -151,6 +152,7 @@ func newFixture(t *testing.T, edit func(map[string]string)) *fixture { inputs: len(names), depEntries: len(fixtureDeps), localCrates: 2, + limits: productionProfile.limits, } f.receipt = map[string]any{ "domain": f.p.receipt.domain, @@ -328,6 +330,12 @@ func (f *fixture) env() environment { f.onRead(key(path)) } }, + beforeList: func(dir string) { + f.record("list:" + key(dir)) + if f.onList != nil { + f.onList(key(dir)) + } + }, beforeRecheck: func() { if f.onRecheck != nil { f.onRecheck() @@ -957,6 +965,7 @@ func TestProductionProfileBindsOneFixedUpdate(t *testing.T) { "counts": {fmt.Sprintf("%d %d %d", p.inputs, p.depEntries, p.localCrates), "126 116 8"}, "guard": {guardExecutable, `D:\CC\Tools\client-records\scrub391.exe`}, "origins": {strings.Join(p.origins, " "), "https://github.com/autodevjapan/cc-pc-tools-rs.git https://github.com/autodevjapan/cc-pc-tools-rs git@github.com:autodevjapan/cc-pc-tools-rs.git"}, + "limits": {fmt.Sprintf("%+v", p.limits), "{packs:64 packEntries:4096 indexBytes:268435456 readBytes:536870912 deltaDepth:4095 commits:65536 treeEntries:65536 pathBytes:16777216}"}, } for name, pair := range checks { if pair[0] != pair[1] { diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 0c55f3e6..1d845af8 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -21,10 +21,12 @@ import ( // environment is the reader's only connection to the outside: the deployed // guard's CLI. The reader runs no Git; it reads the repository's files itself. -// Tests replace the guard with a recorded fake and observe every read. +// Tests replace the guard with a recorded fake and observe every read and +// directory listing. type environment struct { guard func(args ...string) (stdout []byte, exitCode int, ran bool) beforeRead func(path string) + beforeList func(dir string) beforeRecheck func() } @@ -293,7 +295,7 @@ func (s *session) checkSource(p profile) (map[string]treeEntry, string) { if reason != "" { return nil, reason } - store, reason := s.openObjects(src) + store, reason := s.openObjects(src, p.limits) if reason != "" { return nil, reason } From 7a008ac9b9d7bc6c695102fa98d136553caabdbb Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 07:29:32 +0900 Subject: [PATCH 14/24] fix: read commits as Git does and bound the object reads in the ToDo3190 reader - parseCommit follows Git's parse_commit_buffer: the tree line needs more bytes after it, and a "parent " line is read only while more than its length follows, as "parent \n". A shorter "parent " line, which Git reads as no parent, refuses the commit, so every commit that the reader accepts has the parents that Git reads. - The pack directory is listed through the handle that pinned it, with GetFileInformationByHandleEx, and never reopened by its path. A directory put in its place after the pin is never listed. - A pack entry is rebuilt in the order of Git's unpack_entry. The headers alone are followed down to a whole object, and a chain that returns to an entry or passes the depth limit is refused before anything is inflated. The base is then inflated and the deltas are applied from the innermost one up, each inflated only to apply it. - objectLimits in the profile bounds the packs (64), the pack directory's entries (4,096), the indexes' total (256 MiB), the bytes read, inflated and rebuilt (512 MiB), the delta chain (4,095, the most Git writes), the commits walked (65,536), the tree entries with subtrees (65,536) and the listed paths' total (16 MiB). Measured by the reader itself with the deployed guard, the real source needs 20 entries, 6 packs, 529,584 index bytes, 982,328 bytes read, 3 deltas, 45 commits, 1,657 entries and 53,698 path bytes. The tests now also tell the five read charges apart, allow at most 256 KiB to be allocated while a delta cycle is refused, and bound the indexes' total over two packs, so that removing any one of these checks fails a test. --- .../gitmeta_windows.go | 2 +- .../object_bounds_windows_test.go | 131 ++++++--- .../objects_windows.go | 256 +++++++++++++----- tools/todo-update-reader-3190/pin_windows.go | 77 +++++- .../todo-update-reader-3190/verify_windows.go | 2 +- 5 files changed, 346 insertions(+), 122 deletions(-) diff --git a/tools/todo-update-reader-3190/gitmeta_windows.go b/tools/todo-update-reader-3190/gitmeta_windows.go index 1b02c5e7..7dc0fef5 100644 --- a/tools/todo-update-reader-3190/gitmeta_windows.go +++ b/tools/todo-update-reader-3190/gitmeta_windows.go @@ -99,7 +99,7 @@ func (s *session) readSource(p profile) (sourceObjects, string, string) { if !guardClear(s.env, objects) { return fail("guard_classification") } - if !s.pinDir(objects) { + if _, ok := s.pinDir(objects); !ok { return fail("source_git") } return sourceObjects{dir: objects, shallow: shallow}, mainID, "" diff --git a/tools/todo-update-reader-3190/object_bounds_windows_test.go b/tools/todo-update-reader-3190/object_bounds_windows_test.go index 047d24e5..064bf56e 100644 --- a/tools/todo-update-reader-3190/object_bounds_windows_test.go +++ b/tools/todo-update-reader-3190/object_bounds_windows_test.go @@ -51,6 +51,25 @@ func deflated(t *testing.T, data []byte) []byte { return buf.Bytes() } +// padded is data as one zlib stream that holds blocks empty stored blocks +// before a final stored block of the data. +func padded(data []byte, blocks int) []byte { + n := len(data) + stream := slices.Concat([]byte{0x78, 0x01}, bytes.Repeat([]byte{0, 0, 0, 0xff, 0xff}, blocks), + []byte{1, byte(n), byte(n >> 8), ^byte(n), ^byte(n >> 8)}, data) + return binary.BigEndian.AppendUint32(stream, adler32.Checksum(data)) +} + +// rawName is the 20-byte name of the object of kind with data. +func rawName(t *testing.T, kind string, data []byte) []byte { + t.Helper() + name, err := hex.DecodeString(objectName(kind, data)) + if err != nil { + t.Fatal(err) + } + return name +} + // rawEntry is one entry of a pack that writeRawPack writes: the name that the // index gives it, and its bytes from its type and size to the end of its zlib // stream. @@ -119,10 +138,11 @@ func openRawStore(t *testing.T, objects string, limits objectLimits) *objectStor return store } -// A delta chain that returns to an entry is refused once the entries' headers -// show the loop, before anything is inflated, however large the deltas say -// they are. Inflating each delta on the way down would hold one per step until -// the depth limit: tens of gigabytes for 16 MiB deltas. +// A delta chain that returns to an entry is refused as soon as the entries' +// headers show the loop, before anything is inflated, however large the +// deltas say they are. Inflating each delta on the way down would hold one per +// step up to the depth limit: tens of gigabytes for 16 MiB deltas. Walking +// the headers on to the depth limit would allocate about a megabyte. func TestDeltaCyclesInflateNothing(t *testing.T) { const size = 16 << 10 delta := deflated(t, make([]byte, size)) @@ -147,7 +167,7 @@ func TestDeltaCyclesInflateNothing(t *testing.T) { if ok || store.failure() != "source_git" { t.Fatalf("object = %v, reason %q; want the cycle refused", ok, store.failure()) } - if grown := after.TotalAlloc - before.TotalAlloc; grown > 1<<20 { + if grown := after.TotalAlloc - before.TotalAlloc; grown > 256<<10 { t.Fatalf("refusing the cycle allocated %d bytes", grown) } if store.spent != 0 { @@ -194,35 +214,63 @@ func TestDeltaDepthLimit(t *testing.T) { } } -// The compressed bytes read count against the read limit, so a stream padded -// with empty blocks is refused under a limit that its inflated size meets. -func TestReadLimitCountsCompressedBytes(t *testing.T) { - data := []byte("a blob whose stream is padded with empty blocks\n") - id := objectName("blob", data) - name, err := hex.DecodeString(id) - if err != nil { - t.Fatal(err) - } - // A zlib header, 2^18 empty stored blocks, a final stored block of the - // data and the data's Adler-32 checksum. - n := len(data) - stream := slices.Concat([]byte{0x78, 0x01}, bytes.Repeat([]byte{0, 0, 0, 0xff, 0xff}, 1<<18), - []byte{1, byte(n), byte(n >> 8), ^byte(n), ^byte(n >> 8)}, data) - stream = binary.BigEndian.AppendUint32(stream, adler32.Checksum(data)) - objects := filepath.Join(testLongPath(t, t.TempDir()), "objects") - writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{{name: name, data: slices.Concat(entryHead(objBlob, n), stream)}}) - - store := openRawStore(t, objects, productionProfile.limits) - if kind, got, ok := store.object(id); !ok || kind != "blob" || !bytes.Equal(got, data) { - t.Fatalf("object = %q %q %v", kind, got, ok) - } - if store.spent < int64(len(stream)) { - t.Fatalf("reading a stream of %d bytes counted %d bytes", len(stream), store.spent) +// Every byte that the store reads or produces counts against the read limit: +// the compressed bytes as they are read, each pack entry's inflated size and +// each delta's result, and each loose file's size and inflated size. Each +// object below is read under the production limit, and refused under a limit +// that only the bytes of its case pass. +func TestReadLimitCountsEveryByte(t *testing.T) { + const limit = 512 << 10 + small := []byte("a blob that fits under any limit\n") + large := make([]byte, 1<<20) + base := make([]byte, 64<<10) + // A delta from base to large: the two sizes, then 16 copies of all of base. + delta := slices.Concat([]byte{0x80, 0x80, 0x04, 0x80, 0x80, 0x40}, bytes.Repeat([]byte{0x80}, 16)) + cases := map[string]struct { + data []byte + write func(t *testing.T, objects string) + }{ + "compressed bytes of a pack entry": {small, func(t *testing.T, objects string) { + writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{ + {rawName(t, "blob", small), slices.Concat(entryHead(objBlob, len(small)), padded(small, 1<<18))}, + }) + }}, + "inflated size of a pack entry": {large, func(t *testing.T, objects string) { + writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{ + {rawName(t, "blob", large), slices.Concat(entryHead(objBlob, len(large)), deflated(t, large))}, + }) + }}, + "size that a delta rebuilds": {large, func(t *testing.T, objects string) { + whole := slices.Concat(entryHead(objBlob, len(base)), deflated(t, base)) + writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{ + {rawName(t, "blob", base), whole}, + {rawName(t, "blob", large), slices.Concat(entryHead(objOfsDelta, len(delta)), ofsBase(len(whole)), deflated(t, delta))}, + }) + }}, + "bytes of a loose file": {small, func(t *testing.T, objects string) { + id := objectName("blob", small) + content := slices.Concat([]byte("blob "+strconv.Itoa(len(small))+"\x00"), small) + writeFile(t, filepath.Join(objects, id[:2], id[2:]), string(padded(content, 1<<18))) + }}, + "inflated size of a loose object": {large, func(t *testing.T, objects string) { + writeLooseObject(t, objects, "blob", string(large)) + }}, } - limits := productionProfile.limits - limits.readBytes = int64(len(stream)) / 2 - if _, _, ok := openRawStore(t, objects, limits).object(id); ok { - t.Fatalf("a stream of %d bytes was read under a limit of %d bytes", len(stream), limits.readBytes) + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + objects := filepath.Join(testLongPath(t, t.TempDir()), "objects") + tc.write(t, objects) + id := objectName("blob", tc.data) + if kind, got, ok := openRawStore(t, objects, productionProfile.limits).object(id); !ok || kind != "blob" || !bytes.Equal(got, tc.data) { + t.Fatalf("under the production limit, object = %q of %d bytes, %v", kind, len(got), ok) + } + limits := productionProfile.limits + limits.readBytes = limit + store := openRawStore(t, objects, limits) + if _, _, ok := store.object(id); ok || store.failure() != "source_git" { + t.Fatalf("under a limit of %d bytes, object = %v, reason %q", limit, ok, store.failure()) + } + }) } } @@ -249,6 +297,13 @@ func TestObjectLimits(t *testing.T) { return f.gitOut(append([]string{"-c", "user.name=fixture", "-c", "user.email=fixture@example.invalid", "commit-tree", f.p.tree}, args...)...) } repack := func(f *fixture) { f.gitIn(f.repo, "repack", "-a", "-d", "-q") } + // twoPacks leaves the fixture's objects in one pack and a commit that a tag + // names in a second one. + twoPacks := func(f *fixture) { + repack(f) + f.gitIn(f.repo, "update-ref", "refs/tags/second-pack", commit(f, "-m", "a commit in a second pack")) + f.gitIn(f.repo, "repack", "-d", "-q") + } packFiles := func(f *fixture, pattern string) []string { matches, err := filepath.Glob(filepath.Join(f.common, "objects", "pack", pattern)) if err != nil || len(matches) == 0 { @@ -266,9 +321,7 @@ func TestObjectLimits(t *testing.T) { "index and pack pairs": { func(l *objectLimits, n int64) { l.packs = int(n) }, func(f *fixture) int64 { - repack(f) - f.gitIn(f.repo, "update-ref", "refs/tags/second-pack", commit(f, "-m", "a commit in a second pack")) - f.gitIn(f.repo, "repack", "-d", "-q") + twoPacks(f) return int64(len(packFiles(f, "pack-*.idx"))) }, }, @@ -276,10 +329,14 @@ func TestObjectLimits(t *testing.T) { func(l *objectLimits, n int64) { l.packEntries = int(n) }, func(f *fixture) int64 { repack(f); return int64(len(packFiles(f, "*"))) }, }, + // Each of the two indexes fits under the bound; only their total does not. "index bytes": { func(l *objectLimits, n int64) { l.indexBytes = n }, func(f *fixture) int64 { - repack(f) + twoPacks(f) + if n := len(packFiles(f, "pack-*.idx")); n != 2 { + f.t.Fatalf("%d indexes", n) + } var total int64 for _, idx := range packFiles(f, "pack-*.idx") { info, err := os.Stat(idx) diff --git a/tools/todo-update-reader-3190/objects_windows.go b/tools/todo-update-reader-3190/objects_windows.go index b404aa9f..50b7961a 100644 --- a/tools/todo-update-reader-3190/objects_windows.go +++ b/tools/todo-update-reader-3190/objects_windows.go @@ -7,6 +7,7 @@ import ( "crypto/sha1" "encoding/binary" "encoding/hex" + "errors" "io" "os" "path/filepath" @@ -16,17 +17,14 @@ import ( "strings" ) -// Bounds of the object reads. The commits and trees read here, and the delta -// bases on the way to them, are far smaller than maxObjectBytes. +// Bounds of each object read. The commits and trees read here, and the delta +// bases on the way to them, are far smaller than maxObjectBytes. The profile's +// objectLimits bound the totals. const ( maxObjectBytes = 16 << 20 maxLooseBytes = 16 << 20 - maxIndexBytes = 256 << 20 maxPackBytes = 1 << 40 - maxDeltaChain = 4096 - maxCommits = 1 << 20 maxTreeDepth = 128 - maxTreeEntries = 1 << 20 maxCachedBytes = 64 << 20 ) @@ -68,7 +66,8 @@ type sourceObjects struct { // store. The guard classifies every file before it is opened, each file is // read from a plain single-link handle that denies writes, and every object // is checked against its name before it is used. The result therefore rests -// on the objects' names, and the store's files are not rechecked. +// on the objects' names, and the store's files are not rechecked. The +// profile's objectLimits bound what it lists, reads and rebuilds. type objectStore struct { s *session dir string @@ -78,7 +77,7 @@ type objectStore struct { reason string cache map[packedAt]packedObject cached int - spent int64 + spent int64 // bytes counted against limits.readBytes } type packedAt struct { @@ -114,6 +113,13 @@ type entryHeader struct { baseID []byte } +// deltaStep is a delta on the way down to its whole base: where its entry +// starts, and its header. +type deltaStep struct { + at int64 + h entryHeader +} + type namedEntry struct { name string entry treeEntry @@ -150,9 +156,10 @@ func (q *commitQueue) Pop() any { } // openObjects lists the store's packs and reads their indexes. The guard -// classifies the pack directory, then every index and pack, before any is -// opened. An index without its pack is skipped, as Git skips it, and no other -// file of the directory is opened. +// classifies the pack directory, which is then listed through the handle that +// pins it, and every index and pack before any is opened. An index without its +// pack is skipped, as Git skips it, and no other file of the directory is +// opened. func (s *session) openObjects(src sourceObjects, limits objectLimits) (*objectStore, string) { o := &objectStore{s: s, dir: src.dir, limits: limits, shallow: map[string]bool{}, cache: map[packedAt]packedObject{}} if text, ok := strings.CutSuffix(string(src.shallow), "\n"); ok { @@ -169,35 +176,41 @@ func (s *session) openObjects(src sourceObjects, limits objectLimits) (*objectSt if !guardClear(s.env, dir) { return nil, "guard_classification" } - if !s.pinDir(dir) { + pinned, ok := s.pinDir(dir) + if !ok { return nil, "source_git" } if s.env.beforeList != nil { s.env.beforeList(dir) } - entries, err := os.ReadDir(dir) - if err != nil { + names, ok := pinned.list(limits.packEntries) + if !ok { return nil, "source_git" } present := map[string]bool{} - for _, e := range entries { - present[e.Name()] = true + for _, name := range names { + present[name] = true } var paths []string - for _, e := range entries { - packName := strings.TrimSuffix(e.Name(), ".idx") + ".pack" - if packIndexName.MatchString(e.Name()) && present[packName] { - paths = append(paths, filepath.Join(dir, e.Name()), filepath.Join(dir, packName)) + for _, name := range names { + packName := strings.TrimSuffix(name, ".idx") + ".pack" + if packIndexName.MatchString(name) && present[packName] { + paths = append(paths, filepath.Join(dir, name), filepath.Join(dir, packName)) } } + if len(paths) > 2*limits.packs { + return nil, "source_git" + } if !s.classifyPaths(paths) { return nil, "guard_classification" } + var indexed int64 for i := 0; i < len(paths); i += 2 { - f, ok := s.openObject(paths[i], maxIndexBytes) + f, ok := s.openObject(paths[i], limits.indexBytes-indexed) if !ok { return nil, "source_git" } + indexed += f.size data, _, ok := s.read(f) f.file.Close() p := &pack{path: paths[i+1]} @@ -234,6 +247,34 @@ func (o *objectStore) failure() string { return o.reason } +// charge counts n more bytes read from the store, inflated or rebuilt from +// deltas, and refuses the source once the total would pass the read limit. +func (o *objectStore) charge(n int64) bool { + if n > o.limits.readBytes-o.spent { + return false + } + o.spent += n + return true +} + +// errReadLimit stops an inflation whose compressed bytes pass the read limit. +var errReadLimit = errors.New("object store read limit reached") + +// metered counts the compressed bytes that an inflation reads against the +// read limit as it reads them, so padding that inflates to nothing counts too. +type metered struct { + r io.Reader + o *objectStore +} + +func (m metered) Read(b []byte) (int, error) { + n, err := m.r.Read(b) + if !m.o.charge(int64(n)) { + return 0, errReadLimit + } + return n, err +} + // parseIndex reads a version 2 pack index: the fanout table, the sorted // object names, their CRCs and offsets, the large offsets and the checksum of // the pack. The index's own trailing checksum must match its bytes. @@ -385,10 +426,15 @@ func (p *pack) header(off int64) (entryHeader, bool) { return h, h.size <= maxObjectBytes } -// inflate decompresses the zlib stream at at. It must hold exactly size bytes -// and end there with a matching checksum. -func (p *pack) inflate(at, size int64) ([]byte, bool) { - z, err := zlib.NewReader(io.NewSectionReader(p.file.file, at, p.file.size-20-at)) +// inflate decompresses the zlib stream at at of p. It must hold exactly size +// bytes and end there with a matching checksum. The size counts against the +// read limit before anything is inflated, and the compressed bytes as they +// are read. +func (o *objectStore) inflate(p *pack, at, size int64) ([]byte, bool) { + if !o.charge(size) { + return nil, false + } + z, err := zlib.NewReader(metered{io.NewSectionReader(p.file.file, at, p.file.size-20-at), o}) if err != nil { return nil, false } @@ -405,46 +451,91 @@ func inflateExact(z io.Reader, size int64) ([]byte, bool) { return data, n == 0 && err == io.EOF } -// entry inflates the entry at off and applies its deltas against bases in the -// same pack, as Git's unpack_entry does for a pack stored on disk. -func (o *objectStore) entry(p *pack, off int64, depth int) (int, []byte, bool) { - if cached, ok := o.cache[packedAt{p, off}]; ok { - return cached.kind, cached.data, true - } - h, ok := p.header(off) - if !ok || depth > maxDeltaChain { - return 0, nil, false - } - data, ok := p.inflate(h.data, h.size) - if !ok { - return 0, nil, false - } - kind := h.kind - if kind == objOfsDelta || kind == objRefDelta { - base := h.base - if kind == objRefDelta { +// entry rebuilds the object at off against bases in the same pack, as Git's +// unpack_entry does for a pack stored on disk. It first follows the entries' +// headers alone down to a whole object, or to one already rebuilt: an entry +// reached twice, or a chain of more deltas than the depth limit, is refused +// before anything is inflated. It then inflates the base and applies the +// deltas from the innermost one up, inflating each delta only to apply it, so +// it holds the object being rebuilt, one delta and the result at a time. +func (o *objectStore) entry(p *pack, off int64) (int, []byte, bool) { + var ( + deltas []deltaStep + reached = map[int64]bool{} + kind int + data []byte + ) + at := off + for { + if cached, ok := o.cache[packedAt{p, at}]; ok { + kind, data = cached.kind, cached.data + break + } + if reached[at] { + return 0, nil, false + } + reached[at] = true + h, ok := p.header(at) + if !ok { + return 0, nil, false + } + if h.kind != objOfsDelta && h.kind != objRefDelta { + if data, ok = o.inflate(p, h.data, h.size); !ok { + return 0, nil, false + } + kind = h.kind + o.remember(p, at, kind, data) + break + } + if len(deltas) == o.limits.deltaDepth { + return 0, nil, false + } + deltas = append(deltas, deltaStep{at, h}) + if at = h.base; h.kind == objRefDelta { i, in := p.find(h.baseID) if !in { return 0, nil, false } - if base, ok = p.offset(i); !ok { + if at, ok = p.offset(i); !ok { return 0, nil, false } } - baseKind, baseData, ok := o.entry(p, base, depth+1) + } + for i := len(deltas) - 1; i >= 0; i-- { + step := deltas[i] + delta, ok := o.inflate(p, step.h.data, step.h.size) if !ok { return 0, nil, false } - if data, ok = applyDelta(baseData, data); !ok { + size, ok := deltaResult(delta) + if !ok || !o.charge(size) { return 0, nil, false } - kind = baseKind + if data, ok = applyDelta(data, delta); !ok { + return 0, nil, false + } + o.remember(p, step.at, kind, data) } + return kind, data, true +} + +// remember keeps a rebuilt object for later reads while the cache has room. +func (o *objectStore) remember(p *pack, at int64, kind int, data []byte) { if o.cached+len(data) <= maxCachedBytes { - o.cache[packedAt{p, off}] = packedObject{kind, data} + o.cache[packedAt{p, at}] = packedObject{kind, data} o.cached += len(data) } - return kind, data, true +} + +// deltaResult reads the size of the object that a delta rebuilds: the second +// of the delta's two leading sizes. +func deltaResult(delta []byte) (int64, bool) { + _, rest, ok := deltaSize(delta) + if !ok { + return 0, false + } + size, _, ok := deltaSize(rest) + return int64(size), ok && size <= maxObjectBytes } // applyDelta rebuilds an object from its base and a delta, as Git's @@ -554,12 +645,13 @@ func (o *objectStore) packed(p *pack, i int) (string, []byte, bool) { if !ok || !o.openPack(p) { return "", nil, false } - kind, data, ok := o.entry(p, off, 0) + kind, data, ok := o.entry(p, off) return objectKinds[kind], data, ok } // loose reads the loose file of id. A missing file means that the store does -// not hold the object. +// not hold the object. The file's size, then the size that it inflates to, +// count against the read limit before they are read. func (o *objectStore) loose(id string) (string, []byte, bool) { path := filepath.Join(o.dir, id[:2], id[2:]) if _, err := os.Lstat(path); err != nil { @@ -573,17 +665,22 @@ func (o *objectStore) loose(id string) (string, []byte, bool) { if !ok { return "", nil, false } + if !o.charge(f.size) { + f.file.Close() + return "", nil, false + } data, _, ok := o.s.read(f) f.file.Close() if !ok { return "", nil, false } - return inflateLoose(data) + return inflateLoose(data, o.charge) } // inflateLoose reads a loose object file: one zlib stream of -// " \x00" and exactly size bytes. -func inflateLoose(data []byte) (string, []byte, bool) { +// " \x00" and exactly size bytes. The size is charged before the +// bytes are inflated. +func inflateLoose(data []byte, charge func(int64) bool) (string, []byte, bool) { z, err := zlib.NewReader(bytes.NewReader(data)) if err != nil { return "", nil, false @@ -602,7 +699,7 @@ func inflateLoose(data []byte) (string, []byte, bool) { } kind, text, _ := strings.Cut(string(header), " ") size, err := strconv.ParseInt(text, 10, 64) - if err != nil || size < 0 || size > maxObjectBytes || strconv.FormatInt(size, 10) != text || !knownKind(kind) { + if err != nil || size < 0 || size > maxObjectBytes || strconv.FormatInt(size, 10) != text || !knownKind(kind) || !charge(size) { return "", nil, false } content, ok := inflateExact(z, size) @@ -636,25 +733,29 @@ func (o *objectStore) commit(id string) (string, []string, int64, bool) { return tree, parents, when, true } -// parseCommit reads the header Git requires of a commit: the tree line first, -// then the parent lines. The committer line's time only orders the walk. +// parseCommit reads a commit's tree and parents as Git's parse_commit_buffer +// does: a "tree \n" line that more bytes follow, then, while a line starts +// with "parent " and more than such a line's length follows, "parent \n". +// Git reads a shorter "parent " line as no parent; the reader refuses that +// commit, so every commit that it reads has the parents that Git reads. The +// committer line's time only orders the walk. func parseCommit(data []byte) (string, []string, int64, bool) { - header, _, _ := strings.Cut(string(data), "\n\n") - lines := strings.Split(header, "\n") - tree, ok := strings.CutPrefix(lines[0], "tree ") - if !ok || !hexDigest(tree, 40) { + const treeLine, parentLine = len("tree \n") + 40, len("parent \n") + 40 + text := string(data) + if len(text) <= treeLine || !strings.HasPrefix(text, "tree ") || text[treeLine-1] != '\n' || !hexDigest(text[5:treeLine-1], 40) { return "", nil, 0, false } + tree := text[5 : treeLine-1] var parents []string - for lines = lines[1:]; len(lines) > 0 && strings.HasPrefix(lines[0], "parent "); lines = lines[1:] { - id := strings.TrimPrefix(lines[0], "parent ") - if !hexDigest(id, 40) { + for text = text[treeLine:]; strings.HasPrefix(text, "parent "); text = text[parentLine:] { + if len(text) <= parentLine || text[parentLine-1] != '\n' || !hexDigest(text[7:parentLine-1], 40) { return "", nil, 0, false } - parents = append(parents, id) + parents = append(parents, text[7:parentLine-1]) } + header, _, _ := strings.Cut(text, "\n\n") var when int64 - for _, line := range lines { + for _, line := range strings.Split(header, "\n") { if who, ok := strings.CutPrefix(line, "committer "); ok { if i := strings.LastIndexByte(who, '>'); i >= 0 { if fields := strings.Fields(who[i+1:]); len(fields) > 0 { @@ -669,12 +770,15 @@ func parseCommit(data []byte) (string, []string, int64, bool) { // reaches reports whether target is tip or one of its ancestors. Like Git's // merge-base walk it reads each commit when it is queued and takes the newest -// first; a shallow boundary has no parents. A commit it cannot read refuses -// the source. +// first; a shallow boundary has no parents. A commit it cannot read, or more +// commits than the limit, refuse the source. func (o *objectStore) reaches(tip, target string) bool { q := &commitQueue{} seen := map[string]bool{} queue := func(id string) bool { + if len(seen) == o.limits.commits { + return false + } _, parents, when, ok := o.commit(id) if ok { seen[id] = true @@ -685,7 +789,7 @@ func (o *objectStore) reaches(tip, target string) bool { if !queue(tip) { return false } - for q.Len() > 0 && len(seen) <= maxCommits { + for q.Len() > 0 { c := heap.Pop(q).(queuedCommit) if c.id == target { return true @@ -701,9 +805,11 @@ func (o *objectStore) reaches(tip, target string) bool { // listTree lists every entry below the tree id as `git ls-tree -r --full-tree` // does: subtrees are entered and not listed; blobs, links and submodule -// commits are listed under their path. +// commits are listed under their path. The entries read, subtrees included, +// and the listed paths' total length are bounded by the limits. func (o *objectStore) listTree(id string) (map[string]treeEntry, bool) { out := map[string]treeEntry{} + entries, pathBytes := 0, int64(0) var walk func(id, prefix string, depth int) bool walk = func(id, prefix string, depth int) bool { kind, data, ok := o.object(id) @@ -714,12 +820,13 @@ func (o *objectStore) listTree(id string) (map[string]treeEntry, bool) { o.refuse("source_git") return false } - entries, ok := parseTreeObject(data) - if !ok { + read, ok := parseTreeObject(data) + if !ok || len(read) > o.limits.treeEntries-entries { o.refuse("source_git") return false } - for _, e := range entries { + entries += len(read) + for _, e := range read { path := prefix + e.name if e.entry.kind == "tree" { if !walk(e.entry.object, path+"/", depth+1) { @@ -727,10 +834,11 @@ func (o *objectStore) listTree(id string) (map[string]treeEntry, bool) { } continue } - if _, dup := out[path]; dup || len(out) == maxTreeEntries { + if _, dup := out[path]; dup || int64(len(path)) > o.limits.pathBytes-pathBytes { o.refuse("source_git") return false } + pathBytes += int64(len(path)) out[path] = e.entry } return true diff --git a/tools/todo-update-reader-3190/pin_windows.go b/tools/todo-update-reader-3190/pin_windows.go index 2995e776..ee221d01 100644 --- a/tools/todo-update-reader-3190/pin_windows.go +++ b/tools/todo-update-reader-3190/pin_windows.go @@ -2,12 +2,14 @@ package main import ( "crypto/sha256" + "encoding/binary" "encoding/hex" "io" "os" "path/filepath" "strings" "syscall" + "unicode/utf16" "unsafe" ) @@ -19,7 +21,20 @@ const ( fileReadAttributes = 0x00000080 ) -var procGetFinalPathNameByHandleW = syscall.NewLazyDLL("kernel32.dll").NewProc("GetFinalPathNameByHandleW") +// GetFileInformationByHandleEx returns FILE_FULL_DIR_INFO records: the offset +// of the next record, the name's length in bytes, then the name. +const ( + fileFullDirectoryInfo = 14 + fileFullDirectoryRestartInfo = 15 + dirInfoNameLength = 60 + dirInfoName = 68 +) + +var ( + kernel32 = syscall.NewLazyDLL("kernel32.dll") + procGetFinalPathNameByHandleW = kernel32.NewProc("GetFinalPathNameByHandleW") + procGetFileInformationByHandleEx = kernel32.NewProc("GetFileInformationByHandleEx") +) // pinnedFile is one evidence file held open for reading with writes, deletes // and renames denied to everyone else until the session closes. @@ -30,8 +45,8 @@ type pinnedFile struct { size int64 } -// pinnedDir keeps the identity of the source root. Others may still create -// files in it, so only its identity and physical path are rechecked. +// pinnedDir keeps the identity of a directory. Others may still create files +// in it, so only its identity and physical path are rechecked. type pinnedDir struct { handle syscall.Handle path string @@ -229,27 +244,71 @@ func (s *session) unchanged(f *pinnedFile) bool { return ok && strings.EqualFold(actual, f.path) && plain && info.Mode().IsRegular() } -func (s *session) pinDir(path string) bool { +// pinDir opens a directory without following a reparse point and keeps it open +// until the session closes. Its entries are listed through the returned handle. +func (s *session) pinDir(path string) (pinnedDir, bool) { if !s.plainAncestors(path) || !s.plainDir(path) { - return false + return pinnedDir{}, false } name, err := syscall.UTF16PtrFromString(path) if err != nil { - return false + return pinnedDir{}, false } h, err := syscall.CreateFile(name, fileListDirectory|fileReadAttributes, syscall.FILE_SHARE_READ|syscall.FILE_SHARE_WRITE|syscall.FILE_SHARE_DELETE, nil, syscall.OPEN_EXISTING, fileFlagBackupSemantics|fileFlagOpenReparsePoint, 0) if err != nil { - return false + return pinnedDir{}, false } d := pinnedDir{handle: h, path: path} actual, ok := finalPath(h) if syscall.GetFileInformationByHandle(h, &d.info) != nil || reparse(d.info.FileAttributes) || d.info.FileAttributes&syscall.FILE_ATTRIBUTE_DIRECTORY == 0 || !ok || !strings.EqualFold(actual, path) { syscall.CloseHandle(h) - return false + return pinnedDir{}, false } s.dirs = append(s.dirs, d) - return true + return d, true +} + +// list returns the names of the pinned directory's entries, read through its +// handle, so a directory that takes the path after the pin is never listed. +// A directory of more than limit entries is refused. +func (d pinnedDir) list(limit int) ([]string, bool) { + buf := make([]uint64, 8<<10) // the records start at 8-byte boundaries + raw := unsafe.Slice((*byte)(unsafe.Pointer(&buf[0])), 8*len(buf)) + var names []string + for class := uintptr(fileFullDirectoryRestartInfo); ; class = fileFullDirectoryInfo { + r, _, err := procGetFileInformationByHandleEx.Call(uintptr(d.handle), class, uintptr(unsafe.Pointer(&buf[0])), uintptr(len(raw))) + if r == 0 { + return names, err == syscall.ERROR_NO_MORE_FILES + } + for at := 0; ; { + if at > len(raw)-dirInfoName { + return nil, false + } + next := int(binary.LittleEndian.Uint32(raw[at:])) + size := int(binary.LittleEndian.Uint32(raw[at+dirInfoNameLength:])) + if size%2 != 0 || size > len(raw)-at-dirInfoName { + return nil, false + } + units := make([]uint16, size/2) + for i := range units { + units[i] = binary.LittleEndian.Uint16(raw[at+dirInfoName+2*i:]) + } + if name := string(utf16.Decode(units)); name != "." && name != ".." { + if len(names) == limit { + return nil, false + } + names = append(names, name) + } + if next == 0 { + break + } + if next < dirInfoName { + return nil, false + } + at += next + } + } } func dirUnchanged(d pinnedDir) bool { diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index 1d845af8..e7c5e3dc 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -81,7 +81,7 @@ func verifyUpdate(requested string, p profile, env environment) (qualification, return qualification{}, "baseline_mismatch" } - if !s.pinDir(p.root) || !guardClear(env, p.root) { + if _, ok := s.pinDir(p.root); !ok || !guardClear(env, p.root) { return qualification{}, "source_root" } tree, reason := s.checkSource(p) From b3c0b718840f68ec91844bfe457a00984d7645eb Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 07:34:34 +0900 Subject: [PATCH 15/24] docs: record the ToDo3513 review and the bounded object reads The specification describes the commit parsing that follows Git, the pack directory listed through its pinned handle, the delta chain walked by its headers first, and the object limits with what the real source needs. It records review ToDo3513, the Red and Green runs, the 24 mutants, the pre-merge reader run on the real files and the CI of the previous head. --- docs/todo-update-reader-3190.md | 104 ++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 6 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index b43f6828..e2342267 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -83,14 +83,24 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト Gitは実行しない。objectsもreader自身が読む。読むのは共有repoのobjectsディレクトリにあるloose objectと、 `objects/pack` の `pack-<40桁>.idx`（version 2）と対になる `.pack`（version 2か3）だけである。 alternates・commit-graph・multi-pack-index・replace refは読まない。guardが `objects` と `objects/pack` の - ディレクトリ、各index・pack・loose objectを1件ずつ clear と分類してから開く。indexは末尾のchecksumを確かめ、 - packは先頭の件数と末尾のchecksumがindexの記録と一致する場合だけ使う。packの各entryは、そのままの形か、 - offset delta・名前を指すdeltaとして同じpackの中から組み立てる。どのobjectも、型と中身から計算したSHA-1の + ディレクトリ、各index・pack・loose objectを1件ずつ clear と分類してから開く。`objects/pack` の項目は、 + 分類の後で固定したhandleから `GetFileInformationByHandleEx` で列挙し、パスから開き直さない。固定の後で + このディレクトリが改名され、同じ名前にjunctionが置かれても、列挙するのは固定したディレクトリの項目だけである。 + indexは末尾のchecksumを確かめ、packは先頭の件数と末尾のchecksumがindexの記録と一致する場合だけ使う。 + packの各entryは、そのままの形か、offset delta・名前を指すdeltaとして同じpackの中から組み立てる。 + 組み立てはGitの `unpack_entry` と同じ順で行う。まずentryの見出しだけを基底までたどり、同じentryに戻る + 循環と、深さの上限を超える連鎖は、何も展開せずに拒否する。次に基底を展開し、内側のdeltaから1つずつ展開して当てる。 + 同時に持つのは、組み立て中のobject、delta 1つ、その結果だけである。どのobjectも、型と中身から計算したSHA-1の 名前が求めた名前と一致する場合だけ使う。そのため、照合中に共有repoのobjectが増えたり、別のstoreを指す - ファイルが一時的に置かれたりしても、readerが読む中身は変わらない。objectsで確かめるのは次の3つ。 + ファイルが一時的に置かれたりしても、readerが読む中身は変わらない。読取の量には上限を置く（次の節）。 + objectsで確かめるのは次の3つ。 - 固定HEADのcommitのtreeが、固定treeと一致する。 - `refs/remotes/origin/main` のcommitから親をたどると、固定HEADに着く。`shallow` にあるcommitは、 Gitのshallow cloneと同じく親を持たないものとして扱う。読めない親があれば拒否する。 + commitはGitの `parse_commit_buffer` と同じに解析する。`tree <40桁>` と改行の後に1byte以上が要る。 + `parent ` で始まる行は、その後にこの行の長さ（48byte）を超えるbytesが続く場合だけ、 + `parent <40桁>` と改行の形の親として読む。Gitはそれより短い `parent ` の行を親として読まないので、 + readerはその形のcommitを拒否する。こうしてreaderが受け付けるcommitの親は、Gitが読む親と一致する。 - 固定treeを再帰的にたどった一覧。`git ls-tree -r --full-tree` と同じく、通常ファイル・link・submoduleの entryを並べ、directoryは並べない。 @@ -112,9 +122,33 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト この再確認が拒否できるのは、再確認の時点で残っている変更である。途中で作られて再確認の前に消えたファイルは 検出できない。そのためreaderは、alternates・grafts・worktreeごとの設定を読む処理を持たず、Gitも実行しない。 objectは名前で中身を確かめるので、再確認の対象にしない。 + `objects/pack` は固定したhandleから列挙したので、列挙した項目は再確認するディレクトリのものである。 + 固定の後でこのディレクトリが改名されれば、実パスが変わるので再確認で拒否する。 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。共有repoのconfigやrefは ほかのworktreeが書き換えることがあり、照合中に書き換わると `recheck` で拒否される。その場合は再実行する。 +## objectsの読取の上限 + +objectsを読む量にはprofileの `objectLimits` で上限を置き、超えれば `source_git` で拒否する。 +共有repoのobjectsが、メモリや読取の量を使い尽くす形に細工されていても、readerが使う量は上限の中に収まる。 +実物の必要量は、reader自身の数え方で、配布済みguardの分類を通して測った値である（計測用の一時的な版で1回）。 + +| 上限 | 値 | 実物の必要量 | +| --- | --- | --- | +| `objects/pack` の項目 | 4,096 | 20 | +| indexとpackの組 | 64 | 6 | +| indexの合計 | 256MiB | 529,584 bytes | +| 読取byte | 512MiB | 982,328 bytes | +| deltaの連鎖 | 4,095（Gitが書く最大） | 3 | +| 祖先をたどって読むcommit | 65,536 | 45 | +| 固定treeの下のentry（directoryを含む） | 65,536 | 1,657 | +| 一覧のパスの合計 | 16MiB | 53,698 bytes | + +読取byteには、packから読んだ圧縮bytes（読んだ時点で数える）、packのentryとdeltaで組み立てたobjectの大きさ、 +loose objectのファイルの大きさと展開後の大きさを数える。大きさは、そのbytesを確保する前に数える。 +中身の無いブロックで水増しした圧縮データも、読んだ分だけ数えるので上限で止まる。 +1つのobjectは16MiBまで、cacheに残す組み立て済みのobjectは合計64MiBまでで、これは以前と同じである。 + ## compiler closure の確認 dep-info はコンパイラが読んだローカルファイルの一覧である。次の条件をすべて満たす場合だけ、 @@ -139,7 +173,7 @@ receiptの126件の入力が、原dep-infoの116件と manifest/lock の10件に テストは `tools/todo-update-reader-3190` の `go test`。合成したGit repo（実物と同じく、別の場所のrepoに属する linked worktree）、ファイル、receiptと、guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 理由コードも照合し、別の理由による偶然の拒否を合格にしない。objectの読取は、Git自身の出力 -（`git cat-file --batch-all-objects`、`git ls-tree`、`git rev-list`）と比べる。 +（`git cat-file --batch-all-objects`、`git ls-tree`、`git rev-list`、`git log`、`git merge-base`、`git verify-pack`）と比べる。 | 条件 | テスト | | --- | --- | @@ -157,6 +191,9 @@ linked worktree）、ファイル、receiptと、guardのCLI応答を模した | Gitを実行せず、設定・別のstore・作業ファイルを読まない。読む全ファイルをguardが先に分類する。入力以外の保護された追跡ファイルは結果を左右しない | `TestNeverReadsWorkingFiles`、`TestReadsNoGitConfiguration`、`TestObjectsIgnoreAlternates`、`TestQualifiesFixedUpdate`、`TestRejectsProtectedResources` | | Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs`、`TestPackedRefsAsGitReadsThem` | | objectをGitと同じに読む（loose、pack、offset delta、名前を指すdelta、commit・tree・祖先） | `TestObjectStoreAgreesWithGit`、`TestQualifiesFromPacks` | +| commitの親をGitと同じに読む。Gitが親として読まない `parent ` の行と、Gitが拒否する形を拒否する | `TestParseCommitAgreesWithGit`、`TestRefusesParentsGitDoesNotRead` | +| `objects/pack` を固定したhandleから列挙し、差し替えたjunctionの先を列挙・分類・読取しない | `TestListsOnlyThePinnedPackDirectory` | +| deltaの循環を何も展開せずに拒否し、連鎖の深さ・読取byte・packの数・項目数・indexの合計・commit数・treeのentry数・パスの合計に上限を置く | `TestDeltaCyclesInflateNothing`、`TestDeltaDepthLimit`、`TestReadLimitCountsEveryByte`、`TestReadLimit`、`TestObjectLimits` | | 壊れたobject・別のobject・保護されたobjectのファイルを拒否する | `TestRejectsDamagedObjects`、`TestApplyDeltaRefusesMalformedDeltas` | | 入力のbytesが固定treeのblob（そのままか、LFをすべてCRLFにした形）である | `TestInputsMustBeTheCommittedBytes`、`TestCheckedOutForms` | | 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | @@ -283,6 +320,56 @@ multi-pack-index・replace refは読まないので、照合中に別のstoreを - 実物の共有repoのobjects（loose 558件、pack 6件とそのindex、multi-pack-index）は、すべてリンク数1だった。 固定HEADのcommitはpackの中にある。 +### 別モデルの再レビュー（ToDo3513） + +gpt-6.1-sol・推論max（job `20261024-061453-todo-3513`、7分2秒、終了コード0）が、base `8b4f34b9`・head `206181d4` の +変更12ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のMust 2件と文書の指摘は +解消と判定された。新たな指摘は Must 3件で、どれもコードとGitの一次資料からの判断であり、再現テストは伴わない。 +SuggestionとNitはない。 + +- Must: 改行で終わらない `parent ` の行を親として読んでいた。`tree <固定tree>` と `parent <固定HEAD>` だけで末尾に + 改行のないcommitを正しい名前で置き、`origin/main` にすると、readerは固定HEADを祖先と判定する。 + Git 2.54.0はその行を親として読まないので、判定が食い違う。 +- Must: `objects/pack` のhandleは改名を許し、列挙ではパスから開き直していた。固定の後で元のディレクトリを改名し、 + 同じ名前にjunctionを置くと、guardが分類していない場所の項目名を列挙できる。最後の再確認では、列挙したことを取り消せない。 +- Must: 展開したdeltaを持ったまま基底へ再帰していた。自分自身を基底にする16MiBのdeltaで、深さの上限までに + 約64GiBを確保し得る。64MiBのcacheの上限は、展開途中のデータを数えない。indexは全件を保持し、packの数と + indexの合計にも上限がない。 + +### 4回目のMustの修正 + +3件とも直した（検証の順序の5と10、objectsの読取の上限）。 + +- commitをGitの `parse_commit_buffer` と同じ長さと改行の判定で解析する。Gitが親として読まない短い `parent ` の行は、 + commitごと拒否する。Gitより受け付ける範囲を狭めるだけにした。 +- `objects/pack` を、固定したhandleから `GetFileInformationByHandleEx` で列挙する。パスから開き直さない。 +- deltaをGitの `unpack_entry` と同じ順で組み立てる。見出しだけで基底までたどり、循環と深さの超過は何も展開せずに + 拒否する。上限はprofileの `objectLimits` にまとめ、productionの値を固定値のテストで照合する。 + +結果は次のとおり。 + +- Red（commit `9fb6c2b5`）: 上限の構造体、列挙の直前に呼ぶ差し込み口、`openObjects` の引数だけを本体に足し、 + 動作は変えていない。`go test -race -count=1 ./...` は終了コード1、192秒。上位32件が合格、8件が失敗し、 + subtest込みで合格186、失敗23、skip 0、ビルド失敗0、data race 0。失敗した8件はすべて新しいテストで、意図した理由による。 + - `parseCommit` は、Gitが親として読まないか拒否する5つの形を受け付けた。tree行だけ、tree行の改行なし、 + 改行のないparent行、parent行で終わるcommit、2つ目のparent行の改行なしである。Git側の読み方は11の形すべてで想定と一致した。 + - 末尾に改行のない `parent <固定HEAD>` のcommitを `origin/main` にすると、Gitの `merge-base --is-ancestor` は + 終了コード1（祖先ではない）なのに、readerは資格を出した。 + - 列挙の直前に `objects/pack` をjunctionへ差し替えると、junctionの先にある `pack-ffff…` の名前がguardの分類まで届いた。 + - 自分自身を基底にするdeltaと、互いを基底にする2つのdelta（どちらも16KiB）で、拒否までに約251MBを確保した。 + - 深さ、読取byte、packの数、項目数、indexの合計、commit数、treeのentry数、パスの合計の上限は、どれも効いていなかった。 +- Green（commit `7a008ac9`）: `go test -race -count=1 ./...` は終了コード0、195秒。上位40件、subtest込み214件が合格し、 + 失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。Greenではテストを3点強めた。循環の拒否で確保してよい量を + 1MiBから256KiBにした（見出しを深さの上限までたどる版を検出するため）。読取byteの計上を5つ（packの圧縮bytes、packの + 展開後の大きさ、deltaの結果、looseのファイル、looseの展開後の大きさ）に分け、どれか1つを外せば失敗するようにした。 + indexの合計は2組のpackで確かめ、1ファイルごとの上限と区別した。 +- 照合の要所を壊した24の版を `go test -overlay` で当て、すべて対応するテストが失敗した。新しい照合では、循環の検出、 + 深さの上限、読取byteの5つの計上、handleでの列挙、項目数、packの数、indexの合計、commit数、treeのentry数、 + directoryの計上、パスの合計、parent行の長さ、tree行の長さ、parent行の改行を壊した。以前の照合では、objectの名前、 + indexとpackのguard分類、packのchecksum、treeの再帰、祖先の判定、deltaの複写範囲を壊した。 +- 途中、Greenの全体の実行が487秒かかった。同じ時間帯に同じテストをRedの本体とGreenで交互に2回ずつ動かすと、 + Greenは37〜41秒、Redは42〜47秒で、遅れはPCの混雑によるものだった。最終の実行は195秒である。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -304,6 +391,10 @@ source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力12 `data_access_granted:false`）を返した。Gitは実行せず、reader自身が実物のpackから固定HEADのcommitを読み、 `origin/main` からの祖先とtreeの一覧を求めた。 +4回目の修正のcommit `7a008ac9` から `go build -trimpath` で生成したreader（SHA256 `efde5add…21a9`、vcs.modified=false） +でも実物を1回照合した。終了コード0、15.0秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。`objects/pack` は固定したhandleから列挙し、読取の量は上限の中に収まった。 + いずれもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 対象の置換、guardの変更、ToDoの状態変更はしていない。 @@ -316,5 +407,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d 修正前のhead `d9edfd4d` では、専用workflow（windows、run 39994455364）とCodeQL（run 39994455228）が成功した。 1回目の修正のhead `308cb24d` でも、専用workflow（windows、run 40002129622）とCodeQL（run 40002129620）が成功した。 2回目の修正のhead `8b4f34b9` でも、専用workflow（windows、run 40015024286）とCodeQL（run 40015024382）が成功した。 -3回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +3回目の修正のhead `206181d4` でも、専用workflow（windows、run 40025021344）とCodeQL（run 40025021288）が成功した。 +4回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる 実物の照合、親3190への結果の返却が残る。 From edbf9629216b047046b07b11ac148b43c74473eb Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:14:36 +0900 Subject: [PATCH 16/24] test: require the ToDo3190 reader to read every commit Git reads and keep one type per name Review 3531 found that the reader accepts a main commit whose tree line names the fixed commit. Git refuses it, because it cannot use one object as both a tree and a commit (object.c, object_as_type). Git's merge-base also reads every commit that main reaches and the fixed commit does not, with their parents, before it answers (commit-reach.c, paint_down_to_common); the reader stops at the fixed commit. TestRefusesNamesOfTwoTypes and TestReadsEveryCommitGitReads run git merge-base --is-ancestor on each history and require the reader to refuse the source where Git exits 128 and to qualify it where Git exits 0. The merges name each side branch before and after the fixed commit, so a walk that stops at the fixed commit misses it in whatever order it reads the parents. TestDeltaDepthCountsKeptBases requires the delta depth limit to count a chain that reaches a base kept by an earlier read. Red (go test -race, these three tests): six histories fail as intended, the reader qualifying where Git exits 128, and a chain of two deltas is read under a limit of one. A parent line that names the fixed tree is already refused, and a side branch of whole commits is qualified in both orders where Git exits 0. No data race, panic or build failure. --- .../commit_walk_windows_test.go | 115 ++++++++++++++++++ .../object_bounds_windows_test.go | 32 +++++ 2 files changed, 147 insertions(+) create mode 100644 tools/todo-update-reader-3190/commit_walk_windows_test.go diff --git a/tools/todo-update-reader-3190/commit_walk_windows_test.go b/tools/todo-update-reader-3190/commit_walk_windows_test.go new file mode 100644 index 00000000..7cedd65a --- /dev/null +++ b/tools/todo-update-reader-3190/commit_walk_windows_test.go @@ -0,0 +1,115 @@ +package main + +import ( + "errors" + "os/exec" + "path/filepath" + "testing" +) + +// Commit times before and after the fixture's commit, which Git makes at the +// current time. +const ( + earlierDates = "author A 1700000000 +0000\ncommitter C 1700000000 +0000\n" + laterDates = "author A 4000000000 +0000\ncommitter C 4000000000 +0000\n" +) + +// historyCase stores commits with put and returns the one that origin/main +// is to name, and Git's exit status for git merge-base --is-ancestor from the +// fixed commit to it: 0 if Git finds the fixed commit, 128 if it refuses the +// history. +type historyCase struct { + status int + main func(f *fixture, put func(body string) string) string +} + +// checkHistories points origin/main at each case's commit and checks that +// Git exits with the case's status there, and that the reader qualifies the +// fixed update where Git finds the fixed commit and refuses the source where +// Git refuses the history. +func checkHistories(t *testing.T, cases map[string]historyCase) { + t.Helper() + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + f := newFixture(t, nil) + objects := filepath.Join(f.common, "objects") + main := tc.main(f, func(body string) string { return writeLooseObject(t, objects, "commit", body) }) + writeFile(t, filepath.Join(f.common, "refs", "remotes", "origin", "main"), main+"\n") + out, err := testGit(t, f.root, "merge-base", "--is-ancestor", f.p.head, main) + status := 0 + var exit *exec.ExitError + if errors.As(err, &exit) { + status = exit.ExitCode() + } else if err != nil { + t.Fatalf("git merge-base: %v\n%s", err, out) + } + if status != tc.status { + t.Fatalf("git merge-base --is-ancestor %s %s exited %d, want %d\n%s", f.p.head, main, status, tc.status, out) + } + if status != 0 { + expectReason(t, f, "source_git") + return + } + if got, reason := f.verify(); !got.Qualified || reason != "" { + t.Fatalf("verify = qualified %v reason %q where Git finds the fixed commit", got.Qualified, reason) + } + }) + } +} + +// Git keeps one type for each object name that it looks up or reads, and +// refuses to use that object as another type (object.c, object_as_type). A +// commit's tree line looks its name up as a tree and each parent line looks +// its name up as a commit, so merge-base refuses a history in which one name +// is both. The reader refuses the source there too. +func TestRefusesNamesOfTwoTypes(t *testing.T) { + checkHistories(t, map[string]historyCase{ + "tree line names the fixed commit, also the parent": {128, func(f *fixture, put func(string) string) string { + return put("tree " + f.p.head + "\nparent " + f.p.head + "\n" + laterDates + "\nmain\n") + }}, + "tree line names the commit of the parent line": {128, func(f *fixture, put func(string) string) string { + side := put("tree " + f.p.tree + "\nparent " + f.p.head + "\n" + earlierDates + "\nside\n") + return put("tree " + side + "\nparent " + side + "\n" + laterDates + "\nmain\n") + }}, + "parent line names the fixed tree": {128, func(f *fixture, put func(string) string) string { + return put("tree " + f.p.tree + "\nparent " + f.p.head + "\nparent " + f.p.tree + "\n" + laterDates + "\nmain\n") + }}, + }) +} + +// Git's merge-base walk does not stop at the fixed commit. Before it answers, +// it reads every commit that main reaches and the fixed commit does not, and +// the parents of each (commit-reach.c, paint_down_to_common), so a commit +// there that Git cannot read, or that gives a name a second type, refuses the +// history. The reader refuses the source there too. Each side branch below is +// older than the fixed commit, and the merge names it before and after the +// fixed commit, so a walk that stopped at the fixed commit would not read it +// whether it took the newest commit or either parent first. +func TestReadsEveryCommitGitReads(t *testing.T) { + sides := map[string]historyCase{ + "side branch of whole commits": {0, func(f *fixture, put func(string) string) string { + root := put("tree " + f.p.tree + "\n" + earlierDates + "\nroot\n") + return put("tree " + f.p.tree + "\nparent " + root + "\n" + earlierDates + "\nside\n") + }}, + "side branch reaches a missing commit": {128, func(f *fixture, put func(string) string) string { + return put("tree " + f.p.tree + "\nparent 1111111111111111111111111111111111111111\n" + earlierDates + "\nside\n") + }}, + "side branch reaches a tree line that names the fixed commit": {128, func(f *fixture, put func(string) string) string { + root := put("tree " + f.p.head + "\n" + earlierDates + "\nroot\n") + return put("tree " + f.p.tree + "\nparent " + root + "\n" + earlierDates + "\nside\n") + }}, + } + orders := map[string]func(head, side string) string{ + "fixed commit first": func(head, side string) string { return "parent " + head + "\nparent " + side + "\n" }, + "side branch first": func(head, side string) string { return "parent " + side + "\nparent " + head + "\n" }, + } + cases := map[string]historyCase{} + for name, side := range sides { + for order, parents := range orders { + cases[name+", "+order] = historyCase{side.status, func(f *fixture, put func(string) string) string { + return put("tree " + f.p.tree + "\n" + parents(f.p.head, side.main(f, put)) + laterDates + "\nmerge\n") + }} + } + } + checkHistories(t, cases) +} diff --git a/tools/todo-update-reader-3190/object_bounds_windows_test.go b/tools/todo-update-reader-3190/object_bounds_windows_test.go index 064bf56e..99371660 100644 --- a/tools/todo-update-reader-3190/object_bounds_windows_test.go +++ b/tools/todo-update-reader-3190/object_bounds_windows_test.go @@ -214,6 +214,38 @@ func TestDeltaDepthLimit(t *testing.T) { } } +// The depth limit bounds a delta's whole chain, also where the chain reaches +// an object that an earlier read rebuilt and kept: a delta based on a kept +// delta is one deeper than that delta. +func TestDeltaDepthCountsKeptBases(t *testing.T) { + base := []byte("0123456789abcdef") + first := append(slices.Clone(base), '1') + second := append(slices.Clone(first), '2') + // grow is a delta that copies all of from and adds one byte. + grow := func(from []byte, add byte) []byte { + return []byte{byte(len(from)), byte(len(from) + 1), 0x90, byte(len(from)), 1, add} + } + whole := slices.Concat(entryHead(objBlob, len(base)), deflated(t, base)) + onBase := slices.Concat(entryHead(objOfsDelta, 6), ofsBase(len(whole)), deflated(t, grow(base, '1'))) + onFirst := slices.Concat(entryHead(objOfsDelta, 6), ofsBase(len(onBase)), deflated(t, grow(first, '2'))) + objects := filepath.Join(testLongPath(t, t.TempDir()), "objects") + writeRawPack(t, filepath.Join(objects, "pack"), []rawEntry{ + {rawName(t, "blob", base), whole}, {rawName(t, "blob", first), onBase}, {rawName(t, "blob", second), onFirst}, + }) + for _, limit := range []int{2, 1} { + limits := productionProfile.limits + limits.deltaDepth = limit + store := openRawStore(t, objects, limits) + if _, got, ok := store.object(objectName("blob", first)); !ok || !bytes.Equal(got, first) { + t.Fatalf("with %d deltas allowed, reading a chain of 1 delta = %q, %v", limit, got, ok) + } + _, got, ok := store.object(objectName("blob", second)) + if ok != (limit == 2) || ok && !bytes.Equal(got, second) { + t.Fatalf("with %d deltas allowed and the first delta kept, reading a chain of 2 deltas = %q, %v", limit, got, ok) + } + } +} + // Every byte that the store reads or produces counts against the read limit: // the compressed bytes as they are read, each pack entry's inflated size and // each delta's result, and each loose file's size and inflated size. Each From c2ff858448ed0fd9eeba0b855089d1786394cd63 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:26:46 +0900 Subject: [PATCH 17/24] fix: read every commit Git reads and keep one type per name in the ToDo3190 reader The reader now gives each object name one type, as Git's object table does: the type of each object that it reads, and the tree type for each commit's tree line. A name given a second type refuses the source, as Git refuses to use one object as two types (object.c, object_as_type). The walk from main reads every commit that main reaches, down to the shallow boundary, and no longer stops at the fixed commit. Before Git's merge-base answers, it reads every commit that main reaches and the fixed commit does not, with their parents (commit-reach.c, paint_down_to_common), so a commit there that Git cannot read, or that types a name twice, refuses the source here too. The walk no longer needs commit times, and parseCommit no longer reads the committer line. The delta depth limit now counts a chain that reaches a base kept by an earlier read: each kept object records the deltas of its own chain. go test -race -count=1 ./...: exit 0, 268 s beside the mutation runs; 43 top-level tests and 226 with subtests pass, with no failure, skip or data race. go vet and gofmt are clean. All 34 mutants are killed. The 10 new ones stop the walk at the fixed commit, follow first parents only, leave tree lines or reads untyped, never let types conflict, type a tree line as a commit, ignore the kept depth, keep it as zero, compare it one off, or walk past shallow boundaries. Measured once on the real source, with a measuring overlay through the installed guard's classification: 1,291 commits walked, 6,755,668 bytes read, delta chains of 3 at most and 2,663 typed names, all under the production limits. --- .../commit_parse_windows_test.go | 2 +- .../objects_windows.go | 179 ++++++++---------- .../objects_windows_test.go | 2 +- .../todo-update-reader-3190/verify_windows.go | 2 +- 4 files changed, 86 insertions(+), 99 deletions(-) diff --git a/tools/todo-update-reader-3190/commit_parse_windows_test.go b/tools/todo-update-reader-3190/commit_parse_windows_test.go index 092f76df..810e4cc2 100644 --- a/tools/todo-update-reader-3190/commit_parse_windows_test.go +++ b/tools/todo-update-reader-3190/commit_parse_windows_test.go @@ -90,7 +90,7 @@ func TestParseCommitAgreesWithGit(t *testing.T) { if gitRead != (tc.git != nil) || gitRead && (gitTree != tree || !slices.Equal(gitParents, tc.git)) { t.Fatalf("Git read tree %q parents %q (%v), want parents %q", gitTree, gitParents, gitRead, tc.git) } - gotTree, gotParents, _, ok := parseCommit([]byte(tc.body)) + gotTree, gotParents, ok := parseCommit([]byte(tc.body)) if ok != (tc.reader != nil) || ok && (gotTree != gitTree || !slices.Equal(gotParents, tc.reader)) { t.Fatalf("parseCommit read tree %q parents %q (%v), want parents %q", gotTree, gotParents, ok, tc.reader) } diff --git a/tools/todo-update-reader-3190/objects_windows.go b/tools/todo-update-reader-3190/objects_windows.go index 50b7961a..6512a903 100644 --- a/tools/todo-update-reader-3190/objects_windows.go +++ b/tools/todo-update-reader-3190/objects_windows.go @@ -3,7 +3,6 @@ package main import ( "bytes" "compress/zlib" - "container/heap" "crypto/sha1" "encoding/binary" "encoding/hex" @@ -74,6 +73,7 @@ type objectStore struct { limits objectLimits packs []*pack shallow map[string]bool + kinds map[string]string // the one type of each object name (typed) reason string cache map[packedAt]packedObject cached int @@ -86,8 +86,9 @@ type packedAt struct { } type packedObject struct { - kind int - data []byte + kind int + data []byte + depth int // the deltas of the entry's chain: none for a whole object } // pack is one pack file and its version 2 index. @@ -125,43 +126,13 @@ type namedEntry struct { entry treeEntry } -type queuedCommit struct { - id string - parents []string - when int64 - order int -} - -// commitQueue holds queued commits newest first, then in the order queued. -type commitQueue []queuedCommit - -func (q commitQueue) Len() int { return len(q) } - -func (q commitQueue) Less(i, j int) bool { - if q[i].when != q[j].when { - return q[i].when > q[j].when - } - return q[i].order < q[j].order -} - -func (q commitQueue) Swap(i, j int) { q[i], q[j] = q[j], q[i] } - -func (q *commitQueue) Push(x any) { *q = append(*q, x.(queuedCommit)) } - -func (q *commitQueue) Pop() any { - old := *q - last := old[len(old)-1] - *q = old[:len(old)-1] - return last -} - // openObjects lists the store's packs and reads their indexes. The guard // classifies the pack directory, which is then listed through the handle that // pins it, and every index and pack before any is opened. An index without its // pack is skipped, as Git skips it, and no other file of the directory is // opened. func (s *session) openObjects(src sourceObjects, limits objectLimits) (*objectStore, string) { - o := &objectStore{s: s, dir: src.dir, limits: limits, shallow: map[string]bool{}, cache: map[packedAt]packedObject{}} + o := &objectStore{s: s, dir: src.dir, limits: limits, shallow: map[string]bool{}, kinds: map[string]string{}, cache: map[packedAt]packedObject{}} if text, ok := strings.CutSuffix(string(src.shallow), "\n"); ok { for _, id := range strings.Split(text, "\n") { o.shallow[id] = true @@ -455,20 +426,25 @@ func inflateExact(z io.Reader, size int64) ([]byte, bool) { // unpack_entry does for a pack stored on disk. It first follows the entries' // headers alone down to a whole object, or to one already rebuilt: an entry // reached twice, or a chain of more deltas than the depth limit, is refused -// before anything is inflated. It then inflates the base and applies the -// deltas from the innermost one up, inflating each delta only to apply it, so -// it holds the object being rebuilt, one delta and the result at a time. +// before anything is inflated. A chain counts the deltas below an object +// already rebuilt too. It then inflates the base and applies the deltas from +// the innermost one up, inflating each delta only to apply it, so it holds +// the object being rebuilt, one delta and the result at a time. func (o *objectStore) entry(p *pack, off int64) (int, []byte, bool) { var ( deltas []deltaStep reached = map[int64]bool{} kind int data []byte + below int // the deltas of the base's own chain ) at := off for { if cached, ok := o.cache[packedAt{p, at}]; ok { - kind, data = cached.kind, cached.data + kind, data, below = cached.kind, cached.data, cached.depth + if len(deltas)+below > o.limits.deltaDepth { + return 0, nil, false + } break } if reached[at] { @@ -484,7 +460,7 @@ func (o *objectStore) entry(p *pack, off int64) (int, []byte, bool) { return 0, nil, false } kind = h.kind - o.remember(p, at, kind, data) + o.remember(p, at, kind, data, 0) break } if len(deltas) == o.limits.deltaDepth { @@ -514,15 +490,16 @@ func (o *objectStore) entry(p *pack, off int64) (int, []byte, bool) { if data, ok = applyDelta(data, delta); !ok { return 0, nil, false } - o.remember(p, step.at, kind, data) + o.remember(p, step.at, kind, data, below+len(deltas)-i) } return kind, data, true } -// remember keeps a rebuilt object for later reads while the cache has room. -func (o *objectStore) remember(p *pack, at int64, kind int, data []byte) { +// remember keeps a rebuilt object, and the number of deltas of its chain, for +// later reads while the cache has room. +func (o *objectStore) remember(p *pack, at int64, kind int, data []byte, depth int) { if o.cached+len(data) <= maxCachedBytes { - o.cache[packedAt{p, at}] = packedObject{kind, data} + o.cache[packedAt{p, at}] = packedObject{kind, data, depth} o.cached += len(data) } } @@ -613,7 +590,8 @@ func deltaSize(d []byte) (uint64, []byte, bool) { } // object returns the type and bytes of the object named id, taken from a pack -// or a loose file of the store and checked against id. +// or a loose file of the store and checked against id. The name takes the +// object's type. func (o *objectStore) object(id string) (string, []byte, bool) { raw, err := hex.DecodeString(id) if err != nil || !hexDigest(id, 40) { @@ -630,15 +608,32 @@ func (o *objectStore) object(id string) (string, []byte, bool) { return o.checked(id, kind, data, ok) } -// checked accepts an object only if its name is id. +// checked accepts an object only if its name is id and the name has no other +// type. func (o *objectStore) checked(id, kind string, data []byte, ok bool) (string, []byte, bool) { if !ok || kind == "" || objectName(kind, data) != id { o.refuse("source_git") return "", nil, false } + if !o.typed(id, kind) { + return "", nil, false + } return kind, data, true } +// typed gives the object name id the type kind, as Git's object table does +// when it reads the object or looks the name up: a name keeps its first type, +// and Git refuses to use its object as another (object.c, object_as_type). A +// name given a second type refuses the source. +func (o *objectStore) typed(id, kind string) bool { + if had, ok := o.kinds[id]; ok && had != kind { + o.refuse("source_git") + return false + } + o.kinds[id] = kind + return true +} + // packed reads the object at index position i of p. func (o *objectStore) packed(p *pack, i int) (string, []byte, bool) { off, ok := p.offset(i) @@ -715,92 +710,84 @@ func knownKind(kind string) bool { return false } -// commit reads a commit's tree, its parents (none for a shallow boundary, as -// Git's shallow file defines it) and its committer time. -func (o *objectStore) commit(id string) (string, []string, int64, bool) { +// commit reads a commit's tree and its parents: none for a shallow boundary, +// as Git's shallow file defines it. The tree line gives the tree's name the +// tree type, as Git's parse_commit_buffer looks it up as a tree. The parents +// take the commit type when the walk reads them; Git does not look up a +// shallow boundary's parents, and the walk does not read them. +func (o *objectStore) commit(id string) (string, []string, bool) { kind, data, ok := o.object(id) if !ok { - return "", nil, 0, false + return "", nil, false } - tree, parents, when, parsed := parseCommit(data) + tree, parents, parsed := parseCommit(data) if kind != "commit" || !parsed { o.refuse("source_git") - return "", nil, 0, false + return "", nil, false + } + if !o.typed(tree, "tree") { + return "", nil, false } if o.shallow[id] { parents = nil } - return tree, parents, when, true + return tree, parents, true } // parseCommit reads a commit's tree and parents as Git's parse_commit_buffer // does: a "tree \n" line that more bytes follow, then, while a line starts // with "parent " and more than such a line's length follows, "parent \n". // Git reads a shorter "parent " line as no parent; the reader refuses that -// commit, so every commit that it reads has the parents that Git reads. The -// committer line's time only orders the walk. -func parseCommit(data []byte) (string, []string, int64, bool) { +// commit, so every commit that it reads has the parents that Git reads. +func parseCommit(data []byte) (string, []string, bool) { const treeLine, parentLine = len("tree \n") + 40, len("parent \n") + 40 text := string(data) if len(text) <= treeLine || !strings.HasPrefix(text, "tree ") || text[treeLine-1] != '\n' || !hexDigest(text[5:treeLine-1], 40) { - return "", nil, 0, false + return "", nil, false } tree := text[5 : treeLine-1] var parents []string for text = text[treeLine:]; strings.HasPrefix(text, "parent "); text = text[parentLine:] { if len(text) <= parentLine || text[parentLine-1] != '\n' || !hexDigest(text[7:parentLine-1], 40) { - return "", nil, 0, false + return "", nil, false } parents = append(parents, text[7:parentLine-1]) } - header, _, _ := strings.Cut(text, "\n\n") - var when int64 - for _, line := range strings.Split(header, "\n") { - if who, ok := strings.CutPrefix(line, "committer "); ok { - if i := strings.LastIndexByte(who, '>'); i >= 0 { - if fields := strings.Fields(who[i+1:]); len(fields) > 0 { - when, _ = strconv.ParseInt(fields[0], 10, 64) - } - } - break - } - } - return tree, parents, when, true + return tree, parents, true } -// reaches reports whether target is tip or one of its ancestors. Like Git's -// merge-base walk it reads each commit when it is queued and takes the newest -// first; a shallow boundary has no parents. A commit it cannot read, or more -// commits than the limit, refuse the source. +// reaches reports whether target is tip or one of its ancestors. It reads +// every commit that tip reaches, down to the shallow boundary, whose commits +// have no parents, and does not stop at target: before Git's merge-base +// answers, it reads every commit that tip reaches and target does not, with +// the parents of each (commit-reach.c, paint_down_to_common). A commit there +// that Git cannot read, or that gives a name a second type, makes Git refuse +// the history; the walk reads it too, wherever it lies, and refuses the +// source. A commit that the walk cannot read, or more commits than the limit, +// refuse the source. func (o *objectStore) reaches(tip, target string) bool { - q := &commitQueue{} - seen := map[string]bool{} - queue := func(id string) bool { - if len(seen) == o.limits.commits { + seen := map[string]bool{tip: true} + todo := []string{tip} + found := false + for read := 0; len(todo) > 0; read++ { + if read == o.limits.commits { return false } - _, parents, when, ok := o.commit(id) - if ok { - seen[id] = true - heap.Push(q, queuedCommit{id: id, parents: parents, when: when, order: len(seen)}) - } - return ok - } - if !queue(tip) { - return false - } - for q.Len() > 0 { - c := heap.Pop(q).(queuedCommit) - if c.id == target { - return true + id := todo[len(todo)-1] + todo = todo[:len(todo)-1] + _, parents, ok := o.commit(id) + if !ok { + return false } - for _, parent := range c.parents { - if !seen[parent] && !queue(parent) { - return false + found = found || id == target + for _, parent := range parents { + if !seen[parent] { + seen[parent] = true + todo = append(todo, parent) } } } - return false + return found } // listTree lists every entry below the tree id as `git ls-tree -r --full-tree` diff --git a/tools/todo-update-reader-3190/objects_windows_test.go b/tools/todo-update-reader-3190/objects_windows_test.go index ce0b8195..5238ac24 100644 --- a/tools/todo-update-reader-3190/objects_windows_test.go +++ b/tools/todo-update-reader-3190/objects_windows_test.go @@ -346,7 +346,7 @@ func TestObjectStoreAgreesWithGit(t *testing.T) { } head := git("rev-parse", "HEAD") - tree, parents, _, ok := store.commit(head) + tree, parents, ok := store.commit(head) wantParents := strings.Fields(git("rev-list", "--parents", "-n", "1", "HEAD"))[1:] if !ok || tree != git("rev-parse", "HEAD^{tree}") || !slices.Equal(parents, wantParents) { t.Fatalf("commit %s: tree %s parents %v (%v)", head, tree, parents, ok) diff --git a/tools/todo-update-reader-3190/verify_windows.go b/tools/todo-update-reader-3190/verify_windows.go index e7c5e3dc..268d4eb3 100644 --- a/tools/todo-update-reader-3190/verify_windows.go +++ b/tools/todo-update-reader-3190/verify_windows.go @@ -300,7 +300,7 @@ func (s *session) checkSource(p profile) (map[string]treeEntry, string) { return nil, reason } defer store.close() - if tree, _, _, ok := store.commit(p.head); !ok || tree != p.tree { + if tree, _, ok := store.commit(p.head); !ok || tree != p.tree { return nil, store.failure() } if !store.reaches(mainID, p.head) { From 4ed83158598b696d9eff21dc3b2913a5c0d3e593 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:29:34 +0900 Subject: [PATCH 18/24] docs: record the ToDo3531 review and the reader that reads every commit Git reads The specification now describes the walk that reads every commit main reaches, the one type kept for each object name, and the delta depth counted through kept bases. It records review 3531, the fifth fix with its Red, Green and mutation results, the real source's needs measured after the fix, the pre-merge observation with the reader built from c2ff8584, and the CI of the previous head b3c0b718. --- docs/todo-update-reader-3190.md | 84 +++++++++++++++++++++++++++++---- 1 file changed, 75 insertions(+), 9 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index e2342267..d2c5edb7 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -89,14 +89,20 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト indexは末尾のchecksumを確かめ、packは先頭の件数と末尾のchecksumがindexの記録と一致する場合だけ使う。 packの各entryは、そのままの形か、offset delta・名前を指すdeltaとして同じpackの中から組み立てる。 組み立てはGitの `unpack_entry` と同じ順で行う。まずentryの見出しだけを基底までたどり、同じentryに戻る - 循環と、深さの上限を超える連鎖は、何も展開せずに拒否する。次に基底を展開し、内側のdeltaから1つずつ展開して当てる。 + 循環と、深さの上限を超える連鎖は、何も展開せずに拒否する。前の読取で組み立ててcacheに残したobjectに着いたときも、 + そのobjectの連鎖の深さを足して数える。次に基底を展開し、内側のdeltaから1つずつ展開して当てる。 同時に持つのは、組み立て中のobject、delta 1つ、その結果だけである。どのobjectも、型と中身から計算したSHA-1の 名前が求めた名前と一致する場合だけ使う。そのため、照合中に共有repoのobjectが増えたり、別のstoreを指す ファイルが一時的に置かれたりしても、readerが読む中身は変わらない。読取の量には上限を置く（次の節）。 objectsで確かめるのは次の3つ。 - 固定HEADのcommitのtreeが、固定treeと一致する。 - - `refs/remotes/origin/main` のcommitから親をたどると、固定HEADに着く。`shallow` にあるcommitは、 - Gitのshallow cloneと同じく親を持たないものとして扱う。読めない親があれば拒否する。 + - `refs/remotes/origin/main` のcommitから届くcommitの中に、固定HEADがある。readerは固定HEADに着いた後も止まらず、 + 届くcommitを全部読む。Gitの `merge-base --is-ancestor` は、答える前に、`origin/main` から届いて固定HEADからは + 届かないcommitを全部と、それぞれの親を読む（`commit-reach.c` の `paint_down_to_common`）。そのどこかに + Gitが読めないcommitがあれば、Gitは履歴を拒否する（終了コード128）。これは実物と同じくcommit-graphが無い場合で、 + commit-graphがあれば、Gitが読むcommitはさらに少ない。readerが読む範囲はどちらの場合も含むので、readerも拒否する。 + `shallow` にあるcommitは、Gitのshallow cloneと同じく親を持たないものとして扱い、その先は読まない。 + 読めない親があれば拒否する。 commitはGitの `parse_commit_buffer` と同じに解析する。`tree <40桁>` と改行の後に1byte以上が要る。 `parent ` で始まる行は、その後にこの行の長さ（48byte）を超えるbytesが続く場合だけ、 `parent <40桁>` と改行の形の親として読む。Gitはそれより短い `parent ` の行を親として読まないので、 @@ -104,6 +110,15 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト - 固定treeを再帰的にたどった一覧。`git ls-tree -r --full-tree` と同じく、通常ファイル・link・submoduleの entryを並べ、directoryは並べない。 + Gitは、調べたobjectの名前ごとに型を1つだけ記録し、同じ名前を別の型として使うと拒否する（`object.c` の + `object_as_type`）。commitのtree行は名前をtreeとして、親の行は名前をcommitとして調べるので、1つの名前がtree行と + 親の行の両方に現れる履歴などは、Gitが拒否する。readerも名前ごとに型を1つだけ持つ。読んだobjectの型と、 + commitのtree行が指す名前のtree型を記録し、2つ目の型が付けば拒否する。親は、たどるときにcommitとして読むので、 + そのとき型が付く。shallowの境界のcommitの親は、Gitが調べないので、readerも読まず、型も付けない。 + 正しいrepoでは名前ごとの型は1つなので、この確認が正しい履歴を拒否することはない。readerが読む範囲はGitより広い。 + そのため、固定HEADの祖先のうちGitが読まずに済んだcommitが壊れていれば、Gitが受け付ける履歴でもreaderは拒否する。 + 広げたのは拒否する側だけである。 + Gitが設定・別のstore・作業ファイルを読むことはない。guardは1件ずつ分類するので、rootがclearでも、 分類していないファイルは読まない。 6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 @@ -131,16 +146,18 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト objectsを読む量にはprofileの `objectLimits` で上限を置き、超えれば `source_git` で拒否する。 共有repoのobjectsが、メモリや読取の量を使い尽くす形に細工されていても、readerが使う量は上限の中に収まる。 -実物の必要量は、reader自身の数え方で、配布済みguardの分類を通して測った値である（計測用の一時的な版で1回）。 +実物の必要量は、reader自身の数え方で、配布済みguardの分類を通して測った値である（5回目の修正の後、 +計測用の一時的な版で1回。`origin/main` は `c6b5ceaa`）。readerは `origin/main` から届くcommitを全部読むので、 +commit数と読取byteは `origin/main` が進むと増える。どちらの上限も、実物の必要量の50倍以上ある。 | 上限 | 値 | 実物の必要量 | | --- | --- | --- | | `objects/pack` の項目 | 4,096 | 20 | | indexとpackの組 | 64 | 6 | | indexの合計 | 256MiB | 529,584 bytes | -| 読取byte | 512MiB | 982,328 bytes | -| deltaの連鎖 | 4,095（Gitが書く最大） | 3 | -| 祖先をたどって読むcommit | 65,536 | 45 | +| 読取byte | 512MiB | 6,755,668 bytes | +| deltaの連鎖（cacheに残したobjectの連鎖を含む） | 4,095（Gitが書く最大） | 3 | +| `origin/main` から届いて読むcommit | 65,536 | 1,291 | | 固定treeの下のentry（directoryを含む） | 65,536 | 1,657 | | 一覧のパスの合計 | 16MiB | 53,698 bytes | @@ -148,6 +165,7 @@ objectsを読む量にはprofileの `objectLimits` で上限を置き、超え loose objectのファイルの大きさと展開後の大きさを数える。大きさは、そのbytesを確保する前に数える。 中身の無いブロックで水増しした圧縮データも、読んだ分だけ数えるので上限で止まる。 1つのobjectは16MiBまで、cacheに残す組み立て済みのobjectは合計64MiBまでで、これは以前と同じである。 +型を記録する名前の数は、読むcommit、そのtree行、固定treeの下のentryの上限で決まる。実物では2,663件だった。 ## compiler closure の確認 @@ -192,8 +210,10 @@ linked worktree）、ファイル、receiptと、guardのCLI応答を模した | Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs`、`TestPackedRefsAsGitReadsThem` | | objectをGitと同じに読む（loose、pack、offset delta、名前を指すdelta、commit・tree・祖先） | `TestObjectStoreAgreesWithGit`、`TestQualifiesFromPacks` | | commitの親をGitと同じに読む。Gitが親として読まない `parent ` の行と、Gitが拒否する形を拒否する | `TestParseCommitAgreesWithGit`、`TestRefusesParentsGitDoesNotRead` | +| 1つの名前に2つの型を付ける履歴を、Gitと同じく拒否する | `TestRefusesNamesOfTwoTypes` | +| 固定HEADに着いた後も、Gitの `merge-base` が読むcommitを読み、Gitが拒否する履歴を拒否する。正しい側枝は受け付ける | `TestReadsEveryCommitGitReads` | | `objects/pack` を固定したhandleから列挙し、差し替えたjunctionの先を列挙・分類・読取しない | `TestListsOnlyThePinnedPackDirectory` | -| deltaの循環を何も展開せずに拒否し、連鎖の深さ・読取byte・packの数・項目数・indexの合計・commit数・treeのentry数・パスの合計に上限を置く | `TestDeltaCyclesInflateNothing`、`TestDeltaDepthLimit`、`TestReadLimitCountsEveryByte`、`TestReadLimit`、`TestObjectLimits` | +| deltaの循環を何も展開せずに拒否し、連鎖の深さ（cacheに残したobjectの連鎖を含む）・読取byte・packの数・項目数・indexの合計・commit数・treeのentry数・パスの合計に上限を置く | `TestDeltaCyclesInflateNothing`、`TestDeltaDepthLimit`、`TestDeltaDepthCountsKeptBases`、`TestReadLimitCountsEveryByte`、`TestReadLimit`、`TestObjectLimits` | | 壊れたobject・別のobject・保護されたobjectのファイルを拒否する | `TestRejectsDamagedObjects`、`TestApplyDeltaRefusesMalformedDeltas` | | 入力のbytesが固定treeのblob（そのままか、LFをすべてCRLFにした形）である | `TestInputsMustBeTheCommittedBytes`、`TestCheckedOutForms` | | 実物の固定値 | `TestProductionProfileBindsOneFixedUpdate` | @@ -370,6 +390,47 @@ SuggestionとNitはない。 - 途中、Greenの全体の実行が487秒かかった。同じ時間帯に同じテストをRedの本体とGreenで交互に2回ずつ動かすと、 Greenは37〜41秒、Redは42〜47秒で、遅れはPCの混雑によるものだった。最終の実行は195秒である。 +### 別モデルの再レビュー（ToDo3531） + +gpt-6.1-sol・推論max（job `20261024-074948-todo-3531`、5分42秒、終了コード0）が、base `206181d4`・head `b3c0b718` の +変更10ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のMust 3件のうち、 +`objects/pack` の列挙とdeltaの組み立ては解消、commitの解析は「一部解消」と判定された。新たな指摘は Must 1件と +Suggestion 1件で、どちらもコードとGitの一次資料からの判断であり、再現テストは伴わない。Nitはない。 + +- Must: commitが指す名前の型の食い違いを確かめていない。`tree <固定HEAD>` と `parent <固定HEAD>` のcommitを + 正しい名前で置いて `origin/main` にすると、readerは固定HEADに着いて資格を出す。Gitは固定HEADをtreeとcommitの + 両方としては扱えないので拒否する（`commit.c` のcommitの解析、`object.c` の型の照合）。 +- Suggestion: cacheに当たるとたどるのをやめるので、深さの上限は展開していない部分にしか効かない。上限1でも、 + 深さ1のobjectを先にcacheへ入れると、それを基底にする深さ2のobjectを受け付ける。 + +### 5回目のMustの修正 + +Mustを直し、同じ種類の食い違いも合わせて直した。Suggestionも直した（検証の順序の5、objectsの読取の上限）。 + +- Gitと同じく、名前ごとに型を1つだけ持つ。読んだobjectの型と、tree行が指す名前のtree型を記録し、2つ目の型が + 付けば拒否する。 +- Gitの `merge-base` は、固定HEADに着いた後も、`origin/main` から届いて固定HEADから届かないcommitを全部読む。 + これまでのreaderは固定HEADに着くと止まったので、その先にあるGitが読めないcommitや、型の食い違うcommitを + 見逃していた。readerは `origin/main` から届くcommitを全部読むようにした。たどる順序に頼らないので、 + commitの日時は読まない。 +- cacheに残すobjectに、その連鎖の深さを持たせる。cacheに当たった場合も、連鎖全体の深さで上限を確かめる。 + +結果は次のとおり。 + +- Red（commit `edbf9629`）: 新しい3つのテストを `go test -race -count=1` で実行し、終了コード1、14秒。失敗はすべて + 意図した理由による。Gitの `merge-base --is-ancestor` が終了コード128で拒否する6つの履歴で、readerは資格を出した。 + 指摘の形（tree行と親の行が固定HEADを指す）、tree行と親の行が同じcommitを指す形、固定HEADより古い側枝の先に + 無いcommitがある形、側枝の先でtree行が固定HEADを指す形である。側枝の2つは、合流commitの親の並びを2通りにした。 + 上限1で、cacheに残った深さ1のdeltaを基底にする深さ2の連鎖を読んだ。親の行が固定treeを指す履歴は既に拒否されていた。 + 正しい側枝は、両方の並びでGitが終了コード0を返し、readerも資格を出した。data race・panic・ビルド失敗は0。 +- Green（commit `c2ff8584`）: `go test -race -count=1 ./...` は終了コード0、268秒（変異テストと同時に実行）。 + 上位43件、subtest込み226件が合格し、失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。 +- 照合の要所を壊した34の版を `go test -overlay` で当て、すべて対応するテストが失敗した。新しい10の版は、 + 固定HEADで止まる、最初の親だけをたどる、tree行の型を記録しない、読んだobjectの型を記録しない、型の食い違いを許す、 + tree行をcommit型として記録する、cacheの深さを足さない、cacheの深さを0として残す、深さの比較を1つずらす、 + shallowの境界の親をたどる、である。固定HEADで止まる版は、側枝を先に挙げた合流でだけ検出された。 + 親の並びを2通りにしたのはこのためである。前の24の版は、新しいコードに合わせて当て直し、すべて検出された。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -395,6 +456,10 @@ source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力12 でも実物を1回照合した。終了コード0、15.0秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 `data_access_granted:false`）を返した。`objects/pack` は固定したhandleから列挙し、読取の量は上限の中に収まった。 +5回目の修正のcommit `c2ff8584` から `go build -trimpath` で生成したreader（SHA256 `341d866a…9bec`、vcs.modified=false） +でも実物を1回照合した。終了コード0、15.2秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。`origin/main`（`c6b5ceaa`）から届く1,291件のcommitを全部読み、拒否はなかった。 + いずれもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 対象の置換、guardの変更、ToDoの状態変更はしていない。 @@ -408,5 +473,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d 1回目の修正のhead `308cb24d` でも、専用workflow（windows、run 40002129622）とCodeQL（run 40002129620）が成功した。 2回目の修正のhead `8b4f34b9` でも、専用workflow（windows、run 40015024286）とCodeQL（run 40015024382）が成功した。 3回目の修正のhead `206181d4` でも、専用workflow（windows、run 40025021344）とCodeQL（run 40025021288）が成功した。 -4回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +4回目の修正のhead `b3c0b718` でも、専用workflow（windows、run 40034507365）とCodeQL（run 40034507248）が成功した。 +5回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる 実物の照合、親3190への結果の返却が残る。 From 59414fcdf2520cd28cd5c7ef42c5f449c8b27220 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:46:55 +0900 Subject: [PATCH 19/24] test: keep Git's automatic maintenance out of the ToDo3190 reader's fixtures CI run 40040005613 failed in TestObjectsIgnoreAlternates: the fixture's loose commit 178adb4b... was gone before the test moved it. Git 2.54.0 starts maintenance after each commit, detached, and its default strategy (geometric) repacks the loose objects and removes their files as soon as objects/17 holds two of them: too_many_loose_objects in builtin/gc.c samples that one directory against ceil(100 / 256). A fixture whose commit name starts with 17 could thus lose its loose objects while its test read them. Locally, a repository with two loose objects under 17 lost that directory within 0.5 s of a plain commit and gained a pack; with -c maintenance.auto=false it kept them. Every Git command that the tests run through testGit and gitRaw now passes -c maintenance.auto=false. The reader itself is unchanged. --- .../objects_windows_test.go | 7 +------ .../reader_windows_test.go | 20 +++++++++++++++---- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/tools/todo-update-reader-3190/objects_windows_test.go b/tools/todo-update-reader-3190/objects_windows_test.go index 5238ac24..27009fe2 100644 --- a/tools/todo-update-reader-3190/objects_windows_test.go +++ b/tools/todo-update-reader-3190/objects_windows_test.go @@ -6,12 +6,10 @@ import ( "fmt" "maps" "os" - "os/exec" "path/filepath" "slices" "strconv" "strings" - "syscall" "testing" ) @@ -184,10 +182,7 @@ func TestPackedRefsAsGitReadsThem(t *testing.T) { // gitRaw returns Git's standard output exactly. func gitRaw(t *testing.T, dir string, args ...string) []byte { t.Helper() - cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) - cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} - cmd.Env = childEnvironment(os.Environ()) - out, err := cmd.Output() + out, err := gitCommand(dir, args...).Output() if err != nil { t.Fatalf("git %v: %v", args, err) } diff --git a/tools/todo-update-reader-3190/reader_windows_test.go b/tools/todo-update-reader-3190/reader_windows_test.go index 778e21e8..dcfa9d86 100644 --- a/tools/todo-update-reader-3190/reader_windows_test.go +++ b/tools/todo-update-reader-3190/reader_windows_test.go @@ -208,12 +208,24 @@ func (f *fixture) input(rel string) fixtureInput { return fixtureInput{Path: rel, Blob: f.gitOut("rev-parse", "HEAD:"+rel), SHA256: sha(data)} } -func testGit(t *testing.T, root string, args ...string) (string, error) { - t.Helper() - cmd := exec.Command("git", append([]string{"-C", root}, args...)...) +// gitCommand is git run in dir for a test, with Git's automatic maintenance +// off, so that a synthetic repository keeps the objects its test wrote as the +// test wrote them. Git 2.54.0, which the PC and its CI runner use, starts +// maintenance after a commit, detached, and its default strategy (geometric) +// repacks the loose objects and removes their files as soon as objects/17 +// holds two of them: too_many_loose_objects in builtin/gc.c samples that one +// directory against ceil(100 / 256). A fixture whose commit name started +// with 17 lost its loose commit that way while its test read it. +func gitCommand(dir string, args ...string) *exec.Cmd { + cmd := exec.Command("git", append([]string{"-C", dir, "-c", "maintenance.auto=false"}, args...)...) cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} cmd.Env = childEnvironment(os.Environ()) - out, err := cmd.CombinedOutput() + return cmd +} + +func testGit(t *testing.T, root string, args ...string) (string, error) { + t.Helper() + out, err := gitCommand(root, args...).CombinedOutput() return strings.TrimSpace(string(out)), err } From 64b28eb235fdc039a3c2a722c1159106e648f2fb Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:48:20 +0900 Subject: [PATCH 20/24] test: require the ToDo3190 reader to give shallow names the commit type, as Git does Review 3545 found that the reader keeps the shallow file's names only as boundaries. When Git first parses a commit it reads the shallow file and looks each name up as a commit (shallow.c, register_shallow), whether or not the history reaches it, so Git refuses a tree line or a directory of the fixed tree that names one of them. TestRefusesNamesOfTwoTypes gains a main commit whose tree line names a shallow boundary that main does not reach: Git's merge-base exits 128. The new TestRefusesShallowNamesOfTrees names a directory of the fixed tree in the shallow file: git ls-tree -r fails. Both require the reader to refuse the source. Red (go test -race, these two tests): both new cases fail as intended, the reader qualifying where Git refuses. The other three type cases pass. No data race, panic or build failure. --- .../commit_walk_windows_test.go | 24 +++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/tools/todo-update-reader-3190/commit_walk_windows_test.go b/tools/todo-update-reader-3190/commit_walk_windows_test.go index 7cedd65a..bfd6142d 100644 --- a/tools/todo-update-reader-3190/commit_walk_windows_test.go +++ b/tools/todo-update-reader-3190/commit_walk_windows_test.go @@ -60,8 +60,10 @@ func checkHistories(t *testing.T, cases map[string]historyCase) { // Git keeps one type for each object name that it looks up or reads, and // refuses to use that object as another type (object.c, object_as_type). A // commit's tree line looks its name up as a tree and each parent line looks -// its name up as a commit, so merge-base refuses a history in which one name -// is both. The reader refuses the source there too. +// its name up as a commit, and when Git first parses a commit it gives each +// name in the shallow file the commit type (shallow.c, register_shallow), +// whether or not the history reaches it. So merge-base refuses a history in +// which one name is both. The reader refuses the source there too. func TestRefusesNamesOfTwoTypes(t *testing.T) { checkHistories(t, map[string]historyCase{ "tree line names the fixed commit, also the parent": {128, func(f *fixture, put func(string) string) string { @@ -74,9 +76,27 @@ func TestRefusesNamesOfTwoTypes(t *testing.T) { "parent line names the fixed tree": {128, func(f *fixture, put func(string) string) string { return put("tree " + f.p.tree + "\nparent " + f.p.head + "\nparent " + f.p.tree + "\n" + laterDates + "\nmain\n") }}, + "tree line names a shallow boundary that main does not reach": {128, func(f *fixture, put func(string) string) string { + const boundary = "2222222222222222222222222222222222222222" + writeFile(f.t, filepath.Join(f.common, "shallow"), boundary+"\n") + return put("tree " + boundary + "\nparent " + f.p.head + "\n" + laterDates + "\nmain\n") + }}, }) } +// A name in the shallow file has the commit type in git ls-tree too, so a +// directory of the fixed tree that the shallow file names makes Git refuse +// the listing. The reader refuses the source. +func TestRefusesShallowNamesOfTrees(t *testing.T) { + f := newFixture(t, nil) + directory := f.gitOut("rev-parse", "HEAD:app") + writeFile(t, filepath.Join(f.common, "shallow"), directory+"\n") + if out, err := testGit(t, f.root, "ls-tree", "-r", "--full-tree", f.p.head); err == nil { + t.Fatalf("git ls-tree listed a tree whose directory the shallow file names:\n%s", out) + } + expectReason(t, f, "source_git") +} + // Git's merge-base walk does not stop at the fixed commit. Before it answers, // it reads every commit that main reaches and the fixed commit does not, and // the parents of each (commit-reach.c, paint_down_to_common), so a commit From bb3bb1be54df62503d20e3f21ec2a2888b6b0af1 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:55:13 +0900 Subject: [PATCH 21/24] fix: give every shallow name the commit type in the ToDo3190 reader, as Git does When Git first parses a commit it reads the shallow file and looks each name up as a commit (shallow.c, register_shallow), whether or not the history reaches it. The reader now gives each name of the shallow file the commit type when it opens the store, so a tree line or a directory of the fixed tree that names one of them refuses the source, as Git refuses it. Git registers the names after it has looked up the first commit's tree and only reports a conflict with that tree, so a shallow file that names the fixed tree itself is refused by the reader alone. go test -race -count=1 ./...: exit 0, 255 s beside the mutation runs; 44 top-level tests and 228 with subtests pass, with no failure, skip or data race. go vet and gofmt are clean. All 35 mutants are killed, among them the new one that leaves shallow names untyped. --- tools/todo-update-reader-3190/objects_windows.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/todo-update-reader-3190/objects_windows.go b/tools/todo-update-reader-3190/objects_windows.go index 6512a903..c756a70d 100644 --- a/tools/todo-update-reader-3190/objects_windows.go +++ b/tools/todo-update-reader-3190/objects_windows.go @@ -133,9 +133,13 @@ type namedEntry struct { // opened. func (s *session) openObjects(src sourceObjects, limits objectLimits) (*objectStore, string) { o := &objectStore{s: s, dir: src.dir, limits: limits, shallow: map[string]bool{}, kinds: map[string]string{}, cache: map[packedAt]packedObject{}} + // When Git first parses a commit it reads the shallow file and looks each + // name up as a commit (shallow.c, register_shallow), whether or not the + // history reaches it, so each name has the commit type from the start. if text, ok := strings.CutSuffix(string(src.shallow), "\n"); ok { for _, id := range strings.Split(text, "\n") { o.shallow[id] = true + o.kinds[id] = "commit" } } dir := filepath.Join(src.dir, "pack") From d30bc929561d2219ca3a6393061d054d6f56cff7 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 08:56:32 +0900 Subject: [PATCH 22/24] docs: record the CI failure from Git's maintenance, review 3545 and the typed shallow names The specification now gives shallow names the commit type, as Git's register_shallow does, notes that a repack by automatic maintenance in the shared repository can refuse a run, which is then rerun, and explains why the tests keep Git's automatic maintenance off. It records CI run 40040005613, whose one failure came from that maintenance in a test fixture, review 3545, the sixth fix with its Red, Green and mutation results, and the pre-merge observation with the reader built from bb3bb1be. --- docs/todo-update-reader-3190.md | 59 +++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index d2c5edb7..db6332a3 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -119,6 +119,14 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト そのため、固定HEADの祖先のうちGitが読まずに済んだcommitが壊れていれば、Gitが受け付ける履歴でもreaderは拒否する。 広げたのは拒否する側だけである。 + shallowファイルの名前も型を持つ。Gitは最初にcommitを解析するときにshallowファイルを読み、そこにある名前を、 + 履歴から届くかどうかに関係なく、すべてcommitとして調べる（`shallow.c` の `register_shallow`）。そのため、 + tree行や固定treeのdirectoryがshallowの名前を指すと、Gitは拒否する（`merge-base` は終了コード128、 + `ls-tree` は `object … is a commit, not a tree` を表示して異常終了する）。readerは、shallowの全ての名前に、 + storeを開く時点でcommit型を付ける。Gitはこの登録を最初のcommitのtree行を調べた後で行い、そのtreeとの + 食い違いは表示するだけで先へ進む。そのため、固定treeそのものがshallowに載っている形は、Gitは受け付けるが + readerは拒否する。 + Gitが設定・別のstore・作業ファイルを読むことはない。guardは1件ずつ分類するので、rootがclearでも、 分類していないファイルは読まない。 6. receipt・log・dep-info・各入力・新候補は、それぞれ同じ方法で開いて保持し、guardの clear を確かめてから読む。 @@ -141,6 +149,8 @@ callerからmanifest、hash、source、root、ToDo番号、許可ディレクト 固定の後でこのディレクトリが改名されれば、実パスが変わるので再確認で拒否する。 入力は書込を共有しないhandleで保持しているので、照合中には変えられない。共有repoのconfigやrefは ほかのworktreeが書き換えることがあり、照合中に書き換わると `recheck` で拒否される。その場合は再実行する。 + 共有repoで自動maintenanceがrepackすると、照合中にloose objectやpackが消え、`source_git` で拒否されることもある。 + この拒否も安全な側の結果で、再実行すればよい。 ## objectsの読取の上限 @@ -192,6 +202,9 @@ receiptの126件の入力が、原dep-infoの116件と manifest/lock の10件に linked worktree）、ファイル、receiptと、guardのCLI応答を模した関数で、実物の配布先・guard・親の作業場所を使わずに確かめる。 理由コードも照合し、別の理由による偶然の拒否を合格にしない。objectの読取は、Git自身の出力 （`git cat-file --batch-all-objects`、`git ls-tree`、`git rev-list`、`git log`、`git merge-base`、`git verify-pack`）と比べる。 +テストが起動するGitには `-c maintenance.auto=false` を付け、合成repoをテストが書いたとおりに保つ。Git 2.54.0は +commitの後に自動maintenanceを切り離して動かし、既定のstrategy（geometric）は、`objects/17` にloose objectが +2つあるだけでrepackして、looseのファイルを消す（`builtin/gc.c` の `too_many_loose_objects`）。 | 条件 | テスト | | --- | --- | @@ -210,7 +223,7 @@ linked worktree）、ファイル、receiptと、guardのCLI応答を模した | Gitの設定と参照をreader自身がguardの分類後に読み、Gitと同じ意味に解釈する | `TestReadsGitMetadataItself`、`TestConfigParserAgreesWithGit`、`TestConfigParserRefusesWhatItDoesNotRead`、`TestPackedRefs`、`TestPackedRefsAsGitReadsThem` | | objectをGitと同じに読む（loose、pack、offset delta、名前を指すdelta、commit・tree・祖先） | `TestObjectStoreAgreesWithGit`、`TestQualifiesFromPacks` | | commitの親をGitと同じに読む。Gitが親として読まない `parent ` の行と、Gitが拒否する形を拒否する | `TestParseCommitAgreesWithGit`、`TestRefusesParentsGitDoesNotRead` | -| 1つの名前に2つの型を付ける履歴を、Gitと同じく拒否する | `TestRefusesNamesOfTwoTypes` | +| 1つの名前に2つの型を付ける履歴（shallowの名前を含む）を、Gitと同じく拒否する | `TestRefusesNamesOfTwoTypes`、`TestRefusesShallowNamesOfTrees` | | 固定HEADに着いた後も、Gitの `merge-base` が読むcommitを読み、Gitが拒否する履歴を拒否する。正しい側枝は受け付ける | `TestReadsEveryCommitGitReads` | | `objects/pack` を固定したhandleから列挙し、差し替えたjunctionの先を列挙・分類・読取しない | `TestListsOnlyThePinnedPackDirectory` | | deltaの循環を何も展開せずに拒否し、連鎖の深さ（cacheに残したobjectの連鎖を含む）・読取byte・packの数・項目数・indexの合計・commit数・treeのentry数・パスの合計に上限を置く | `TestDeltaCyclesInflateNothing`、`TestDeltaDepthLimit`、`TestDeltaDepthCountsKeptBases`、`TestReadLimitCountsEveryByte`、`TestReadLimit`、`TestObjectLimits` | @@ -431,6 +444,42 @@ Mustを直し、同じ種類の食い違いも合わせて直した。Suggestion shallowの境界の親をたどる、である。固定HEADで止まる版は、側枝を先に挙げた合流でだけ検出された。 親の並びを2通りにしたのはこのためである。前の24の版は、新しいコードに合わせて当て直し、すべて検出された。 +### 正式CIの失敗（run 40040005613）と別モデルの再レビュー（ToDo3545） + +5回目の修正のhead `4ed83158` で、専用workflow（windows、run 40040005613）が失敗した。CodeQL（run 40040005526）は成功した。 +失敗したのは `TestObjectsIgnoreAlternates` の1件で、fixtureのloose commit（`178adb4b…`）のファイルが、テストが動かす前に +消えていた。readerの変更ではなく、テストの合成repoで動いたGitの自動maintenanceが原因だった。Git 2.54.0は、 +commitの後にmaintenanceを切り離して動かす。既定のstrategy（geometric）は、`objects/17` だけを数えてloose objectの +総数を見積もり（`builtin/gc.c` の `too_many_loose_objects`。閾値は100を256で割って切り上げた1）、`17` で始まる名前の +loose objectが2つあればrepackしてlooseのファイルを消す。fixtureのcommitの名前は時刻で変わるので、まれにこの条件に当たる。 +手元でも、`objects/17` に2つのloose objectを置いたrepoで普通にcommitすると、0.5秒以内に `objects/17` が消えて +packが1つできた。`-c maintenance.auto=false` を付けると、5秒後も残った。 + +gpt-6.1-sol・推論max（job `20261024-083553-todo-3545`、4分30秒、終了コード0）が、base `b3c0b718`・head `4ed83158` の +変更7ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のSuggestionは解消、 +前回のMustは「一部解消」と判定された。新たな指摘は Must 1件で、コードとGitの一次資料からの判断であり、再現テストは +伴わない。SuggestionとNitはない。 + +- Must: shallowの名前を境界の印として持つだけで、commit型を付けていない。Gitは `register_shallow` で、履歴から + 届かない名前にもcommit型を付ける。届かないshallowの名前をtree行が指すcommitを正しい名前で置いて `origin/main` に + すると、Gitは型の食い違いで終了コード128を返すが、readerは資格を出す。 + +### 6回目の修正 + +- テストが起動するGitに `-c maintenance.auto=false` を付けた（commit `59414fcd`）。readerは変えていない。 +- shallowファイルの全ての名前に、storeを開く時点でcommit型を付ける（検証の順序の5）。 + +結果は次のとおり。 + +- Red（commit `64b28eb2`）: 新しい2つの形を `go test -race -count=1` で実行し、終了コード1。どちらも意図した理由で + 失敗した。届かないshallowの名前をtree行が指す履歴では、Gitの `merge-base --is-ancestor` が終了コード128なのに、 + readerは資格を出した。固定treeのdirectoryをshallowに載せると、Gitの `ls-tree -r` は失敗したのに、readerは資格を出した。 + ほかの3つの型の形は合格した。data race・panic・ビルド失敗は0。 +- Green（commit `bb3bb1be`）: `go test -race -count=1 ./...` は終了コード0、255秒（変異テストと同時に実行）。 + 上位44件、subtest込み228件が合格し、失敗0・skip 0・data race 0。`go vet ./...` とgofmtも差分なし。 +- 照合の要所を壊した35の版を `go test -overlay` で当て、すべて対応するテストが失敗した。新しい1つの版は、 + shallowの名前に型を付けない。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -460,6 +509,10 @@ source `0e16e817…` / tree `31d45a74…`、receipt `20345f44…087c`、入力12 でも実物を1回照合した。終了コード0、15.2秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 `data_access_granted:false`）を返した。`origin/main`（`c6b5ceaa`）から届く1,291件のcommitを全部読み、拒否はなかった。 +6回目の修正のcommit `bb3bb1be` から `go build -trimpath` で生成したreader（SHA256 `a148a4c9…525e`、vcs.modified=false） +でも実物を1回照合した。終了コード0、15.2秒で、同じ固定値の `qualified:true`（入力126件・dep-info 116件・crate 8件、 +`data_access_granted:false`）を返した。 + いずれもmerge前のsourceによる観測で、最終の結果はmerge後のsourceから生成したreaderで取り直す。 対象の置換、guardの変更、ToDoの状態変更はしていない。 @@ -474,5 +527,7 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d 2回目の修正のhead `8b4f34b9` でも、専用workflow（windows、run 40015024286）とCodeQL（run 40015024382）が成功した。 3回目の修正のhead `206181d4` でも、専用workflow（windows、run 40025021344）とCodeQL（run 40025021288）が成功した。 4回目の修正のhead `b3c0b718` でも、専用workflow（windows、run 40034507365）とCodeQL（run 40034507248）が成功した。 -5回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +5回目の修正のhead `4ed83158` では、CodeQL（run 40040005526）が成功し、専用workflow（run 40040005613）は +テストの合成repoで動いた自動maintenanceのために失敗した（前の節）。 +6回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる 実物の照合、親3190への結果の返却が残る。 From 3cd052b5a35f94e89794642c5ca712f3ac76a776 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 09:31:51 +0900 Subject: [PATCH 23/24] test: start Git for the config listing through gitCommand gitConfigList started Git with exec.Command directly, so it ran without -c maintenance.auto=false, unlike every other Git the tests start. Start it through gitCommand and keep the variables that stop Git from reading the system and global configuration. git config --file lists only that file, so the values TestConfigParserAgreesWithGit compares are unchanged. The reader itself is unchanged. Suggestion from the review in ToDo3548 (gpt-6.1-sol, max). --- tools/todo-update-reader-3190/gitmeta_windows_test.go | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/tools/todo-update-reader-3190/gitmeta_windows_test.go b/tools/todo-update-reader-3190/gitmeta_windows_test.go index 2f63c5c8..065f3ab9 100644 --- a/tools/todo-update-reader-3190/gitmeta_windows_test.go +++ b/tools/todo-update-reader-3190/gitmeta_windows_test.go @@ -2,11 +2,9 @@ package main import ( "os" - "os/exec" "path/filepath" "slices" "strings" - "syscall" "testing" ) @@ -18,9 +16,8 @@ func gitConfigList(t *testing.T, config string) []string { if err := os.WriteFile(name, []byte(config), 0o644); err != nil { t.Fatal(err) } - cmd := exec.Command("git", "config", "--file", name, "--null", "--list") - cmd.SysProcAttr = &syscall.SysProcAttr{HideWindow: true, CreationFlags: createNoWindow} - cmd.Env = append(childEnvironment(os.Environ()), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null") + cmd := gitCommand(filepath.Dir(name), "config", "--file", name, "--null", "--list") + cmd.Env = append(cmd.Env, "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null") out, err := cmd.Output() if err != nil { t.Fatalf("git config --list: %v", err) From 0a63ca865352161b89861261e0e473f3564dbfe6 Mon Sep 17 00:00:00 2001 From: mashfromband Date: Sat, 24 Oct 2026 09:32:44 +0900 Subject: [PATCH 24/24] docs: record review 3548, its suggestion and the passing CI of d30bc929 Review ToDo3548 (gpt-6.1-sol, max) found no Must and judged the previous one resolved. Its one Suggestion, starting the config listing in the tests through gitCommand, is applied in 3cd052b5. The formal CI of d30bc929 passed (windows run 40042426529, CodeQL run 40042426578). --- docs/todo-update-reader-3190.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/todo-update-reader-3190.md b/docs/todo-update-reader-3190.md index db6332a3..fa9908f4 100644 --- a/docs/todo-update-reader-3190.md +++ b/docs/todo-update-reader-3190.md @@ -480,6 +480,24 @@ gpt-6.1-sol・推論max（job `20261024-083553-todo-3545`、4分30秒、終了 - 照合の要所を壊した35の版を `go test -overlay` で当て、すべて対応するテストが失敗した。新しい1つの版は、 shallowの名前に型を付けない。 +### 別モデルの再レビュー（ToDo3548）と提案の反映 + +gpt-6.1-sol・推論max（job `20261024-090457-todo-3548`、5分25秒、終了コード0）が、base `4ed83158`・head `d30bc929` の +変更5ファイルを差分と全文で読んだ。レビュー担当はテスト・ビルド・CIを実行していない。前回のMustは解消と判定された。 +新たな指摘は Suggestion 1件で、コードからの判断であり、再現テストは伴わない。MustとNitはない。 + +- Suggestion: `gitmeta_windows_test.go` の `gitConfigList` は、Gitを `exec.Command` で直接起動していて、 + `-c maintenance.auto=false` が付かない。「テストが起動するGitに付ける」という説明とずれる。設定の読取はfixtureを + 変えないので、CIの失敗を再発させるものではない。 + +`gitConfigList` も `gitCommand` からGitを起動するように直した（commit `3cd052b5`）。Gitにシステムとglobalの設定を +読ませない環境変数（`GIT_CONFIG_NOSYSTEM=1`、`GIT_CONFIG_GLOBAL=/dev/null`）はそのまま付ける。readerは変えていない。 + +- `go test -race -count=1 -run 'TestConfigParser|TestPackedRefs' ./...` は合格した（4.1秒）。`git config --file` は + 指定したファイルだけを列挙するので、`TestConfigParserAgreesWithGit` が比べる内容は変わらない。`-c` で渡した値が + 列挙に混ざれば、この比較が失敗する。`go vet ./...` とgofmtも差分なし。 +- テストの中でGitを直接起動する箇所は、`gitCommand` の1か所だけになった。 + ### 実物への事前の観測（merge前） 最初の実装から生成した検査用readerで、実物に対して1回照合した。終了コード0、9.6秒で、 @@ -529,5 +547,6 @@ SHA256 `d4392e63…0718`、同じサイズと更新時刻で、`target/release/d 4回目の修正のhead `b3c0b718` でも、専用workflow（windows、run 40034507365）とCodeQL（run 40034507248）が成功した。 5回目の修正のhead `4ed83158` では、CodeQL（run 40040005526）が成功し、専用workflow（run 40040005613）は テストの合成repoで動いた自動maintenanceのために失敗した（前の節）。 -6回目の修正後のheadの正式CI、修正の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる +6回目の修正のhead `d30bc929` では、専用workflow（windows、run 40042426529）とCodeQL（run 40042426578）が成功した。 +提案を反映した後のheadの正式CI、追加した差分の別モデル再レビュー、通常merge、merge後のsourceから生成したreaderによる 実物の照合、親3190への結果の返却が残る。