Skip to content

br: fix fake-gcs test - #67645

Merged
ti-chi-bot[bot] merged 2 commits into
pingcap:masterfrom
JoyC-dev:fix_test
Apr 9, 2026
Merged

ti-chi-bot[bot] merged 2 commits into
pingcap:masterfrom
JoyC-dev:fix_test

Conversation

@JoyC-dev

@JoyC-dev JoyC-dev commented Apr 9, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67623

Problem Summary:

What changed and how does it work?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Summary by CodeRabbit

  • Bug Fixes
    • Updated GCS backup/restore endpoint to use the base host/port (removing the embedded path) for improved compatibility.
    • Enhanced service-account credential format with additional required fields for GCS operations.
    • Legacy backup/restore steps are now optional and run only when explicitly enabled, reducing unintended legacy runs.

@pantheon-ai

pantheon-ai Bot commented Apr 9, 2026 •

Copy link
Copy Markdown

@JoyC-dev I've received your pull request and will start the review. I'll conduct a thorough review covering code quality, potential issues, and implementation details.

⏳ This process typically takes 10-30 minutes depending on the complexity of the changes.

ℹ️ Learn more details on Pantheon AI.

@ti-chi-bot ti-chi-bot Bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed do-not-merge/needs-tests-checked do-not-merge/needs-linked-issue labels Apr 9, 2026
@tiprow

tiprow Bot commented Apr 9, 2026

Copy link
Copy Markdown

Hi @JoyC-dev. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@coderabbitai

coderabbitai Bot commented Apr 9, 2026 •

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Updated GCS test scripts to include project_id and private_key_id fields in generated Google service-account JSON, remove the /storage/v1 suffix from GCS endpoint URLs used in backup/restore and prepared targets, and make legacy bin/brv4.0.8 backup/restore and its cleanup/verification conditional on TEST_LEGACY_BR_GCS=true.

Changes

Cohort / File(s) Summary
Primary BR GCS test
br/tests/br_gcs/run.sh
Added project_id and private_key_id to the generated service-account JSON; changed GCS endpoint args from http://$GCS_HOST:$GCS_PORT/storage/v1 to http://$GCS_HOST:$GCS_PORT; gated legacy bin/brv4.0.8 backup/restore and the second cleanup/row-count verification behind TEST_LEGACY_BR_GCS=true.
Docker-compatible GCS scripts
br/tests/docker_compatible_gcs/_run.sh, br/tests/docker_compatible_gcs/prepare.sh
Added project_id and private_key_id to the generated service-account JSON; updated --gcs.endpoint / endpoint query parameter to http://$GCS_HOST:$GCS_PORT (removed /storage/v1/).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 I nibbled keys and fixed a sigh,

project_id and private_key_id, oh my!
Trimmed the path that caused the fuss,
legacy hops now gated for us.
Hop—GCS tests pass, carrot plus! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is largely incomplete with minimal details about what changed and how it works despite following the template structure. Add a detailed 'What changed and how does it work?' section explaining the GCS endpoint changes, credential field additions, and the legacy BR v4.0.8 gating logic to clarify the fix.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'br: fix fake-gcs test' is directly related to the main change—fixing the BR fake-gcs integration test by updating GCS endpoint configurations and adding missing credential fields.
Linked Issues check ✅ Passed The PR changes address the core requirement from issue #67623: adding missing 'project_id' and 'private_key_id' fields to the fake GCS service-account JSON and updating GCS endpoint values to the base URL.
Out of Scope Changes check ✅ Passed All changes are scoped to BR GCS test files, consistently addressing the GCS endpoint configuration and credential field issues. The legacy BR v4.0.8 gating is a supportive change for test stability.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
br/tests/br_gcs/run.sh (1)

92-92: Quote "$PD_ADDR" consistently in BR invocations.

These three commands still use unquoted --pd $PD_ADDR; quoting avoids shell splitting/globbing pitfalls.

Suggested patch
-run_br --pd $PD_ADDR backup full -s "gcs://$BUCKET/$DB?endpoint=http://$GCS_HOST:$GCS_PORT"
+run_br --pd "$PD_ADDR" backup full -s "gcs://$BUCKET/$DB?endpoint=http://$GCS_HOST:$GCS_PORT"

-run_br restore full -s "gcs://$BUCKET/$DB?" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
+run_br restore full -s "gcs://$BUCKET/$DB?" --pd "$PD_ADDR" --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false

-run_br restore full -s "gcs://$BUCKET/${DB}_old" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
+run_br restore full -s "gcs://$BUCKET/${DB}_old" --pd "$PD_ADDR" --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false

Also applies to: 110-110, 138-138

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@br/tests/br_gcs/run.sh` at line 92, The BR invocations call run_br with an
unquoted PD address (e.g., the command using run_br --pd $PD_ADDR backup full
...); update each invocation to quote the variable to prevent shell
word-splitting/globbing by changing --pd $PD_ADDR to --pd "$PD_ADDR" in the
run_br calls found around the lines with run_br --pd $PD_ADDR (also apply same
fix at the other occurrences mentioned).
br/tests/docker_compatible_gcs/_run.sh (1)

45-45: Quote "$PD_ADDR" in the BR command.

--pd $PD_ADDR is currently unquoted; quote it to avoid shell word splitting/globbing edge cases. This is inconsistent with the quoting of other variables like "$BUCKET", "$GCS_HOST", and "$GCS_PORT" on the same line.

Suggested patch
-    bin/br restore db --db test -s "gcs://$BUCKET/bk${TAG}" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
+    bin/br restore db --db test -s "gcs://$BUCKET/bk${TAG}" --pd "$PD_ADDR" --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@br/tests/docker_compatible_gcs/_run.sh` at line 45, The BR restore invocation
uses an unquoted --pd $PD_ADDR which risks shell word-splitting/globbing; update
the command that calls bin/br restore db (the line with --pd $PD_ADDR) to quote
the PD_ADDR variable as --pd "$PD_ADDR" so it matches the quoting style used for
"$BUCKET", "$GCS_HOST", and "$GCS_PORT".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@br/tests/br_gcs/run.sh`:
- Line 92: The BR invocations call run_br with an unquoted PD address (e.g., the
command using run_br --pd $PD_ADDR backup full ...); update each invocation to
quote the variable to prevent shell word-splitting/globbing by changing --pd
$PD_ADDR to --pd "$PD_ADDR" in the run_br calls found around the lines with
run_br --pd $PD_ADDR (also apply same fix at the other occurrences mentioned).

In `@br/tests/docker_compatible_gcs/_run.sh`:
- Line 45: The BR restore invocation uses an unquoted --pd $PD_ADDR which risks
shell word-splitting/globbing; update the command that calls bin/br restore db
(the line with --pd $PD_ADDR) to quote the PD_ADDR variable as --pd "$PD_ADDR"
so it matches the quoting style used for "$BUCKET", "$GCS_HOST", and
"$GCS_PORT".

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: dbc1ebb0-1645-4778-ac64-3ee03cff7bca

📥 Commits

Reviewing files that changed from the base of the PR and between de09871 and ad7bd37.

📒 Files selected for processing (3)
  • br/tests/br_gcs/run.sh
  • br/tests/docker_compatible_gcs/_run.sh
  • br/tests/docker_compatible_gcs/prepare.sh

@codecov

codecov Bot commented Apr 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.6548%. Comparing base (e0b4e44) to head (b07efb5).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #67645        +/-   ##
================================================
+ Coverage   77.5935%   78.6548%   +1.0612%     
================================================
  Files          1981       1983         +2     
  Lines        547950     548035        +85     
================================================
+ Hits         425174     431056      +5882     
+ Misses       121966     115955      -6011     
- Partials        810       1024       +214     
Flag Coverage Δ
integration 44.3882% <ø> (+10.0485%) ⬆️
unit 76.6500% <ø> (+0.3067%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 61.5065% <ø> (ø)
parser ∅ <ø> (∅)
br 66.0183% <ø> (+5.5799%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JoyC-dev

JoyC-dev commented Apr 9, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Apr 9, 2026

Copy link
Copy Markdown

@JoyC-dev: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@joechenrh joechenrh changed the title fix fake-gcs test br: fix fake-gcs test Apr 9, 2026
@ti-chi-bot ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 9, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
br/tests/br_gcs/run.sh (1)

138-139: Log text is misleading for the executed restore binary.

The message says v4.0.8 version restore start..., but this block runs run_br restore (current BR). Consider renaming the log to reflect “restore legacy backup with current BR” for clearer diagnostics.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@br/tests/br_gcs/run.sh` around lines 138 - 139, The log message "v4.0.8
version restore start..." is misleading because the script actually invokes the
current BR via the run_br restore command; update the echo to clearly state that
the current BR is restoring a legacy v4.0.8 backup (e.g., change the message to
something like "restoring v4.0.8 backup using current BR...") so diagnostics
reflect that run_br is used; locate the echo string near the run_br restore
invocation and adjust its text accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@br/tests/br_gcs/run.sh`:
- Line 111: The restore command uses a storage URL with a dangling query marker
'-s "gcs://$BUCKET/$DB?"' which adds an empty query component; update the
invocation in run_br (the line calling run_br restore full) to pass the storage
URL without the trailing '?' (use 'gcs://$BUCKET/$DB') so the GCS URL is clean
and avoids parser edge cases across BR versions.

---

Nitpick comments:
In `@br/tests/br_gcs/run.sh`:
- Around line 138-139: The log message "v4.0.8 version restore start..." is
misleading because the script actually invokes the current BR via the run_br
restore command; update the echo to clearly state that the current BR is
restoring a legacy v4.0.8 backup (e.g., change the message to something like
"restoring v4.0.8 backup using current BR...") so diagnostics reflect that
run_br is used; locate the echo string near the run_br restore invocation and
adjust its text accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 1e21c7fb-2221-440d-9860-492b59d6e097

📥 Commits

Reviewing files that changed from the base of the PR and between ad7bd37 and b07efb5.

📒 Files selected for processing (1)
  • br/tests/br_gcs/run.sh

Comment thread br/tests/br_gcs/run.sh
# new version restore full
echo "restore start..."
run_br restore full -s "gcs://$BUCKET/$DB?" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT/storage/v1/" --check-requirements=false
run_br restore full -s "gcs://$BUCKET/$DB?" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Remove the dangling ? in the restore storage URL.

-s "gcs://$BUCKET/$DB?" carries an empty query component. It’s safer to pass gcs://$BUCKET/$DB directly to avoid parser edge behavior across BR versions.

💡 Proposed fix
-run_br restore full -s "gcs://$BUCKET/$DB?" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
+run_br restore full -s "gcs://$BUCKET/$DB" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
run_br restore full -s "gcs://$BUCKET/$DB?" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
run_br restore full -s "gcs://$BUCKET/$DB" --pd $PD_ADDR --gcs.endpoint="http://$GCS_HOST:$GCS_PORT" --check-requirements=false
🧰 Tools
🪛 Shellcheck (0.11.0)

[info] 111-111: Double quote to prevent globbing and word splitting.

(SC2086)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@br/tests/br_gcs/run.sh` at line 111, The restore command uses a storage URL
with a dangling query marker '-s "gcs://$BUCKET/$DB?"' which adds an empty query
component; update the invocation in run_br (the line calling run_br restore
full) to pass the storage URL without the trailing '?' (use 'gcs://$BUCKET/$DB')
so the GCS URL is clean and avoids parser edge cases across BR versions.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Apr 9, 2026
@ti-chi-bot

ti-chi-bot Bot commented Apr 9, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 9, 2026
@ti-chi-bot

ti-chi-bot Bot commented Apr 9, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-04-09 10:26:29.891607345 +0000 UTC m=+1038395.096967402: ☑️ agreed by D3Hunter.
  • 2026-04-09 10:27:05.745709257 +0000 UTC m=+1038430.951069324: ☑️ agreed by Leavrth.

@ti-chi-bot
ti-chi-bot Bot merged commit 997e75c into pingcap:master Apr 9, 2026
34 of 36 checks passed
premal pushed a commit to premal/tidb that referenced this pull request Apr 30, 2026
jebter pushed a commit to YuJuncen/tidb that referenced this pull request May 11, 2026
@v01dstar v01dstar mentioned this pull request Jun 16, 2026
2 of 4 tasks
bb7133 added a commit to bb7133/tidb that referenced this pull request Sep 20, 2026
Port upstream pingcap#67645 (997e75c). Update service-account fields and base endpoints; retain legacy BR v4.0.8 checks behind TEST_LEGACY_BR_GCS=true, matching upstream.
@expxiaoli

Copy link
Copy Markdown
Contributor

/cherry-pick feature/release-8.5-read_only

@ti-chi-bot ti-chi-bot mentioned this pull request Sep 21, 2026
1 of 13 tasks
@ti-chi-bot

Copy link
Copy Markdown
Member

@expxiaoli: new pull request created to branch feature/release-8.5-read_only: #71427.

Details

In response to this:

/cherry-pick feature/release-8.5-read_only

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@Leavrth

Leavrth commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

fixed by #67142 in release-8.5

@gengliqi

Copy link
Copy Markdown
Contributor

/cherry-pick release-8.5-20260608-v8.5.6

@ti-chi-bot

Copy link
Copy Markdown
Member

@gengliqi: new pull request created to branch release-8.5-20260608-v8.5.6: #71932.

Details

In response to this:

/cherry-pick release-8.5-20260608-v8.5.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot mentioned this pull request Oct 13, 2026
1 of 13 tasks
@Leavrth

Leavrth commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

NO NEED cherry-pick to feature branch, you can add | tikv={branch-name} to the end of the PR title to specify the tikv version instead of v8.5.8, like this #71838.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

br: br_gcs integration test fails with missing private_key_id in fake GCS credentials

6 participants