Seed zero-value quota metric on bucket create / quota enable by delthas · Pull Request #6221 · scality/cloudserver

delthas · 2026-07-06T14:53:05Z

Context

Bucket/account quota enforcement reads utilization metrics from Scuba, which serves the __infostore collection. Those documents are written today only by the s3utils count-items cron (every 6–24h). A freshly created bucket — or a bucket that has a quota enabled while empty — has no metric document until the next cron run, so validateQuotas → _evaluateQuotas gets NoSuchEntity from Scuba and fails open: the quota is not enforced for up to 24h (ARTESCA-17063).

Change

Seed a zero-value bucket metric document in __infostore at the moments CloudServer knows the bucket is empty, so the quota is enforceable immediately:

On createBucket — the bucket is empty by definition, so seed unconditionally.
On bucketUpdateQuota — seed only when the bucket is verifiably empty (a maxKeys:1 DelimiterVersions listing). Never default to zero for a bucket that may hold uncounted data (that would silently under-enforce).

Both paths:

are gated on config.isQuotaEnabled();
are best-effort / non-fatal — a seeding failure is logged and never fails the request (quota simply stays disabled until count-items, i.e. today's behavior);
use the new arsenal MetadataWrapper.initializeBucketCapacity (idempotent $setOnInsert, so an existing count-items document is never overwritten), keyed by the bucket's metastore creationDate — exactly what _evaluateQuotas queries with, so the create-window lookup is an exact match.

Account-level metrics are intentionally out of scope: a brand-new bucket's account may hold data in other, not-yet-counted buckets, so seeding account_<id>=0 would under-report. Empty-account coverage stays with count-items (S3UTILS-224).

Dependency

Requires ARSN-610 (arsenal#2666). The first commit here temporarily pins arsenal to that branch commit so the new method resolves; it should be replaced with a released arsenal 8.5 tag once ARSN-610 merges.

Tests

New unit tests: bucketUpdateQuota seeds on an empty bucket, does not seed on a non-empty bucket, still succeeds when seeding errors, and does not seed when quotas are disabled; bucketPut seeds on creation, stays non-fatal on seed error, and does not seed when quotas are disabled.

Issue: CLDSRV-949

bert-e · 2026-07-06T14:53:10Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-07-06T14:54:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

codecov · 2026-07-06T15:06:15Z

Codecov Report

❌ Patch coverage is 95.71429% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.01%. Comparing base (9314dc5) to head (96ee768).
⚠️ Report is 16 commits behind head on development/9.4.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/api/apiUtils/bucket/bucketCreation.js	96.25%	3 Missing ⚠️
lib/api/bucketUpdateQuota.js	95.00%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/api/apiUtils/bucket/bucketCreation.js	`96.96% <96.25%> (+3.58%)`	⬆️
lib/api/bucketUpdateQuota.js	`96.05% <95.00%> (-0.03%)`	⬇️

... and 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6221      +/-   ##
===================================================
+ Coverage            85.99%   86.01%   +0.02%     
===================================================
  Files                  212      212              
  Lines                14415    14451      +36     
===================================================
+ Hits                 12396    12430      +34     
- Misses                2019     2021       +2

Flag	Coverage Δ
file-ft-tests	`69.46% <66.42%> (-0.11%)`	⬇️
file-ft-tests-null-compat	`69.99% <66.42%> (-0.01%)`	⬇️
kmip-ft-tests	`28.03% <37.14%> (+0.03%)`	⬆️
mongo-v0-ft-tests	`70.62% <66.42%> (-0.05%)`	⬇️
mongo-v1-ft-tests	`70.62% <66.42%> (-0.04%)`	⬇️
multiple-backend	`36.03% <37.14%> (+0.01%)`	⬆️
s3c-ft-tests-v0	`64.85% <72.14%> (-0.05%)`	⬇️
s3c-ft-tests-v0-null-compat	`64.90% <72.14%> (-0.05%)`	⬇️
s3c-ft-tests-v1	`64.83% <72.14%> (-0.05%)`	⬇️
sur-tests	`35.05% <61.42%> (+0.09%)`	⬆️
sur-tests-inflights	`37.77% <61.42%> (+0.06%)`	⬆️
unit	`73.26% <87.85%> (+0.28%)`	⬆️
utapi-v2-tests	`35.22% <42.85%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

+                    if (err) {
+                        return callback(err);
+                    }
+                    return callback(null, data);
+                });
+            },
        },
-    },
-    // Function to run upon finishing both parallel requests
-    (err, results) => {
-        if (err) {
-            return cb(err);


Isolates prettier reformatting of files that were already prettier-dirty on development/9.4, so the functional change commits that follow stay prettier-clean. Issue: CLDSRV-949

bert-e · 2026-07-07T14:17:15Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

bert-e · 2026-07-08T09:58:03Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following reviewers are expecting changes from the author, or must review again:

@DarkIsDude

DarkIsDude

LGTM

bert-e · 2026-07-15T10:06:01Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

+
+async function freshStartCreateBucket(bucket, canonicalID, log, callback) {
+    if (callback) {
+        return freshStartCreateBucket(bucket, canonicalID, log).then(() => callback(null), callback);


-function cleanUpBucket(bucketMD, canonicalID, log, callback) {
+async function cleanUpBucket(bucketMD, canonicalID, log, callback) {
+    if (callback) {
+        return cleanUpBucket(bucketMD, canonicalID, log).then(() => callback(null), callback);


+        return bucketUpdateQuota(authInfo, request, log).then(
+            corsHeaders => callback(null, corsHeaders),
+            err => callback(err, err.code, err.additionalResHeaders),
+        );


benzekrimaha

I think that the current ordering still allows usage to be silently under-reported =>

the quota-enable path has a check-then-act race between the empty listing, metric initialization, and quota persistence;
bucket creation clears the transient/deleted flag before initializing the metric, exposing a similar race;
The callback trampolines also leave the new CodeQL .then() findings open; async helpers should be promise-only, with callback adaptation at their callers IMO

benzekrimaha · 2026-07-20T19:28:53Z

+    }
+    const bucketName = bucket.getName();
+    try {
+        const list = await listObjects(bucketName, { maxKeys: 1, listingType: 'DelimiterVersions' }, log);


this is check-then-act: an object can be written after this empty listing and before the quota is persisted. Since quota tracking is still disabled, that object is not reflected in Scuba, while $setOnInsert leaves the zero metric in place. This silently under-enforces the newly enabled quota until count-items runs — the same problem this PR is meant to fix. How do we prevent this race?

You're right that it's check-then-act, and there's no atomic fix here without a bucket lock, since the bucket is already live (unlike the creation path, which has the transient-flag gate we can seed behind).

We're accepting this deliberately: the seed is a best-effort floor, not an authoritative count — count-items is the source of truth and reconciles at its next run. The failure mode is one bucket briefly under-enforced until then, which is far narrower than the bug this PR fixes (Scuba's deep-health failing → quotas disabled cluster-wide, indefinitely). I've documented it inline at the seed call.

If you'd want a stronger guarantee it'd need a lock; otherwise I think accept-and-self-heal is the right tradeoff. Happy to discuss — leaving this open for your call.

bert-e · 2026-07-20T19:29:03Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following reviewers are expecting changes from the author, or must review again:

@benzekrimaha

Pull in MetadataWrapper.initializeBucketCapacity (ARSN-610) for the empty-bucket quota metric seeding. Issue: CLDSRV-949

Seed a zero-value bucket metric document in __infostore when the bucket is known to be empty, so bucket quota checks are enforceable immediately instead of waiting up to 24h for the periodic count-items job (ARTESCA-17063): - on createBucket, the bucket is empty by definition, so seed it; - on bucketUpdateQuota, seed only when the bucket is verifiably empty (a maxKeys:1 DelimiterVersions listing), never defaulting to zero for a bucket that may hold uncounted data. Both paths are gated on config.isQuotaEnabled() and are best-effort: a seeding failure is logged and never fails the request. The write uses the new arsenal MetadataWrapper.initializeBucketCapacity (idempotent $setOnInsert), keyed by the bucket's metastore creationDate to match the enforcement lookup. Issue: CLDSRV-949

Migrate bucketUpdateQuota and the bucket-creation seeding helpers (seedBucketQuotaCapacity, freshStartCreateBucket, cleanUpBucket) to async/await, using a callback trampoline so existing callers keep working. Seed the zero metric before persisting the quota to close the fail-open window, probe the MPU shadow bucket so a bucket with only in-progress uploads is not seeded to zero, log seeding failures as non-fatal warnings, and declare the promisified metadata helpers once at module scope. Issue: CLDSRV-949

scality deleted a comment from bert-e Jul 6, 2026

claude Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread package.json Outdated

github-advanced-security AI found potential problems Jul 6, 2026

View reviewed changes

Comment thread lib/api/apiUtils/bucket/bucketCreation.js Fixed

Comment thread lib/api/bucketUpdateQuota.js Fixed

Comment thread lib/api/bucketUpdateQuota.js Fixed

delthas force-pushed the improvement/CLDSRV-949/seed-empty-bucket-quota-metric branch from c463226 to 64c60e4 Compare July 6, 2026 15:13

github-advanced-security AI found potential problems Jul 6, 2026

View reviewed changes

claude Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread lib/api/bucketUpdateQuota.js Outdated

delthas force-pushed the improvement/CLDSRV-949/seed-empty-bucket-quota-metric branch 2 times, most recently from 7930f2c to 815f61c Compare July 7, 2026 08:36

github-advanced-security AI found potential problems Jul 7, 2026

View reviewed changes

Comment thread lib/api/apiUtils/bucket/bucketCreation.js Fixed

delthas force-pushed the improvement/CLDSRV-949/seed-empty-bucket-quota-metric branch 3 times, most recently from 7d8d376 to 049c059 Compare July 7, 2026 14:02

Prettier reformat files touched by CLDSRV-949

ecae692

Isolates prettier reformatting of files that were already prettier-dirty on development/9.4, so the functional change commits that follow stay prettier-clean. Issue: CLDSRV-949

delthas force-pushed the improvement/CLDSRV-949/seed-empty-bucket-quota-metric branch from 049c059 to b0c3fc9 Compare July 7, 2026 14:17

scality deleted a comment from bert-e Jul 7, 2026

delthas marked this pull request as ready for review July 7, 2026 14:21

delthas mentioned this pull request Jul 8, 2026

Add initializeBucketCapacity to seed empty-bucket metrics scality/Arsenal#2666

Merged

delthas requested review from a team, DarkIsDude and benzekrimaha July 8, 2026 09:39

delthas force-pushed the improvement/CLDSRV-949/seed-empty-bucket-quota-metric branch from b0c3fc9 to 7e7a71a Compare July 8, 2026 09:41

DarkIsDude requested changes Jul 8, 2026

View reviewed changes