Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions .agents/skills/pgpm/references/ci-cd.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,13 @@ env:
PGPASSWORD: password
```

For MinIO/S3 testing (uploads, storage):
For RustFS/S3 testing (uploads, storage):

```yaml
env:
MINIO_ENDPOINT: http://localhost:9000
AWS_ACCESS_KEY: minioadmin
AWS_SECRET_KEY: minioadmin
OBJECT_STORE_ENDPOINT: http://localhost:9000
AWS_ACCESS_KEY: constructive
AWS_SECRET_KEY: constructive-dev-secret
AWS_REGION: us-east-1
BUCKET_NAME: test-bucket
```
Expand Down Expand Up @@ -320,22 +320,22 @@ steps:
fi
```

## MinIO Service Container
## RustFS Service Container

For testing uploads and S3-compatible storage:

```yaml
services:
minio_cdn:
image: minio/minio:edge-cicd
rustfs_cdn:
image: rustfs/rustfs:1.0.0-rc.5
env:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
RUSTFS_ACCESS_KEY: constructive
RUSTFS_SECRET_KEY: constructive-dev-secret
RUSTFS_ADDRESS: ":9000"
ports:
- 9000:9000
- 9001:9001
options: >-
--health-cmd "curl -f http://localhost:9000/minio/health/live || exit 1"
--health-cmd "curl -f http://localhost:9000/health || exit 1"
--health-interval 10s
--health-timeout 5s
--health-retries 5
Expand Down
4 changes: 2 additions & 2 deletions .agents/skills/pgpm/references/environment-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,12 @@ const deployOptions = getDeploymentEnvOptions();

| Variable | Description |
|----------|-------------|
| `BUCKET_PROVIDER` | Storage provider (s3, minio, rustfs, gcs) — defaults to minio |
| `BUCKET_PROVIDER` | Storage provider (s3, minio, rustfs, gcs) — `minio` is path-style S3-compatible storage (RustFS, MinIO) |
| `BUCKET_NAME` | Bucket name |
| `AWS_REGION` | AWS region |
| `AWS_ACCESS_KEY_ID` | AWS access key |
| `AWS_SECRET_ACCESS_KEY` | AWS secret key |
| `MINIO_ENDPOINT` | S3-compatible endpoint URL (MinIO or RustFS; both listen on 9000) |
| `OBJECT_STORE_ENDPOINT` | S3-compatible endpoint URL (RustFS or MinIO; both listen on 9000) |

### Jobs Configuration

Expand Down
20 changes: 10 additions & 10 deletions .github/workflows/run-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ concurrency:
# Service tiers (avoids spinning up unneeded containers):
# unit-tests → no services (pure JS/TS)
# pg-tests → PostgreSQL only
# integration-tests → PostgreSQL + MinIO
# integration-tests → PostgreSQL + RustFS
# ai-tests → PostgreSQL + Ollama
# ---------------------------------------------------------------------------

Expand Down Expand Up @@ -333,7 +333,7 @@ jobs:
done

# =========================================================================
# TIER 3 – Integration tests (PostgreSQL + MinIO)
# TIER 3 – Integration tests (PostgreSQL + RustFS)
# =========================================================================
integration-tests:
needs: build
Expand All @@ -356,8 +356,8 @@ jobs:
PGUSER: postgres
PGPASSWORD: password
CDN_ENDPOINT: http://localhost:9000
AWS_ACCESS_KEY: minioadmin
AWS_SECRET_KEY: minioadmin
AWS_ACCESS_KEY: constructive
AWS_SECRET_KEY: constructive-dev-secret
AWS_REGION: us-east-1
# uploads/s3-streamer reads BUCKET_NAME; harmless for the others.
BUCKET_NAME: test-bucket
Expand All @@ -379,16 +379,16 @@ jobs:
ports:
- 5432:5432

minio_cdn:
image: minio/minio:edge-cicd
rustfs_cdn:
image: rustfs/rustfs:1.0.0-rc.5
env:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
RUSTFS_ACCESS_KEY: constructive
RUSTFS_SECRET_KEY: constructive-dev-secret
RUSTFS_ADDRESS: ":9000"
ports:
- 9000:9000
- 9001:9001
options: >-
--health-cmd "curl -f http://localhost:9000/minio/health/live || exit 1"
--health-cmd "curl -f http://localhost:9000/health || exit 1"
--health-interval 3s
--health-timeout 5s
--health-retries 10
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ React Query Hooks or Prisma-like ORM Client
| `graphile/` | PostGraphile plugins - filters, i18n, meta-schema, PostGIS, search, uploads, settings |
| `postgres/` | PostgreSQL utilities - introspection, testing (pgsql-test), seeding, AST, query context |
| `packages/` | Shared utilities - CLI (`cnc`), ORM base, query builder, server utils, client |
| `uploads/` | File streaming - S3/MinIO, ETags, content-type detection, UUID hashing |
| `uploads/` | File streaming - RustFS (or any S3-compatible store), ETags, content-type detection, UUID hashing |

### Key Packages & CLIs

Expand Down Expand Up @@ -115,7 +115,7 @@ Tests require PostgreSQL. Standard PG env vars:
- `PGHOST` (default: localhost), `PGPORT` (default: 5432)
- `PGUSER` (default: postgres), `PGPASSWORD` (default: password)

For S3/MinIO tests: `MINIO_ENDPOINT`, `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, `AWS_REGION`
For S3/RustFS tests: `OBJECT_STORE_ENDPOINT`, `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, `AWS_REGION`

## Build System

Expand Down
29 changes: 3 additions & 26 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,34 +12,12 @@ services:
networks:
- constructive-net

minio:
container_name: minio
image: minio/minio
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
ports:
- "9000:9000"
- "9001:9001"
expose:
- "9000"
- "9001"
command: server /data --console-address ":9001"
networks:
- constructive-net

# RustFS is an alternative to the `minio` service above: same S3 API on 9000
# and console on 9001, so the two cannot run together. Behind a profile:
# `docker compose up postgres` keeps MinIO, `docker compose --profile rustfs
# up postgres rustfs` swaps it in. Pair it with BUCKET_PROVIDER=rustfs.
rustfs:
container_name: rustfs
image: rustfs/rustfs:latest
profiles:
- rustfs
image: rustfs/rustfs:1.0.0-rc.5
environment:
- RUSTFS_ACCESS_KEY=minioadmin
- RUSTFS_SECRET_KEY=minioadmin
- RUSTFS_ACCESS_KEY=constructive
- RUSTFS_SECRET_KEY=constructive-dev-secret
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ADDRESS=:9001
- RUSTFS_CONSOLE_ENABLE=true
Expand All @@ -49,7 +27,6 @@ services:
expose:
- "9000"
- "9001"
command: /data
networks:
- constructive-net

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
/**
* Integration tests for s3-signer against a real MinIO instance.
* Integration tests for s3-signer against a real RustFS instance.
*
* These tests exercise the presigned URL pipeline end-to-end:
* 1. generatePresignedPutUrl → PUT a file via the presigned URL
* 2. headObject → verify the file exists with correct metadata
* 3. generatePresignedGetUrl → GET the file via the presigned URL
*
* Requires MinIO running on localhost:9000 (docker-compose or CI service).
* Requires RustFS running on localhost:9000 (docker-compose or CI service).
*/

import { S3Client } from '@aws-sdk/client-s3';
Expand All @@ -21,12 +21,12 @@ import {
} from '../src/s3-signer';
import type { S3Config } from '../src/types';

// --- MinIO config (matches docker-compose.yml + CI env) ---
// --- RustFS config (matches docker-compose.yml + CI env) ---

const MINIO_ENDPOINT = process.env.CDN_ENDPOINT || 'http://localhost:9000';
const OBJECT_STORE_ENDPOINT = process.env.CDN_ENDPOINT || 'http://localhost:9000';
const AWS_REGION = process.env.AWS_REGION || 'us-east-1';
const AWS_ACCESS_KEY = process.env.AWS_ACCESS_KEY || 'minioadmin';
const AWS_SECRET_KEY = process.env.AWS_SECRET_KEY || 'minioadmin';
const AWS_ACCESS_KEY = process.env.AWS_ACCESS_KEY || 'constructive';
const AWS_SECRET_KEY = process.env.AWS_SECRET_KEY || 'constructive-dev-secret';
const TEST_BUCKET = 'presigned-url-test-bucket';

// --- S3 client + config ---
Expand All @@ -37,14 +37,14 @@ const s3Client = new S3Client({
secretAccessKey: AWS_SECRET_KEY,
},
region: AWS_REGION,
endpoint: MINIO_ENDPOINT,
endpoint: OBJECT_STORE_ENDPOINT,
forcePathStyle: true,
});

const s3Config: S3Config = {
client: s3Client,
bucket: TEST_BUCKET,
endpoint: MINIO_ENDPOINT,
endpoint: OBJECT_STORE_ENDPOINT,
region: AWS_REGION,
forcePathStyle: true,
};
Expand Down Expand Up @@ -101,7 +101,7 @@ async function downloadFromPresignedUrl(url: string): Promise<{

// --- Tests ---

describe('s3-signer integration (MinIO)', () => {
describe('s3-signer integration (RustFS)', () => {
describe('generatePresignedPutUrl', () => {
it('should generate a presigned PUT URL that accepts a valid upload', async () => {
const key = 'test-put-basic.txt';
Expand Down
6 changes: 3 additions & 3 deletions graphile/graphile-presigned-url-plugin/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export interface BucketConfig {
max_file_size: number | null;
allow_custom_keys: boolean;
/**
* The physical S3/MinIO bucket name recorded by reconciliation. NULL until
* The physical S3-compatible (RustFS, MinIO) bucket name recorded by reconciliation. NULL until
* reconciliation completes. Once set, it is the source of truth for the
* physical bucket — reads never reconstruct the name.
*/
Expand Down Expand Up @@ -179,11 +179,11 @@ export interface S3Config {
client: S3Client;
/** S3 bucket name (the actual S3 bucket, not the logical bucket key) */
bucket: string;
/** S3 endpoint URL (for MinIO/custom S3) */
/** S3 endpoint URL (for RustFS, MinIO, or custom S3) */
endpoint?: string;
/** S3 region */
region?: string;
/** Whether to use path-style URLs (required for MinIO) */
/** Whether to use path-style URLs (required for path-style S3-compatible storage) */
forcePathStyle?: boolean;
/** Public URL prefix for generating download URLs */
publicUrlPrefix?: string;
Expand Down
21 changes: 11 additions & 10 deletions graphile/graphile-settings/src/upload-resolver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@
* ENV VARS (S3 connection only):
* BUCKET_PROVIDER - 'minio' | 's3' (default: 'minio')
* AWS_REGION - AWS region (default: 'us-east-1')
* AWS_ACCESS_KEY - access key (default: 'minioadmin')
* AWS_SECRET_KEY - secret key (default: 'minioadmin')
* CDN_ENDPOINT - S3-compatible endpoint (default: 'http://localhost:9000')
* Defaults come from `pgpmDefaults.cdn` (dev-only values; set these in production).
* AWS_ACCESS_KEY - access key
* AWS_SECRET_KEY - secret key
* CDN_ENDPOINT - S3-compatible endpoint
*/

import { getEnvOptions } from '@constructive-io/graphql-env';
Expand Down Expand Up @@ -61,21 +62,21 @@ let streamer: Streamer | null = null;
function getStreamer(): Streamer {
if (streamer) return streamer;

const { cdn = {} } = getEnvOptions();
const { cdn } = getEnvOptions();

if (process.env.NODE_ENV === 'production' && (!cdn.awsAccessKey || !cdn.awsSecretKey)) {
log.warn('[upload-resolver] WARNING: Using default credentials in production.');
log.warn('[upload-resolver] WARNING: CDN credentials not configured in production.');
}

const provider = cdn.provider || 'minio';
const provider = cdn.provider;
log.info(`[upload-resolver] Initializing: provider=${provider}`);

streamer = new Streamer({
provider,
awsRegion: cdn.awsRegion || 'us-east-1',
awsAccessKey: cdn.awsAccessKey || 'minioadmin',
awsSecretKey: cdn.awsSecretKey || 'minioadmin',
endpoint: cdn.endpoint || 'http://localhost:9000',
awsRegion: cdn.awsRegion,
awsAccessKey: cdn.awsAccessKey,
awsSecretKey: cdn.awsSecretKey,
endpoint: cdn.endpoint,
});

return streamer;
Expand Down
4 changes: 2 additions & 2 deletions graphql/env/__tests__/__snapshots__/merge.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ exports[`getEnvOptions merges pgpm defaults, graphql defaults, config, env, and
"routingSchema": "routing_public",
},
"cdn": {
"awsAccessKey": "minioadmin",
"awsAccessKey": "constructive",
"awsRegion": "us-east-1",
"awsSecretKey": "minioadmin",
"awsSecretKey": "constructive-dev-secret",
"bucketName": "test-bucket",
"endpoint": "http://localhost:9000",
"provider": "minio",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
* (`buckets`/`files`), because the plane *is* that database's storage. Both must
* produce an upload surface, so this pins the two together: one schema build over
* both planes has to expose `uploadAppFile` AND `uploadFile`, and the unprefixed
* one has to work end to end — presigned PUT to MinIO, and a physical bucket
* one has to work end to end — presigned PUT to RustFS, and a physical bucket
* recorded on the tenant's own bucket row.
*
* Uses real MinIO (the `minio_cdn` service in CI, localhost:9000 locally).
* Uses real RustFS (the `rustfs_cdn` service in CI, localhost:9000 locally).
*
* pnpm test -- --testPathPattern=db-scope-upload
*/
Expand Down Expand Up @@ -103,8 +103,8 @@ describe('database-scope upload surface', () => {
provider: 'minio',
region: 'us-east-1',
endpoint: 'http://localhost:9000',
accessKeyId: 'minioadmin',
secretAccessKey: 'minioadmin'
accessKeyId: 'constructive',
secretAccessKey: 'constructive-dev-secret'
});

const post = (
Expand Down Expand Up @@ -170,7 +170,7 @@ describe('database-scope upload surface', () => {
let contentHash: string;
let uploadUrl: string;

// MinIO uses path-style URLs: http://host:9000/<bucket>/<key>?...
// RustFS uses path-style URLs: http://host:9000/<bucket>/<key>?...
const bucketFromPresignedUrl = (url: string): string =>
new URL(url).pathname.replace(/^\/+/, '').split('/')[0];

Expand Down
8 changes: 4 additions & 4 deletions graphql/server-test/__tests__/upload.integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
* Exercises the file-centric upload pipeline:
* uploadAppFile mutation -> presigned PUT URL -> PUT to S3
*
* Uses real MinIO (available in CI as minio_cdn service) and reconciled
* Uses real RustFS (available in CI as rustfs_cdn service) and reconciled
* physical bucket fixtures.
*
* Three actors (single beforeAll, single server -- stays fast):
Expand Down Expand Up @@ -284,8 +284,8 @@ describe('Integration tests (uploads, tenant isolation, RLS)', () => {
provider: 'minio',
region: 'us-east-1',
endpoint: 'http://localhost:9000',
accessKeyId: 'minioadmin',
secretAccessKey: 'minioadmin'
accessKeyId: 'constructive',
secretAccessKey: 'constructive-dev-secret'
});

const ensureS3Buckets = async (bucketNames: string[]): Promise<void> => {
Expand Down Expand Up @@ -506,7 +506,7 @@ describe('Integration tests (uploads, tenant isolation, RLS)', () => {
return res.rows[0]?.physical_name ?? null;
};

// MinIO uses path-style URLs: http://host:9000/<bucket>/<key>?...
// RustFS uses path-style URLs: http://host:9000/<bucket>/<key>?...
const bucketFromPresignedUrl = (url: string): string =>
new URL(url).pathname.replace(/^\/+/, '').split('/')[0];

Expand Down
Loading
Loading