Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 15 additions & 20 deletions browser/data-browser/src/chunks/PluginRuns/FileImport.tsx
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
import { useEffect, useId, useState } from 'react';
import {
acceptFor,
core,
dataBrowser,
ensureSchema,
executeServerPlugin,
pluginConfigFor,
pluginConfigProblems,
readUpload,
useStore,
DEFAULT_ACCEPT_MAX_BYTES,
type DeclaredAccept,
Expand All @@ -28,9 +30,11 @@ import { RunPluginDialog } from './RunPluginDialog';
* The generic entry point for a plugin that declares `accepts`: choose a file,
* preview what the plugin proposes for it, approve.
*
* The host owns acquisition: it enforces the declared size, decodes the file
* and hands it over as `input.upload`. The plugin runs on the server (it may
* need `ctx.query`/`ctx.read` to recognise earlier imports), and nothing is
* The host owns acquisition: it enforces the declared size on the file's
* bytes, reads the file as its `accepts` entry declares (decoded text, or
* base64 of the exact bytes) and hands it over as `input.upload`. The plugin
* runs on the server (it may need `ctx.query`/`ctx.read` to recognise earlier
* imports), and nothing is
* written until the proposal is approved in {@link RunPluginDialog}. A plugin
* that also declares a `destination` gets a Set up step first, which creates
* that table and stores it as the plugin's config.
Expand Down Expand Up @@ -105,23 +109,23 @@ export function FileImport({
setImported(false);

try {
const max = maxBytes(accepts);
const accept = acceptFor(file, accepts);
if (!accept) throw new Error('This plugin does not accept files.');
// The server bounds an upload by the entries sharing its encoding.
const max = maxBytes(
accepts.filter(other => (other.as ?? 'text') === (accept.as ?? 'text')),
);
if (file.size > max)
throw new Error(
`This file is ${formatBytes(file.size)}; this importer accepts at most ${formatBytes(max)}. Export a shorter period.`,
);
const text = decode(await file.arrayBuffer());
const upload = readUpload(file, await file.arrayBuffer(), accept);
const result = await executeServerPlugin(store, {
drive,
plugin: resource.subject,
source,
input: {
upload: {
name: file.name,
mediaType: file.type,
size: file.size,
text,
},
upload,
config,
trigger: {
kind: 'manual',
Expand Down Expand Up @@ -233,15 +237,6 @@ function formatBytes(bytes: number): string {
return `${bytes} bytes`;
}

/** UTF-8 when the file is valid UTF-8; older bank exports are often Windows-1252. */
export function decode(bytes: ArrayBuffer): string {
try {
return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
return new TextDecoder('windows-1252').decode(bytes);
}
}

async function storedConfig(
store: Store,
drive: string,
Expand Down
8 changes: 8 additions & 0 deletions browser/lib/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ export {
type DeclaredConfig,
type DeclaredConfigField,
type DeclaredAccept,
type AcceptEncoding,
type DeclaredDestination,
DEFAULT_ACCEPT_MAX_BYTES,
ACCEPT_MAX_BYTES_CEILING,
Expand All @@ -214,6 +215,13 @@ export {
type PluginManifest,
type PluginManifestV2,
} from './plugin-manifest.js';
export {
acceptFor,
readUpload,
decodeUploadText,
encodeBase64,
type PluginUpload,
} from './plugin-upload.js';
export {
pluginConfigFor,
pluginConfigProblems,
Expand Down
38 changes: 29 additions & 9 deletions browser/lib/src/plugin-manifest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ export interface DeclaredConfig {
required?: string[];
}

/** How an accepted file reaches the plugin; see {@link DeclaredAccept}. */
export type AcceptEncoding = 'text' | 'base64';

/** What the host accepts when no `maxBytes` is declared: 5 MiB. */
export const DEFAULT_ACCEPT_MAX_BYTES = 5 * 1024 * 1024;
/**
Expand All @@ -75,17 +78,29 @@ export const ACCEPT_MAX_BYTES_CEILING = 20 * 1024 * 1024;
/**
* A file a plugin can be handed by the host, instead of fetching data itself.
*
* The host draws the picker, enforces `maxBytes`, decodes the file and passes
* it as `input.upload` = `{ name, mediaType, size, text }`. `extensions` and
* `mediaTypes` only filter the picker; the plugin must still validate the
* content it is given. Only `as: 'text'` exists so far: UTF-8, falling back to
* Windows-1252.
* The host draws the picker, enforces `maxBytes` on the file's raw size and
* passes it as `input.upload` (`ctx.upload` in `run`), in the declared
* encoding, under a field named after it:
*
* - `as: 'text'` (the default when `as` is left out):
* `{ name, mediaType, size, text }`, decoded as UTF-8 and falling back to
* Windows-1252 when the file is not valid UTF-8.
* - `as: 'base64'`: `{ name, mediaType, size, base64 }`, the file's exact bytes
* base64-encoded (standard alphabet, padded), with no charset detection.
*
* `size` is always the byte size of the file. `extensions` and `mediaTypes`
* only filter the picker; the plugin must still validate the content it is
* given. See `PluginUpload` in `plugin-upload.ts`.
*/
export interface DeclaredAccept {
/** Lower-case, with the leading dot: `.xml`. */
extensions?: string[];
mediaTypes?: string[];
as: 'text';
/**
* How the host hands over the file. Left out, it is `text`, and it stays
* left out when serialized, so a release's id does not change.
*/
as?: AcceptEncoding;
/** Bytes, at most {@link ACCEPT_MAX_BYTES_CEILING}. */
maxBytes?: number;
}
Expand Down Expand Up @@ -662,8 +677,12 @@ export function validateManifest(raw: unknown): PluginManifest {
const accepts = list(entry.accepts, 'accepts').map(value => {
const accept = object(value, 'accepts entry');
known(accept, ['extensions', 'mediaTypes', 'as', 'maxBytes']);
if (accept.as !== 'text')
throw new Error('accepts entries must be read `as` text');
if (
accept.as !== undefined &&
accept.as !== 'text' &&
accept.as !== 'base64'
)
throw new Error('accepts entries must be read `as` text or base64');
const extensions = list(accept.extensions, 'accepts.extensions').map(
extension => {
if (
Expand Down Expand Up @@ -702,7 +721,8 @@ export function validateManifest(raw: unknown): PluginManifest {
return {
...(extensions.length ? { extensions } : {}),
...(mediaTypes.length ? { mediaTypes } : {}),
as: 'text' as const,
// Kept exactly as declared: an absent `as` stays absent.
...(accept.as !== undefined ? { as: accept.as as AcceptEncoding } : {}),
...(accept.maxBytes !== undefined
? { maxBytes: accept.maxBytes as number }
: {}),
Expand Down
92 changes: 92 additions & 0 deletions browser/lib/src/plugin-upload.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import { describe, expect, it } from 'vitest';
import { acceptFor, readUpload } from './plugin-upload.js';
import type { DeclaredAccept } from './plugin-manifest.js';

const everyByte = Uint8Array.from({ length: 256 }, (_, i) => i);

describe('readUpload', () => {
it('hands a base64 plugin the exact bytes, every value 0–255', () => {
const upload = readUpload(
{ name: 'drop.willow', type: '' },
everyByte.buffer,
{ as: 'base64' },
);

expect(upload).toEqual({
name: 'drop.willow',
mediaType: '',
size: 256,
base64: Buffer.from(everyByte).toString('base64'),
});
expect('text' in upload).toBe(false);
expect(
Uint8Array.from(
Buffer.from((upload as { base64: string }).base64, 'base64'),
),
).toEqual(everyByte);
});

it('encodes a file larger than one chunk without losing bytes', () => {
const large = Uint8Array.from({ length: 100_003 }, (_, i) => (i * 7) & 255);
const upload = readUpload({ name: 'x', type: '' }, large, {
as: 'base64',
}) as { base64: string };

expect(Uint8Array.from(Buffer.from(upload.base64, 'base64'))).toEqual(
large,
);
});

it('keeps text mode unchanged for MT940, with `as` given or left out', () => {
const mt940 =
':20:STARTUMSE\r\n:25:NL91ABNA0417164300\r\n:28C:00001/001\r\n' +
':60F:C240101EUR1000,00\r\n:86:Café\r\n';
const utf8 = new TextEncoder().encode(mt940);
// The same statement exported as Windows-1252: é is the single byte 0xE9.
const cp1252 = Uint8Array.from(mt940, c => c.charCodeAt(0));

for (const accept of [{ as: 'text' }, {}] as DeclaredAccept[]) {
for (const bytes of [utf8, cp1252]) {
const upload = readUpload(
{ name: 'statement.sta', type: 'text/plain' },
bytes,
accept,
);
expect(upload).toEqual({
name: 'statement.sta',
mediaType: 'text/plain',
size: bytes.byteLength,
text: mt940,
});
}
}
});
});

describe('acceptFor', () => {
const accepts: DeclaredAccept[] = [
{ extensions: ['.sta'], as: 'text' },
{
extensions: ['.willow'],
mediaTypes: ['application/x-willow'],
as: 'base64',
},
];

it('picks the entry that names the file', () => {
expect(acceptFor({ name: 'A.STA', type: '' }, accepts)).toBe(accepts[0]);
expect(acceptFor({ name: 'drop.willow', type: '' }, accepts)).toBe(
accepts[1],
);
expect(
acceptFor({ name: 'drop', type: 'application/x-willow' }, accepts),
).toBe(accepts[1]);
});

it('falls back to the first entry when none names the file', () => {
expect(acceptFor({ name: 'other.bin', type: '' }, accepts)).toBe(
accepts[0],
);
expect(acceptFor({ name: 'x', type: '' }, [])).toBeUndefined();
});
});
70 changes: 70 additions & 0 deletions browser/lib/src/plugin-upload.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import type { DeclaredAccept } from './plugin-manifest.js';

/**
* A file as a plugin receives it, in `input.upload` (`ctx.upload` in `run`).
* The field that carries the content is named after the encoding the
* matching `accepts` entry declared, so a plugin can tell them apart: `text`
* for `as: 'text'` (the default), `base64` for `as: 'base64'`. `size` is the
* file's byte size either way.
*/
export type PluginUpload = {
name: string;
mediaType: string;
size: number;
} & ({ text: string } | { base64: string });

/**
* The `accepts` entry a chosen file falls under: the first that names its
* extension, then the first that names its media type, else the first entry
* (the picker's filter is only a hint, so a file may match none).
*/
export function acceptFor(
file: { name: string; type: string },
accepts: DeclaredAccept[],
): DeclaredAccept | undefined {
const name = file.name.toLowerCase();

return (
accepts.find(accept =>
accept.extensions?.some(extension => name.endsWith(extension)),
) ??
accepts.find(
accept => file.type && accept.mediaTypes?.includes(file.type),
) ??
accepts[0]
);
}

/** Builds `input.upload` from a file's bytes, as `accept` declares. */
export function readUpload(
file: { name: string; type: string },
bytes: ArrayBuffer | Uint8Array,
accept: DeclaredAccept,
): PluginUpload {
const view = bytes instanceof Uint8Array ? bytes : new Uint8Array(bytes);
const meta = { name: file.name, mediaType: file.type, size: view.byteLength };

return accept.as === 'base64'
? { ...meta, base64: encodeBase64(view) }
: { ...meta, text: decodeUploadText(view) };
}

/** UTF-8 when the file is valid UTF-8; older bank exports are often Windows-1252. */
export function decodeUploadText(bytes: ArrayBuffer | Uint8Array): string {
try {
return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
return new TextDecoder('windows-1252').decode(bytes);
}
}

/** Standard, padded base64 of exact bytes. Chunked: files run to megabytes. */
export function encodeBase64(bytes: Uint8Array): string {
const chunk = 0x8000;
let binary = '';

for (let i = 0; i < bytes.length; i += chunk)
binary += String.fromCharCode(...bytes.subarray(i, i + chunk));

return btoa(binary);
}
5 changes: 5 additions & 0 deletions docs/src/plugins/creating-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,11 @@ Every field except `schemaVersion` is optional; unknown fields and malformed dec
- `secrets`, `operations`, `actions`: as in schema version 1. Secrets name an exact origin a credential may be sent to; operations are exact endpoints with an `effect` of `read` or `write`; actions reference operations.
- `network.origins`: exact origins (no wildcards, paths or ports beyond the origin) for packages that call the host `fetch` without an operation id. It never widens what `operations` grant.
- `configSchema`, `defaultConfig`: objects, as in `plugin.json`.
- `accepts`: files the host may hand the plugin, up to eight entries of `{ extensions?, mediaTypes?, as?, maxBytes? }`. `extensions` (lower-case, with the dot) and `mediaTypes` only filter the file picker. `maxBytes` (default 5 MiB, at most 20 MiB) bounds the file's raw byte size. `as` says how the file arrives in `ctx.upload`, in a field named after it:
- `"text"`, the default when `as` is left out: `{ name, mediaType, size, text }`, decoded as UTF-8, falling back to Windows-1252.
- `"base64"`: `{ name, mediaType, size, base64 }`, the file's exact bytes in standard padded base64, with no charset detection. `atob(ctx.upload.base64)` gives it back as a binary string, one character per byte.

`size` is the byte size of the file in both cases. Leaving `as` out does not add it to the stored manifest, so the release id stays the same.
- `name`, `namespace`, `version`, `description`, `author`: metadata. `name` and `namespace` must be safe path segments.

A schema version 1 manifest is still accepted and is read as version 2 with `runtime: atomic-js/1`, `world: extension` and `entrypoints: { run: true }`.
Expand Down
Loading
Loading