Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6aa5166
refactor(network): introduce shared egress pipeline
johntmyers Jul 14, 2026
939c6e5
test(network): cover shared proxy egress paths
johntmyers Jul 14, 2026
918a72d
refactor(network): make destination authorization explicit
johntmyers Jul 15, 2026
f90a4fc
refactor(network): pin proxy relay policy context
johntmyers Jul 15, 2026
fae4b5e
test(network): lock relay generation contracts
johntmyers Jul 15, 2026
956e4fd
test(network): establish phase zero compatibility baseline
johntmyers Jul 16, 2026
8c458ab
feat(policy): detect ambiguous network endpoints
johntmyers Jul 20, 2026
64d7781
feat(sandbox): fail closed on invalid policy updates
johntmyers Jul 20, 2026
0745cd9
refactor(network): invalidate relays on policy changes
johntmyers Jul 20, 2026
fbe6356
docs(policy): document validation failure posture
johntmyers Jul 20, 2026
d823923
test(network): cover validation and middleware egress
johntmyers Jul 20, 2026
438627e
fix(config): move policy failure mode to gateway toml
johntmyers Jul 20, 2026
0e7efe2
test(network): name proxy contracts by behavior
johntmyers Jul 21, 2026
e593f8c
fix(network): align overlap validation with endpoint selection
johntmyers Jul 23, 2026
20ab44b
test(network): expect hard loopback denial
johntmyers Jul 23, 2026
6c1022f
test(network): match declared endpoint denial
johntmyers Jul 23, 2026
8b5e728
fix(policy): preserve path-specific endpoint overrides
johntmyers Jul 23, 2026
ad25d9e
test(network): respect hard-blocked host gateways
johntmyers Jul 23, 2026
b913aa0
fix(network): reconcile proxy refactor with main
johntmyers Jul 27, 2026
7d9470c
fix(network): preserve CONNECT policy generation
johntmyers Jul 27, 2026
5844386
fix(policy): cover runtime endpoint glob semantics
johntmyers Jul 27, 2026
5adac68
fix(server): reject ambiguous policies before persistence
johntmyers Jul 27, 2026
b2e79aa
docs(policy): explain ambiguity preflight behavior
johntmyers Jul 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,30 @@ The middleware service must start before the gateway and be reachable from both

At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the request, while `fail_open` bypasses only that stage and emits a detection finding. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event.

For network policy validation failures, first distinguish a gateway mutation
rejection from a supervisor runtime rejection. Direct policy updates,
incremental merges and approvals, provider attachments, and provider-profile
fanout are validated against the complete effective policy before persistence
when the gateway knows the affected sandbox scope. A `FAILED_PRECONDITION`
ambiguity response means no invalid revision or partial fanout was stored.
Supervisor validation remains defense in depth for startup, races, and policy
sources outside those mutation paths.

Runtime rejection behavior is configured only in `gateway.toml`:

```toml
[openshell.gateway]
policy_validation_failure_mode = "fail_closed"
```

The default `fail_closed` mode deactivates the previous generation, closes
pinned relays, and quarantines new egress until a valid generation loads.
`retain_last_valid` explicitly keeps the previous valid policy active; without
one it still fails closed. Restart the gateway after changing this field.
Inspect sandbox OCSF configuration and finding events for the validation
rationale, configured and effective modes, active generation, and the explicit
`previous_policy_active` state.

### Step 4: Check Docker-Backed Gateways

```bash
Expand Down Expand Up @@ -400,6 +424,8 @@ openshell logs <sandbox-name>
| Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `builtin` and `user` when intended |
| Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response |
| Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields |
| Policy mutation returns `FAILED_PRECONDITION` for endpoint ambiguity | Equally specific effective endpoint selectors disagree on connection or request-processing metadata | CLI error, base and provider-composed policy, affected profile attachments; confirm no new revision was stored |
| Supervisor enters policy quarantine | A runtime candidate failed validation while `policy_validation_failure_mode = "fail_closed"` | Sandbox OCSF config/finding events, validation rationale, active generation, `previous_policy_active` |
| HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` |
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
Expand Down
7 changes: 5 additions & 2 deletions .agents/skills/generate-sandbox-policy/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,10 @@ Only needed for the **Moderate** and **Full** tiers. Translate API path paramete
| `/api/v1/models/{model_id}/versions/{version}` | `/api/v1/models/*/versions/*` |
| All sub-paths under `/api/v1/` | `/api/v1/**` |

Remember: `*` does not cross `/` boundaries. Use `**` for recursive matching across path segments.
Path matching uses the runtime `glob` engine. Both `*` and `**` may cross `/`
boundaries; `?` matches one character, and bracket classes such as `[0-9]` and
`[!0]` are supported. Prefer segment-shaped patterns such as
`/repos/*/issues` for readability, but do not rely on `*` to stop at `/`.

### Building the Explicit Rules List

Expand Down Expand Up @@ -439,7 +442,7 @@ The policy needs to go somewhere. Determine which mode applies:

2. **Check for conflicts**:
- Does a policy with the same key already exist? If so, ask the user whether to **replace** it, **merge** new endpoints/binaries into it, or use a different key.
- Does an existing policy already cover the same host:port? Warn the user — overlapping endpoint coverage across policies causes OPA evaluation errors (complete rule conflict).
- Does an existing endpoint selector overlap the new selector? Compatible overlaps are allowed and can intentionally aggregate allow and deny rules. Reject or revise equally specific overlaps that disagree on connection or request-processing metadata, including TLS, destination constraints, protocol/parser behavior, enforcement, or credential handling. A more-specific path selector may override broader request-processing metadata.

3. **Apply the change**:
- **Adding a new policy**: Insert the new policy block under `network_policies`, maintaining the file's existing indentation and style.
Expand Down
4 changes: 3 additions & 1 deletion .agents/skills/generate-sandbox-policy/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -727,7 +727,9 @@ An exact IP is treated as `/32` — only that specific address is permitted.
**Agent workflow**:

1. Read `sandbox-policy.yaml`
2. Check that no existing policy already covers `api.github.com:443` — if one does, warn about overlap
2. Check existing selectors for `api.github.com:443`. Compatible overlaps may
aggregate request rules; revise equally specific overlaps that disagree on
TLS, destination, protocol/parser, enforcement, or credential behavior.
3. Check that the key `github_readonly` doesn't already exist
4. Insert the new policy under `network_policies`:

Expand Down
14 changes: 14 additions & 0 deletions .agents/skills/openshell-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,13 @@ Edit `current-policy.yaml` to allow the blocked actions. **For policy content au
openshell policy set dev --policy current-policy.yaml --wait
```

The gateway validates the complete effective candidate—including attached
provider-profile policy—before it stores a direct update, incremental merge,
approved proposal, provider attachment, or profile update that affects attached
sandboxes. An ambiguity failure returns `FAILED_PRECONDITION`; the rejected
candidate does not create a policy revision or partially update affected
sandboxes. Fix the conflicting endpoint selectors and submit again.

The `--wait` flag blocks until the sandbox confirms the policy is loaded (polls every second). Exit codes:
- **0**: Policy loaded successfully
- **1**: Policy load failed
Expand Down Expand Up @@ -569,6 +576,13 @@ openshell settings set --global --key providers_v2_enabled --value true

Global mutations prompt for confirmation. Use `--yes` only in reviewed automation.

`policy_validation_failure_mode` is gateway startup configuration, not a
mutable `openshell settings` key. Set it under `[openshell.gateway]` in
`gateway.toml` and restart the gateway. The security-first default is
`fail_closed`; `retain_last_valid` is an explicit availability tradeoff. OCSF
configuration events state whether the previous generation is active after a
runtime validation failure.

## Workflow 10: Service Access

Use `forward` for local access and `service` for a gateway-managed HTTP endpoint:
Expand Down
15 changes: 15 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,21 @@ unsafe internal destinations, and evaluates the active policy. On Linux, it
maps an accepted proxy connection back to the workload socket by matching the
complete local-to-remote TCP tuple before resolving every process that owns the
socket inode.

CONNECT and absolute-form forward HTTP are explicit-proxy adapters over the same
egress pipeline. Each adapter normalizes its request into an egress intent, and
the shared authorization result carries the process evidence used by destination
validation and relay selection. During the compatibility migration, endpoint
state is hydrated at the adapters' existing policy query points; it is not yet
one atomic, generation-consistent authorization result. Destination validation
returns an unopened connector so adapters retain their existing response and
upstream-dial timing. CONNECT prepares a generation-pinned relay context before
entering shared TLS-terminated or plaintext HTTP relays; non-HTTP traffic uses
the shared raw byte relay after the existing adapter gates. Forward HTTP retains
its guarded single-request relay while sharing authorization, request context,
policy-pinning, and destination boundaries.
Adapter-specific response and OCSF event shapes remain at the protocol boundary.

For inspected HTTP traffic, the proxy can enforce REST method/path rules,
WebSocket upgrade and text-message rules, GraphQL operation rules, and
MCP method, tool, and supported params rules or generic JSON-RPC method rules
Expand Down
41 changes: 34 additions & 7 deletions architecture/security-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,9 @@ metadata before forwarding. The proxy also supports credential injection on
terminated HTTP streams when policy allows the endpoint.

Raw streams and long-lived response bodies are connection scoped. Policy
reloads affect the next connection or the next parsed HTTP request; they do not
rewrite bytes already being relayed. HTTP upgrades switch to raw relay by
default. A `protocol: rest` endpoint can opt in to
generation changes close relays pinned to the previous generation instead of
allowing them to continue under stale authorization. HTTP upgrades switch to
raw relay by default. A `protocol: rest` endpoint can opt in to
`websocket_credential_rewrite` for client-to-server WebSocket text messages
after an allowed `101` upgrade; server-to-client traffic and all other upgraded
protocols remain raw passthrough.
Expand All @@ -98,10 +98,37 @@ supervisor polls for config revisions and attempts to load new dynamic policy
into the in-process OPA engine; CLI reads of the latest sandbox policy use the
same effective configuration path.

If a new policy fails validation or loading, the supervisor reports the failure
and keeps the last-known-good policy. Static controls, such as filesystem
allowlists and process identity, require a new sandbox because they are applied
before the child process starts.
The supervisor validates complete effective policy generations before
activation. Overlapping endpoint selectors may contribute request allow and
deny rules only when their connection and request-processing metadata agree;
conflicting TLS, destination, credential, parser, or enforcement metadata
rejects the complete generation. Plain L4 endpoints do not contribute
request-processing metadata, so they may overlap an L7 endpoint when their
connection metadata agrees. When request paths overlap, a path endpoint with a
higher specificity rank deterministically overrides broader request-processing
metadata. Equally specific overlapping endpoints must agree.

Gateway mutation paths validate the complete effective candidate before
persistence when the affected sandbox scope is known. Direct replacements,
incremental merges and approvals, provider attachment, and profile fanout reject
ambiguity atomically, without creating an invalid revision or partially
activating an update. Supervisor validation remains the defense-in-depth
boundary for startup, concurrent changes, and sources outside those mutations.

The `[openshell.gateway] policy_validation_failure_mode` configuration controls
candidates rejected by supervisor runtime validation. Gateway preflight
rejections never become generations and leave the active policy unchanged. The
runtime mode defaults to `fail_closed`, which publishes a quarantine generation,
denies new egress, invalidates existing relays, and leaves the previous policy
inactive. Operators may explicitly select
`retain_last_valid`, which keeps the previous generation active. With no
previous valid generation, the effective mode remains `fail_closed` regardless
of the configured mode. The gateway distributes this startup configuration to
sandbox supervisors with each effective policy snapshot. OCSF configuration and finding events state the
candidate version, validation rationale, configured and effective modes, active
generation, and whether the previous policy is active. Static controls,
such as filesystem allowlists and process identity, require a new sandbox
because they are applied before the child process starts.

Gateway-global policy can override sandbox-scoped policy. Use it sparingly
because it changes the effective access model for every sandbox on the gateway.
Expand Down
64 changes: 60 additions & 4 deletions crates/openshell-core/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,43 @@ pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker";
/// Default domain used for browser-facing sandbox service URLs.
pub const DEFAULT_SERVICE_ROUTING_DOMAIN: &str = "openshell.localhost";

/// Gateway posture when a sandbox rejects a candidate policy generation.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum PolicyValidationFailureMode {
/// Deactivate the previous policy and deny new egress until a valid
/// generation is loaded.
#[default]
FailClosed,
/// Keep the last valid generation active when a newer candidate fails
/// validation. Startup still fails closed when no valid generation exists.
RetainLastValid,
}

impl PolicyValidationFailureMode {
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::FailClosed => "fail_closed",
Self::RetainLastValid => "retain_last_valid",
}
}
}

impl FromStr for PolicyValidationFailureMode {
type Err = String;

fn from_str(value: &str) -> Result<Self, Self::Err> {
match value {
"fail_closed" => Ok(Self::FailClosed),
"retain_last_valid" => Ok(Self::RetainLastValid),
_ => Err(format!(
"invalid policy validation failure mode '{value}'; expected fail_closed or retain_last_valid"
)),
}
}
}

/// Default OCI repository for the supervisor image (no tag).
pub const DEFAULT_SUPERVISOR_IMAGE_REPO: &str = "ghcr.io/nvidia/openshell/supervisor";

Expand Down Expand Up @@ -396,6 +433,9 @@ pub struct Config {
/// Log level (trace, debug, info, warn, error).
pub log_level: String,

/// Security posture for rejected sandbox policy generations.
pub policy_validation_failure_mode: PolicyValidationFailureMode,

/// TLS configuration. When `None`, the server listens on plaintext HTTP.
pub tls: Option<TlsConfig>,

Expand Down Expand Up @@ -737,6 +777,7 @@ impl Config {
health_bind_address: None,
metrics_bind_address: None,
log_level: default_log_level(),
policy_validation_failure_mode: PolicyValidationFailureMode::default(),
tls,
oidc: None,
auth: GatewayAuthConfig::default(),
Expand Down Expand Up @@ -981,10 +1022,10 @@ mod tests {
use super::{
ComputeDriverKind, Config, DEFAULT_SERVICE_ROUTING_DOMAIN, GatewayInterceptorBindingPolicy,
GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, GatewayJwtConfig,
GatewayProviderProfileSourceConfig, detect_docker_socket_from_candidates, detect_driver,
detect_podman_socket_from_candidates, docker_host_unix_socket_path, docker_socket_responds,
is_unix_socket, normalize_compute_driver_name, podman_socket_candidates_from_env,
podman_socket_responds,
GatewayProviderProfileSourceConfig, PolicyValidationFailureMode,
detect_docker_socket_from_candidates, detect_driver, detect_podman_socket_from_candidates,
docker_host_unix_socket_path, docker_socket_responds, is_unix_socket,
normalize_compute_driver_name, podman_socket_candidates_from_env, podman_socket_responds,
};
#[cfg(unix)]
use std::io::{Read as _, Write as _};
Expand Down Expand Up @@ -1020,6 +1061,21 @@ mod tests {
assert!(err.contains("unsupported compute driver 'firecracker'"));
}

#[test]
fn policy_validation_failure_mode_is_secure_by_default() {
assert_eq!(
Config::new(None).policy_validation_failure_mode,
PolicyValidationFailureMode::FailClosed
);
assert_eq!(
"retain_last_valid"
.parse::<PolicyValidationFailureMode>()
.unwrap(),
PolicyValidationFailureMode::RetainLastValid
);
assert!("keep_old".parse::<PolicyValidationFailureMode>().is_err());
}

#[test]
fn compute_driver_name_normalization_accepts_builtin_and_custom_names() {
assert_eq!(normalize_compute_driver_name(" VM ").unwrap(), "vm");
Expand Down
37 changes: 37 additions & 0 deletions crates/openshell-core/src/grpc_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -780,6 +780,8 @@ pub struct SettingsPollResult {
pub supervisor_middleware_services: Vec<crate::proto::SupervisorMiddlewareService>,
/// Workspace the sandbox belongs to.
pub workspace: String,
/// Gateway-configured posture for rejected policy generations.
pub policy_validation_failure_mode: crate::PolicyValidationFailureMode,
}

fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> SettingsPollResult {
Expand All @@ -795,6 +797,41 @@ fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> Settin
provider_env_revision: inner.provider_env_revision,
supervisor_middleware_services: inner.supervisor_middleware_services,
workspace: inner.workspace,
policy_validation_failure_mode: inner
.policy_validation_failure_mode
.parse()
.unwrap_or_default(),
}
}

#[cfg(test)]
mod settings_poll_tests {
use super::settings_poll_result;
use crate::PolicyValidationFailureMode;
use crate::proto::GetSandboxConfigResponse;

#[test]
fn validation_failure_mode_round_trips_from_gateway_config() {
let result = settings_poll_result(GetSandboxConfigResponse {
policy_validation_failure_mode: "retain_last_valid".to_string(),
..Default::default()
});
assert_eq!(
result.policy_validation_failure_mode,
PolicyValidationFailureMode::RetainLastValid
);
}

#[test]
fn unknown_validation_failure_mode_fails_closed() {
let result = settings_poll_result(GetSandboxConfigResponse {
policy_validation_failure_mode: "future_mode".to_string(),
..Default::default()
});
assert_eq!(
result.policy_validation_failure_mode,
PolicyValidationFailureMode::FailClosed
);
}
}

Expand Down
2 changes: 1 addition & 1 deletion crates/openshell-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ pub use config::{
ComputeDriverKind, Config, GatewayAuthConfig, GatewayInterceptorBindingOverride,
GatewayInterceptorBindingPolicy, GatewayInterceptorConfig, GatewayInterceptorFailurePolicy,
GatewayInterceptorPhaseConfig, GatewayJwtConfig, GatewayProviderProfileSourceConfig,
MtlsAuthConfig, OidcConfig, TlsConfig,
MtlsAuthConfig, OidcConfig, PolicyValidationFailureMode, TlsConfig,
};
pub use error::{ComputeDriverError, Error, Result};
pub use metadata::{
Expand Down
Loading
Loading