@chamvi/sandbox-k8s

One pod per workspace, for chamvi agents on Kubernetes.
Root inside so package managers work, a few capabilities, no service account token, resource limits,
found again by label after a restart or on another replica.

Table of Contents

• Installation
• Usage
• RBAC
• The pod
• Image
• Security
• Timeouts and abort
• Garbage collection
• Options
• API
• License

Installation

npm install @chamvi/sandbox-k8s @chamvi/sandbox @chamvi/core ai --save

ESM only, Node.js >=26.0.0. @chamvi/core and @chamvi/sandbox are peer dependencies. The API server is reached through @kubernetes/client-node 2 (CoreV1Api and Exec over a WebSocket). It declares @types/node ^26 as a runtime dependency, so a project on older Node.js types gets a nested copy; that is harmless with skipLibCheck.

Usage

import type { KubeConfig } from '@kubernetes/client-node';
import type { ISandboxProvider } from '@chamvi/sandbox';
import { createKubernetesSandbox } from '@chamvi/sandbox-k8s';

export function createSandbox(kubeConfig: KubeConfig): ISandboxProvider {
    // one pod per workspace in this namespace; nothing is contacted before the first create()
    return createKubernetesSandbox({ namespace: 'agents', kubeConfig });
}

Without kubeConfig the provider calls loadFromDefault(): the in-cluster service account when chamvi runs in a pod, else KUBECONFIG or ~/.kube/config. Pass the provider to createWorkspaceManager({ provider }) from @chamvi/sandbox. Creating it contacts nothing; create() creates the pod, waits until it runs (polling every second, at most two minutes, failing fast on ErrImagePull, ImagePullBackOff, InvalidImageName, CreateContainerConfigError and CreateContainerError), then checks out.

RBAC

The identity chamvi runs as needs, in the sandbox namespace only:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: chamvi-sandbox
  namespace: agents
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "list", "delete"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["get", "create"]
  # only with persistence
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["create", "get", "list", "delete"]

Pod Security: by default the pods run as root with a writable root filesystem and the capabilities CHOWN, DAC_OVERRIDE, FOWNER, FSETID, SETUID, SETGID and KILL, which the Pod Security level baseline admits and restricted refuses. Label the sandbox namespace accordingly:

kubectl label namespace agents pod-security.kubernetes.io/enforce=baseline

Where only restricted is allowed, use the hardened mode, createKubernetesSandbox({ namespace, user: { uid: 1000, gid: 1000 }, readOnlyRoot: true }): non-root with runAsNonRoot and fsGroup, a read-only root, no capability (apt then fails; toolchains still install into /cache with mise). A shared cache.claimName must honour fsGroup (NFS-style volume plugins ignore it) or be world-writable, else the hardened user cannot write it.

pods/exec needs both verbs: the client opens the exec WebSocket with a GET, which the API server authorizes as get, while servers with the WebSocket-upgrade authorization change ask for create. Keep the sandbox namespace apart from chamvi's own, so the pods never share a namespace with its secrets.

The pod

Each workspace is a bare pod chamvi-ws-<id> with one container sandbox:

A pod is used rather than a Job: exec needs a pod name known at create time, and a Job's controller, retries and generated names add nothing the pod spec does not already enforce.

limits.pids has no per-pod field in Kubernetes: the kubelet's podPidsLimit applies, so set it on the nodes that run sandboxes.

Image

The default image is ghcr.io/chamvi/sandbox-dev:trixie (DEFAULT_SANDBOX_IMAGE): Debian, git, build tools, ripgrep, jq and mise for any toolchain, built for linux/amd64 and linux/arm64 from images/sandbox-dev; tags trixie (rebuilt on changes and weekly) and trixie-<short sha>. imagePullPolicy (the provider image's) defaults to 'IfNotPresent'; use 'Always' to follow the mutable tag, or pin a digest in production (image: 'ghcr.io/chamvi/sandbox-dev@sha256:...').

A run may pick an image (create_workspace({ image })) only when the manager has an allow-list (allowImage, or CHAMVI_SANDBOX_IMAGES in sandboxModule()); imagePullPolicy applies to every image, the provider's, the manager's and one a run asked for. An image needs sh, git and a GNU timeout that takes -k: create probes them once the pod runs and fails with a SandboxError naming what is missing, before any checkout (and deletes the pod).

Cache

Every pod mounts /cache and gets the cache variables (CACHE_ENV: XDG_CACHE_HOME, npm_config_cache, PIP_CACHE_DIR, GOMODCACHE, GOCACHE, COMPOSER_CACHE_DIR; package downloads only, mise keeps toolchains in HOME). Without cache that is an emptyDir of the pod (counted against its ephemeral-storage), so installs work but start cold every time. With cache: { claimName } every workspace mounts the same ReadWriteMany claim (NFS, CephFS, EFS, Azure Files), so workspaces sharing the claim reuse each other's package downloads (a miss, an eviction or concurrent installs may still fetch again; toolchains stay per workspace, in HOME).

Risk: a shared claim is shared by every run and agent of the namespace. A malicious package one run fetched can sit in another run's cache; npm verifies integrity hashes, pip, Go's build cache and others less. Keep runs that must not influence each other on different claims, or without one.

Security

• Not a network boundary. Without a NetworkPolicy a workspace pod reaches every service in the cluster by DNS, the chamvi API and its database included, and cloud metadata. Select the pods by app.kubernetes.io/component=sandbox (or your podLabels) and deny all egress except what the work needs (GitHub, the npm registry), the chamvi namespace and 169.254.169.254 explicitly denied. The Helm chart will ship this policy.
• Root inside, not on the node: the default root user holds only the package-manager capabilities, no privilege escalation, seccomp RuntimeDefault, no host path and no privileged mode; the hardened mode drops root too.
• No credentials: no service account token is mounted, no service links are injected, and no chamvi environment variable reaches the pod.
• Prompt injection: what the model runs follows what it reads. Narrow allowRepo on the manager and put bash behind an approval where needed.

Timeouts and abort

Every command runs as timeout -k 5 <seconds> <argv>, which signals its whole process group. A client-side guard stops waiting 15 s after the timeout and reports exit 137 with the output so far. Kubernetes sends an exec's exit status only once every process holding its output closed it, so the command reports its exit on stderr itself: one that exits while a background job it started holds its output (node server.js & echo started) returns about 2 s after its exit with its own exit code, and the job runs on until the pod is deleted. An abort (signal) detaches: the exec WebSocket closes, and the command runs on until its timeout fires or the pod is deleted. Every settled run disposes its workspaces (manager.attach(engine)). A dispose deletes the pod without a grace period.

Stdin is closed only on the v5.channel.k8s.io exec protocol (Kubernetes 1.30+); on v4 it is written but never closed, so a command waiting for stdin EOF hangs until its timeout. writeFile does not depend on it (head -c <bytes>).

Garbage collection

find(runId, { deployment }) lists the run's pods by label and rebuilds the workspaces of the running ones (a pod that failed, finished or is being deleted is gone), so a run finds its workspace after a restart or on another replica. collectGarbage({ deployment, isSettled }) lists the deployment's pods with the managed labels and deletes those past their chamvi.dev/expires-at annotation, those in phase Failed or Succeeded and, with isSettled, those of settled runs (open workspaces of this process are disposed first). The manager calls it from acquire at most every 10 minutes; it is the only reaper.

Persistence (persistence: { storageClassName?, size? }, CHAMVI_SANDBOX_PERSISTENCE=true with sandboxModule()): create makes a ReadWriteOnce claim chamvi-ws-<id> (the pod's labels and annotations, size default limits.storage, storageClassName default the cluster's) before the pod, which mounts it at /workspace and /home/sandbox. With isSuspendable the collection deletes the running pod of a run that waited long enough and keeps the claim; find rebuilds the pod of a live claim without a running one (the image and expiry from the claim, env and limits from the provider and the manager, activeDeadlineSeconds what is left of the lifetime), waits like create and fails with a SandboxError when it does not start. Dispose, expiry and a settled run delete the claim too. Lost on suspension: background jobs, /tmp, writes outside the two mounts and an emptyDir /cache. A node-local storage class pins the rebuilt pod to its node. A pod that cannot be removed does not stop the pass: the others go, and one SandboxError with the count and the first failure is thrown at the end.

Options

createKubernetesSandbox(options):

Misconfiguration throws core InvalidAgentDefinitionError; API and pod failures are SandboxErrors.

API

Types: KubernetesSandboxOptions, KubernetesPersistence, PodUser, IPodClient, PodExecInput, CreatePodClientInput.

License

Published under Apache License 2.0.