Paired web applications

Sandbox can start an Ubuntu desktop and a separate application container (today: Draw.io), then link them as one pair. Chrome on the desktop is given a token-protected Sandbox URL. When Chrome opens it, Sandbox checks that the token belongs to the right companion and pair, then forwards the request to Draw.io inside Kubernetes. Draw.io itself has no public address.

This page is the step-by-step platform view of that design, plus the current desktop-to-companion network route and its in-VPC fix.

Intended path (step by step)

1. Create the pair — POST /sandbox/sandboxes/v1/environments/pairs with immutable desktop and companion pins (application_version + digest).
2. Poll — GET /sandbox/sandboxes/v1/environments/pairs/{pair_id} until status=ready (or failed).
3. Open the launch URL — when ready, launch_url points at the gateway, not at the companion pod IP and not at 127.0.0.1.
4. Chrome on the trainer desktop loads that URL over HTTPS.
5. Companion proxy authenticates the ticket, resolves the pair from the companion sandbox record, and reverse-proxies to the private Draw.io container.
6. Assets reuse an HttpOnly cookie scoped to the gateway-visible companion path (uses X-Forwarded-Prefix).

Refresh reconnects to the same pair. A new attempt (attempt_key) gets a fresh pair and clean state.

URL and auth (shareable facts)

Example launch URL (shape only):

https://sandbox-preprod.turing.com/sandbox/sandboxes/v1/<companion-sandbox-id>/companion/?token=<pair-scoped-ticket>

What works in code today

API surface: Sandboxes API — pairs.

Why the desktop cannot use the public gateway

Trainer desktops are GCE VMs in a subnet with no external IP and no Cloud NAT (by design: "Runtime desktops continue to use the primary subnet without internet NAT"). The public gateway is an internet-facing load balancer, so a desktop that resolves its hostname still times out connecting to :443. Preprod tests from two running desktops showed exactly this. No amount of proxy work helps until Chrome has a route it can actually dial.

The desktop subnet can reach resources inside the VPC: the firewall already allows the desktop subnet to GKE nodes, and the Sandbox service reaches desktops the same way over :5000 / :8080. So the fix is an in-VPC address for the companion route, which Sandbox owns end to end (Helm + config), with no change to the public gateway or Cloud NAT.

Trainer Chrome (desktop VM, no internet)
        │  HTTP, in-VPC only
        ▼
Internal LB  sandbox-sandboxes-internal   (source range = desktop subnet)
        │
        ▼
sandbox-sandboxes pod  →  /v1/<companion-id>/companion/  (token-checked)
        │  private, in-cluster
        ▼
Draw.io pod (no public address)

How it is configured

Everything lives in one values block per environment, sandboxes.companionInternalLb (defaults in infra/helm/sandbox/values.yaml):

Rendering fails if the block is enabled without loadBalancerIP or baseUrl, so an environment can never hand desktops the unreachable public URL by accident.

Why a reserved address: GKE does not reserve the address you set with loadBalancerIP (it can be released when the Service is deleted), and an ephemeral address "might" change when the Service is updated. Both are in the GKE LoadBalancer Service parameters reference. A launch URL that points at a moved address would silently stop working.

The base URL must be an http(s) origin with no path and never a loopback host; the service rejects anything else. A direct in-VPC request has no gateway prefix, so the cookie is scoped to /v1/<companion-id>/companion (_companion_cookie_prefix in app.py).

Rolling it out in an environment (no DevOps ticket)

1. Reserve the address first (once per environment). The command fails if the address is already taken, which is the check you want:

   gcloud compute addresses create sandbox-companion-internal \
     --project turing-agios-<env> --region us-central1 \
     --subnet turing-agios-<env>-us-central1 --addresses <loadBalancerIP>

   Do not reuse a name that starts with k8s- or contains the Service name.

2. Deploy once. GKE creates the internal load balancer on that address and the sandboxes pod gets the matching base URL in the same rollout.

3. Create a pair. It only becomes ready after the desktop itself fetches the companion route, so a wrong address fails closed with desktop could not reach companion gateway URL.

Enabling a new environment (for example prod)

1. Confirm desktops have no internet route: no Cloud NAT covering their subnets.
2. List every desktop subnet CIDR from SANDBOX_GCE_UBUNTU_GNOME_PLACEMENTS (gcloud compute networks subnets list --project <project>) and put them in sourceRanges; set globalAccess: true if any are outside the cluster's region.
3. Pick an address in the cluster subnet that is not used by any VM, forwarding rule or reservation, reserve it (step 1 above), set loadBalancerIP, set enabled: true, deploy.

Known limits

• Plain HTTP inside the VPC. The launch URL carries a bearer token over HTTP between a desktop and the load balancer. The traffic never leaves the VPC, but it is not encrypted. Mitigations today: the token is scoped to one companion and expires with the pair, and after the first request it moves into a path-scoped cookie. TLS on the internal route is a follow-up.
• Chrome blocks downloads from an HTTP origin. Saving from a companion app (for example Draw.io's File → Save) fails with "Insecure download blocked". The desktop image's Chrome wrapper therefore passes --unsafely-treat-insecure-origin-as-secure for the companion origin. Task code can write the launch-url file in the workspace, so the wrapper never trusts it to name an origin. It trusts only origins listed in the root-owned /etc/sandbox/companion-secure-origins, and only when launch-url is a plain, non-symlink file with exactly http://<private IPv4>:<port>/v1/<sandbox>/companion/... for that origin. The image builder writes the allowlist from SANDBOX_COMPANION_DESKTOP_BASE_URL for every image that carries a companion app (the build worker needs that variable, and the chart sets it when companionInternalLb is enabled). The origin is part of the image's content hash, so an image built for another environment is never reused. Without the allowlist the wrapper trusts nothing and downloads stay blocked, and a build with an HTTP origin that is not a private IPv4 address fails. This needs a desktop image rebuilt with the updated wrapper; Chrome shows an "unsupported command-line flag" bar while it is active.
• The whole Sandboxes service is reachable from the desktop subnet. The load balancer cannot filter by path, so the service is exposed to sourceRanges. Application authentication (companion token or API key) is still enforced on every route, and the firewall limits callers to desktop subnets. A dedicated companion-only listener or Deployment would remove this exposure and also isolate Draw.io traffic from the control plane; it is a follow-up.
• Scope. This covers the paired Draw.io route. It does not prove the validator or other app templates; each needs its own end-to-end run.

Scaling and tuning (hundreds of trainers)

The companion proxy is built so that many trainers can use their own Draw.io at the same time:

• Route cache. The first request for a pair resolves its upstream (a control-plane lookup); the result is cached and shared by concurrent requests, so a trainer loading 200 assets costs one lookup, not 200. A 502 from the upstream drops the cached route. Only ready routes are cached. Deleting a pair (or stopping or deleting its companion) drops the cached route in the pod that handled the request at once, including a lookup that is running at that moment (it is never cached). The cache is per pod, so other Sandboxes replicas can keep serving the route for up to the TTL, or until the pair's expiry if that comes first. Lower the TTL, or set it to 0, if that window is too long for your environment. Tune with SANDBOX_COMPANION_ROUTE_CACHE_TTL_SEC (default 15, range 0-120; 0 disables caching).
• Pooled, streamed upstream calls. One pooled HTTP client per worker streams responses through unchanged (compressed assets stay compressed). Tune with SANDBOX_COMPANION_PROXY_MAX_CONNECTIONS (default 256). When the pool is full the proxy answers 503 with Retry-After: 1 instead of queueing forever.
• Per-profile contract cache. A profile's private-web contract is read from the catalog once per process. Rebuilding the catalog on every lookup cost about 200 ms of CPU and was the main limit under load.
• Measured (single process, load generator in the same process, fake upstream). 100 trainers (2,100 requests): no errors, asset p99 about 0.4 s, one route lookup per pair. 200 trainers: no errors, asset p99 about 1.1 s. 500 trainers shares one core with the load generator and degrades (first-load p99 above 10 s), so run several Sandboxes replicas for that scale. The chart's CPU HPA already covers the sandboxes Deployment (autoscaling in infra/helm/sandbox/values-*.yaml: dev 1-2, preprod 2-8, prod 3-12 replicas). Before a large session, raise minReplicas so capacity is warm instead of waiting on the HPA, and confirm it with a load test on that environment. Reproduce with SANDBOX_LOAD_TEST=1 pytest -s sandbox/services/sandboxes/tests/test_companion_load.py.
• Admission quota. Each pair is two sandboxes. Check the project's concurrent sandbox quota (SANDBOX_PROJECT_MAX_CONCURRENT; 10 by default) covers 2 x trainers before a large session.

Status

Related canary signals (dev)

Recent sandbox-dev canaries also hit deploy/IAM issues that can fail the desktop half of the pair before Chrome even starts (for example desktop credential storage HTTP 403, or a crash-looping sandboxes revision). Fix those separately so both children stay running.

Unblock checklist

You are done with the connectivity canary when:

• The address is reserved (gcloud compute addresses describe sandbox-companion-internal --region us-central1) and kubectl -n sandbox get svc sandbox-sandboxes-internal shows that EXTERNAL-IP.
• From a running trainer desktop, curl -sS --max-time 15 -o /dev/null -w '%{http_code}\n' http://<EXTERNAL-IP>:8784/healthz connects (no timeout).
• Pair reaches status=ready with a non-loopback launch_url on that address.
• Opening launch_url in desktop Chrome loads Draw.io HTML (not 401 / 409 / 502).
• A follow-up asset request works via the companion cookie (no token in the query).
• Refresh reconnects to the same pair without resetting trainer work.

Operator verify sketch

export SANDBOX_GATEWAY_URL=https://sandbox-preprod.turing.com   # or sandbox-dev
export SANDBOX_API_KEY=…
AUTH=(-H "X-Api-Key: $SANDBOX_API_KEY" -H "Content-Type: application/json" \
      -H "X-Sandbox-Tenant-Id: ctp" -H "X-Sandbox-Project-Id: sandbox")

# 1) Create pair (pins must match catalog)
curl -fsS "${AUTH[@]}" -d @pair-request.json \
  "$SANDBOX_GATEWAY_URL/sandbox/sandboxes/v1/environments/pairs"

# 2) Poll until ready / failed
curl -fsS "${AUTH[@]}" \
  "$SANDBOX_GATEWAY_URL/sandbox/sandboxes/v1/environments/pairs/$PAIR_ID"

# 3) On the desktop VM (not your laptop), prove the in-VPC route
curl -vI --max-time 15 "$SANDBOX_COMPANION_DESKTOP_BASE_URL/healthz"

# 4) Only then open launch_url in desktop Chrome

If step 3 times out, stop and fix the route (internal LB, source range, base URL) — proxy debugging will not help.

Deep reference (repo)

MVP companion is Draw.io (state_model=file). Stateful companions (for example Overleaf) remain rejected until a dedicated design and canary.