Analytics and reporting

Tag every workload with labels (flat string map, max 20 keys) and reuse one x-correlation-id across Harbor, Agent, CodeEdit, and Sandboxes in the same workflow.

Labels contract

Key (examples)	Purpose
`trainer`	Human or team running the eval
`pass`	Attempt number in a multi-pass program
`program`	Product or curriculum id
`datapoint_id`	Your task identifier

Harbor metadata stays for task location (task_archive_url, retry_of). Do not put trainer tags in metadata.

Per service

Harbor — `POST /sandbox/harbor/v2/jobs/execute-tasks`

{
  "task_slug": "my-task",
  "agents": [{"name": "run", "harbor_agent": "oracle", "pass_at_k": 1}],
  "metadata": {"task_archive_url": "https://..."},
  "labels": {"trainer": "alice", "pass": "3", "program": "swe-bench"}
}

Poll GET /sandbox/harbor/v2/jobs/{job_id} — response includes correlation_id, labels, and on terminal jobs result.outcome.attempt_summary / trials_detail.

Agent — `POST /sandbox/agent/v1/runs`

Same labels on the body; poll GET /sandbox/agent/v1/runs/{job_id}.

CodeEdit — `POST /sandbox/codeedit/v1/executions`

Sync response echoes correlation_id and labels. Send the parent run's correlation id when CodeEdit is a sub-step.

Sandboxes — `POST /sandbox/sandboxes/v1`

{
  "profile": "ide-ubuntu",
  "linked_job_id": "optional-harbor-job-id",
  "labels": {"trainer": "alice", "pass": "1"}
}

GET /sandbox/sandboxes/v1/{sandbox_id} returns persisted correlation_id and labels.

BigQuery (operators)

When SANDBOX_ANALYTICS_ENABLED=1, terminal events land in dataset sandbox_analytics. See the operator runbook and interim playbook before BQ is enabled.

Interim (no BQ)

See interim playbook: poll GET routes, maintain your own correlation_id → job_id index, use Langfuse on job_id.

Analytics and reporting

On this page