Metric

Metric API endpoints

Every request needs a credential — see Authentication.

Method	Path	Summary
`POST`	`/sandbox/metric/metric/judge`	Run an agentic judge over an arbitrary subject directory.
`POST`	`/sandbox/metric/metric/judge-batch`	Run an agentic judge over an arbitrary subject directory for N rubrics in parallel.
`POST`	`/sandbox/metric/metric/judge-batch/async`	Enqueue a batched agentic judge job and return immediately.
`POST`	`/sandbox/metric/metric/judge/async`	Enqueue an agentic judge job and return immediately.
`GET`	`/sandbox/metric/metric/judge/async/{job_id}`	Poll an async judge job for status / result.
`POST`	`/sandbox/metric/metric/judge/async/{job_id}/cancel`	Cancel an in-flight async judge job.
`POST`	`/sandbox/metric/metric/prompt-reward`	Alias of `POST /metric/score-prompt-reward`
`POST`	`/sandbox/metric/metric/score-average-reward`	Average reward → 1–10 score
`POST`	`/sandbox/metric/metric/score-behavior-correctness`	Score whether tests check behavior vs implementation details
`POST`	`/sandbox/metric/metric/score-determinism`	Score determinism across reruns
`POST`	`/sandbox/metric/metric/score-documentation`	Score completeness of instruction.md documentation
`POST`	`/sandbox/metric/metric/score-edge-coverage`	Score the fraction of tests that target edge cases
`POST`	`/sandbox/metric/metric/score-false-negative-risk`	Score risk that a correct implementation fails the tests
`POST`	`/sandbox/metric/metric/score-false-positive-risk`	Score risk that a wrong implementation passes the tests
`POST`	`/sandbox/metric/metric/score-leakage-heuristic`	Static heuristic for instruction/tests leakage
`POST`	`/sandbox/metric/metric/score-model-breaking`	Score candidate as `model-breaking`
`POST`	`/sandbox/metric/metric/score-oracle`	Score the oracle agent run
`POST`	`/sandbox/metric/metric/score-prompt-reward`	Infer difficulty + score from oracle vs candidate
`POST`	`/sandbox/metric/metric/score-real-world-applicability`	Score how representative the task is of real-world work
`POST`	`/sandbox/metric/metric/score-test-alignment`	Score how well tests/ aligns with instruction.md
`POST`	`/sandbox/metric/metric/test-quality`	Alias of `POST /metric/score-test-alignment`

Common headers

Accepted on every endpoint on this page.

Name	In	Type	Required	Description
`x-correlation-id`	header	string	no	End-to-end correlation id for this eval run. When set, the gateway and downstream services honor it verbatim in logs, Cloud Trace span attributes, async job payloads, and Langfuse trace.id. When omitted, the gateway generates a ULID and echoes it on the response.

Endpoints

Run an agentic judge over an arbitrary subject directory.

POST /sandbox/metric/metric/judge

Request body

application/json, required — JudgeMetricRequest

Field	Type	Required	Description
`prompt`	string	yes	Full rubric / system prompt the judge agent receives. Written verbatim to `/inputs/rubric.md` inside the judge container.
`rubric_label`	string \| null	no	Short human-readable label surfaced in worker logs and the result envelope (e.g. `"test-alignment"`). Defaults to a label derived from the subject source.
`output_schema`	object \| null	no	Optional JSON Schema validated against the agent's judgment.json inside the verifier. When set, judgment.json MUST validate or the worker returns `JudgeOutputInvalid`.
`extra_context`	string \| null	no	Optional free-form context appended to the agent's instruction.md. Useful for dataset version, locale, etc.
`subject_bundle`	object \| null	no	Inline path → UTF-8 text map. Same shape as `collect_task_bundle_from_task_dir`; skips remote fetch.
`subject_archive_url`	string \| null	no	HTTPS URL returning either a zip archive of the subject tree OR a single file (PDF, image, document, ...). Zip payloads are extracted into `/inputs/subject/` on the worker; single-file payloads are saved at `/inputs/subject/<basename>` so the judge agent can evaluate either shape without code changes. JSON aliases: `subject_url`, `task_archive_url`.
`subject_manifest_url`	string \| null	no	Legacy CTP / GDPval `_manifest.json` URL. Rejected on the platform (`400`): zip the files and send `subject_archive_url`. JSON aliases: `manifest_url`, `task_manifest_url`.
`subject_manifest_expires_at`	string \| null	no	Legacy deadline for `subject_manifest_url`; only valid with it, so also unsupported on the platform.
`subject_storage_resource`	string \| null	no	Legacy Drive / S3 / GCS storage location. Rejected on the platform (`400`): send a presigned HTTPS URL (zip or single file) as `subject_archive_url`. JSON alias: `storage_resource`.
`subject_storage_connector_id`	integer \| null	no	Legacy TI storage connector id; only valid with `subject_storage_resource`, so also unsupported on the platform. JSON alias: `storage_connector_id`.
`subject_task_slug`	string \| null	no	Legacy local task slug. Rejected on the platform (`400`): the worker has no task tree. JSON alias: `task_slug`.
`judge_agent`	string \| null	no	Override for the Harbor agent name to run as the judge (e.g. `claude-code`, `openhands`). Defaults to the worker's configured default.
`model`	string \| null	no	Override for the model id passed to the judge agent. Defaults to the worker's configured default model.
`timeout_sec`	number \| null	no	Wall-clock cap for the judge Harbor invocation. Worker may clamp this to a configured maximum.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-request Harbor sandbox/environment override for this judge trial. Mirrors the agent service's `RunDatapointRequest.sandbox` so callers learn one vocabulary across both endpoints. When omitted, the worker uses its configured `SANDBOX_METRIC_SANDBOX` default. Values: `asg` is a routing hint — by the time the worker runs the trial, `asg` is collapsed to 'no override' (the worker is already the destination). `k8s` selects the K8s-Job sandbox for this trial only (cluster bits must be installed). `daytona` selects the Daytona SDK. `docker` selects the local Docker provider (needs a Docker socket on the worker host).
`credential_source`	string \| null	no	Optional per-request override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`). `llmgateway` prefers `gateway_workspace` + `gateway_project` with platform `llm_egress` M2M (no body `tk_live_`). Legacy inline `credentials.llmgateway.api_key` remains accepted during soak.
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-request credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

Example:

{
  "prompt": "Identify any leakage between `/inputs/subject/instruction.md` and `/inputs/subject/tests/`. Write /outputs/judgment.json.",
  "subject_archive_url": "https://storage.example.com/tasks/candidate.zip",
  "rubric_label": "leakage-check",
  "output_schema": {
    "type": "object",
    "required": [
      "score",
      "explanation",
      "details"
    ],
    "properties": {
      "score": {
        "type": "number",
        "minimum": 0,
        "maximum": 1
      },
      "explanation": {
        "type": "string"
      },
      "details": {
        "type": "object"
      }
    }
  },
  "timeout_sec": 1200
}

Responses

Status	Description	Body
`200`	Agentic judge completed; parsed Judgment + metadata.	JudgeMetricResponse
`400`	Caller-side input is the problem: the subject is not an inline bundle or archive URL (task slugs, manifests and Drive/TI resources are not supported on the platform).	—
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—
`499`	Client disconnected; judge run cancelled.	—
`502`	Worker- or judge-agent-side failure: subject couldn't be materialized, sandbox refused to start, or the judge agent's `judgment.json` didn't validate against `output_schema`.	—
`503`	Worker fleet unavailable, per-pod concurrency cap hit, or admission preflight rejected the requested sandbox.	—

Example response:

{
  "score": 0.85,
  "explanation": "Tests align with instruction in 4 of 5 cases; one assertion checks output formatting not described in instruction.md.",
  "details": {
    "covered_requirements": 4,
    "total_requirements": 5,
    "missing_coverage": [
      "formatting of trailing newline"
    ]
  },
  "subject_label": "test-alignment",
  "judge_agent": "claude-code",
  "model_used": "anthropic/claude-haiku-4-5",
  "duration_sec": 38.7,
  "telemetry": {
    "model": "anthropic/claude-haiku-4-5",
    "prompt_tokens": 12400,
    "completion_tokens": 890,
    "cache_tokens": 3200,
    "total_tokens": 13290,
    "latency_ms": 38700,
    "cost_usd": 0.042
  }
}

Run an agentic judge over an arbitrary subject directory for N rubrics in parallel.

POST /sandbox/metric/metric/judge-batch

Request body

application/json, required — JudgeBatchRequest

Field	Type	Required	Description
`subject`	JudgeBatchSubject	yes	Shared subject under evaluation; one source per request.
`rubrics`	JudgeBatchRubricRequest[]	yes	Ordered list of rubrics. Response preserves this order. Length is capped at the route layer (`metric.judgeBatchMaxRubrics`).
`judge_agent`	string \| null	no	Batch-level default agent; per-rubric overrides take precedence.
`model`	string \| null	no	Batch-level default model; per-rubric overrides take precedence.
`timeout_sec`	number \| null	no	Batch-level default per-rubric timeout.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-batch sandbox override. Same vocabulary as the singleton endpoint; the entire batch runs in one sandbox.
`credential_source`	string \| null	no	Optional per-batch override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`).
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-batch credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

Example:

{
  "subject": {
    "archive_url": "https://storage.example.com/tasks/candidate.zip"
  },
  "rubrics": [
    {
      "label": "test-alignment",
      "prompt": "Score test alignment on 0..1."
    },
    {
      "label": "leakage-check",
      "prompt": "Identify leakage between instruction and tests."
    }
  ],
  "judge_agent": "claude-code",
  "model": "anthropic/claude-haiku-4-5",
  "timeout_sec": 600
}

Responses

Status	Description	Body
`200`	Per-rubric results in request order plus batch-level timing. Per-rubric failures populate `error` slots; the response remains HTTP 200 as long as the batch reached the fan-out phase.	JudgeBatchResponse
`400`	Request rejected at the route layer — typically because `rubrics` exceeds the configured per-batch cap.	—
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—
`499`	Client disconnected; judge run cancelled.	—
`502`	Whole-batch failure (subject can't be materialized or sandbox refused to start). Per-rubric failures are surfaced as `error` slots in a 200 response instead.	—
`503`	Worker fleet unavailable, per-pod concurrency cap hit, or admission preflight rejected the requested sandbox.	—

Example response:

{
  "results": [
    {
      "label": "test-alignment",
      "score": 0.9,
      "explanation": "All instruction requirements have matching test assertions.",
      "details": {
        "covered_requirements": 5,
        "total_requirements": 5
      },
      "judge_agent": "claude-code",
      "model_used": "anthropic/claude-haiku-4-5",
      "duration_sec": 33.4,
      "telemetry": {
        "model": "anthropic/claude-haiku-4-5",
        "prompt_tokens": 12400,
        "completion_tokens": 890,
        "cache_tokens": 3200,
        "total_tokens": 13290,
        "latency_ms": 38700,
        "cost_usd": 0.042
      }
    },
    {
      "label": "leakage-check",
      "score": 0.95,
      "explanation": "No leakage detected between instruction and tests.",
      "details": {
        "leakage_signals": []
      },
      "judge_agent": "claude-code",
      "model_used": "anthropic/claude-haiku-4-5",
      "duration_sec": 28.7,
      "telemetry": {
        "model": "anthropic/claude-haiku-4-5",
        "prompt_tokens": 12400,
        "completion_tokens": 890,
        "cache_tokens": 3200,
        "total_tokens": 13290,
        "latency_ms": 38700,
        "cost_usd": 0.042
      }
    }
  ],
  "subject_label": "hello-world",
  "shared_setup_duration_sec": 12.4,
  "batch_duration_sec": 41.1
}

Enqueue a batched agentic judge job and return immediately.

POST /sandbox/metric/metric/judge-batch/async

Request body

application/json, required — JudgeBatchRequest

Field	Type	Required	Description
`subject`	JudgeBatchSubject	yes	Shared subject under evaluation; one source per request.
`rubrics`	JudgeBatchRubricRequest[]	yes	Ordered list of rubrics. Response preserves this order. Length is capped at the route layer (`metric.judgeBatchMaxRubrics`).
`judge_agent`	string \| null	no	Batch-level default agent; per-rubric overrides take precedence.
`model`	string \| null	no	Batch-level default model; per-rubric overrides take precedence.
`timeout_sec`	number \| null	no	Batch-level default per-rubric timeout.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-batch sandbox override. Same vocabulary as the singleton endpoint; the entire batch runs in one sandbox.
`credential_source`	string \| null	no	Optional per-batch override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`).
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-batch credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

Example:

{
  "subject": {
    "archive_url": "https://storage.example.com/tasks/candidate.zip"
  },
  "rubrics": [
    {
      "label": "test-alignment",
      "prompt": "Score test alignment on 0..1."
    },
    {
      "label": "leakage-check",
      "prompt": "Identify leakage between instruction and tests."
    }
  ],
  "judge_agent": "claude-code",
  "model": "anthropic/claude-haiku-4-5",
  "timeout_sec": 600
}

Responses

Status	Description	Body
`202`	Job accepted; `poll_url` is pre-built for the matching status endpoint. On an idempotent replay the same `job_id` is returned.	JudgeAsyncAcceptedResponse
`400`	Same per-batch cap and validation as the sync routes — only applies to `/metric/judge-batch/async`.	—
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—
`503`	Worker fleet unavailable, per-pod concurrency cap hit, or admission preflight rejected the requested sandbox.	—

Example response:

{
  "job_id": "8b1f3c2a-2c3d-4e5f-9012-abcdef012345",
  "request_id": "65c7f62a-c7aa-457e-9ec7-351510b76827",
  "status": "queued",
  "poll_url": "/metric/judge/async/8b1f3c2a-2c3d-4e5f-9012-abcdef012345"
}

Enqueue an agentic judge job and return immediately.

POST /sandbox/metric/metric/judge/async

Request body

application/json, required — JudgeMetricRequest

Field	Type	Required	Description
`prompt`	string	yes	Full rubric / system prompt the judge agent receives. Written verbatim to `/inputs/rubric.md` inside the judge container.
`rubric_label`	string \| null	no	Short human-readable label surfaced in worker logs and the result envelope (e.g. `"test-alignment"`). Defaults to a label derived from the subject source.
`output_schema`	object \| null	no	Optional JSON Schema validated against the agent's judgment.json inside the verifier. When set, judgment.json MUST validate or the worker returns `JudgeOutputInvalid`.
`extra_context`	string \| null	no	Optional free-form context appended to the agent's instruction.md. Useful for dataset version, locale, etc.
`subject_bundle`	object \| null	no	Inline path → UTF-8 text map. Same shape as `collect_task_bundle_from_task_dir`; skips remote fetch.
`subject_archive_url`	string \| null	no	HTTPS URL returning either a zip archive of the subject tree OR a single file (PDF, image, document, ...). Zip payloads are extracted into `/inputs/subject/` on the worker; single-file payloads are saved at `/inputs/subject/<basename>` so the judge agent can evaluate either shape without code changes. JSON aliases: `subject_url`, `task_archive_url`.
`subject_manifest_url`	string \| null	no	Legacy CTP / GDPval `_manifest.json` URL. Rejected on the platform (`400`): zip the files and send `subject_archive_url`. JSON aliases: `manifest_url`, `task_manifest_url`.
`subject_manifest_expires_at`	string \| null	no	Legacy deadline for `subject_manifest_url`; only valid with it, so also unsupported on the platform.
`subject_storage_resource`	string \| null	no	Legacy Drive / S3 / GCS storage location. Rejected on the platform (`400`): send a presigned HTTPS URL (zip or single file) as `subject_archive_url`. JSON alias: `storage_resource`.
`subject_storage_connector_id`	integer \| null	no	Legacy TI storage connector id; only valid with `subject_storage_resource`, so also unsupported on the platform. JSON alias: `storage_connector_id`.
`subject_task_slug`	string \| null	no	Legacy local task slug. Rejected on the platform (`400`): the worker has no task tree. JSON alias: `task_slug`.
`judge_agent`	string \| null	no	Override for the Harbor agent name to run as the judge (e.g. `claude-code`, `openhands`). Defaults to the worker's configured default.
`model`	string \| null	no	Override for the model id passed to the judge agent. Defaults to the worker's configured default model.
`timeout_sec`	number \| null	no	Wall-clock cap for the judge Harbor invocation. Worker may clamp this to a configured maximum.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-request Harbor sandbox/environment override for this judge trial. Mirrors the agent service's `RunDatapointRequest.sandbox` so callers learn one vocabulary across both endpoints. When omitted, the worker uses its configured `SANDBOX_METRIC_SANDBOX` default. Values: `asg` is a routing hint — by the time the worker runs the trial, `asg` is collapsed to 'no override' (the worker is already the destination). `k8s` selects the K8s-Job sandbox for this trial only (cluster bits must be installed). `daytona` selects the Daytona SDK. `docker` selects the local Docker provider (needs a Docker socket on the worker host).
`credential_source`	string \| null	no	Optional per-request override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`). `llmgateway` prefers `gateway_workspace` + `gateway_project` with platform `llm_egress` M2M (no body `tk_live_`). Legacy inline `credentials.llmgateway.api_key` remains accepted during soak.
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-request credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

Example:

{
  "prompt": "Identify any leakage between `/inputs/subject/instruction.md` and `/inputs/subject/tests/`. Write /outputs/judgment.json.",
  "subject_archive_url": "https://storage.example.com/tasks/candidate.zip",
  "rubric_label": "leakage-check",
  "output_schema": {
    "type": "object",
    "required": [
      "score",
      "explanation",
      "details"
    ],
    "properties": {
      "score": {
        "type": "number",
        "minimum": 0,
        "maximum": 1
      },
      "explanation": {
        "type": "string"
      },
      "details": {
        "type": "object"
      }
    }
  },
  "timeout_sec": 1200
}

Responses

Status	Description	Body
`202`	Job accepted; `poll_url` is pre-built for the matching status endpoint. On an idempotent replay the same `job_id` is returned.	JudgeAsyncAcceptedResponse
`400`	Same per-batch cap and validation as the sync routes — only applies to `/metric/judge-batch/async`.	—
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—
`503`	Worker fleet unavailable, per-pod concurrency cap hit, or admission preflight rejected the requested sandbox.	—

Example response:

{
  "job_id": "8b1f3c2a-2c3d-4e5f-9012-abcdef012345",
  "request_id": "65c7f62a-c7aa-457e-9ec7-351510b76827",
  "status": "queued",
  "poll_url": "/metric/judge/async/8b1f3c2a-2c3d-4e5f-9012-abcdef012345"
}

Poll an async judge job for status / result.

GET /sandbox/metric/metric/judge/async/{job_id}

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Always 200 on a successful poll. Job success/failure encoded in the `status` + `error` / `error_class` fields. Use the embedded `result` envelope for the parsed judge outcome on `status='succeeded'`.	JudgeAsyncStatusResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	No `status.json` exists in S3 for this `job_id`.	—
`422`	Validation Error	HTTPValidationError
`503`	Async poll endpoint not wired on this deployment (the agent package is not bundled into this metric image).	—

Example response:

{
  "job_id": "8b1f3c2a-2c3d-4e5f-9012-abcdef012345",
  "status": "queued",
  "updated_at": "2026-05-19T15:04:01.123456+00:00"
}

Cancel an in-flight async judge job.

POST /sandbox/metric/metric/judge/async/{job_id}/cancel

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	S3 `status.json` is now `cancelled`. The worker confirms the cancel on its next watch-loop tick (~5 s).	JudgeAsyncCancelResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	No `status.json` exists in S3 for this `job_id`.	—
`409`	Job already settled in a terminal-non-cancelled state — overwriting the terminal result is refused.	—
`422`	Validation Error	HTTPValidationError
`503`	Async poll endpoint not wired on this deployment (the agent package is not bundled into this metric image).	—

Example response:

{
  "job_id": "8b1f3c2a-2c3d-4e5f-9012-abcdef012345",
  "status": "cancelled",
  "was_active": true
}

Alias of `POST /metric/score-prompt-reward`

POST /sandbox/metric/metric/prompt-reward

Compatibility alias for POST /metric/score-prompt-reward.

Request body

application/json, required — PromptRewardRequest

Field	Type	Required	Description
`oracle_harbor_result`	object \| null	no	Harbor job result for the oracle/reference agent (typically agent_results[0].result).
`candidate_harbor_result`	object \| null	no	Harbor job result for the candidate agent (typically agent_results[1].result).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). Oracle and candidate entries are auto-extracted by name.
`expected_output`	"easy" \| "medium" \| "hard" \| "expert" \| null	no	Optional difficulty label to compare against inferred difficulty. For harbor-task-pipeline jobs, supply this on the metric request body; it is not part of the pipeline input contract.

Example:

{
  "oracle_harbor_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 1
            }
          ],
          "reward_stats": {
            "reward": {
              "1.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  },
  "candidate_harbor_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 0
            }
          ],
          "reward_stats": {
            "reward": {
              "0.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  },
  "expected_output": "hard"
}

Responses

Status	Description	Body
`200`	Prompt reward + inferred difficulty scored.	PromptRewardResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "Oracle mean 1.0, candidate mean 0.0; reward gap 1.0 → inferred 'hard'. Matches expected_output='hard'.",
  "difficulty": "hard",
  "difficulty_score": 1,
  "expected_output": "hard",
  "matches_expected": true,
  "oracle_mean_reward": 1,
  "candidate_mean_reward": 0
}

Average reward → 1–10 score

POST /sandbox/metric/metric/score-average-reward

Single Harbor job, or mean of per-agent job means (multi-agent / datapoint).

Request body

application/json, required — AverageRewardRequest

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	One Harbor job-level result.json (single agent). Omit when sending harbor_job_results.
`harbor_job_results`	object[] \| null	no	One Harbor job per agent: either raw result.json (top-level stats) or a datapoint row with the job nested under result (same shape as RunDatapointResponse.agent_results[]). Score is the mean of each job's average reward, then mapped to 1–10. Do not combine with harbor_job_result.
`verifier_ctrf`	object \| null	no	Optional CTRF test report (from verifier/ctrf.json) for the single harbor_job_result. When present, per-test pass/fail details are included in the explanation.
`verifier_ctrfs`	object \| null[] \| null	no	Optional list of CTRF reports parallel to harbor_job_results (one per agent). When present, per-test pass/fail details are included in the explanation.
`artifact_download_urls`	ArtifactDownloadUrlsPayload \| null	no	Optional presigned artifact URLs from the datapoint run; echoed in score-average-reward `explanation` for convenience.

Example:

{
  "harbor_job_results": [
    {
      "stats": {
        "evals": {
          "a": {
            "metrics": [
              {
                "mean": 1
              }
            ],
            "reward_stats": {
              "reward": {
                "1.0": [
                  "t1"
                ]
              }
            },
            "n_errors": 0
          }
        }
      }
    },
    {
      "stats": {
        "evals": {
          "b": {
            "metrics": [
              {
                "mean": 0
              }
            ],
            "reward_stats": {
              "reward": {
                "0.0": [
                  "t1"
                ]
              }
            },
            "n_errors": 0
          }
        }
      }
    }
  ]
}

Responses

Status	Description	Body
`200`	Average reward scored (single job or multi-agent pool).	AverageRewardResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "Mean reward 1.0 across 1 trial; mapped to 10.0 on the 1–10 scale.",
  "mean_reward": 1,
  "n_rewards": 1,
  "had_error": false
}

Score whether tests check behavior vs implementation details

POST /sandbox/metric/metric/score-behavior-correctness

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "Most tests assert observable behavior; few coupling to internal state or magic numbers.",
  "details": {
    "implementation_coupled_tests": [],
    "magic_number_tests": [],
    "behavior_correct_tests": [
      "test_output_lines",
      "test_exit_code",
      "test_handles_empty_input"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Score determinism across reruns

POST /sandbox/metric/metric/score-determinism

Request body

application/json, required — DeterminismRequest

Field	Type	Required	Description
`harbor_run_results`	object[] \| null	no	Pre-assembled list of Harbor job result.json objects (main + reruns).
`agent_results`	object[] \| null	no	Full agent_results array from RunDatapointResponse. The first entry's `result` key is used as the main run for determinism comparison.
`determinism_rerun_results`	object[] \| null	no	Extra Harbor job result.json objects from determinism reruns of the first agent (top-level field from RunDatapointResponse).

Example:

{
  "harbor_run_results": [
    {
      "stats": {
        "n_trials": 1,
        "n_errors": 0,
        "evals": {
          "default": {
            "metrics": [
              {
                "mean": 1
              }
            ],
            "reward_stats": {
              "reward": {
                "1.0": [
                  "t1"
                ]
              }
            },
            "n_errors": 0
          }
        }
      }
    },
    {
      "stats": {
        "n_trials": 1,
        "n_errors": 0,
        "evals": {
          "default": {
            "metrics": [
              {
                "mean": 1
              }
            ],
            "reward_stats": {
              "reward": {
                "1.0": [
                  "t1"
                ]
              }
            },
            "n_errors": 0
          }
        }
      }
    },
    {
      "stats": {
        "n_trials": 1,
        "n_errors": 0,
        "evals": {
          "default": {
            "metrics": [
              {
                "mean": 1
              }
            ],
            "reward_stats": {
              "reward": {
                "1.0": [
                  "t1"
                ]
              }
            },
            "n_errors": 0
          }
        }
      }
    }
  ]
}

Responses

Status	Description	Body
`200`	Determinism stability scored.	DeterminismResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "All 3 runs returned mean_reward=1.0; fully deterministic and passing.",
  "run_count": 3,
  "mean_rewards": [
    1,
    1,
    1
  ],
  "all_means_identical": true,
  "all_means_one": true
}

Score completeness of instruction.md documentation

POST /sandbox/metric/metric/score-documentation

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "Instruction lists every assertion in tests/, with input/output examples.",
  "details": {
    "missing_requirements": [],
    "undocumented_assertions": []
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Score the fraction of tests that target edge cases

POST /sandbox/metric/metric/score-edge-coverage

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "8 of 12 tests cover edge cases (66%); the test suite covers boundaries, empty inputs, and one concurrency case.",
  "details": {
    "edge_case_count": 8,
    "total_test_count": 12,
    "edge_fraction": 0.66,
    "edge_tests": [
      "test_empty_input",
      "test_max_int_boundary",
      "test_concurrent_writes"
    ],
    "missing_edge_cases": [
      "zero-length file",
      "unicode names"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Score risk that a correct implementation fails the tests

POST /sandbox/metric/metric/score-false-negative-risk

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "Tests don't over-specify implementation details; a correct alternative implementation would pass.",
  "details": {
    "risk": "low",
    "top_risks": [
      "test depends on the exact order of dict keys in stdout (Python ≥3.7 only)"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Score risk that a wrong implementation passes the tests

POST /sandbox/metric/metric/score-false-positive-risk

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "Tests assert specific behavior; unlikely to pass a buggy implementation.",
  "details": {
    "risk": "low",
    "top_risks": [
      "test relies on print formatting only — could miss a mis-rounded float"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Static heuristic for instruction/tests leakage

POST /sandbox/metric/metric/score-leakage-heuristic

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	Heuristic leakage score (static; no LLM).	ScoreOnlyResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "No shared identifiers between instruction.md and the tests/ directory."
}

Score candidate as `model-breaking`

POST /sandbox/metric/metric/score-model-breaking

Request body

application/json, required — ModelBreakingRequest

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	Full Harbor job-level result.json object (candidate / model-under-test run).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). The candidate entry is auto-extracted by name.

Example:

{
  "harbor_job_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 0
            }
          ],
          "reward_stats": {
            "reward": {
              "0.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  }
}

Responses

Status	Description	Body
`200`	Model-breaking metric scored.	ModelBreakingResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "Candidate mean reward 0.0 over 1 trial; below the 0.05 threshold (model-breaking).",
  "pass_rate": 0,
  "n_trials": 1,
  "n_success_trials": 0,
  "threshold": 0.05,
  "had_error": false,
  "below_threshold": true
}

Score the oracle agent run

POST /sandbox/metric/metric/score-oracle

Request body

application/json, required — OracleMetricRequest

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	Full Harbor job-level result.json object (oracle / reference run).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). The oracle entry is auto-extracted by name.

Example:

{
  "harbor_job_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 1
            }
          ],
          "reward_stats": {
            "reward": {
              "1.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  }
}

Responses

Status	Description	Body
`200`	Oracle metric scored.	OracleMetricResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "Oracle mean reward 1.0 over 1 trial; no Harbor/eval errors.",
  "mean_reward": 1,
  "n_trials": 1,
  "pass_rate": 1,
  "had_error": false
}

Infer difficulty + score from oracle vs candidate

POST /sandbox/metric/metric/score-prompt-reward

Request body

application/json, required — PromptRewardRequest

Field	Type	Required	Description
`oracle_harbor_result`	object \| null	no	Harbor job result for the oracle/reference agent (typically agent_results[0].result).
`candidate_harbor_result`	object \| null	no	Harbor job result for the candidate agent (typically agent_results[1].result).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). Oracle and candidate entries are auto-extracted by name.
`expected_output`	"easy" \| "medium" \| "hard" \| "expert" \| null	no	Optional difficulty label to compare against inferred difficulty. For harbor-task-pipeline jobs, supply this on the metric request body; it is not part of the pipeline input contract.

Example:

{
  "oracle_harbor_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 1
            }
          ],
          "reward_stats": {
            "reward": {
              "1.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  },
  "candidate_harbor_result": {
    "stats": {
      "n_trials": 1,
      "n_errors": 0,
      "evals": {
        "default": {
          "metrics": [
            {
              "mean": 0
            }
          ],
          "reward_stats": {
            "reward": {
              "0.0": [
                "t1"
              ]
            }
          },
          "n_errors": 0
        }
      }
    }
  },
  "expected_output": "hard"
}

Responses

Status	Description	Body
`200`	Prompt reward + inferred difficulty scored.	PromptRewardResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`422`	Request body failed Pydantic validation.	—

Example response:

{
  "score": 10,
  "explanation": "Oracle mean 1.0, candidate mean 0.0; reward gap 1.0 → inferred 'hard'. Matches expected_output='hard'.",
  "difficulty": "hard",
  "difficulty_score": 1,
  "expected_output": "hard",
  "matches_expected": true,
  "oracle_mean_reward": 1,
  "candidate_mean_reward": 0
}

Score how representative the task is of real-world work

POST /sandbox/metric/metric/score-real-world-applicability

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 8,
  "explanation": "Task involves multi-file orchestration similar to real microservice deployments; only minor synthetic shortcuts.",
  "details": {
    "realism_factors": [
      "multiple cooperating containers",
      "network-bound state"
    ],
    "weak_points": [
      "tests/ never exercises a transient network failure"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Score how well tests/ aligns with instruction.md

POST /sandbox/metric/metric/score-test-alignment

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "All 4 instructional requirements have at least one matching test assertion; only minor coverage gaps.",
  "details": {
    "coverage_matrix": [
      {
        "requirement": "prints integers 1..10 one per line",
        "tests": [
          "test_output_lines"
        ]
      },
      {
        "requirement": "exits 0 on success",
        "tests": [
          "test_exit_code"
        ]
      }
    ],
    "gaps": [
      "no test asserts ``\\n`` separator vs ``\\r\\n``"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Alias of `POST /metric/score-test-alignment`

POST /sandbox/metric/metric/test-quality

Compatibility alias for POST /metric/score-test-alignment.

Request body

application/json, required — TaskMetricRequest

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

Example:

{
  "task_archive_url": "https://storage.example.com/tasks/exported-task.zip"
}

Responses

Status	Description	Body
`200`	LLM rubric scored.	LlmScoreResponse
`401`	Missing or invalid credentials: a Keycloak bearer token, or a platform API key (`SANDBOX_API_KEYS` / `SANDBOX_API_KEY`); same auth as every Sandbox service.	—
`404`	Task bundle could not be assembled. `task_slug` did not match anything on disk, `task_archive_url` 404'd, or `storage_resource` was unreachable.	—
`422`	Request body failed Pydantic validation.	—
`502`	Upstream LLM or task-fetch failure — the request was well-formed but the metric service couldn't complete it. Safe to retry.	—

Example response:

{
  "score": 9,
  "explanation": "All 4 instructional requirements have at least one matching test assertion; only minor coverage gaps.",
  "details": {
    "coverage_matrix": [
      {
        "requirement": "prints integers 1..10 one per line",
        "tests": [
          "test_output_lines"
        ]
      },
      {
        "requirement": "exits 0 on success",
        "tests": [
          "test_exit_code"
        ]
      }
    ],
    "gaps": [
      "no test asserts ``\\n`` separator vs ``\\r\\n``"
    ]
  },
  "task_slug": "hello-world",
  "model_used": "anthropic/claude-haiku-4-5"
}

Schemas

The object shapes referenced above.

ArtifactDownloadUrlsPayload

Optional presigned artifact URLs (same shape as RunDatapointResponse).

Field	Type	Required	Description
`test_stdout_url`	string \| null	no	Presigned GET when a single verifier stdout exists.
`job_archive_url`	string \| null	no	Presigned GET for zipped Harbor job directory.
`test_stdout_artifacts`	VerifierStdoutArtifactPayload[]	no	Per-trial verifier stdout URLs when multiple files exist.

Provide either harbor_job_result (one Harbor job) or a non-empty harbor_job_results list (one result.json per agent, e.g. datapoint agent_results[].result). Both fields together, or neither, yields HTTP 422. Aliases: harbor_result, harbor_results. Request payloads use Harbor reward units (typically 0–1 per trial); the response field score is on a 1–10 scale.

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	One Harbor job-level result.json (single agent). Omit when sending harbor_job_results.
`harbor_job_results`	object[] \| null	no	One Harbor job per agent: either raw result.json (top-level stats) or a datapoint row with the job nested under result (same shape as RunDatapointResponse.agent_results[]). Score is the mean of each job's average reward, then mapped to 1–10. Do not combine with harbor_job_result.
`verifier_ctrf`	object \| null	no	Optional CTRF test report (from verifier/ctrf.json) for the single harbor_job_result. When present, per-test pass/fail details are included in the explanation.
`verifier_ctrfs`	object \| null[] \| null	no	Optional list of CTRF reports parallel to harbor_job_results (one per agent). When present, per-test pass/fail details are included in the explanation.
`artifact_download_urls`	ArtifactDownloadUrlsPayload \| null	no	Optional presigned artifact URLs from the datapoint run; echoed in score-average-reward `explanation` for convenience.

AverageRewardResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`mean_reward`	number \| null	no	Single job: mean of per-trial rewards (or metrics mean). Multi job: mean of each job's average reward (unweighted across agents).
`n_rewards`	integer	yes	Single job: trial count (or 1 for metrics fallback). Multi job: number of agent jobs that contributed a mean_reward to the pool.
`had_error`	boolean	yes	Whether Harbor or eval reported errors.

DeterminismRequest

Provide either a pre-assembled harbor_run_results list or agent_results + determinism_rerun_results (the first agent's main run is extracted from agent_results[0].result and prepended to the reruns).

Field	Type	Required	Description
`harbor_run_results`	object[] \| null	no	Pre-assembled list of Harbor job result.json objects (main + reruns).
`agent_results`	object[] \| null	no	Full agent_results array from RunDatapointResponse. The first entry's `result` key is used as the main run for determinism comparison.
`determinism_rerun_results`	object[] \| null	no	Extra Harbor job result.json objects from determinism reruns of the first agent (top-level field from RunDatapointResponse).

DeterminismResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`run_count`	integer	yes	Number of Harbor results compared.
`mean_rewards`	number \| null[]	yes	Per-run mean rewards aligned with input order.
`all_means_identical`	boolean	yes	True when every run has the same mean reward.
`all_means_one`	boolean	yes	True when all means exist and equal 1.0.

HTTPValidationError

Field	Type	Required	Description
`detail`	ValidationError[]	no

HarborAgentTelemetry

Normalized LLM usage for one Harbor agent invocation (possibly multi-trial).

Field	Type	Required
`model`	string \| null	no
`prompt_tokens`	integer \| null	no
`completion_tokens`	integer \| null	no
`cache_tokens`	integer \| null	no
`total_tokens`	integer \| null	no
`latency_ms`	number \| null	no
`cost_usd`	number \| null	no
`api_calls`	integer \| null	no

JobCredentialsSpec

Optional per-job credential overrides keyed by provider slug.

Field	Type	Required
`anthropic`	ProviderCredentialRef \| null	no
`openai`	ProviderCredentialRef \| null	no
`xai`	ProviderCredentialRef \| null	no
`google`	ProviderCredentialRef \| null	no
`fireworks`	ProviderCredentialRef \| null	no
`tinker`	ProviderCredentialRef \| null	no
`github`	ProviderCredentialRef \| null	no
`llmgateway`	LlmGatewayCredentialRef \| null	no

JudgeArtifactUrls

Presigned S3 GET URLs for the judge Harbor job archive and verifier stdout. Present only when the artifacts bucket is configured on the worker.

Field	Type	Required	Description
`job_archive_url`	string \| null	no	Presigned URL for `jobs/{job_id}/job-archive.zip` (zip of the judge Harbor job tree). Expires per S3 SigV4 (max 7 days); use the agent service `GET /jobs/{job_id}/artifacts/archive` for a stable URL.
`test_stdout_url`	string \| null	no	Presigned URL for the verifier `test-stdout.txt` when exactly one trial produced one. `None` otherwise.

JudgeAsyncAcceptedResponse

202 envelope for the RFC 0005 async judge / judge-batch enqueue routes.

Field	Type	Required	Description
`job_id`	string	yes	Worker job id — the same value the worker uses in S3 (`jobs/{job_id}/status.json` and `jobs/{job_id}/result.json`) and the value the poll / cancel routes accept in the URL path.
`request_id`	string	no	Echo of the inbound `X-Request-ID` header for log / trace correlation across the metric pod, the queue hop, and the agent worker. Default: "".
`status`	string	no	Always `queued` on the 202 response. The async-poll endpoint reports the progression to `running` / `succeeded` / `failed` / `cancelled`. Default: "queued".
`poll_url`	string	yes	Path of the async-poll endpoint for this `job_id` (e.g. `/metric/judge/async/8b1f...`). Pre-built so callers don't have to know the URL shape.

JudgeAsyncCancelResponse

Outcome of an async judge cancel. Always 200 on success; see was_active to distinguish first-cancel from idempotent re-cancel.

Field	Type	Required	Description
`job_id`	string	yes	Echoes the path parameter.
`status`	string	no	Always `cancelled` on a 200 response. Default: "cancelled".
`was_active`	boolean	yes	True when the cancel flipped a non-terminal (`queued` / `running`) job to `cancelled`; False on an idempotent re-cancel of an already-cancelled job.

JudgeAsyncStatusResponse

Status snapshot for an async judge job. Always 200 on a successful poll; failure / cancellation surfaces in the body, not the HTTP status.

Field	Type	Required	Description
`job_id`	string	yes	Echoes the path parameter for client logging.
`status`	"queued" \| "running" \| "succeeded" \| "failed" \| "cancelled"	yes	Worker-reported status. `queued` and `running` are non-terminal (continue polling); the other three are terminal.
`updated_at`	string \| null	no	ISO-8601 timestamp of the last status transition write, from `status.json`. Used by client watchdogs to detect wedged workers (RFC 0004).
`heartbeat_at`	string \| null	no	ISO-8601 timestamp of the most recent worker liveness pulse, from `heartbeat.json`. `null` when the job hasn't started running yet, when the worker predates RFC 0004, or when the heartbeat-write feature flag is off.
`error`	string \| null	no	Worker-side error message when `status` is `failed` or `cancelled`. `null` for non-terminal and `succeeded` jobs.
`error_class`	string \| null	no	Qualified Python class name of the worker exception that produced `error` (e.g. `StorageNotFoundError`). The synchronous route uses this to map worker failures to precise HTTP statuses; on the async path it is surfaced verbatim so callers can apply the same mapping themselves.
`result`	object \| null	no	Parsed `result.json` envelope for the job — a `JudgeMetricResponse` (singleton route) or `JudgeBatchResponse` (batch route). Populated only when `status == "succeeded"`; `null` otherwise. Surfaced as a free-form dict because the same poll endpoint serves both job types and the per-route response model is known at the call site, not at the schema level.

JudgeBatchRequest

Batched agentic-judge request: one subject, N rubrics. Materialized once, evaluated in parallel inside one sandbox.

Field	Type	Required	Description
`subject`	JudgeBatchSubject	yes	Shared subject under evaluation; one source per request.
`rubrics`	JudgeBatchRubricRequest[]	yes	Ordered list of rubrics. Response preserves this order. Length is capped at the route layer (`metric.judgeBatchMaxRubrics`).
`judge_agent`	string \| null	no	Batch-level default agent; per-rubric overrides take precedence.
`model`	string \| null	no	Batch-level default model; per-rubric overrides take precedence.
`timeout_sec`	number \| null	no	Batch-level default per-rubric timeout.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-batch sandbox override. Same vocabulary as the singleton endpoint; the entire batch runs in one sandbox.
`credential_source`	string \| null	no	Optional per-batch override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`).
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-batch credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

JudgeBatchResponse

Per-rubric results in request order plus batch-level timing.

Field	Type	Required	Description
`results`	JudgeBatchRubricResponse[]	yes	Per-rubric results in request order.
`subject_label`	string	yes	Resolved label for the shared subject.
`shared_setup_duration_sec`	number	no	Wall-clock seconds spent on shared work (subject materialization, sandbox startup) before any rubric began. Default: 0.
`batch_duration_sec`	number	yes	Total wall-clock seconds end-to-end.

JudgeBatchRubricError

Per-rubric error slot in :class:JudgeBatchResponse.

Field	Type	Required	Description
`error_class`	string	yes	Stable string class for the failure (e.g. `timeout`, `schema_validation`, `llm_error`, `parse_error`, `unknown`).
`message`	string	no	Human-readable failure message. Default: "".

JudgeBatchRubricRequest

One rubric inside a batched judge request.

Field	Type	Required	Description
`label`	string	yes	Stable, unique-within-batch identifier for this rubric.
`prompt`	string	yes	Full rubric prompt for this rubric.
`output_schema`	object \| null	no	Per-rubric JSON Schema for the expected `Judgment.details` shape.
`extra_context`	string \| null	no	Per-rubric free-form context appended to the prompt.
`judge_agent`	string \| null	no	Per-rubric override for the agent; falls back to batch default.
`model`	string \| null	no	Per-rubric override for the model; falls back to batch default.
`timeout_sec`	number \| null	no	Per-rubric wall-clock cap; falls back to batch default.

JudgeBatchRubricResponse

One slot in :class:JudgeBatchResponse.results.

Exactly one of (score, explanation, details) and error is populated. Both being None or both being set is a server-side bug — clients should treat that as the rubric having failed with class unknown.

Field	Type	Required	Description
`label`	string	yes	Echoes the request rubric's label.
`score`	number \| null	no	0..1 quality score on success; None on per-rubric failure.
`explanation`	string \| null	no	Human-readable summary on success; None on failure.
`details`	object \| null	no	Rubric-specific structured fields on success; None on failure.
`judge_agent`	string	yes	Agent that ran (or attempted to run) this rubric.
`model_used`	string	yes	Model used (or attempted to be used) for this rubric.
`duration_sec`	number	yes	Wall-clock seconds the rubric's LLM call consumed; 0.0 if it never started.
`error`	JudgeBatchRubricError \| null	no	Set when the rubric failed; None on success.
`telemetry`	HarborAgentTelemetry \| null	no	LLM usage from Harbor's trial-level `result.json` for this rubric's judge run.

JudgeBatchSubject

Subject under evaluation for a batch. Provide exactly one source. On the platform only bundle and archive_url are accepted; manifest_url, storage_resource and task_slug return 400.

Field	Type	Required	Description
`bundle`	object \| null	no	Inline path → UTF-8 text map; skips remote fetch.
`archive_url`	string \| null	no	HTTPS URL returning either a zip archive of the subject tree OR a single file (PDF, image, document, ...). Zip payloads are extracted; single-file payloads are saved at `/inputs/subject/<basename>`. Alias: `task_archive_url`.
`manifest_url`	string \| null	no	Legacy CTP / GDPval `_manifest.json` URL. Rejected on the platform (`400`); send `archive_url`. Alias: `task_manifest_url`.
`manifest_expires_at`	string \| null	no	Legacy deadline for `manifest_url`; only valid with it, so also unsupported on the platform.
`storage_resource`	string \| null	no	Legacy Drive / S3 / GCS storage location. Rejected on the platform (`400`); send a presigned HTTPS URL as `archive_url`.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id; only valid with `storage_resource`, so also unsupported on the platform.
`task_slug`	string \| null	no	Legacy local task slug. Rejected on the platform (`400`).

JudgeMetricRequest

Agentic judge metric: a rubric prompt + a candidate subject. The metric service dispatches a Harbor trial to the worker fleet; the inner agent (default claude-code) reads the rubric and subject and writes a Judgment JSON the verifier validates. Provide exactly one of subject_bundle or subject_archive_url (https or sandbox-object; a zip or a single file). subject_manifest_url, subject_storage_resource and subject_task_slug are rejected with 400 on the platform.

Field	Type	Required	Description
`prompt`	string	yes	Full rubric / system prompt the judge agent receives. Written verbatim to `/inputs/rubric.md` inside the judge container.
`rubric_label`	string \| null	no	Short human-readable label surfaced in worker logs and the result envelope (e.g. `"test-alignment"`). Defaults to a label derived from the subject source.
`output_schema`	object \| null	no	Optional JSON Schema validated against the agent's judgment.json inside the verifier. When set, judgment.json MUST validate or the worker returns `JudgeOutputInvalid`.
`extra_context`	string \| null	no	Optional free-form context appended to the agent's instruction.md. Useful for dataset version, locale, etc.
`subject_bundle`	object \| null	no	Inline path → UTF-8 text map. Same shape as `collect_task_bundle_from_task_dir`; skips remote fetch.
`subject_archive_url`	string \| null	no	HTTPS URL returning either a zip archive of the subject tree OR a single file (PDF, image, document, ...). Zip payloads are extracted into `/inputs/subject/` on the worker; single-file payloads are saved at `/inputs/subject/<basename>` so the judge agent can evaluate either shape without code changes. JSON aliases: `subject_url`, `task_archive_url`.
`subject_manifest_url`	string \| null	no	Legacy CTP / GDPval `_manifest.json` URL. Rejected on the platform (`400`): zip the files and send `subject_archive_url`. JSON aliases: `manifest_url`, `task_manifest_url`.
`subject_manifest_expires_at`	string \| null	no	Legacy deadline for `subject_manifest_url`; only valid with it, so also unsupported on the platform.
`subject_storage_resource`	string \| null	no	Legacy Drive / S3 / GCS storage location. Rejected on the platform (`400`): send a presigned HTTPS URL (zip or single file) as `subject_archive_url`. JSON alias: `storage_resource`.
`subject_storage_connector_id`	integer \| null	no	Legacy TI storage connector id; only valid with `subject_storage_resource`, so also unsupported on the platform. JSON alias: `storage_connector_id`.
`subject_task_slug`	string \| null	no	Legacy local task slug. Rejected on the platform (`400`): the worker has no task tree. JSON alias: `task_slug`.
`judge_agent`	string \| null	no	Override for the Harbor agent name to run as the judge (e.g. `claude-code`, `openhands`). Defaults to the worker's configured default.
`model`	string \| null	no	Override for the model id passed to the judge agent. Defaults to the worker's configured default model.
`timeout_sec`	number \| null	no	Wall-clock cap for the judge Harbor invocation. Worker may clamp this to a configured maximum.
`sandbox`	"asg" \| "k8s" \| "daytona" \| "docker" \| "prism" \| null	no	Per-request Harbor sandbox/environment override for this judge trial. Mirrors the agent service's `RunDatapointRequest.sandbox` so callers learn one vocabulary across both endpoints. When omitted, the worker uses its configured `SANDBOX_METRIC_SANDBOX` default. Values: `asg` is a routing hint — by the time the worker runs the trial, `asg` is collapsed to 'no override' (the worker is already the destination). `k8s` selects the K8s-Job sandbox for this trial only (cluster bits must be installed). `daytona` selects the Daytona SDK. `docker` selects the local Docker provider (needs a Docker socket on the worker host).
`credential_source`	string \| null	no	Optional per-request override of `SANDBOX_METRIC_CREDENTIAL_SOURCE` (`registry` \| `prism` \| `registry_with_prism_fallback` \| `litellm` \| `llmgateway` \| `registry_with_llmgateway_fallback`). `llmgateway` prefers `gateway_workspace` + `gateway_project` with platform `llm_egress` M2M (no body `tk_live_`). Legacy inline `credentials.llmgateway.api_key` remains accepted during soak.
`gateway_workspace`	string \| null	no	LLM Gateway workspace slug (`x-llmcp-workspace`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_project` for M2M egress without a body `tk_live_`.
`gateway_project`	string \| null	no	LLM Gateway project slug (`x-llmcp-project`). Stamped by Prism when `credential_source=llmgateway`; required with `gateway_workspace` for M2M egress without a body `tk_live_`.
`credentials`	JobCredentialsSpec \| null	no	Per-request credential overrides (RFC 0010 §6). Inline `env` / `api_key` are never persisted to S3. Prefer omitting `credentials.llmgateway.api_key` and stamping gateway workspace/project for platform M2M; legacy `api_key` + `base_url` remain accepted during soak.

JudgeMetricResponse

Result of an agentic judge run. The output shape is defined by the caller's output_schema: score is optional and, when present, follows whatever range the schema declares. The full judgment object is echoed in output.

Field	Type	Required	Description
`score`	number \| null	no	Optional quality score. Presence and range are defined by the caller's `output_schema`; null when the judgment carries no score.
`explanation`	string	no	Human-readable summary of the judgment. Default: "".
`details`	object	no	Rubric-specific structured fields (free-form).
`output`	object	no	The full judgment object the agent produced, verbatim. Use this when the contract is defined by `output_schema` rather than the score/explanation/details convention.
`subject_label`	string	yes	Resolved label for the evaluated subject (slug, archive hash, or rubric_label echo) used for logs / audit.
`judge_agent`	string	yes	Harbor agent that ran the judgment (e.g. `claude-code`).
`model_used`	string	yes	Model identifier the judge agent ran with.
`duration_sec`	number	yes	Wall-clock seconds the judge run consumed end-to-end.
`telemetry`	HarborAgentTelemetry \| null	no	LLM usage from Harbor's trial-level `result.json` for the judge agent run.
`artifact_download_urls`	JudgeArtifactUrls \| null	no	Presigned S3 GET URLs for the judge Harbor job output (`job-archive.zip` + verifier stdout). Populated only when the worker has `SANDBOX_METRIC_JOB_ARTIFACTS_BUCKET` configured; `None` otherwise. The same zip is reachable long-term via the agent service `GET /jobs/{job_id}/artifacts/archive` keyed by the worker `job_id`.

LlmGatewayCredentialRef

Prism / LLM Gateway credential stamp (api_key + base_url preferred).

Field	Type	Required
`api_key`	string \| null	no
`base_url`	string \| null	no
`secret_ref`	string \| null	no
`env`	object \| null	no

LlmScoreResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`details`	object	no	Structured fields returned by the LLM judge.
`task_slug`	string	yes	Task label used for this evaluation.
`model_used`	string	yes	Resolved LLM model id used for the judge call.

ModelBreakingRequest

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	Full Harbor job-level result.json object (candidate / model-under-test run).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). The candidate entry is auto-extracted by name.

ModelBreakingResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`pass_rate`	number \| null	no
`n_trials`	integer	yes	Trial count from Harbor summary.
`n_success_trials`	integer	yes	Trials counted as success for pass-rate fallback.
`threshold`	number	no	Internal threshold reference (model-breaking scoring). Default: 0.05.
`had_error`	boolean	yes	Whether Harbor or eval reported errors.
`below_threshold`	boolean	yes	True when candidate accuracy (mean_reward or pass_rate) is at or below the 5% failure-guarantee threshold.

OracleMetricRequest

Field	Type	Required	Description
`harbor_job_result`	object \| null	no	Full Harbor job-level result.json object (oracle / reference run).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). The oracle entry is auto-extracted by name.

OracleMetricResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`mean_reward`	number \| null	no	Mean reward from Harbor eval stats, if present.
`n_trials`	integer	yes	Trial count from Harbor summary.
`pass_rate`	number \| null	no	Pass rate proxy when mean reward is missing.
`had_error`	boolean	yes	Whether Harbor or eval reported errors.

PromptRewardRequest

Provide either oracle_harbor_result + candidate_harbor_result (individual dicts) or a harbor_job_results array (full agent_response / model_response); oracle and candidate are auto-extracted by name. Supply expected_output here directly when comparing against a known difficulty label; the harbor-task-pipeline request no longer carries that label.

Field	Type	Required	Description
`oracle_harbor_result`	object \| null	no	Harbor job result for the oracle/reference agent (typically agent_results[0].result).
`candidate_harbor_result`	object \| null	no	Harbor job result for the candidate agent (typically agent_results[1].result).
`harbor_job_results`	object[] \| null	no	Full agent results array (same shape as model_response / agent_response). Oracle and candidate entries are auto-extracted by name.
`expected_output`	"easy" \| "medium" \| "hard" \| "expert" \| null	no	Optional difficulty label to compare against inferred difficulty. For harbor-task-pipeline jobs, supply this on the metric request body; it is not part of the pipeline input contract.

PromptRewardResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.
`difficulty`	"easy" \| "medium" \| "hard" \| "expert" \| null	no	Inferred difficulty from reward gap.
`difficulty_score`	number \| null	no	Numeric mapping of difficulty tier.
`expected_output`	"easy" \| "medium" \| "hard" \| "expert" \| null	no	Expected label from request, if any.
`matches_expected`	boolean \| null	no	Whether inferred difficulty equals expected_output when both are set.
`oracle_mean_reward`	number \| null	no	Mean reward from oracle Harbor result.
`candidate_mean_reward`	number \| null	no	Mean reward from candidate Harbor result.
`oracle_error`	string \| null	no	Error detail when oracle run had Harbor/eval issues.
`candidate_error`	string \| null	no	Error detail when candidate run had issues.

ProviderCredentialRef

One provider's credential source on a job request.

Field	Type	Required	Description
`secret_ref`	string \| null	no
`env`	object \| null	no

ScoreOnlyResponse

Field	Type	Required	Description
`score`	number	yes	Primary metric score for this endpoint.
`explanation`	string	yes	Human-readable rationale.

TaskMetricRequest

Task rubric (LLM) and static metrics: provide one of: inline task_bundle; task_archive_url (HTTPS zip); or a task_slug under SANDBOX_METRIC_TASKS_ROOT (empty unless tasks are mounted). Google Drive / TI sources (storage_resource, storage_connector_id) are rejected on the platform. task_slug is optional with a URL. Optional model and extra_context apply to LLM judge routes.

Field	Type	Required	Description
`task_slug`	string	no	Task directory under `SANDBOX_METRIC_TASKS_ROOT`. Optional when `task_bundle` or `task_archive_url` is provided; then used as the cache key and display label. Default: "".
`task_bundle`	object \| null	no	Inline path→text map (same shape as collect_task_bundle); skips disk lookup.
`model`	string \| null	no	LLM judge model id; defaults to METRIC_JUDGE_MODEL when omitted.
`extra_context`	string \| null	no	Optional extra text passed to the judge.
`storage_resource`	string \| null	no	Legacy Google Drive / TI location. Rejected on the platform (`400`): send `task_archive_url` or `task_bundle` instead.
`storage_connector_id`	integer \| null	no	Legacy TI storage connector id. Ignored on the platform; Drive / TI sources are not fetched — send `task_archive_url` or `task_bundle`.
`task_archive_url`	string \| null	no	HTTPS URL to a task zip. JSON alias: `task_url`.

ValidationError

Field	Type	Required
`loc`	string \| integer[]	yes
`msg`	string	yes
`type`	string	yes
`input`	any	no
`ctx`	object	no

VerifierStdoutArtifactPayload

Matches agent API VerifierStdoutArtifactItem.

Field	Type	Required	Description
`relative_path`	string	yes
`url`	string \| null	no

Machine-readable spec: OpenAPI YAML.

Metric

On this page