Diagnosis from recorded history

Back to README

Before concluding that a mechanism is or is not the cause, query the recorded history for that mechanism's signature.

Throughput, concurrency, regression, and "why is this slow" questions are answered from work_attempts history, never from /api/v1/state. The state endpoint is a point-in-time snapshot. It can answer what Detent sees now, but it cannot establish a trend, a sustained ceiling, or what happened before the sample.

The dashboard's Hourly concurrency card is the first stop for throughput. It uses a rolling 24-hour interval sweep of work_attempts and shows fleet-wide and per-project median, p90, and maximum concurrency. Use the queries below when the dashboard needs corroboration or a different window.

Recovery operating procedure

Recovery is an operator responsibility through verification and handoff. A tag, green CI, installed binary, occupied slot, claim or lease heartbeat proves only its own boundary. None establishes that affected projects are delivering work. Keep monitoring after deployment; use scoped claims such as “installed” or “landing path verified” until the delivery evidence supports a broader result. This procedure adds no runtime gate, polling service or automatic lane writer.

Establish a comparable baseline

Choose closed UTC windows of equal length before and after the intervention. Record project scope, window bounds, deployed builds, available routing and workload differences. Separate autonomous delivery from operator merges and state corrections. Do not compare an incomplete current window with a complete baseline or pool unlike projects into an unexplained fleet average.

Use existing supported API/MCP projections, aggregates and bounded histories. Start with summaries, then fetch receipts for the affected cohort. Follow cursors only until the selected window or question is covered; retain that cursor and reuse the captured evidence instead of repeatedly downloading large histories. Declare partial, omitted and unavailable data. A partial page with no landing is not evidence of zero landings. Never bypass application scope or permissions by copying tenant databases or inspecting hidden credentials.

Repair and verify the failing path

1. Reproduce the current failure on its actual source/build and name the existing owner. Match an existing issue or failure fingerprint before filing. Simplify that owner or remove the unnecessary restriction before adding checks. Preserve live work, published source and genuine human holds; use supported revision-aware operations for authorized historical corrections.
2. For this repository, resume an active, unshipped frozen candidate instead of starting a competing release. With no active candidate, if the prior series has shipped and a new full release is requested, cut the next release/vX.Y from current develop once and freeze it. Fixes land on develop first, then reviewed cherry-pick -x PRs reach the release. Its green full suite permits an immutable tag, staging smoke and production smoke. Follow the release contract; other projects retain their chosen contract. Do not restart successful validation or chase a moving candidate instead of stabilizing the active one.
3. Verify the installed Hub and every relevant runner's version, commit and observation time. Requested, downloaded or saved updates are not applied updates. Check terminal operation receipts and fresh readback. Track offline runners as exceptions; on return, verify their build, current lease generation and publication custody before relying on them. Do not clear live leases or discard work to make occupancy look correct.
4. Follow formerly failing work after rollout from its exact retained or fresh source through publication, security audit, validation/acceptance and actual landing. Identify each receipt's version/base/head and producing build. A new unrelated issue landing does not prove the repaired path, and a pre-rollout failure does not establish a current regression. If a required stage is not observed, leave that part of verification pending.
5. Compare the equal windows again: delivered changes, repeat failure classes, stage distributions and outstanding holds. Explain changes in workload, topology or evidence coverage before attributing improvement to the repair. A higher slot count without more landings can be more waiting or repeated work.

Reuse applicable exact-source host receipts only where the project's existing completion contract permits it. Stop duplicate validation caused solely by polling, restarts or a new operator session; a changed source, landing base or required environment still needs its applicable checks. Keep diagnostics focused and process budgets coordinated across the host. Retain the original failure and every check's scope rather than presenting a short gate as a full suite.

Continue or hand off explicitly

Publish observations, source identities, decisions, cursors and evidence links to the selected tracker or Change/PR, never a repository incident notebook. Report what is verified, what remains unknown, and each known exception's owner, next action and observation trigger. Continue the existing monitoring cadence or hand it to an accepting operator; deployment alone does not end recovery. Keep unrelated projects moving. A genuine hold or one offline runner does not make the whole fleet blocked, and successful work elsewhere does not clear it. Use project delivery guidance when changing a project's gates, concurrency or release policy.

Authority by question

Read the served build directly from the live page:

curl -fsS http://127.0.0.1:4000/ | rg -o 'data-detent-served-version="[^"]+"' -m 1

Activity classification uses the provider's existing parsed command actions when available, preserving the original command fingerprint. Each tool span retains an ordered actions list with fixed types/categories, hashed type/name/ path identities, action fingerprints, repeats and instruction candidates. A mixed command retains one timed span: actions have no independent durations or outcomes. Sort spans by elapsed_seconds in the existing workflow timeline API to identify the largest observed operations; compare actions[].fingerprint and repeat under the same workspace head to identify repeated work. Native action fingerprints include cwd; raw tool command fingerprints remain unchanged. The first available native list from start or completion is retained once. Compare actions[].sources using path_ref, sha256, observed_at and matched_line to distinguish source versions and inferred instruction candidates. Do not sum parallel command durations as sequential elapsed time or assign a compound command's total to every action.

Native instruction reads use the provider's path and working directory (the workspace when cwd is absent). Native snapshots cover only WORKFLOW.md, AGENTS.md and CLAUDE.md regular files inside the workspace, including nested files. The existing recorder goroutine performs these bounded local reads; dispatch and worker commands never wait for them. Snapshots are limited to 64 read requests and 64 source records per profile, 256 KiB per file. The queue retains at most 32 actions and 8 KiB of native metadata including cwd per event; actions_dropped reports truncation. Each action retains at most four inferred instruction candidates. source_coverage explains missing paths, external files, unavailable snapshots and exhausted limits. File contents, commands, arguments, output and provider names/paths are excluded from persisted records.

A read request is evidence that an instruction file was requested, not that it caused later work or succeeded. recorder_snapshot_after_native_read records the version available when telemetry processed the event, not a proof of the bytes the provider read; edits before queue consumption may change it. Prior versions remain in instructions, while text matching uses the latest local snapshot. Exact command matches are inferred_text_match. Existing native events supply no instruction-origin provenance, so causal_attribution remains unknown_provider_origin, including observed reads and inferred matches. coverage_notes, dropped/unpaired counts and unobserved intervals explain partial coverage. Older profiles lack structured actions and cannot be repaired from discarded evidence; absent causal attribution is unknown.

Hub API latency

Every authenticated /api/v2 response carries a Server-Timing header with db, compute, encode and total durations in milliseconds. db is the time spent executing SQLite statements and stepping their rows on any pool, encode is JSON encoding, and compute is the remainder of total. The db span leaves out BEGIN, COMMIT and the wait for a pool connection; that time falls into compute. Reads made of several sub-reads add one named span per sub-read after total, such as reports_analytics, reports_diagnostics, reports_completion, reports_residence, reports_prior and the analytics_* parts of the analytics read; spans of concurrent project reads add up. The hub request timing log line of a slow request carries db_ms and the same spans in spans_ms. Neither appears in any UI. Unauthenticated responses carry no timing. One header is a single sample; it never establishes a trend.

The tenant request-metrics store records per-route latency history: route template, status class, request count, p50, p95 and max, in hourly buckets kept for 90 days. Requests only update an in-memory batch, which is flushed to the metrics store and never to the tenant writer. Organization administrators read the history through route_latency in the time_series MCP tool and the request diagnostics API (Hub API). Slowdown and regression claims cite this history.

TestHotReadQueryPlans runs EXPLAIN QUERY PLAN on every statement that Activity, the work-items list, board lanes, projects, Reports and Diagnostics issue. A hot read never fully scans native_attempts, collaboration_events, change_requests, attempt_usage, analytics_hourly_rollups or the rollup views, and never sorts rows of those tables in a temporary B-tree. TestAnalyticsRollupReportPlans applies the same rules to Reports over production-shaped tenants and logs their times. A statement that currently breaks a rule is listed in the test's known-failing table with the issue that fixes it, and the entry is removed when that fix lands.

GitHub usage timing and privacy

The existing github rest scope usage event now includes a bounded timings aggregate alongside its unchanged REST request_count, reserve_refused, fanout_deferred and steps. Correlate existing project/refresh/scope identity with project refresh timing and sub-step completion events for wall elapsed. GraphQL HTTP attempts are separate from REST requests and quota queries/points; request count alone cannot establish latency or a saving.

Each timing key uses the existing fixed stage, step, endpoint family and outcome, plus Boundary and an allowlisted GraphQL QueryPurpose. Unknown GraphQL purpose or operation names become graphql in this aggregate without changing quota accounting. Each completed observed boundary reports AttemptCount, TimedCount, ElapsedSumNS and ElapsedMaxNS; elapsed values are nanoseconds. Coverage is TimedCount / AttemptCount within that observed key, never inferred coverage of all local work. An unavailable timing observation has an attempt but no timed count or duration. Current instrumented completed attempts are all timed; in-flight attempts are published when their boundary finishes. Drain clears counts and timings together; a completion after a drain belongs to the next outside-refresh cohort. No per-request intervals or rows are retained.

The HTTP clock excludes response logging, progress callbacks, header accounting, decoding and recursive auth retries between its measured segments. Existing resource cleanup and request sequencing stay unchanged. Bounded text reads time only the consumption/close they actually perform; RESTTextWithSize includes its full-body counting pass. HTTP outcome is the existing fixed HTTP status classification (200, 304, 429, error), with body/close failures recorded as timing error without rewriting legacy REST outcomes. HTTP 200 does not prove GraphQL logical success. Token 200 means a nonempty token was resolved, not that a later HTTP request succeeded. Token failures or local budget/backoff refusal create no fabricated HTTP observation.

Direct token retrieval outside these request entries, JWT preparation, JSON encoding/decoding, actor queues and other local work are unmeasured. There is no independent connection-pool/actor-queue owner. Do can include connection-pool wait, DNS/TLS and server time, so this elapsed is HTTP/transport time, not pure network time. Sums are overlapping work: two concurrent requests can sum to more than wall elapsed, and inclusive token resolution can contain installation HTTP. Never add token and HTTP as disjoint elapsed, subtract their sums from stage wall, or infer critical path, busy-wall, savings or percentage attribution.

Timing is automatic in the existing aggregate event and requires no detailed request logs, extra API call, configuration, persistence table or polling. Persist/log only existing project/refresh/scope identity, fixed names and numeric aggregates. Raw URLs, query/variables, bodies, headers, credentials/hashes, command/environment, prompts and private operator criteria are excluded. Existing unrelated debug behavior is unchanged; it is not needed for timing.

Runtime acceptance belongs to Detent integration and the existing release owner after a normal authorized rollout. Record the deployed version/integration SHA and one naturally completed refresh's scope aggregate together with its stage and whole-refresh wall elapsed. Report REST and GraphQL HTTP/token attempt and timed counts, sums/maxima by fixed key, installation HTTP if observed, and any absent/unobserved boundaries. Explicitly leave actor queue, local work and pure network components unknown. Until that completed refresh is recorded, runtime acceptance remains pending; source diagnostics do not establish rollout evidence.

Token spend

The operator health metric is tokens per merged PR per project per UTC day, across both hosts (mac-studio and prometheus in the September 2026 audit). Use recorded database history, never the dashboard, for this number. Include spend on unsuccessful and still-open work in the numerator; merged-only spend answers a different question. Sum host numerators and deduplicate merged identifiers across hosts before dividing; do not average host ratios. Record the window, hosts, token source, and merge definition with every result.

Authority, joins, and time windows

usage_events is the authoritative complete source for recorded session token usage, equivalent to workflow_phase_events filtered to phase_type = 'agent_session' in the 2026-09-07..14 audit. Do not add those two sources together. Complete here means recorded session usage: restart-abandoned sessions can lack a phase/usage row. work_attempts.metrics_json is not an alternative authority: rows written before the #2710 fix may retain only the last checkpoint/recovery segment, including on session_duration_exceeded. Terminal writes now use cumulative session usage and the session turn count; historical rows are not backfilled. For attempt outcome dimensions, use coalesce(phase_event.total_tokens, metrics_json.total_tokens, 0) and report the fallback coverage. Preserve validator/routine sessions that have no attempt when computing all-project spend. Missing metrics mean unknown spend; the fallback zero is not evidence of free work.

The join keys are:

• work_attempts.detent_session_id = workflow_phase_events.session_id = codex_sessions.id.
• work_attempts.provider_session_id is the Codex rollout session ID, not the Detent session ID. Use it to locate the provider transcript.

Stored timestamps are ISO strings containing T. SQLite datetime('now', '-24 hours') emits a space: comparing it directly to these strings can silently match every timestamp on the cutoff date, including hours before the cutoff. Bind ISO literals such as '2026-09-14T00:00:00Z', or convert both sides to epoch values. The copied audit queries below use the explicit lower bound '2026-09-07T00:00:00Z'. Replace every occurrence for another audit; for a closed window, add the same exclusive ISO upper bound to each underlying source. Their original window is open-ended, so rerunning later includes newer rows. Daily spend is assigned by session/attempt start time, not prorated.

What total tokens means

Codex counts the whole submitted context again on every model call. total_tokens is cumulative processed input plus output, not unique text or final-answer size. The September 7–14 audit measured roughly 96–98% cached prefix and 0.3% output. These are observations, not guaranteed ratios. Calls per session multiplied by context length per call explains most token volume; a Detent turn may contain many model calls.

Cached reads still meter at one tenth of uncached input for the audited Astra and Sol rates. Weight tokens by the model's applicable rate before comparing models; equal token counts do not imply equal spend. Consult the Codex pricing page for current rates, billing units, and speed multipliers. With rates per million, compute ((input_tokens - cached_input_tokens) * input_rate + cached_input_tokens * cached_rate + output_tokens * output_rate) / 1000000. Cached tokens are already included in input; reasoning output is already part of output. Do not charge either twice or present credits as dollars.

Rollout layout

On each audited host, transcripts live in ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl; the worker's isolated CODEX_HOME sessions directory was empty. Inspect the host user's sessions rather than assuming the worker directory contains the rollouts.

Filter JSONL session_meta records to payload.originator == "detent-orchestrator". Sub-agent threads carry parent_thread_id and duplicate parent history: keep their lineage and exclude copied history when aggregating calls, text, or cumulative counters. Do not sum parent and child transcript histories as independent spend.

event_msg records with payload.type == "token_count" carry payload.info.total_token_usage (cumulative) and last_token_usage (last model call). Do not sum cumulative snapshots; use the final cumulative total or validated deltas, accounting for inherited history. In response_item records, the first user message after the AGENTS.md item is the Detent prompt, and payload.type == "custom_tool_call_output" denotes tool output. These locations let an audit separate prompt overhead from tool-output growth.

Ready-to-paste audit queries

The setup and queries below are copied from the operator's ~/.detent/token-audit/baseline-2026-09-14/token-audit-appendix.md, MAC-DB (Q0a/Q0e, Q2b, Q3b/Q3f, Q4b, Q5a, Q6d). The appendix abbreviates merged; its full definition below comes from the companion audit_db.py, with the window literal substituted. Semicolons are added for interactive SQLite use. No access to the operator's audit directory is needed to run these copies.

Open a fresh connection on each host and paste the setup once, then any query:

sqlite3 -readonly "$HOME/.detent/detent.db"

.headers on
.mode column

Only temporary views are created; the database remains read-only. att.tokens uses phase-event totals with attempt-metrics fallback; pe.total_tokens uses only recorded agent sessions. The merge definition counts distinct identifiers that reached Done, were delivered, or had a successful merge attempt with the terminal status message. This is the audit's merged-PR proxy, not an independent GitHub merge verification; Done can also represent an operator closure.

create temp view att as
select w.id, w.project_id, w.identifier, w.pr_number, w.lane, w.attempt_number, w.status,
  w.started_at, w.completed_at, substr(w.started_at,1,10) day,
  w.terminal_state, w.error_class, w.error_message, w.status_message, w.wait_reason, w.ci_state,
  w.detent_session_id,
  json_extract(w.worker_metadata_json,'$.run_mode') run_mode,
  json_extract(w.worker_metadata_json,'$.work_product_pushed') pushed,
  json_extract(w.worker_metadata_json,'$.completion_progress.outcome') cp_outcome,
  json_extract(w.worker_metadata_json,'$.completion_progress.reason') cp_reason,
  json_extract(w.worker_metadata_json,'$.completion_progress.progress_kinds') cp_kinds,
  json_extract(w.worker_metadata_json,'$.completion_progress.workspace_diffstat.files_changed') cp_files,
  json_extract(w.worker_metadata_json,'$.completion_progress.tracker_state') cp_tracker,
  json_extract(w.worker_metadata_json,'$.cancellation.reason') cancel_reason,
  json_extract(w.worker_metadata_json,'$.dispatch_loop_start.fingerprint.lane') fp_lane,
  json_extract(w.worker_metadata_json,'$.dispatch_loop_start.fingerprint.pr_head_sha') fp_pr_head,
  json_extract(w.worker_metadata_json,'$.dispatch_loop_start.fingerprint.workspace_head') fp_ws_head,
  coalesce(json_extract(w.worker_metadata_json,'$.dispatch_loop_start.fingerprint.pr_head_sha'),
           json_extract(w.worker_metadata_json,'$.dispatch_loop_start.fingerprint.workspace_head')) dispatch_head,
  json_extract(w.worker_metadata_json,'$.pr_head_sha') end_pr_head,
  json_extract(w.metrics_json,'$.total_tokens') wa_tokens,
  json_extract(w.metrics_json,'$.turns') wa_turns,
  json_extract(w.metrics_json,'$.runtime_seconds') wa_runtime,
  p.id pe_id, p.total_tokens pe_tokens, p.input_tokens, p.output_tokens, p.cached_input_tokens,
  p.reasoning_output_tokens, p.turns pe_turns, p.duration_seconds, p.model_context_window,
  coalesce(p.total_tokens, json_extract(w.metrics_json,'$.total_tokens'), 0) tokens,
  c.agent_role
from work_attempts w
left join workflow_phase_events p on p.session_id = w.detent_session_id and p.phase_type = 'agent_session'
left join codex_sessions c on c.id = w.detent_session_id
where w.started_at >= '2026-09-07T00:00:00Z';

create temp view pe as
select p.*, substr(p.started_at,1,10) day, c.agent_role, c.final_state, c.work_attempt_id
from workflow_phase_events p left join codex_sessions c on c.id = p.session_id
where p.phase_type = 'agent_session' and p.started_at >= '2026-09-07T00:00:00Z';

create temp view merged as
select distinct project_id, identifier from (
  select project_id, identifier from workflow_phase_events where phase_type='lane' and phase_name='Done' and started_at >= '2026-09-07T00:00:00Z' and identifier is not null
  union
  select project_id, identifier from att where terminal_state='delivered'
  union
  select project_id, identifier from att where run_mode='merge' and terminal_state='success' and status_message='worker reached terminal state'
);

Source reconciliation

Report phase coverage and fallback totals before using attempt dimensions.

select count(*) attempts, sum(detent_session_id is not null) with_session,
  sum(pe_id is not null) with_phase_event, sum(wa_tokens is not null) with_wa_metrics,
  sum(wa_tokens) sum_wa_metrics_tokens, sum(pe_tokens) sum_pe_tokens, sum(tokens) sum_coalesced_tokens
from att;

Usage and phase equivalence

The original audit query checks usage rows against their matching phases. It does not detect phase-only sessions; run the independent check below too.

select count(*) usage_rows, sum(u.total_tokens) usage_tokens, sum(p.total_tokens) pe_tokens,
  sum(u.total_tokens = p.total_tokens) equal_rows, sum(p.id is null) usage_without_pe
from usage_events u left join workflow_phase_events p on p.session_id = u.session_id and p.phase_type='agent_session'
where u.started_at >= '2026-09-07T00:00:00Z';

Check source totals independently and report both unmatched directions and non-unique session keys before treating the sources as equivalent. Totals alone can conceal offsetting differences; inspect any nonzero mismatch count.

with u as (
  select session_id, total_tokens from usage_events
  where started_at >= '2026-09-07T00:00:00Z'
), p as (
  select session_id, total_tokens from workflow_phase_events
  where phase_type='agent_session' and started_at >= '2026-09-07T00:00:00Z'
)
select (select count(*) from u) usage_rows,
       (select count(*) from p) phase_rows,
       (select sum(total_tokens) from u) usage_tokens,
       (select sum(total_tokens) from p) phase_tokens,
       (select count(*) from u where not exists
         (select 1 from p where p.session_id=u.session_id)) usage_without_phase,
       (select count(*) from p where not exists
         (select 1 from u where u.session_id=p.session_id)) phase_without_usage,
       (select count(*) from u join p using (session_id)
         where u.total_tokens is not p.total_tokens) token_mismatches,
       (select count(*) from
         (select session_id from u group by session_id having count(*)>1)) usage_duplicate_keys,
       (select count(*) from
         (select session_id from p group by session_id having count(*)>1)) phase_duplicate_keys;

Tokens per merged PR per project

This original window-wide query uses all attempt tokens, including fallback, and separately shows merged-only spend. It excludes sessions without attempts; use the daily all-session query below for the operator health metric.

with proj as (
  select project_id, sum(tokens) all_tokens, count(*) all_attempts, sum(pe_id is not null) all_sessions from att group by 1),
mg as (
  select a.project_id, count(distinct a.identifier) merged_prs, sum(a.tokens) merged_tokens, count(*) merged_attempts, sum(a.pe_id is not null) merged_sessions
  from att a join merged m on m.project_id=a.project_id and m.identifier=a.identifier group by 1)
select p.project_id, coalesce(mg.merged_prs,0) merged_prs, p.all_tokens, mg.merged_tokens,
  round(p.all_tokens*1.0/nullif(mg.merged_prs,0)) tokens_per_merged_pr_all_basis,
  round(mg.merged_tokens*1.0/nullif(mg.merged_prs,0)) tokens_per_merged_pr_merged_only,
  p.all_sessions, round(p.all_sessions*1.0/nullif(mg.merged_prs,0),2) sessions_per_merged_pr_all_basis,
  round(mg.merged_sessions*1.0/nullif(mg.merged_prs,0),2) sessions_per_merged_pr_merged_only,
  p.all_attempts
from proj p left join mg on mg.project_id=p.project_id order by p.all_tokens desc;

Repeat attempts per identifier and head SHA

The dispatch head prefers the PR head and falls back to the workspace head. Unknown heads are excluded. The copied query groups by a ten-character head prefix for display; inspect full hashes if prefixes collide. Its sequence is diagnostic text, not a guaranteed chronological ordering.

with ranked as (
  select id, project_id, identifier, dispatch_head, lane, terminal_state, tokens, started_at,
    row_number() over (partition by identifier, dispatch_head order by started_at) rn
  from att where dispatch_head is not null)
select project_id, identifier, substr(dispatch_head,1,10) head, count(*) attempts, sum(case when rn>1 then tokens else 0 end) repeat_tokens, sum(tokens) all_tokens,
  group_concat(lane || '/' || coalesce(terminal_state,'active'), ' > ') sequence
from ranked group by 1,2,3 having count(*) > 1 order by repeat_tokens desc;

Tokens by terminal state and error class

Uses att.tokens, including the metrics fallback.

select terminal_state, error_class, count(*) n, sum(tokens) tokens, round(100.0*sum(tokens)/(select sum(tokens) from att),1) pct from att group by 1,2 order by tokens desc;

Cache ratio per project

Uses recorded phase events only; the denominator is input tokens.

select project_id, count(*) sessions, sum(input_tokens) input, sum(cached_input_tokens) cached, round(1.0*sum(cached_input_tokens)/nullif(sum(input_tokens),0),3) cache_ratio,
  sum(output_tokens) output, round(1.0*sum(output_tokens)/nullif(sum(total_tokens),0),4) output_share,
  sum(reasoning_output_tokens) reasoning, round(1.0*sum(reasoning_output_tokens)/nullif(sum(total_tokens),0),4) reasoning_share,
  sum(cached_input_tokens is null) cached_null
from pe group by 1 order by input desc;

Sessions above 30 million tokens

Uses recorded phase events; missing-phase abandoned sessions are not included.

select p.project_id, p.agent_role, p.session_id, p.identifier, p.total_tokens, p.turns, p.duration_seconds, p.model_context_window, a.lane, a.terminal_state, a.error_class
from pe p left join att a on a.detent_session_id = p.session_id where p.total_tokens > 30000000 order by p.total_tokens desc;

Service-restart abandonment clusters

Uses att.tokens and groups completion timestamps by minute. tokens_lost is the audit alias for spend on abandoned attempts, not proof that all work was discarded. Correlate clusters with service logs; infrastructure failures belong to the instance, never the issue.

select substr(completed_at,1,16) restart_minute, count(*) attempts_abandoned, sum(tokens) tokens_lost, group_concat(distinct project_id) projects
from att where terminal_state='abandoned' and error_class='service_restart' group by 1 order by 1;

Daily health metric and host combination

This daily extension of Q1b/Q2b includes all recorded sessions plus attempt metrics only where a phase event is missing. It counts merge evidence by its completion/Done day, independently of when the attempt started. Run after the setup above. Change the start literal here along with the setup. A null ratio means no merges that day; keep the spend visible.

create temp view merged_days as
select project_id, identifier, substr(min(merged_at),1,10) day
from (
  select project_id, identifier, started_at merged_at
  from workflow_phase_events
  where phase_type='lane' and phase_name='Done'
    and started_at >= '2026-09-07T00:00:00Z'
  union all
  select project_id, identifier, completed_at
  from work_attempts
  where completed_at >= '2026-09-07T00:00:00Z'
    and (terminal_state='delivered' or
      (json_extract(worker_metadata_json,'$.run_mode')='merge'
       and terminal_state='success'
       and status_message='worker reached terminal state'))
)
where identifier is not null
group by project_id, identifier;

create temp view spend_days as
select project_id, day, sum(recorded_tokens) recorded_tokens,
       sum(fallback_tokens) fallback_tokens
from (
  select project_id, day, total_tokens recorded_tokens, 0 fallback_tokens
  from pe
  union all
  select project_id, day, 0, tokens from att where pe_id is null
)
group by project_id, day;

with days as (
  select project_id, day from spend_days
  union
  select project_id, day from merged_days
), merges as (
  select project_id, day, count(*) merged_prs
  from merged_days group by project_id, day
)
select d.project_id, d.day,
       coalesce(s.recorded_tokens,0) recorded_tokens,
       coalesce(s.fallback_tokens,0) fallback_tokens,
       coalesce(m.merged_prs,0) merged_prs,
       (coalesce(s.recorded_tokens,0) + coalesce(s.fallback_tokens,0)) * 1.0
         / nullif(m.merged_prs,0) tokens_per_merged_pr
from days d
left join spend_days s using (project_id, day)
left join merges m using (project_id, day)
order by d.project_id, d.day;

Run against each host's local database, not an aggregate copy that repeats the same sessions. Export spend_days and merged_days from the same connection:

select * from spend_days order by project_id, day;
select * from merged_days order by project_id, identifier;

For the both-host report, sum recorded_tokens and fallback_tokens by (project_id, day) across the host exports. Union the merge exports, group by (project_id, identifier), and take the earliest day so a merge seen by both hosts counts once. Count those identifiers by project/day and divide summed spend by that count. Retain the two token columns so every result states the source and fallback contribution. Session IDs are host-local; do not join numeric IDs between hosts. If replicated history is used, deduplicate it by its originating host/session before summing. Compare like windows and flag partial days rather than treating today's unfinished work as a regression.

Codex startup stalls

A backend_startup_timeout identifies a missed handshake deadline, not a provider outage. The codex app-server startup failed log event contains payload-free startup evidence even for admission runs. Issue attempts also persist it in work_attempts.worker_metadata_json.backend_startup.

Compare before_cleanup with process (after cleanup). An absent before_cleanup means the older runtime or transport did not capture it; do not interpret absent counters as observed zeroes. Counters are cumulative for that app-server process, including initialization and optional account checks. They contain no methods, arguments, credentials, prompts, or stderr contents.

Response waits use one read_timeout_ms deadline across notifications and server requests. Streaming turns retain their separate activity/stall rules. A close timeout is retained in the error chain, but cannot turn an explicit startup rejection or EOF into a startup timeout for capacity classification. Retries retain workspace state and the existing startup failure breaker. Admission startup failures retain their error in failed-run records instead of becoming generic capacity deferrals. Quota and provider-overload errors continue to defer admission without marking model output malformed.

For a recurrence, capture only the affected worker, without restarting or signaling the live Detent service:

1. Record the attempt/session IDs, startup event, executable version, and failed stage. Correlate the PID with codex_sessions.worker_pid and worker_started_at and verify its executable and start time before sampling; a stale PID may belong to a different process.
2. During the stall, take a one-second macOS sample <worker-pid> 1 1 -file "$TMPDIR/codex-startup.sample" with an outer five-second command deadline. Record ps -p <worker-pid> -o pid,ppid,lstart,state,%cpu,rss alongside host memory/CPU pressure. Keep the raw sample private; report only relevant stack symbols and aggregate metrics. Existing worker concurrency and free disk space cannot exclude memory pressure or per-process blocking.
3. Reproduce initialize, initialized, and thread/start in an isolated app-server process with an absolute temporary workspace, no prompt or turn/start, a fixed request deadline, and guaranteed process-group cleanup. Match the installed protocol schema and runtime model/config. Compare a minimal Codex home with the configured plugins, hooks, and skills restored one category at a time. Keep credentials local and never print RPC payloads.
4. For launchd-only failures, compare an isolated temporary launch agent with a direct process. A matching TCC denial or blocking filesystem stack supports fixing access to that particular skill/config path; a successful direct launch alone does not. Do not mutate operator-owned links during diagnosis.
5. Apply a remedy only to the established cause: correct a rejected startup setting; remove or repair the isolated failing plugin/hook; repair the proven launchd access restriction; reduce spawn pressure when correlated resource evidence supports it; or use the provider's recovery guidance when remote response/connection evidence establishes an outage. Preserve existing work and let the existing instance breaker limit repeated pre-turn failures. Do not increase the timeout or restart the service to substitute for diagnosis.

September 9, 2026 incident (#2376)

Read-only recorded history shows attempts 4959, 4960, and 4961 initialized within 533–791 ms, then spent 5000–5001 ms waiting for thread/start. Attempt 4964 initialized within 338 ms and still failed after 30000 ms. All four recorded one concurrent startup, with 8–10 active workers, and exited with signal: killed after a further close deadline. Sessions 5844–5846 and 5849 were subsequently recorded as already exited by worker cleanup. Admission session 5848 failed during the same interval and was also reaped. Its admission run 9529 (00:45:00–00:46:10 UTC) instead recorded deferred, agent_backend_capacity, and no error text. That loss of startup failure attribution is reproduced by the admission regression and corrected here.

Attempt 4966 began at 00:48:20 UTC and obtained a provider identity around 00:49, before the unsuccessful timeout trial was reverted around 00:50. The unpushed workspace commit was preserved across the failed retries. The incident therefore establishes intermittent post-initialization silence and bounded termination, not a cause or a timeout-setting remedy. The recorded history lacks pre-cleanup I/O counters and process samples, so it cannot distinguish provider deadlock, configuration/plugin I/O, resource pressure, or external service failure retroactively. The notification deadline renewal and cleanup-error misclassification regressions reproduce independently; neither is claimed as the cause of these four silent waits.

Workspace scan deadlines (#2657)

A workspace scan command stage=wait error means the command started but the scan did not complete before cancellation. A short start_elapsed measures cmd.Start, not completion of executable initialization. When cancellation returns before cmd.Wait, output_bytes=unknown means the waiter still owns the output buffer. Older builds reported output_bytes=0 in that case even when a child had written output; do not infer silence or a startup stall from that historical value.

The reported Go 1.27.1 darwin/arm64 failure reached its individual 30-second scan deadline during scratch verification. Bounded reproduction on the same platform passed three runs of TestReapScratchProcessesOutsideWorkspace (including all legacy cases). Three direct lsof -O -w -d cwd -F pn scans with file-backed output exited successfully in 1.21–2.28 seconds with 208–211 KB of output. This does not establish the cause of the historical timeout. A separate inherited-pipe regression reproduces the misleading zero-byte report deterministically; correcting that diagnostic preserves the existing deadline and cancellation behavior, without adding a recovery mechanism.

For a recurrence, record the error and command PID, then take a bounded process sample while the affected command is still running. Keep draining stdout concurrently, or redirect an isolated probe to a temporary file: pausing a pipe reader while sampling can itself block lsof in __write_nocancel. Such probe-induced backpressure is not evidence of the original failure. Record completed output size, exit status, elapsed time, and relevant stack symbols, keeping process paths and raw samples private. Do not increase the lifecycle deadline or attribute an unproven host failure to the issue.

Ready-to-paste SQLite queries

These queries default to a rolling 24-hour window. Replace '-24 hours' or add a project_id predicate when investigating a narrower incident.

Hourly maximum-concurrency interval sweep

This event sweep counts overlapping attempt intervals without relying on a single observation. Active attempts end at their recorded lease or heartbeat when no completion exists, so an old incomplete row does not extend forever.

WITH RECURSIVE
params AS (
  SELECT unixepoch(strftime('%Y-%m-%dT%H:00:00Z', 'now'), '-24 hours') AS from_at,
         unixepoch(strftime('%Y-%m-%dT%H:00:00Z', 'now')) AS to_at
),
hours(hour_start) AS (
  SELECT from_at FROM params
  UNION ALL
  SELECT hour_start + 3600
  FROM hours, params
  WHERE hour_start + 3600 < to_at
),
project_hours AS (
  SELECT DISTINCT attempts.project_id, hours.hour_start,
         MIN(hours.hour_start + 3600, params.to_at) AS hour_end
  FROM work_attempts AS attempts
  CROSS JOIN hours
  CROSS JOIN params
  WHERE unixepoch(attempts.started_at) < MIN(hours.hour_start + 3600, params.to_at)
    AND unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                          attempts.heartbeat_at, attempts.started_at)) > hours.hour_start
),
raw_events AS (
  SELECT ph.project_id, ph.hour_start, ph.hour_start AS event_at,
         SUM(CASE WHEN unixepoch(attempts.started_at) < ph.hour_start
                   AND unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                                         attempts.heartbeat_at, attempts.started_at)) > ph.hour_start
                  THEN 1 ELSE 0 END) AS delta
  FROM project_hours AS ph
  LEFT JOIN work_attempts AS attempts ON attempts.project_id = ph.project_id
  GROUP BY ph.project_id, ph.hour_start
  UNION ALL
  SELECT ph.project_id, ph.hour_start, unixepoch(attempts.started_at), 1
  FROM project_hours AS ph
  JOIN work_attempts AS attempts ON attempts.project_id = ph.project_id
  WHERE unixepoch(attempts.started_at) >= ph.hour_start
    AND unixepoch(attempts.started_at) < ph.hour_end
    AND unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                          attempts.heartbeat_at, attempts.started_at)) > unixepoch(attempts.started_at)
  UNION ALL
  SELECT ph.project_id, ph.hour_start,
         unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                            attempts.heartbeat_at, attempts.started_at)), -1
  FROM project_hours AS ph
  JOIN work_attempts AS attempts ON attempts.project_id = ph.project_id
  WHERE unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                          attempts.heartbeat_at, attempts.started_at)) > ph.hour_start
    AND unixepoch(COALESCE(attempts.completed_at, attempts.lease_expires_at,
                          attempts.heartbeat_at, attempts.started_at)) < ph.hour_end
    AND unixepoch(attempts.started_at) < ph.hour_end
),
events AS (
  SELECT project_id, hour_start, event_at, SUM(delta) AS delta
  FROM raw_events
  GROUP BY project_id, hour_start, event_at
),
points AS (
  SELECT project_id, hour_start, event_at,
         SUM(delta) OVER (
           PARTITION BY project_id, hour_start
           ORDER BY event_at
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
         ) AS concurrency
  FROM events
)
SELECT project_id, datetime(hour_start, 'unixepoch') AS hour_start,
       MAX(concurrency) AS max_concurrency
FROM points
GROUP BY project_id, hour_start
ORDER BY hour_start, project_id;

Correlate a pinned result with the mechanism signature before attributing it:

SELECT project_id,
       strftime('%Y-%m-%dT%H:00:00Z', decision_at) AS hour,
       COUNT(*) AS sampled_refusals
FROM scheduler_decisions
WHERE unixepoch(decision_at) >= unixepoch('now', '-24 hours')
  AND result = 'skipped'
  AND wait_reason = 'provider_rate_window_backpressure'
GROUP BY project_id, hour
ORDER BY hour, project_id;

Per-issue dispatch counts

SELECT project_id,
       COALESCE(NULLIF(TRIM(identifier), ''), NULLIF(TRIM(issue_id), ''),
                NULLIF(TRIM(issue_url), '')) AS issue,
       COUNT(*) AS dispatches,
       MIN(started_at) AS first_dispatch,
       MAX(started_at) AS latest_dispatch
FROM work_attempts
WHERE unixepoch(started_at) >= unixepoch('now', '-24 hours')
GROUP BY project_id, issue
HAVING issue IS NOT NULL
ORDER BY dispatches DESC, latest_dispatch DESC;

Lane and project distribution

SELECT project_id,
       COALESCE(NULLIF(TRIM(lane), ''), '(unrecorded)') AS lane,
       COUNT(*) AS dispatches,
       ROUND(SUM(MAX(
         0,
         unixepoch(MIN(COALESCE(completed_at, lease_expires_at, heartbeat_at,
                                started_at), datetime('now')))
         - unixepoch(MAX(started_at, datetime('now', '-24 hours')))
       )) / 3600.0, 2) AS agent_hours
FROM work_attempts
WHERE unixepoch(started_at) < unixepoch('now')
  AND unixepoch(COALESCE(completed_at, lease_expires_at, heartbeat_at, started_at))
      > unixepoch('now', '-24 hours')
GROUP BY project_id, lane
ORDER BY agent_hours DESC, dispatches DESC, project_id, lane;

Blocked-cause breakdown by recovery reason

workflow_phase_events.reason is the durable recovery reason on the recorded entry into Blocked; the alias below matches the operator-facing field name.

SELECT project_id,
       COALESCE(NULLIF(TRIM(reason), ''), '(unrecorded)') AS recovery_reason,
       COUNT(*) AS blocked_entries,
       MIN(started_at) AS first_seen,
       MAX(started_at) AS latest_seen
FROM workflow_phase_events
WHERE phase_type = 'lane'
  AND lower(phase_name) = 'blocked'
  AND status = 'entered'
  AND unixepoch(started_at) >= unixepoch('now', '-24 hours')
GROUP BY project_id, recovery_reason
ORDER BY blocked_entries DESC, project_id, recovery_reason;

Security audit waits (#2716)

security_audit_runs stores completed results, not launch records. Its completed_at field is required. Absence of a row cannot establish that an audit did not start. Correlate the wait timestamp with codex_sessions where agent_role = 'security_audit', including started_at and completed_at. Audit scratch paths alone are insufficient: allowance triage also uses the security-audit scratch root.

On prometheus after the September 14 21:12Z restart, session 12329 for digitaldrywood/pyroapex#2127 started at 23:49:18Z and completed at 23:58:06Z. The 23:54:51Z security_audit_wait decision fell inside that interval. Completed audit row 29 has the same timestamps, successful execution, and a fail verdict. This evidence contradicts the reported absence of audit launches; it does not establish the cause of other cards' allowance parks.

The in-memory stage registry prevents duplicate launches while a reviewer is active. It is initialized empty on restart. Missing exact-base/head results request an audit on the existing auto-promote pass, including Merging; a trusted completed result remains usable after restart. Re-running a completed passing audit merely because the process restarted is unnecessary. TestSecurityAuditLifecycleAcrossRestart covers those boundaries with a held reviewer, without wall-clock sleeps. No additional startup reconciliation is needed to discard a registry that is never persisted.

Validation lock cost

For historical lock investigations, use scripts/validation-report.sql alongside the work-attempt throughput queries above. Before the lock was removed from the Make targets, make check and make check-fast retained a synced event stream at $(git rev-parse --path-format=absolute --git-common-dir)/detent-validation-events.jsonl. This survives worktree removal. The stream includes host, unique run ID, issue/workspace attribution, wait and hold seconds, command runtime, and queue observations. It contains local filesystem paths; handle it like other local operational history. Validation before checklock starts (including Make's preflight and compiling the wrapper) is outside these measurements.

The following read-only report uses Python's standard library and SQLite 3.42+ (for subsecond timestamps). Set database to that host's Detent database, events to the retained JSONL file, host to its recorded OS hostname, worker_host to the corresponding work_attempts.worker_host, and day to a UTC date (YYYY-MM-DD). Hostnames and configured worker-host names need not be equal. For local attempts whose worker_host is null, set worker_host to the empty string and use that originating host's database (not an aggregate copy). Run from the repository root. The connection cannot write the history database; all imported records and report tables are temporary.

python3 - "$database" "$events" "$host" "$worker_host" "$day" <<'PY'
import json
import pathlib
import sqlite3
import sys

history, events, host, worker, day = sys.argv[1:]
if sqlite3.sqlite_version_info < (3, 42, 0):
    raise SystemExit("SQLite 3.42+ is required for subsecond timing")
uri = pathlib.Path(history).resolve().as_uri() + "?mode=ro"
with sqlite3.connect(uri, uri=True) as db:
    db.row_factory = sqlite3.Row
    db.execute("CREATE TEMP TABLE validation_events(payload TEXT)")
    with open(events, encoding="utf-8") as stream:
        db.executemany("INSERT INTO validation_events VALUES (?)",
                       ((line.rstrip("\n"),) for line in stream if line.strip()))
    db.execute("CREATE TEMP TABLE validation_report_params(host, worker_host, day)")
    db.execute("INSERT INTO validation_report_params VALUES (?, ?, ?)",
               (host, worker, day))
    sections = pathlib.Path("scripts/validation-report.sql").read_text().split("-- Output ")
    db.executescript(sections[0])
    for section in sections[1:]:
        title, query = section.split("\n", 1)
        print(title)
        for row in db.execute(query):
            print(json.dumps(dict(row), sort_keys=True))
PY

The outputs are:

• Coverage and observed wait fraction: uniquely attributed waiting seconds / observed dispatched code-session seconds for that host/day. Attempts are clipped to the UTC day and current time; incomplete attempts end at their last heartbeat, not a future lease. Multiple simultaneous gates in one attempt are unioned before summing. Matching requires the same issue and host with the gate start inside exactly one attempt interval. Ambiguous, standalone, and unmatched runs are counted, not guessed into an attempt. No matched measurements produces a null fraction. Even a non-null fraction is a lower bound when instrumentation coverage is incomplete.
• Mean, median, p90, and maximum queue depth weighted by seconds with waiters, reconstructed from observed wait intervals (including registration). Idle periods are excluded from these busy-depth statistics. This describes the lock represented by the input file, not unrelated repositories on the host. The separately reported max_observed_queue_depth comes from queue-position lookups and includes older clients that have no retained timing stream; max_instrumented_queue_depth and the time-weighted statistics cover only recorded runs. During rollout these can differ substantially: neither the instrumented maximum nor its typical depth describes the whole queue until coverage is complete. Simultaneous interval edges are combined. Incomplete runs contribute only their observed wait lower bounds, never an assumed wait through midnight.
• Whole-run wait/hold durations and completeness flags for runs observed in the day. Hold values on nonterminal records are not final measurements. These whole-run values must not be summed as clipped daily totals.

Malformed records and legacy records without run IDs are counted explicitly. Duplicate imports with the same run ID do not multiply measurements. A killed writer, a telemetry write error, an uninstrumented gate, or an incomplete event file prevents claims of complete coverage. The old worktree-local streams lack host/run attribution and cannot establish typical or worst-case historical queue depth. The #2962 observation (46m31.845s waiting, 11m14.254s running) is one run, not a host baseline. Collect a complete busy UTC day after rollout before using these statistics to choose a remedy; report the window, host mapping, coverage counts, and whether the day is partial.

Deadline and allowance accounting

Waiting can consume a configured worker session deadline. In internal/runner/agent.go, runAgentTurn starts sessionLimit before the provider turn; withAgentDurationLimit uses a context deadline. Running a shell command that waits in checklock does not suspend that deadline. Checklock also has its own no-progress and absolute queue-wait deadlines, which are distinct from the parent session deadline. After acquisition those queue deadlines no longer limit command runtime. A parent deadline can still cancel the command.

Waiting does not consume an additional attempt by itself, but the already started code/rework session counts toward the issue's attempt allowance. countSessionsWithoutMerge in internal/orchestrator/attempt_allowance.go counts started sessions and excludes specifically classified infrastructure failures, merge routing, and external waits. Validation-lock waiting has no special exclusion. A session that exhausts its duration while queued can therefore finish without running validation and still consume an allowance slot. This documents current accounting; #2977 changes neither those policies nor validation serialization.