Observability

Velnor exposes local, read-only operator views. They answer different questions and have different retention. A healthy local view does not prove that GitHub accepted a result.

Navigation: ← Operator guide · Index · Troubleshooting

First checks

velnorctl status
velnorctl status --json
velnorctl get hosts
velnorctl get instances
velnorctl get slots
velnorctl get runners
velnorctl get jobs
velnorctl get runs
velnorctl get queue
velnorctl events
velnorctl logs JOB_UID_OR_RUN_ID
velnorctl telemetry
velnorctl top host
velnorctl top instances
velnorctl top slots
velnorctl top jobs
velnorctl top storage
velnorctl doctor --url https://github.com/ORG --name NAME --slots N
velnorctl preflight --config-dir DIR --work-dir DIR
velnorctl capabilities export
velnorctl capabilities check JOB_MESSAGE.json
velnorctl run list
velnorctl run view RUN_ID
velnorctl run watch RUN_ID
velnorctl run logs RUN_ID

These are read-only views. velnorctl top TARGET requires one of host, instances, slots, jobs, or storage; it queries that resource collection and is not a metrics backend. velnorctl wait takes a snapshot, watches for changes, and resnapshots when its bounded cursor expires.

RUN_ID is a numeric workflow-run ID. JOB_UID_OR_RUN_ID is either that bare numeric run ID or the exact job UID; neither subject uses a run/ or job/ prefix.

The control socket is normally /run/velnor/<instance>/control.sock. The admin socket is separate, but mutation routes are currently unsupported; /v1/info reports mutations:false. Reconciliation, storage mutation, diagnostics-bundle, and run mutation commands therefore cannot be used as operator actions today.

doctor checks registered runners against GitHub. preflight checks local configuration and execution prerequisites. Capability export/check inspects the strict capability manifest and a job message. Run list/view/watch/logs are read-only; run cancel, rerun, download, dispatch, and open are unavailable. Run watch currently treats its run identifier as a cursor and does not filter events by run identity.

Health

The guardian writes an atomic health.json snapshot. An optional health.sock provides a one-request fast path; if it is absent or unusable, read the file. A missing or invalid file means health is unavailable, not healthy.

For the packaged single-instance service, the exact paths are /var/lib/velnor/health.json and /var/lib/velnor/health.sock. For a templated instance named NAME, they are /var/lib/velnor-NAME/health.json and /var/lib/velnor-NAME/health.sock. These paths come from systemd's StateDirectory; custom --config-dir or storage settings can change the runtime directory, so use the instance's configured state directory when those settings are in use.

Health includes control and journal state, GitHub connectivity, routing and runner-group proof, desired/actual/ready capacity, executor/backend readiness, queue and outbox age, lifecycle state, and alert codes.

Interpret states carefully:

• ready: control, journal, routing, registration, executor, and capacity checks permit work.
• degraded: Velnor is alive but a dependency, routing proof, registration, or capacity condition is impaired.
• not_ready: Velnor must not accept work, for example when control/journal is unavailable or no slot/permit is ready.

controller-metrics.json is a current local snapshot: process counts, reconcile timing, journal/WAL bytes, and CPU totals. Phase CPU buckets are currently zero; do not treat them as measured per-phase metrics. There is no Prometheus endpoint, dashboard, SSE endpoint, or HTTP health endpoint.

Jobs, events, logs, and telemetry

Resource queries cover hosts, instances, slots, runners, jobs, runs, queue, events, reservations, and leases. Use the repository and workflow/job identity shown in a job or run record to correlate with GitHub. When local records are ambiguous, correlate timestamps with the GitHub workflow run and job ID; Velnor does not provide a separate dashboard correlation view.

velnorctl events reads a bounded event stream. Watches retain at most 4,096 items. Event cursors are opaque and expire; resnapshot before continuing.

velnorctl logs JOB_UID_OR_RUN_ID reads bounded local log records (up to 16,384 records or 16 MiB). Log cursors expire with retention. --step and --failed are accepted by the CLI but currently do not filter results. Runner forensic files are lifecycle.log, broker.log, registry.log, and daemon.log; each has an active file and one rotated .1 generation. These writers are best effort.

GitHub receives separate log forms: live masked lines over the Results Service WebSocket, uploaded step/job blobs, and timeline feed updates. Local logs can show that publishing was attempted; GitHub is the authority for what it actually received and rendered.

velnorctl telemetry returns versioned records with opaque cursors. A cursor is a continuation position, not a timestamp or record ID to interpret. When a page has more records, next_cursor tells the next request where to continue; dropped_before reports the oldest available position when retention has already discarded earlier records. Example response shape:

{
  "records": [{"cursor": "opaque", "kind": "..."}],
  "next_cursor": "opaque",
  "dropped_before": "opaque"
}

The default bounds are 4,096 records and an 8 MiB file. Rotation drops old generations; telemetry is not an archive. If a cursor expires, start with a fresh snapshot/page. The optional trace file is <config>/logs/trace.jsonl, rotated at 32 MiB with one .1. OTLP (OpenTelemetry Protocol) export is optional: set VELNOR_OTLP_ENDPOINT to send traces to an OTLP collector; export failure does not stop the runner.

Durability and visibility limits

The SQLite control journal durably stores events and recovery intent. Normal query, log, and storage projections are not fully rehydrated after service recreation, so a reopened service can have durable events while resource or log views are empty. Treat an empty projection after restart as a visibility limitation, not proof that no work occurred.

Completion recovery uses local outbox state plus remote acknowledgement. An outbox entry is retained until GitHub acknowledges completion. Inspect local outbox age and completion-related events when execution succeeded locally but GitHub remains pending.

There is no currently available diagnostics bundle command. Fixture reports and host-doctor scripts are maintainer/evidence tooling, use synthetic inputs in some cases, and are not live-service observability.

Correlating a GitHub job

1. Copy the GitHub repository, workflow run, job, and queued/start timestamps.
2. Query velnorctl get runs, get jobs, and events.
3. Confirm runner, slot, labels, backend, admission, and completion state.
4. Inspect velnorctl logs JOB_UID_OR_RUN_ID and velnorctl telemetry for the same subject and time window.
5. Compare GitHub's rendered logs/timeline/result with Velnor's local terminal and outbox evidence.

The broker delivers a job reference; the run service supplies the full job. Therefore a broker-delivery event without an acquired job is not execution, and local execution completion without remote acknowledgement is not GitHub completion.