# Observability (/operations/observability) # Observability Velnor exposes local, read-only operator views. They answer different questions and have different retention. A healthy local view does not prove that GitHub accepted a result. > Navigation: [← Operator guide](/guides/operator) · [Index](/) · [Troubleshooting](/troubleshooting) ## First checks ```sh velnorctl status velnorctl status --json velnorctl get hosts velnorctl get instances velnorctl get slots velnorctl get runners velnorctl get jobs velnorctl get runs velnorctl get queue velnorctl events velnorctl logs JOB_UID_OR_RUN_ID velnorctl telemetry velnorctl top host velnorctl top instances velnorctl top slots velnorctl top jobs velnorctl top storage velnorctl doctor --url https://github.com/ORG --name NAME --slots N velnorctl preflight --config-dir DIR --work-dir DIR velnorctl capabilities export velnorctl capabilities check JOB_MESSAGE.json velnorctl run list velnorctl run view RUN_ID velnorctl run watch RUN_ID velnorctl run logs RUN_ID ``` These are read-only views. `velnorctl top TARGET` requires one of `host`, `instances`, `slots`, `jobs`, or `storage`; it queries that resource collection and is not a metrics backend. `velnorctl wait` takes a snapshot, watches for changes, and resnapshots when its bounded cursor expires. `RUN_ID` is a numeric workflow-run ID. `JOB_UID_OR_RUN_ID` is either that bare numeric run ID or the exact job UID; neither subject uses a `run/` or `job/` prefix. The control socket is normally `/run/velnor//control.sock`. The admin socket is separate, but mutation routes are currently unsupported; `/v1/info` reports `mutations:false`. Reconciliation, storage mutation, diagnostics-bundle, and run mutation commands therefore cannot be used as operator actions today. `doctor` checks registered runners against GitHub. `preflight` checks local configuration and execution prerequisites. Capability export/check inspects the strict capability manifest and a job message. Run list/view/watch/logs are read-only; run cancel, rerun, download, dispatch, and open are unavailable. Run watch currently treats its run identifier as a cursor and does not filter events by run identity. ## Health The guardian writes an atomic `health.json` snapshot. An optional `health.sock` provides a one-request fast path; if it is absent or unusable, read the file. A missing or invalid file means health is unavailable, not healthy. For the packaged single-instance service, the exact paths are `/var/lib/velnor/health.json` and `/var/lib/velnor/health.sock`. For a templated instance named `NAME`, they are `/var/lib/velnor-NAME/health.json` and `/var/lib/velnor-NAME/health.sock`. These paths come from systemd's `StateDirectory`; custom `--config-dir` or storage settings can change the runtime directory, so use the instance's configured state directory when those settings are in use. Health includes control and journal state, GitHub connectivity, routing and runner-group proof, desired/actual/ready capacity, executor/backend readiness, queue and outbox age, lifecycle state, and alert codes. Interpret states carefully: * `ready`: control, journal, routing, registration, executor, and capacity checks permit work. * `degraded`: Velnor is alive but a dependency, routing proof, registration, or capacity condition is impaired. * `not_ready`: Velnor must not accept work, for example when control/journal is unavailable or no slot/permit is ready. `controller-metrics.json` is a current local snapshot: process counts, reconcile timing, journal/WAL bytes, and CPU totals. Phase CPU buckets are currently zero; do not treat them as measured per-phase metrics. There is no Prometheus endpoint, dashboard, SSE endpoint, or HTTP health endpoint. ## Jobs, events, logs, and telemetry Resource queries cover hosts, instances, slots, runners, jobs, runs, queue, events, reservations, and leases. Use the repository and workflow/job identity shown in a job or run record to correlate with GitHub. When local records are ambiguous, correlate timestamps with the GitHub workflow run and job ID; Velnor does not provide a separate dashboard correlation view. `velnorctl events` reads a bounded event stream. Watches retain at most 4,096 items. Event cursors are opaque and expire; resnapshot before continuing. `velnorctl logs JOB_UID_OR_RUN_ID` reads bounded local log records (up to 16,384 records or 16 MiB). Log cursors expire with retention. `--step` and `--failed` are accepted by the CLI but currently do not filter results. Runner forensic files are `lifecycle.log`, `broker.log`, `registry.log`, and `daemon.log`; each has an active file and one rotated `.1` generation. These writers are best effort. GitHub receives separate log forms: live masked lines over the Results Service WebSocket, uploaded step/job blobs, and timeline feed updates. Local logs can show that publishing was attempted; GitHub is the authority for what it actually received and rendered. `velnorctl telemetry` returns versioned records with opaque cursors. A cursor is a continuation position, not a timestamp or record ID to interpret. When a page has more records, `next_cursor` tells the next request where to continue; `dropped_before` reports the oldest available position when retention has already discarded earlier records. Example response shape: ```json { "records": [{"cursor": "opaque", "kind": "..."}], "next_cursor": "opaque", "dropped_before": "opaque" } ``` The default bounds are 4,096 records and an 8 MiB file. Rotation drops old generations; telemetry is not an archive. If a cursor expires, start with a fresh snapshot/page. The optional trace file is `/logs/trace.jsonl`, rotated at 32 MiB with one `.1`. OTLP (OpenTelemetry Protocol) export is optional: set `VELNOR_OTLP_ENDPOINT` to send traces to an OTLP collector; export failure does not stop the runner. ## Durability and visibility limits The SQLite control journal durably stores events and recovery intent. Normal query, log, and storage projections are not fully rehydrated after service recreation, so a reopened service can have durable events while resource or log views are empty. Treat an empty projection after restart as a visibility limitation, not proof that no work occurred. Completion recovery uses local outbox state plus remote acknowledgement. An outbox entry is retained until GitHub acknowledges completion. Inspect local outbox age and completion-related events when execution succeeded locally but GitHub remains pending. There is no currently available diagnostics bundle command. Fixture reports and host-doctor scripts are maintainer/evidence tooling, use synthetic inputs in some cases, and are not live-service observability. ## Correlating a GitHub job 1. Copy the GitHub repository, workflow run, job, and queued/start timestamps. 2. Query `velnorctl get runs`, `get jobs`, and `events`. 3. Confirm runner, slot, labels, backend, admission, and completion state. 4. Inspect `velnorctl logs JOB_UID_OR_RUN_ID` and `velnorctl telemetry` for the same subject and time window. 5. Compare GitHub's rendered logs/timeline/result with Velnor's local terminal and outbox evidence. The broker delivers a job reference; the run service supplies the full job. Therefore a broker-delivery event without an acquired job is not execution, and local execution completion without remote acknowledgement is not GitHub completion.