Observability

Metrics, structured logs, and OpenTelemetry traces — opt-in so a quiet process pays nothing until you turn a channel on. The same span tree that powers the dev-bar flamegraph exports to your collector in production.

Three signals, one process

Prometheus at /_metrics, NDJSON on stdout, OTLP/HTTP to a collector. Pair JSON logs with traces so trace_id joins them in Grafana or Datadog. Env-var tables live on Configuration; this page is the operator guide.

Signal	Enable	Where it goes
Metrics	`SOLI_METRICS=1`	Prometheus text at `GET /_metrics`
Logs	`SOLI_LOG=…` (+ optional `SOLI_LOG_FORMAT=json`)	stdout / stderr
Traces	`SOLI_OTEL=1` or `OTEL_EXPORTER_OTLP_*`	OTLP/HTTP JSON to your collector
Health	always on	`GET /_health`, `GET /_ready`

Quick start

APP_ENV=production \
SOLI_METRICS=1 \
SOLI_LOG=access \
SOLI_LOG_FORMAT=json \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
OTEL_SERVICE_NAME=myapp \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production \
soli serve

Boot prints a one-line banner for each signal that is active (worker count, OTEL endpoint, JSON log format).

Health and readiness

Always available — nothing to enable. Use them for load balancers and orchestrators.

Endpoint	Meaning	Answers
`GET /_health`	Liveness	`200 ok` while the process runs, including mid-drain
`GET /_ready`	Readiness	`200 ready`, or `503 starting` / `503 draining`

Point liveness at /_health and readiness at /_ready. On SIGTERM readiness fails first so the LB stops routing, in-flight requests finish, then the process exits (bounded by SOLI_SHUTDOWN_GRACE_SECS, default 25s). Full drain walkthrough: Configuration → Health checks.

Metrics (`/_metrics`)

Collection is opt-in via SOLI_METRICS=1 (or true). Until that is set, counters stay at zero and the hot path skips the per-operation clocks.

SOLI_METRICS=1 soli serve
curl -s localhost:5011/_metrics

Metric	Meaning
`soli_http_requests_total`	Requests handled
`soli_lexing_duration_seconds` / `_count`	Time in the lexer
`soli_parsing_duration_seconds` / `_count`	Time in the parser
`soli_vm_execution_seconds` / `_count`	Bytecode VM wall time
`soli_template_render_duration_seconds` / `_count`	Views, layouts, partials
`soli_middleware_duration_seconds` / `_count`	Middleware totals
`soli_db_query_duration_seconds` / `_count`	SoliDB / SolidB query time
`soli_vm_handler_demotions_total`	Handlers that fell back from the VM to the tree-walker (cached per worker). `SOLI_ENGINE_LOG=1` prints one line per unique handler; `SOLI_FAIL_ON_VM_DEMOTION=1` exits the process when the VM refuses a handler, so CI cannot ship a new refuse. The bytecode VM only runs outside `--dev`, so neither applies to `soli serve --dev` or `soli test`
`soli_handler_panics_total`	Panics contained by the per-request `catch_unwind` (client got 500; worker stayed up)

soli_handler_panics_total and soli_vm_handler_demotions_total are counted even when SOLI_METRICS is off — rare enough that the atomics are free, and most wanted when nobody thought to enable collection in advance. There is no auth on /_metrics; bind it to a private interface or front it with a proxy that restricts access.

Structured logs

Channels (`SOLI_LOG`)

Comma-separated list. Any detail channel implies access so the block has a request line to hang off. Legacy: SOLI_REQUEST_LOG=1 aliases access.

Channel	What it prints
`access`	Method, path, status, handler ms (+ queue wait)
`query`	AQL with binds + duration (secret-looking bind names redacted)
`http`	Outgoing `HTTP.*` calls (credential-like query params redacted)
`kv`	SoliKV / Cache commands
`timing`	Middleware / view / phase breakdown
`all`	Everything

# Access only
SOLI_LOG=access soli serve

# Full per-request breakdown (noisy — prefer slow mode in prod)
SOLI_LOG=query,http,timing soli serve

Slow requests

SOLI_SLOW_REQUEST_MS emits the full detail block only when queue wait + handler time crosses a threshold. Fast requests stay silent unless you also asked for explicit channels.

SOLI_SLOW_REQUEST_MS=100 soli serve

Format (`SOLI_LOG_FORMAT`)

Default is multi-line human text. Set json for one NDJSON object per event on stdout (errors on stderr) so Loki / CloudWatch / Datadog can ingest without a parser.

SOLI_LOG=access SOLI_LOG_FORMAT=json soli serve

{
  "ts": "2026-08-09T12:00:00.123Z",
  "level": "info",
  "msg": "request",
  "method": "GET",
  "path": "/users",
  "status": 200,
  "duration_ms": 4.2,
  "total_ms": 4.2,
  "request_id": "…",
  "trace_id": "…",
  "span_id": "…"
}

Detail channels (or a slow hit) grow nested db / http / kv / timing arrays on the same object. Production errors use level: "error" and msg: "request_error" with a redacted request snapshot, stack, and env. No file rotation in-process — use your supervisor or container log driver.

Distributed tracing (OpenTelemetry)

Soli does not pull in the heavyweight OTel SDK. It reuses the hierarchical span tree already built for the dev-bar flamegraph and exports it as OTLP/HTTP JSON.

Enable

# Local collector sidecar (defaults to http://127.0.0.1:4318/v1/traces)
SOLI_OTEL=1 soli serve

# Explicit collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
OTEL_SERVICE_NAME=myapp \
soli serve

Variable	Role	Default
`SOLI_OTEL`	Force tracing on (`1` / `true` / `yes`)	unset
`OTEL_EXPORTER_OTLP_ENDPOINT`	Collector base URL; enables tracing. Appends `/v1/traces` unless already present.	unset
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	Full traces URL override	derived
`OTEL_SERVICE_NAME`	`service.name` resource attribute	`soli`
`OTEL_RESOURCE_ATTRIBUTES`	Extra `key=value` pairs, comma-separated	unset
`OTEL_SDK_DISABLED`	Force off when `true`	unset

What is exported

A SERVER root span (GET /path) with http.request.method, url.path, http.response.status_code, soli.request_id.
Nested middleware / before-after actions / controller actions / views / DB / outgoing HTTP spans — the same tree the flamegraph shows under --dev.
Export is async on a dedicated background thread. A full queue drops batches rather than stalling web workers; the first drop and the first POST failure print a one-time warning on stderr.

W3C Trace Context

Inbound traceparent is parsed and becomes the parent of the root span.
Every response carries traceparent so gateways and clients can correlate.
When tracing is on, responses also get X-Request-Id (unless X-Soli-Request-Id was already set in --dev).

Log ↔ trace joins

SOLI_LOG=access SOLI_LOG_FORMAT=json SOLI_OTEL=1 soli serve

JSON access lines include trace_id and span_id matching the exported root span. In Grafana / Datadog / Jaeger, jump from a log line to the full span tree.

Soli always samples when tracing is enabled (flags bit 0x01). Configure sampling, batching, and retention on the collector (Grafana Tempo, Jaeger, Datadog agent, OpenTelemetry Collector, …) rather than in the Soli process.

Dev vs production

	`--dev`	Production
Dev bar (queries, flamegraph, replay)	on	off
Access log	always on	`SOLI_LOG` / `SOLI_REQUEST_LOG`
JSON format	available	available
Span tree	flamegraph	OTLP when OTEL on
Metrics	opt-in	opt-in
Health endpoints	on	on

Limits (honest)

No auto-instrumentation of every third-party client library — only Soli's own request path, ORM, and HTTP.* client.
OTLP export is traces only (not metrics or logs pipelines). Metrics stay on Prometheus /_metrics; logs stay on stdout.
Outbound traceparent injection on every HTTP.* call is not yet automatic; inbound propagation and response echo are.
No in-process sampling UI — put that on the collector.