Documentation
Operations and limits
Operational limits, failure checks, and benchmark context.
Fixed request and render limits
| Limit | Value |
|---|---|
| HTTP request body | 1 MiB |
| Render data | 64 KiB |
| One request asset | 256 KiB |
| All request assets | 512 KiB |
| One published object | 2 MiB |
| One downloaded control-plane object | 4 MiB |
| Concurrent cold catalog loads | 32 |
| Cold catalog load deadline | 30 seconds |
| Worker-pool target | 2 |
| One render per worker | 1 |
| Render deadline | 10 seconds |
Operator configuration
The registry is a local bounded LRU cache, not a catalog mirror. A cold render reads
the version from PostgreSQL and S3, SHA-256-verifies its objects, and then caches it.
Startup does no catalog-wide load and eviction never prevents publication. Local
publish, load, retire, and unretire work is coordinated per version, so slow I/O for
one version does not block warm reads of another. Cross-replica retirement
PostgreSQL LISTEN/NOTIFY invalidates peers, while freshness-bounded
reconciliation repairs missed notifications under ADR 0004.
| Variable | Default | Meaning |
|---|---|---|
REGISTRY_MAX_VERSIONS | 128 | Versions in one renderer cache |
REGISTRY_MAX_BYTES | 67108864 | Source, font, and static bytes in that cache |
MAX_ACTIVE_VERSIONS | 10000 | Deployment-wide published-version limit |
DEFAULT_TEMPLATE_ACTIVE_VERSIONS | 100 | Default template published-version limit |
MAX_STORAGE_BYTES | 1073741824 | Distinct retained-object byte cap |
OBJECT_GC_GRACE_SECONDS | 86400 | Minimum unreferenced-object age before collection |
OBJECT_GC_INTERVAL_SECONDS | 300 | Background collector interval |
WORKER_BINARY | sibling katagami-worker | Dedicated worker executable; production Compose uses /usr/local/bin/katagami-worker |
WORKER_POOL_SIZE | 2 | Target supervised workers; each admits one render |
RENDER_DEADLINE_MS | 10000 | Per-worker render deadline |
WORKER_MEMORY_BYTES | 536870912 | Per-worker address-space limit on Unix except macOS; unsupported native hosts fail startup |
WORKER_MAX_REQUEST_BYTES | 16777216 | Maximum worker IPC request |
WORKER_MAX_RESPONSE_BYTES | 33554432 | Maximum worker IPC response |
WORKER_RESTART_BUDGET | 5 | Deaths allowed in the restart window before the pool degrades |
WORKER_RESTART_WINDOW_MS | 60000 | Restart-budget window |
WORKER_RESTART_BACKOFF_MS | 250 | Replacement-spawn delay |
WORKER_RECOVERY_COOLDOWN_MS | 30000 | Delay before a degraded pool retries recovery |
WORKER_LIVENESS_INTERVAL_MS | 5000 | Worker liveness-probe cadence |
WORKER_LIVENESS_TIMEOUT_MS | 1000 | Liveness deadline; cannot exceed its interval |
All values are positive integers. PostgreSQL applies active-version and storage limits under its transaction-scoped advisory lock. Per-template overrides are SQL-resident operator policy, never publish input:
Compose applies RENDERER_CPUS (default 2.0) and RENDERER_MEMORY_LIMIT
(default 2g) to the HTTP server and its complete worker pool. Size the pool within
those shared cgroup limits. Native startup fails closed on macOS and non-Unix hosts
because WORKER_MEMORY_BYTES cannot be enforced. Linux containers enforce both the
per-worker address-space limit and the container cgroup limit.
UPDATE templates SET active_version_limit = 3 WHERE id = 'invoice';
UPDATE templates SET active_version_limit = NULL WHERE id = 'invoice';
Retirement frees an active slot but retains objects. Unretirement rechecks active capacity and re-verifies the stored pack. It does not free retained-byte capacity.
Legacy rows with object_bytes = NULL are repaired only when a cold render
SHA-256-verifies that one version and records exact bytes. Startup never reloads the
whole catalog for this. New publications require exact bytes. Until retained rows are
accounted, publication fails closed with 503 catalog retained-version byte capacity exceeded or incomplete.
New publications store one object per distinct SHA-256. PostgreSQL reserves unknown bytes before upload and records manifest references at commit. The collector removes only unreferenced global objects after the configured grace period. Legacy per-version keys remain readable and are never collector targets.
Metrics, traces, and privacy
Every renderer has a process-local Prometheus endpoint at GET /metrics on its
dedicated METRICS_ADDR listener, defaulting to 0.0.0.0:9090. The public API
listener returns 404 for /metrics. The supplied Compose files deliberately do not
publish port 9090: scrape http://renderer:9090/metrics and
http://renderer2:9090/metrics from the internal monitoring network instead of a
public proxy or ingress.
| Signal | Stable metric |
|---|---|
| Request rate and errors | katagami_http_requests_total, katagami_http_errors_total |
| HTTP latency | katagami_http_request_duration_seconds |
| Render rate, latency, and admission | katagami_render_requests_total, katagami_render_duration_seconds, katagami_render_active, katagami_render_capacity |
| Worker pool health | katagami_worker_pool_state, katagami_worker_pool_workers, katagami_worker_pool_failures_total |
| Catalog sync | katagami_catalog_sync_age_seconds, katagami_catalog_sync_failures_total |
| PostgreSQL and S3 failures | katagami_dependency_failures_total |
Treat these as views of failures, not independent incident counters. A PostgreSQL
catalog-listener transport error can increment both
katagami_dependency_failures_total{dependency="postgres",operation="receive"}
and katagami_catalog_sync_failures_total{class="receive"}. Count that as
one incident: the first metric identifies the transport cause and the second records
the catalog-sync effect. Do not add the two values together.
Labels are closed operational classifications. Template IDs, versions, request IDs, asset names, raw URLs, request JSON, and other unbounded customer values are never metric labels.
Metrics work without a collector. Set OTEL_EXPORTER_OTLP_ENDPOINT only for an
operator-managed OTLP HTTP/protobuf collector; Katagami posts to its /v1/traces
endpoint. The base URL can use HTTP or HTTPS; HTTPS uses the container’s trusted CA
certificates. Sampling, storage, and retention are operator policy, and exporter
failure is best effort: it cannot change requests, liveness, or readiness. Logs and
traces exclude render JSON, request assets, admin tokens, credentials, source files,
fonts, and PDF bytes.
Prometheus scrape failure and OTLP exporter or collector failure do not stop request handling. An unexpected exit of Katagami’s internal metrics listener is different: Katagami starts graceful shutdown and exits. Run Katagami under production supervision that restarts the process.
First checks
| Symptom | First check |
|---|---|
| Renderer exits at startup | Check PostgreSQL, object storage, and required environment variables |
502 control plane unavailable | Check object hashes and PostgreSQL and storage connectivity |
409 template version already exists or template is unknown | Use a new immutable version name or create the template ID |
409 template version lifecycle state conflicts with requested transition | The version is already in the requested lifecycle state |
410 on render | The version is retired; unretire only if active capacity permits |
413 | Reduce request data, assets, objects, or worker IPC frames |
422 | Check the request JSON shape, validate it against the published schema, then inspect Typst source and declared assets |
502 render worker died while rendering | Inspect the worker exit in renderer logs; clients may retry a read-only render |
503 renderer cannot currently admit work | Retry after saturation, degradation, or shutdown clears |
503 from /health/ready | Keep the renderer out of traffic until catalog freshness and worker admission recover |
503 active template version capacity exceeded | Raise an operator active-version limit or retire an active version |
503 catalog retained-version byte capacity exceeded or incomplete | Raise retained-byte capacity or cold-verify a legacy row; retirement does not help |
503 template catalog load capacity exceeded or timed out | Retry after current cold catalog loads finish; warm cache hits remain available |
The startup log has a closed failure_class. config identifies invalid environment
configuration. public_api_listener and metrics_listener identify bind failures.
postgres_connect, postgres_migrate, storage, and worker_config identify the
failed startup stage without exposing an endpoint, credential, or raw error.
Readiness and recovery
Use GET /health/live only to prove the HTTP process can answer requests. Route
traffic with GET /health/ready: it returns 204 only while catalog reconciliation
is within its freshness bound and the worker pool can admit a render. It returns
503 while the pool is saturated, degraded after its restart budget, stopping, or
while catalog freshness is stale. A worker death is reaped and replaced after the
configured backoff; budget exhaustion enters the configured recovery cooldown before
replacement is retried.
Benchmark context
On an ARM64 Docker host with both renderers capped at 2 CPU and 2 GiB, a small, steady-state invoice workload measured Katagami at 1,440–2,151 requests per second. The equivalent Gotenberg HTML conversion measured 10–14 requests per second. This is a stored-template API comparison, not a general renderer-only claim.
Reproduce it with:
REQUESTS=100 CONCURRENCY=4 CPUS=2 MEMORY=2g \
bash scripts/benchmark-gotenberg.sh