Skip to manual
gluon

Operations

Deploy Gluon and investigate service failures

Run the API, workers, storage and model gateway, and find out why one of them stopped.

Platform operator

Gluon ships as two Docker images. The api image runs the FastAPI service on port 8410, and the same image runs the workers with gluon worker. The web image serves chat on 8080, the designer on 8081 and the admin console on 8082. Around them you run Postgres (a master database and a separate audit database), Elastic for instances and vectors, the LiteLLM gateway for every model call, a key service and an OpenTelemetry collector. This page covers putting that stack up and finding out why a part of it stopped. Deploying an ontology version to people is a separate job.

Configure before you start anything

Every setting is a GLUON_* environment variable. The defaults are laptop values: localhost services, the file key service, the dev tenant. Outside GLUON_ENVIRONMENT=development the API and the worker refuse to start while any of those defaults is left, and they name each variable that still needs a value.

  1. Set the tenant, the environment, both database URLs (GLUON_DATABASE_URL, GLUON_AUDIT_DATABASE_URL), the app hosts and GLUON_OTEL_ENDPOINT, then every other variable the API names when it refuses to start. .pl.yaml shows a working set for Gluon on Pushleft.
  2. Put secrets in the key service and give Gluon references to them (vault: in production), such as GLUON_GATEWAY_KEY_REF. Keys for AI services, the TypeSafe System One key included, are entered in the admin console under AI connections and go to the key service; a key your operators keep in the gateway's environment is named there by its variable instead.
  3. Set the GLUON_PUBLIC_* variables on the web container. They are written to /env-config.js when the container starts, so the apps find each other's hosts at run time.

These settings tune how an Accurate answer searches (Fast has its own GLUON_FAST_* settings). Leave them unset unless an evaluation shows a change helps.

SettingDefaultWhat it does
GLUON_ANSWER_K20The passages an answer reads. A Context plan that sets its own number wins.
GLUON_ANSWER_EXPAND5The top this many documents are read further: up to 12,000 characters around the match.
GLUON_ANSWER_QUERIES3Up to this many extra search queries (at most 8) the model writes besides the question.
GLUON_ANSWER_WIDE0Each search reads this many passages and the model picks the ones the question needs.
GLUON_ANSWER_STEPS4Turns of the search loop (at most 12): the model reads the results, searches again or opens one, then keeps the ones the answer needs. Each turn is one more model call.
GLUON_ANSWER_BY_DOCUMENTtrueEach search keeps one passage per document, so the answer reads more distinct documents.
GLUON_RERANK_CANDIDATES100With an approved reranker Model, the documents it re-orders per search (at most 300, one passage each). 0 turns reranking off.
GLUON_SEARCH_TEXT_WEIGHT2The keyword (BM25) leg's weight against the meaning leg's 1 when results are fused, in both modes. 1 weighs them equally.

Deploy

  1. Migrate the databases as a step of its own:

    Shell
    gluon migrate
    

    It prints migrated master and audit to head.

  2. Start the API and the workers, which read the durable Postgres queue. Gluon runs on Pushleft with a .pl.yaml manifest: one API, one worker and a managed Postgres with a daily backup and point-in-time restore.

  3. Replicas of the API, automatic Postgres failover and a second region are not part of a Gluon deployment yet. They wait for a first customer who needs them.

  4. Point the settings at your search store, gateway and key service.

For a local stack, docker compose -f ops/compose/docker-compose.yml --profile all up starts Postgres on 5442, Elastic on 9210, the gateway on 4410 and the collector on 4417 and 4418.

The gateway keeps its routes in its own database (litellm, created on first start) and encrypts the keys Gluon pushes there with LITELLM_SALT_KEY. Set that once and never change it, and include the gateway's database in your backups and key drill. Gluon pushes routes for the approved Models and the model routes on Admin › Models through their AI connections on every console change and every minute. Routes the gateway's own config file holds (gluon gateway-config, for air-gapped installs) are left alone.

Read the Health page

Open the admin console and go to Health & pipeline › Health. Each row is one indicator with its State (Within threshold or Breached), the value Now, the Threshold and the Runbook to follow. When anything is breached, a Degraded callout at the top counts the indicators past threshold.

IndicatorBreached when
Directory reconciliation ageover 15 min
ACL lag, per mirrored sourceover 15 min, or never synced
Data older than its expected refreshpast the Source's expect
Queue age, per queueover 5 min
Audit buffer fillover 50%
Model route, and the gateway itselfdown
Gateway serves the approved routesdown: a Model has no route, with the reason
Model spend used, per departmentover 85%
Storage adapternot answering
Versions served to the audiencemore than one version on a segment
Probea failed run, or no result for three periods

The worker runs the database, audit-buffer, store, permission and gateway probes every minute, and a synthetic question every ten minutes through chat, API and MCP in turn. A probe showing "not run yet: is the worker running?" usually means just that. The gateway row is checked live on each page load and cached for 10 seconds.

The same data is available to scripts:

Shell
curl -s -H "Authorization: Bearer $TOKEN" $GLUON/api/admin/health | jq '.indicators[] | select(.breached)'

Follow the runbook

Every indicator links to its runbook in ops/runbooks/, with the alert, the impact, how to diagnose, how to mitigate and how to verify recovery. A P1 is acknowledged within 15 minutes. Fix the dependency the row names and reload the page; the row clears on the next read or the next probe run.

Trace a question

The API exports traces over OTLP to GLUON_OTEL_ENDPOINT. One question is one trace: a gluon.chat root with gluon.retrieval, gluon.pdp, gluon.model and gluon.audit spans under it. Span attributes come from an allowlist, so no span carries the question or the person asking. Gluon has no trace viewer of its own; read traces in the tracing backend your collector feeds. A gluon.pdp span covers a whole batch of candidates, so its duration is the batch's time.

The Health page with the All indicators within threshold callout and a table of indicators, states, values, thresholds and runbooks.
Health page with every indicator within threshold and a runbook on every row.