Indara

Personal web archive with BM25 full-text search, built on PostgreSQL + pg_textsearch. An alternative to Hister that keeps the search index in Postgres instead of an in-process bleve index.

Layout

• apps/server — axum API + serves the frontend (/openapi.json for the OpenAPI spec; bring your own viewer)
• apps/worker — pg-jobs worker running the fetch → extract → index pipeline
• apps/migrate — applies application + pg-jobs migrations (k8s pre-upgrade job)
• apps/cli — indara-cli import <hister-dataset.json> through the HTTP API
• crates/indara-core — config, language routing
• crates/indara-db — queries + migrations/
• crates/indara-jobs — pipeline steps, readability extraction, sanitization
• frontend/ — React 19 + Vite SPA
• charts/indara — Helm chart (OCI: oci://forge.netserv.fr/azsiaz/charts/indara)

Dev quickstart

docker compose up -d           # postgres (pg_textsearch preloaded) + dex + minio
cargo run -p indara-migrate
(cd frontend && npm ci && npm run build)
cargo run -p indara-server     # http://localhost:3001 (BIND_ADDR/PUBLIC_URL in .env.local)
cargo run -p indara-worker     # separate terminal

The server uses one restricted PostgreSQL connection through DATABASE_SERVER_URL. It authenticates as indara_control for identity/admin/jobs and executes SET LOCAL ROLE indara_runtime inside every RLS-protected document transaction. Only the migration binary uses the owner connection in DATABASE_URL. The worker connects through DATABASE_WORKER_URL, authenticating as indara_worker, a DML-only role with no membership; its boot guard refuses a superuser or owner credential. A new Compose volume creates both roles and their restricted membership automatically. In local development, use:

DATABASE_SERVER_URL=postgresql://indara_control@localhost:5433/app

For a volume created before that bootstrap existed, run the migration first, then normalize the development roles and membership:

cargo run -p indara-migrate
docker compose exec postgres psql -U postgres -d app \
  -c 'ALTER ROLE indara_control LOGIN NOINHERIT; ALTER ROLE indara_runtime NOLOGIN NOINHERIT; GRANT indara_runtime TO indara_control WITH ADMIN FALSE, INHERIT FALSE, SET TRUE'

Environment is loaded from .env.local via mise. Dev OIDC login (Dex): stef@example.com / password. Set OIDC_ADMIN_GROUP=indara-admins to make that development user an application administrator.

Raw page snapshots go to an S3-compatible bucket. The Compose stack runs MinIO (console at http://localhost:9001, indara / indara-dev-secret) and creates the indara-snapshots bucket on startup. Point the server and worker at it in .env.local:

INDARA_S3_ENDPOINT=http://localhost:9000
INDARA_S3_BUCKET=indara-snapshots
INDARA_S3_ACCESS_KEY_ID=indara
INDARA_S3_SECRET_ACCESS_KEY=indara-dev-secret

Import a Hister dataset with an API key created in the account UI (/me):

cargo run -p indara-cli -- import go-stdlib.json --api-key indara_...

Auth model

Every endpoint requires auth except /healthz, /api/auth/*, /opensearch.xml and the static assets. Browsers use the OIDC session cookie; third-party import tools send Authorization: Bearer <key> or X-API-Key. API keys have the fixed documents:write scope, expire after 30 days, and can only import documents plus read the key owner's basic /api/me identity and groups. Creating, listing and revoking keys requires a browser session.

OIDC profile pictures and group memberships are synchronized at login. OIDC_ADMIN_GROUP names the exact, case-sensitive OIDC group whose members are application administrators; an unset or empty value leaves the application without an administrator. Application admins can use the /api/admin/* identity, key and job views. They do not bypass document permissions.

An application admin may assign one current member as a group's admin, or leave the group without one. A group admin may delete a document assigned to that group or remove it from the group, but otherwise has ordinary user permissions.

Documents imported from the UI or API are private by default. A document is readable by its owner, by members of its owner group, or through an explicit user/group share; a share may expire or remain valid indefinitely. public means visible to every authenticated Indara user, not to anonymous Internet visitors. Only the individual owner can change visibility, assign a document to a group, or manage its shares. Raw fast-path HTML in document_sources is write-only to the HTTP server and is read only by the worker.

PostgreSQL RLS enforces these rules. The HTTP server opens one pool as indara_control, a direct LOGIN NOINHERIT role with identity/admin/jobs privileges and no direct document privilege. Its only membership is indara_runtime with ADMIN FALSE, INHERIT FALSE, SET TRUE. indara_runtime is a NOLOGIN NOINHERIT role whose document queries run only in short, role-local transactions carrying the current user id, authentication method and scopes. Startup validates both roles, their exact membership and their privilege boundaries. Only the migration binary receives the trusted owner DATABASE_URL; the HTTP server must never receive it. The worker connects through DATABASE_WORKER_URL as indara_worker, a DML-only role with no membership, and its boot guard refuses a superuser or owner credential.

Worker SSRF protection

The worker only fetches public destinations. Two layers:

• Application (always on): fetch DNS resolution goes through INDARA_FETCH_DNS (default 1.1.1.1:53,9.9.9.9:53), any non-public IP (loopback, RFC1918, link-local, metadata, ULA…) is rejected, the resolved IP is pinned (anti-rebinding), and redirects are followed manually (INDARA_FETCH_MAX_REDIRECTS, default 5), revalidating each hop.
• Network (K8s, backstop, opt-in): worker.networkPolicy.enabled=true and/or worker.ciliumNetworkPolicy.enabled=true. worker.fetchDns is the single source of truth: it feeds both the app's env and the egress DNS rules. Hardened mode: address the DB by IP/ExternalName and set worker.networkPolicy.allowClusterDns=false so the worker can only talk to public resolvers.

Search

One partial BM25 index per language (documents_bm25_en, …). pg_textsearch has no per-row text config, so unknown languages are normalized to en at write time (v1 indexes everything as English; detection code exists behind DETECT_LANGUAGES). Queries must name the index via to_bm25query and repeat the partial-index predicate — see crates/indara-db/src/documents.rs.

Notes

• .sqlx/ is committed; CI runs cargo sqlx prepare --check --workspace.
• The migrate binary runs the application and pg-jobs migrators; the latter keeps its migration history in the jobs schema.
• The CNPG cluster must preload pg_textsearch (postgresql.shared_preload_libraries) — see charts/indara/README.md.