AzSiAz/Indara
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity

feat: per-domain rate limiting for the document pipeline #1

Merged
AzSiAz merged 24 commits from feat/per-domain-rate-limit into main 2026-07-05 12:40:10 +02:00
Conversation 0 Commits 24 Files changed 88 +8551 -1335
AzSiAz commented 2026-06-14 17:48:50 +02:00
Owner
Copy link

Throttles outbound fetches per host:port so a small self-hosted or VPS
target is not hammered. A Postgres GCRA token bucket (host_throttle, one
row per host) is shared across worker replicas; a new gate step reserves
a slot and reschedules the fetch with GoTo::Delay instead of sleeping, so
no worker slot is held while waiting.

On 429/503 the host is backed off and the job's own run_at is pushed
(honoring Retry-After, otherwise exponential backoff), both with jitter
to avoid stampedes when many requests share a deadline. Stale
host_throttle rows are pruned hourly by a standalone maintenance job.

The pipeline namespace is bumped to v2 because a step was inserted:
apalis dispatches stepped jobs by index, so in-flight jobs from the old
shape must drain (re-enqueue) rather than run under the new chain.

WORKER_CONCURRENCY default raised 2 -> 8 so per-host pacing does not
starve other hosts.

Throttles outbound fetches per host:port so a small self-hosted or VPS target is not hammered. A Postgres GCRA token bucket (host_throttle, one row per host) is shared across worker replicas; a new gate step reserves a slot and reschedules the fetch with GoTo::Delay instead of sleeping, so no worker slot is held while waiting. On 429/503 the host is backed off and the job's own run_at is pushed (honoring Retry-After, otherwise exponential backoff), both with jitter to avoid stampedes when many requests share a deadline. Stale host_throttle rows are pruned hourly by a standalone maintenance job. The pipeline namespace is bumped to v2 because a step was inserted: apalis dispatches stepped jobs by index, so in-flight jobs from the old shape must drain (re-enqueue) rather than run under the new chain. WORKER_CONCURRENCY default raised 2 -> 8 so per-host pacing does not starve other hosts.
AzSiAz self-assigned this 2026-06-14 17:48:50 +02:00 
Throttles outbound fetches per host:port so a small self-hosted or VPS
target is not hammered. A Postgres GCRA token bucket (host_throttle, one
row per host) is shared across worker replicas; a new gate step reserves
a slot and reschedules the fetch with GoTo::Delay instead of sleeping, so
no worker slot is held while waiting.

On 429/503 the host is backed off and the job's own run_at is pushed
(honoring Retry-After, otherwise exponential backoff), both with jitter
to avoid stampedes when many requests share a deadline. Stale
host_throttle rows are pruned hourly by a standalone maintenance job.

The pipeline namespace is bumped to v2 because a step was inserted:
apalis dispatches stepped jobs by index, so in-flight jobs from the old
shape must drain (re-enqueue) rather than run under the new chain.

WORKER_CONCURRENCY default raised 2 -> 8 so per-host pacing does not
starve other hosts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AzSiAz force-pushed feat/per-domain-rate-limit from 22cd15cd1e to d63f7d1732 2026-06-14 18:24:04 +02:00 Compare
AzSiAz force-pushed feat/per-domain-rate-limit from c77edf4272 to 42380de32b 2026-06-17 10:49:26 +02:00 Compare 
The document pipeline becomes a pg-jobs flow (fetch -> metadata/extract
-> finalize) with typed inputs/outputs. Big payloads no longer ride the
queue: fetched HTML is staged in document_sources (kept until document
deletion, so a pipeline retry never refetches) and extracted content is
written straight onto the documents row inside the job transaction.

Per-domain politeness moves off the custom GCRA (host_throttle dropped):
fetch jobs carry the host as pg-jobs group_key with a fleet-wide cap of
1, and 429/503 answers defer via JobError::delayed (Retry-After honored,
escalating, capped at 2h) without burning retry attempts - which also
retires the max_attempts=10_000 hack. The engine has no dead-letter
hook, so every pipeline job preemptively syncs documents.status=failed
on lifecycle-terminating errors.

The hourly purge timer and PurgeThrottle job are replaced by the
built-in GcJob/MetricsRollupJob driven by the pg-jobs Scheduler.
indara-workflow is deleted (host_key moves into indara-jobs); sqlx is
bumped to 0.9 (pg-jobs requirement, apalis was the only blocker) and
pg-jobs is consumed from the forgejo cargo registry. Its migrations
track in jobs._sqlx_migrations, the apalis schema is dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
reqwest 0.13 with default-features off ships no TLS backend, so every
https fetch failed instantly with an opaque error. And since pg-jobs
persists only the top-level message as last_error, the real cause never
reached the job row - bake the whole anyhow chain into the message.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Server: /api/admin/jobs (stats, list with substate filters, detail with
attempts+logs, rollup-backed time-bucketed metrics), requeue actions for
dead jobs and failed flows (FlowBusy -> 409), and a per-document
/api/documents/:id/pipeline flow view. pg-jobs tracking types are not
Serialize, so everything goes through thin DTOs.

Frontend: a Jobs page (queue tiles, substate chips, hand-rolled SVG
throughput chart, filterable job table with requeue) polling every 5s,
and a pipeline DAG on the document page (generic layered SVG renderer,
colored by sub-state) polling while the flow runs, with a retry button
on failed pipelines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mise run dev starts server + worker + vite together (deps run in
parallel); each is also runnable alone, plus a migrate task. Env comes
from .env.local via mise as before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A job's name now opens /jobs/:id — full detail with payload/output JSON
blocks, per-run attempts (durations, errors) and log lines, requeue on
dead jobs, polling while live. A new Flow column and /flows/:flowId page
give a document-independent view of any pipeline (clickable DAG, node
table, retry), backed by a new GET /api/admin/flows/{flow_id}. DAG nodes
navigate to the job detail from both the flow and document pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A jobs/flows segmented toggle groups the current page of jobs by flow:
one collapsible-style group per flow_id (header links to /flows/:id,
derived outcome chip, job count, oldest age), standalone jobs last.
Client-side grouping of the fetched page - a paginated server-side
GET /api/admin/flows list would make it exhaustive later. Shared JobRow
keeps both views on one markup; filters, pager, polling and requeue
work in both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase A kills the centered 56rem column: an app-shell grid with a left
'card-drawer' nav rail (active drawer pulled, pure-CSS fallback to the
top bar under 960px) and a full-width workspace with fluid gutters.

Phase B is the new identity, tokens-only plus the JobsPage witness
screen: neutral ivory bristol + blue-black ink + ruling-blue hairlines,
stamp-red head rules, bottle-green accent (red retreats to danger),
Vollkorn/Archivo/IBM Plex Mono, label-holder chips, sharpened radii.
Chart colors validated for CVD in both modes. Other pages inherit the
tokens; their grids come in the next lots.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase C: JobDetail gets a sticky identity-card aside (facts + requeue)
next to the main column; FlowPage takes the full width with the red
head rule. Phase D: DocPage becomes a reading grid (sticky meta rail,
51rem article, pipeline on the full track), History and Keys move their
forms to sticky filing-card asides, Search aligns hero and results left
(no centered exception). Phase E: sharpened DAG node radii, scrollable
mobile nav, and a pass over responsive tiers, focus-visible,
reduced-motion and dark mode. Behavior untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The empty-query view becomes a real search home: hero centered in the
viewport (the one assumed exception to the left-aligned grid), archive
stat tiles (indexed pages, domains, pipeline links to /jobs), recent
searches as chips (localStorage, recorded only on committed searches -
Enter or result click), a domain: pro tip, and the OpenSearch note
pinned bottom-left like an engine footnote.

Server side: /api/search understands a domain:example.com token (parsed
out, filters WHERE domain, echoed back for the active-filter chip) and
a new GET /api/stats returns per-owner archive counters. A global '/'
shortcut focuses or navigates to the search bar; stat-tile styles move
to base.css now that two pages share them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A search that is only a domain:example.com token had no BM25 query to
rank by and silently returned nothing - it now lists that domain's
indexed documents by recency (zeroed scores, an em-dash in the score
gutter). The OpenSearch footnote was pinned to the viewport's left edge,
underneath the nav rail on desktop: offset it past the rail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The aside is first in the DOM with an explicit grid-column: 2; sparse
auto-placement then pushed the main column (column 1, auto row) down to
row 2, leaving an aside-sized hole. Pin both to row 1, reset on mobile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 100rem content guardrail left a large void on wide displays; drop
it (--content-max: none) and rely on the per-page reading measures for
the things that must stay narrow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nodes of a flow are created in one transaction (identical
created_at), so the API's creation order is effectively random and the
table read alphabetically while the graph read topologically. Reuse the
graph's depth computation (extracted as nodeDepths) to sort the table
by depth, name as tie-break within a column.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same defect as the flow page table, but the jobs list API carries no
edges: approximate the topology with what the rows have - execution
chronology (started_at, exact for the completed part), then claimable
not-yet-started jobs, then waiting_children (blocked = downstream),
name as tie-break.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /api/admin/flows serves Tracking::list_flows - newest-first flow
summaries with server-derived outcome and substate counts, queue and
outcome filters, has_more pagination. The JobsPage flows view now
fetches it instead of grouping the current page of jobs client-side:
one row per flow (name links to /flows/:id, outcome chip, non-zero
count chips), outcome filter chips replace the substate row in that
view, and the groupByFlow/topo-ish approximation code is gone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
view/queue/substate/outcome/page become search params (defaults never
written, invalid values fall back silently), following the HistoryPage
useSearchParams pattern: filtered views are shareable and back/forward
walks the pagination. Filter changes replace the history entry; view
toggling resets the page and drops the other view's filter param.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- FetchDocument answers 429/503 with delayed_group (whole host deferred,
  engine-applied jitter) and the pipeline queue gains a hard per-host rate
  window (1 start per HOST_RATE_INTERVAL_MS, default 5s) on top of the
  group-concurrency cap — the crawl politeness the old GCRA provided.
- documents.status='failed' now flows from the jobs.job_event outbox
  (status_sync consumer, durable cursor + jobs_event LISTEN) instead of
  every handler predicting the engine's terminal conditions; the write is
  guarded (pending-only) against stale redeliveries.
- documents.flow_id is dropped: the pipeline endpoint finds the flow by
  metadata containment, and submit_flow now returns the effective flow id
  on a dedup fold (workaround removed).
- Admin API: GET /api/admin/schedules, schedule_id on job summaries and
  as a list filter, metrics accept runtime queue names via QueueRef.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third view backed by GET /api/admin/schedules: name/kind, job label,
queue, cadence (every-interval or raw cron), next/last fire, enabled
chip. Each row links to the jobs view filtered by schedule_id (new URL
param + API filter), shown there as a removable chip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AzSiAz force-pushed feat/per-domain-rate-limit from 8607a7a06c to ee097b5a71 2026-07-04 20:44:36 +02:00 Compare 
0.6.0 makes the database clock the single time authority (drift-free run_at /
crons / backoff) and stamps lifecycle events with clock_timestamp() instead of
the transaction-pinned now() (migration 0021): Transactional jobs' finished_at
is now the handler's real end, and the phantom job_attempt-window overlaps from
the group-concurrency audit shrink to ~one commit latency. No app code change
(no JobSpec.run_at / defer_group call sites); JobRow.run_at is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish the container's 5432 on host 5433 so a local pg-jobs Postgres can keep
5432 and both run at once (no more stop/start dance). DATABASE_URL lives in the
gitignored .env.local. Recreate with `docker compose up -d postgres` (a plain
`docker start` keeps the old mapping); the postgres-data volume is preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BIND_ADDR + PUBLIC_URL (in .env.local) are the single knob for the server port.
The Vite dev proxy now reads PUBLIC_URL, Dex accepts the 3001 callback (3000 kept
as a fallback), and the CLI's default --api-url follows. mise/README strings
updated. The code default stays 0.0.0.0:3000 (overridden by .env.local in dev).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AzSiAz merged commit 50e6bf47b6 into main 2026-07-05 12:40:10 +02:00
AzSiAz deleted branch feat/per-domain-rate-limit 2026-07-05 12:41:07 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
AzSiAz
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
AzSiAz/Indara!1
No description provided.