pg-jobs

pg-jobs is a PostgreSQL-native job and flow engine. It provides typed jobs, DAG workflows, retries and dead-letter handling, scheduling, queue controls, high-throughput Basic jobs, and observable benchmark contracts without an external broker.

The repository uses a multi-implementation monorepo layout. The production implementation is currently Rust; portable benchmark contracts and campaign orchestration are kept outside it so a future Go worker can share the same evidence model. SQL migrations remain packaged with the current Rust crate, but their versioned database API is the language-neutral worker boundary.

Repository layout

Path	Responsibility
rust/	Rust implementation, derive macro, benchmark library and Rust adapters
benchmark-manager/	Cross-implementation campaign orchestration and immutable benchmark evidence
contracts/	Canonical JSON Schemas, implementation descriptors, vocabularies and fixtures
generated/	Generated bindings for non-Rust consumers
tools/	Language-independent probes and code-generation support
docs/	Design notes, portability conventions and audited decisions

Start with the Rust README to install or use the current library. The benchmark manager guide documents controlled campaigns, while the contract guide explains how shared schemas and generated bindings evolve.

Development

The repository has two Cargo workspaces with independent lockfiles:

• rust/Cargo.toml and rust/Cargo.lock cover the Rust implementation, benchmark runner, contracts bindings and implementation-specific tools;
• benchmark-manager/Cargo.toml and benchmark-manager/Cargo.lock cover the cross-implementation controller.

The lockfiles are not interchangeable: target builds use the implementation lockfile, while building the manager uses its own resolution. Commands below are written from the repository root with an explicit manifest path:

mise install
docker compose up -d
export DATABASE_URL="postgres://postgres:postgres@localhost:5432/pg_jobs"
cargo test --manifest-path rust/Cargo.toml --workspace --all-targets --locked
cargo test --manifest-path benchmark-manager/Cargo.toml --all-targets --locked

Useful fast checks are:

cargo fmt --manifest-path rust/Cargo.toml --all -- --check
cargo fmt --manifest-path benchmark-manager/Cargo.toml --all -- --check
cargo clippy --manifest-path rust/Cargo.toml --workspace --all-targets --locked -- -D warnings
cargo clippy --manifest-path benchmark-manager/Cargo.toml --all-targets --locked -- -D warnings
cargo run --manifest-path rust/Cargo.toml --profile test --locked -p benchmark-contract-codegen -- check
go -C generated/go test ./...

Changes to canonical schemas belong in contracts/; generated Rust and Go files are never edited directly. Database behavior belongs in versioned SQL migrations and explicit functions where possible, so every implementation can reuse the same atomic boundary.

Stable and prerelease bumps are prepared from a clean main checkout. The repository helper requires a strictly increasing canonical SemVer value (build metadata is excluded), then updates the shared Rust version, the exact pg-jobs-derive pin and both lockfiles as one fail-closed operation; CI remains responsible for tagging and publishing:

tools/prepare-release.sh 0.11.0-alpha.1
git diff -- rust/Cargo.toml rust/Cargo.lock benchmark-manager/Cargo.lock

Benchmark builds select an implementation through a closed manager enum. The current rust-pg-jobs adapter is described by the portable v1 fixture under contracts/fixtures/implementations/; its manifest, lockfile, packages and attested paths are data derived by the manager, never a caller-provided build command. The canonical descriptor SHA-256 is 23896c43ce6813cce06c62a3d0334ac0fc8e216792d72d5497e95afe8b2c471c.

Benchmark discipline

Performance decisions use reproducible, immutable evidence rather than one wall-clock run:

• establish a fresh baseline after any source-layout or provenance-contract change;
• run A/A before A/B, then balance decision runs in AB and BA order;
• report jobs/s next to latency deltas so a percentage in milliseconds keeps its operational scale;
• capture host load, PostgreSQL identity and storage provenance, and keep different storage configurations in separate result series;
• do not treat load average as a CPU-frequency control: freeze and record the effective cpuset, core/SMT/cache topology, governor/EPP and boost policy for both the runner and PostgreSQL when a single-worker result is a decision;
• do not bind one physical database identity permanently to each A/B arm across repeated pairs; use fresh per-invocation databases or a pre-registered crossover so database-local cache/storage effects cannot masquerade as code;
• separate a controlled code-regression gate from operational canaries: align checkpoints outside measured invocations for the former, and keep default or intentionally stressed maintenance settings as separately labelled evidence;
• reserve the PostgreSQL reference cluster exclusively for one manager and one campaign, with no foreign client; bind its published port to loopback, disable autovacuum in the controlled profile, and remember that exclusivity remains an operator precondition; profile limits only reduce checkpoint probability, so on a PostgreSQL 18 primary idle-fence and seal fresh pg_stat_checkpointer snapshots immediately around every runner, then invalidate the gate on any server identity, counter, time or reset change;
• treat a wide confidence interval as inconclusive, not as a regression or an improvement;
• monitor long campaigns after startup so a crashed runner cannot waste hours;
• compare medians across repeated cold and warm runs because shared hosts and CI scheduling add noise.

The CI shares Cargo registry downloads, but deliberately does not share a global target/ directory and does not run cargo clean. Compatible steps may reuse a target directory inside one workflow; cross-workflow compiler reuse should use a compiler cache rather than concurrent Cargo artefacts.

License

MIT © Stephan Deumier