Case study
P6 (Pinceau6) — Architecture
A C4 walk-through of P6, my task-orchestration side project: a Python framework that runs any business process needing orchestration as a distributed Task DAG, with or without an associated database. Example workloads range from document indexing and knowledge-base pipelines to LLM inference and image generation. The diagrams below are the actual project documentation, maintained as Mermaid sources alongside the code.
Reading the diagrams
The diagrams follow C4 model conventions — zooming from system context, to containers, to code structure — and use the project's visual identity:
- Pinceau6 system & runtime containers
- Components inside a container
- People / actors
- Brokers & caches (RabbitMQ, Redis)
- Data stores (MongoDB, Elasticsearch, File DB)
- Third-party services (secrets, LLM providers, data sources)
Level 1 — System context
Two kinds of actors use the system: admins operating and monitoring DAGs, and API clients consuming results. P6 persists to MongoDB, queues jobs through RabbitMQ, caches results in Redis, indexes into Elasticsearch, resolves secrets at startup, and runs inference against external LLM/ML providers.
Level 2 — Containers
Three runtime containers: a FastAPI server, a Celery worker, and a React admin SPA. The key runtime loop: the API enqueues DAG runs on RabbitMQ, the worker executes them — and progress flows back not over the Celery result channel but as events published to RabbitMQ via Pika, which the API consumes and fans out to subscribed WebSocket clients.
Level 3 — Package layering
The monorepo is layered strictly bottom-up: each layer only depends on the ones below it. Business applications sit at the top as plugins; the base framework at the bottom knows nothing about the web or messaging layers. The full documentation goes one level deeper — component diagrams for the API server and the worker — but that zoom level is more useful to contributors than to readers.
Dynamic view — remote DAG execution
A DAG run submitted through the API and executed on a remote worker, with live progress streamed back to the browser. Cancellation follows the reverse path.
Key architectural decisions
- ▸
Strict layering. p6_core → p6_workers → p6_fastapi → src/ — lower layers never import from higher ones.
- ▸
RUN_MODE profiles. The same codebase boots as API, WORKER or TEST; TEST swaps MongoDB for a file-backed store and disables secret resolution.
- ▸
Config precedence. P6_-prefixed env vars → local .env → platform/environment .env files; $-prefixed keys resolve through a secrets manager at startup.
- ▸
Event-driven feedback loop. Workers do not respond over the Celery result channel for UX — task lifecycle events are published back through RabbitMQ (Pika) to the API, which fans them out to subscribed WebSocket clients.
- ▸
Applications as plugins. Each src/applications/<name>/ bundles its own models, tasks, DAGs and API endpoints, and registers itself into the shared Task/DAG registry.