Wire Celery Beat periodic sync with ingestion run tracking
This commit is contained in:
17
README.md
17
README.md
@ -97,7 +97,7 @@ docker compose exec web python manage.py createsuperuser
|
||||
- `web` service also builds Tailwind CSS before `collectstatic` when `AUTO_BUILD_TAILWIND=1`.
|
||||
- `web`, `celery_worker`, `celery_beat`, and `tailwind` run as a non-root user inside the image.
|
||||
- `celery_worker` executes background sync work.
|
||||
- `celery_beat` supports scheduled jobs (future scheduling strategy can be added per provider).
|
||||
- `celery_beat` triggers periodic provider sync (`apps.ingestion.tasks.scheduled_provider_sync`).
|
||||
- `tailwind` service runs watch mode for development (`npm run dev`).
|
||||
- nginx proxies web traffic and serves static/media volume mounts.
|
||||
|
||||
@ -222,6 +222,21 @@ trigger_full_sync.delay(provider_namespace="balldontlie")
|
||||
- Run-level status/counters: `IngestionRun`
|
||||
- Structured error records: `IngestionError`
|
||||
- Provider entity mappings + diagnostic payload snippets: `ExternalMapping`
|
||||
- `IngestionRun.error_summary` captures top-level failure/partial-failure context
|
||||
|
||||
### Scheduled sync via Celery Beat
|
||||
|
||||
Configure scheduled sync through environment variables:
|
||||
|
||||
- `INGESTION_SCHEDULE_ENABLED` (`0`/`1`)
|
||||
- `INGESTION_SCHEDULE_CRON` (5-field cron expression, default `*/30 * * * *`)
|
||||
- `INGESTION_SCHEDULE_PROVIDER_NAMESPACE` (optional; falls back to default provider namespace)
|
||||
- `INGESTION_SCHEDULE_JOB_TYPE` (`incremental` or `full_sync`)
|
||||
- `INGESTION_PREVENT_OVERLAP` (`0`/`1`) to skip obvious overlapping runs
|
||||
- `INGESTION_OVERLAP_WINDOW_MINUTES` overlap guard window
|
||||
|
||||
When enabled, Celery Beat enqueues the scheduled sync task on the configured cron.
|
||||
The task uses the existing ingestion service path and writes run/error records in the same tables as manual sync.
|
||||
|
||||
## Provider Backend Selection
|
||||
|
||||
|
||||
Reference in New Issue
Block a user