Worker routing flows

How core and provider workers talk, as sequence and flow diagrams. They complement the provider contract: the contract states the rules, these show them in motion. A pool is every worker instance sharing one worker_name for one provider; a pool name is permanent and bound to one package.

Channel	Subject or key
Jobs	cairn.jobs.<job>.<provider>.<worker_name>
Results	cairn.results.<job>.<provider>.<worker_name>
Discovery	cairn.discover.<provider>.<worker_name>
Account lookup	cairn.accounts.lookup.<provider>.<worker_name>
Core release	KV cairn_core/info
Rate budgets	KV cairn_rate_limits/<provider>.<app>.<window>

Routine import of an account

The scheduler sends the reconcile to the account's pool. An unassigned account takes the provider's only known pool (the registry, not whoever happens to be online) and is then assigned to it. The worker sends its own fetch_source sub-jobs to its own pool and archives every parser input. Core checks each result against the sender's name on the subject, the pool's package, and the account's pool.

Pool registry, retirement and reassignment

A pool is registered the first time its manifest or a result arrives; the name is then bound to that package. Admins retire a pool: it leaves routing and pickers, the name stays reserved, and the same package coming back un-retires it. The owner of a connection may move it to another pool: free within a package, with confirmation across packages, which flags the connection's activities pool_changed for a re-fetch.

Re-parse while a pool is mid-upgrade

Core picks the newest live parser of the source's own package and dataset, and puts the source's current parser version in the job. An older instance that still pulls the job hands it back with growing delay; after ten deliveries it reports parser_too_old, and core keeps the source flagged instead of failing it. Core refuses any result that would lower the version.

Two worker implementations for one provider

Each pool has its own job and result subjects, so the work queue never sees overlapping consumers and a pool cannot publish as another. The matcher allows one source per pool, so an API import and a FIT import of the same ride cluster into one activity. A merge priority entry may name a pool as garmin.garmin-fit; a bare garmin covers every pool.

Two connections for one provider account

A provider account may be connected once per pool; a pool the user does not want is simply not connected. Each connection has its own sync state, queue, tokens and app. A gear id decided on one connection maps the sibling's reference too.

Webhook event to account

A webhook names the provider's owner id. The worker asks on its own pool's lookup subject; the subject (which it is permitted to publish on) names the pool, so core returns that pool's connection and another pool cannot look it up. A connection without a pool matches any pool.

Rate budgets per provider app

OAuth apps belong to connections, and one pool serves many of them. Core names the app in the token reply (app_id, a hash of the client id); the worker attributes usage headers, 429s and reservations to that app's buckets. Core reads the same keys to gate reconcile and to show the budget per connection. A token without an app id falls back to the pool's bucket.

Release compatibility

Core publishes its release in the cairn_core KV bucket (workers can only read it). Before 1.0 a worker needs the same major.minor. From 1.0 on, minor and patch releases are backwards compatible: same major, and the worker's minor not above core's. A worker refuses to start, and stops on its next heartbeat, against an incompatible core; core also flags such workers and ignores them for re-parse and gear routing. Untagged builds skip the check.

Upgrade from 0.4: draining queued jobs

Jobs queued on the old provider-only subjects before the upgrade are not lost: for one minor release the permission template still lets the pool subscribe to them with its old durable name. Results and lookups are pool-scoped only. Removed in 0.6.

Bulk re-parse after a parser release

A new parser flags every source of its pool and dataset as update_available / newer_parser; a new dataset flags newer_dataset (re-fetch only). The Connections page shows the counts, and one action re-parses up to 500 sources without provider quota.