Skip to main content

The write path

Every change to the cache comes from one of four writes: a full refresh, a webhook event, a branch view build or refresh, and a definition change at startup. All four go through the writer of the inventory they change.

One writer per inventory​

Each inventory has one InventoryWriter (crates/sync/src/writer.rs) with one lock. Every write to the inventory, in every view, holds that lock from the moment it reads the stored records to the moment its transaction commits. Reads do not take the lock.

This gives two guarantees:

  • Publication of a full refresh compares the refresh with what is stored, and keeps any webhook observation received after the refresh started. With one writer, no other write can land between that comparison and the transaction.
  • An entity is recomputed from the records of every contribution that shares its HFID. With one writer, those records do not change while the entity is computed.

The lock is per inventory, so a long publication in one inventory does not delay another. Writes to one inventory are serialized: a webhook for an inventory waits while that inventory publishes a refresh, about 1.5 to 2 seconds at 10,000 entities in the recorded scale test.

The note one writer per inventory records what a finer lock would need to keep.

Full refresh​

refresh_contribution (crates/sync/src/refresh.rs) records the start time, sets the health to refreshing if a snapshot was published before, and fetches every object of every object type into memory. If the connector fails, it writes only the contribution's state: degraded, with the error.

When every page is fetched, publish_refresh takes the writer lock and, in one transaction:

  1. Builds a record from every fetched object. It skips an object that a webhook deleted after the refresh started: the writer remembers deletions for 24 hours.
  2. Loads the contribution's stored records in the view.
  3. For a record that is already stored, keeps every stored field observation received after the refresh started.
  4. Deletes the stored records the refresh does not contain, unless they were received after the refresh started.
  5. Recomputes which records of the contribution are excluded for a missing or duplicate HFID.
  6. Collects every affected HFID, the old and new HFID of each changed record, and resolves those entities from all eligible records of the inventory. An HFID with no eligible record left loses its entity.
  7. Writes records, entities and the contribution's state.

The whole refresh is held in memory and written at once. At the validation target, 10,000 records, a publication took 1.5 to 2 seconds. Decision 0002 explains the choice and its limit.

Webhook events​

The receiver (crates/server/src/webhook_routes.rs) only queues the request. The connection's ingest loop (crates/sync/src/ingest.rs) parses it with the connector, which checks the signature, and keeps the last event per object within a 500 millisecond window (crates/sync/src/debounce.rs).

For each event, apply_event (crates/sync/src/events.rs) finds every contribution, in every inventory, that uses the connection and collects the event's object type. It fetches the object with that contribution's paths and writes it through the inventory's writer as a full observation of the record, in one transaction, recomputing the old and new HFID. An event carries no field values, so each inventory applies its own mapping to the object it fetched.

An event for a branch other than the connection's default branch updates the branch view only when the inventory holds that view.

A failed fetch changes nothing; the next full refresh includes the change.

Writes to the default view that branch views read​

A branch view reads the default-view records of the contributions whose source does not have the branch. When such a record changes, the writer also recomputes the affected HFIDs in every branch view of the inventory that reads that contribution's default data.