Keep inventories up to date

Census updates each contribution in two ways: full refreshes on a fixed interval, for every source, and webhooks between refreshes, for Infrahub.

Full refreshes

When Census starts, it runs one full refresh of every contribution. After that, each contribution refreshes on its own refresh_interval, five minutes by default. Contributions refresh independently of each other.

During a full refresh, Census:

1. Fetches every object of the contribution's object types, 100 per request, and holds them in memory.
2. When every request has succeeded, publishes them in one database transaction: it adds and updates records, removes the records of objects the source no longer has, and recomputes the affected entities.

A refresh that fails part-way, because the source cannot be reached, refuses the credentials or returns an error, publishes nothing. The contribution keeps its last published data, its health becomes degraded and last_error holds the error. The next successful refresh sets the health back to ok.

A source that returns zero objects in a successful refresh has zero objects: Census removes the contribution's records.

Contribution health

Health	When
ok	The last refresh was published.
refreshing	A refresh is running and the previous one was published.
degraded	The last refresh failed. The last published data is still served.

Each response gives every contribution's health, its snapshot (published once a refresh was published, otherwise none), last_published_at and last_error.

Receive Infrahub webhooks

An Infrahub webhook tells Census that a node changed, so that the change reaches the inventory before the next full refresh.

1. Choose a secret and set it as the connection's webhook_secret, preferably through an environment variable:

   [[connections]]
   id = "infrahub-prod"
   type = "infrahub"
   endpoint = "https://infrahub.example.net"
   webhook_secret = "${INFRAHUB_WEBHOOK_SECRET}"

2. In Infrahub, configure a webhook that sends node events to https://<census address>/webhooks/infrahub-prod, where the last part is the connection's id, signed with the same secret.

Census checks the signature as Infrahub sends it: an HMAC-SHA256 of {webhook-id}.{webhook-timestamp}.{body}, compared with the webhook-signature header. A secret with the whsec_ prefix is accepted. When the connection has no webhook_secret, Census does not check signatures.

What Census does with a webhook

1. The receiver queues the request and answers at once:

   Status	Meaning
   202	The event is queued.
   404	No connection has this id.
   429	The connection's queue holds 1,024 events. The event is dropped; the next full refresh includes the change.
   503	The queue is closed.

2. Census checks the signature and reads the event: infrahub.node.deleted is a deletion, any other node event is an update. An event with an invalid signature is logged and dropped. The 202 answer does not mean that the signature was valid.

3. Census groups the events of one node received within 500 milliseconds and keeps the last one.

4. For each contribution that uses the connection and collects the node's kind, in every inventory, Census fetches the node with that contribution's paths and stores it. A node that is deleted, or that Infrahub no longer returns, is removed.

The webhook body carries no field values: Census always fetches the node, so each inventory applies its own mapping.

If the fetch fails, Census logs the error and changes nothing. The contribution's health does not change, and the next full refresh includes the change.

Webhooks during a full refresh

A webhook received while a full refresh of the same contribution is running wins over that refresh, for updates and for deletions. When the refresh publishes, Census keeps every value a webhook stored after the refresh started, and does not bring back an object that a webhook deleted.

Webhooks for Infrahub branches

An event for the connection's default branch (branch, main by default) updates the default view. An event for another branch updates that branch's view only if a consumer has queried it recently and Census still holds it. Otherwise Census ignores the event; the view is built from the source when a consumer queries the branch.

NetBox

NetBox contributions have no webhooks. Their data changes at each full refresh. Use a shorter refresh_interval for a NetBox contribution whose data must reach consumers sooner.