Cache layout

Census stores its cache in Memgraph, behind the CacheStore trait of crates/store-api. Memgraph is the only backend. Every backend must pass the conformance checks in crates/store-api/src/conformance/, and apply each change in one transaction.

The cache holds only data that Census can fetch again. There are no migrations: a build with a different layout version empties the cache at startup and refills it from the sources.

Nodes

Every node carries inventory and view. view is empty for the default view, and the branch name in a branch view. There are no relationships: the records of an entity are found by (inventory, view, hfid).

Label	Key	Holds
:Record	{inventory}/{contribution}/{view}/{object_type}/{native_id}	One contribution's observation of one source object: native id, source reference, object type, original and cleaned HFID, exclusion, one observation per field and the location observation.
:Entity	{inventory}/{view}/{hfid}	The resolved entity: one f_<field> property per non-null resolved value, one loc_<level> property per level of the resolved location, the location path, and the full resolution.
:ContributionState	{inventory}/{contribution}/{view}	Snapshot, health, last publication, last attempt, last error, the branch view it supplied and why, and the collection hash.
:InventoryState	inventory	The resolution hash, and ready or rebuilding.
:CacheLayout		The layout version.

The object type is part of a record's key because a source numbers each type separately: a NetBox device and a virtual machine can both have id 5.

Each node stores its whole value as one JSON property (record_json, entity_json, state_json), which reads decode. The other properties exist for queries and indexes.

Observations

A record holds one observation per field the contribution supplied:

Key	Meaning
state	value, unknown or cleared. A field without an entry was not supplied.
value	The normalized, typed value.
outcome	as_reported, mapped, defaulted, unmapped, wrong_type or ambiguous.
original	The value as the source reported it.
path, level_paths	The source path of the field, or of each location level.
observed_at	When Census received the observation. Publication compares it with the refresh start.
confirmed_at	When the source last confirmed the value. Freshness is computed from it.

Keeping the original value lets Census normalize every observation again after a vocabulary or mapping change, without fetching from the sources.

Queries on resolved values

Filters, sorting and counts run on the entity's f_<field> and loc_<level> properties. A null resolved value is not stored, which is why a null never matches a filter. Memgraph indexes are per label and property, so Census creates one index per field name and level name used by any inventory, shared by every inventory that uses the name.

Indexes also exist on the keys, inventory and hfid, and on a record's contribution and excluded properties.

Values computed at read time

Completeness depends on the state of every contribution, and freshness on the time of the request. Both are computed when a query runs, from the contribution states, the count of excluded records and each value's confirmed_at, and are not stored on the entity.