Skip to main content

Add a contribution

A contribution is the use of one connection inside one inventory. It tells Census which source objects to collect, which source field supplies each object's HFID, and which source field supplies each inventory field and location level. An inventory has one contribution per source, under [[inventories.<id>.contributions]].

[[inventories.devices.contributions]]
id = "infrahub"
connection = "infrahub-prod"
object_types = ["InfraDevice"]
hfid = "name.value"
refresh_interval = "5m"

[inventories.devices.contributions.attributes]
role = "role.value"
status = "status.value"

[inventories.devices.contributions.location]
site = "site.node.name.value"

Contribution settings​

SettingRequiredMeaning
idYesUnique within the inventory. Priority lists, API responses and diagnostics name the contribution by this id.
connectionYesThe id of a declared connection.
object_typesYesOne or more object types to collect. For Infrahub, a node kind such as InfraDevice. For NetBox, an API path such as dcim/devices or virtualization/virtual-machines.
hfidYesThe path of the source field that supplies the HFID.
attributesNoA table of inventory field name to source field path. A field that is not listed is never supplied by this contribution.
refresh_intervalNoHow often Census runs a full refresh of this contribution. Default 5m.
values, defaultsNoValue mappings and defaults for fields with a vocabulary. Normalize source values describes them.
location, location_valuesNoThe source field path of each location level, and name mappings. Declare the location hierarchy describes them.

Durations are a number followed by s, m, h or d, for example 30s, 5m or 6h. A number without a unit is a number of seconds.

Write source field paths​

A path is a list of keys joined by dots. Census follows the keys into the JSON object that the source returns:

  • name.value reads {"name": {"value": "fw-01"}}, an Infrahub attribute.
  • site.node.name.value reads the name of the site peer of an Infrahub node.
  • status.value and role.name read nested fields of a NetBox object.
  • serial reads a top-level field.

When a key on the path holds a list, Census takes the first element. tags.name reads the name of the first tag. An empty list means the field is not supplied.

When a key is missing, or a value on the path is null, the contribution does not supply the field. Census then considers the next contribution in priority order. An empty string, 0 and false are values: they are supplied.

To find the paths of an Infrahub node, query it in the Infrahub GraphQL sandbox. To find the paths of a NetBox object, read it at /api/<object type>/<id>/.

Choose the HFID field​

The HFID is how Census decides which records describe the same entity. Pick a source field that holds the same identifier in every source, such as the device name.

Census cleans each HFID by removing the whitespace around it and converting it to lowercase. Every other character is kept:

  • FW-01 and fw-01 are one entity.
  • fw-01 and fw_01 are two entities.
  • fw-01 and fw-01.example.net are two entities.

A number is used as its decimal text. Any other type, an empty string and a missing field give a missing HFID.

Records that Census excludes because of their HFID​

Census stores every record it fetches, but excludes two kinds from resolution:

  • missing_hfid: the record has no HFID.
  • duplicate_hfid: two or more source objects of one contribution have the same cleaned HFID. Census excludes all of them for that contribution. Other contributions can still supply the entity.

Each excluded record is listed in the diagnostics with its source reference, and makes the inventory incomplete. When the source is fixed, the next refresh or webhook makes the record eligible again.

When a source changes an object's HFID, Census moves the record to the new HFID. The entity under the old HFID loses that record, and is removed if no record is left. Census keeps no link from the old HFID to the new one.

Objects that disappear from a source​

When a full refresh succeeds and no longer contains an object, or a webhook reports that it was deleted, Census removes the record. An entity with no record left disappears from lists, and looking it up returns 404.

A failed refresh never removes records: a source outage is not treated as a deletion.