MetricsHub
MetricsHub Community Connectors 1.0.24
-
Home
- Connector Developer Guide Connector Structure 7
Monitors and Jobs
A monitor defines how MetricsHub discovers and collects metrics for one type of component: fan, disk, battery, enclosure, network, etc. Each monitor produces instances — one per physical or logical component found — and each instance is exported as one OpenTelemetry Resource.
This page explains the job models (simple vs discovery + collect), instance identity (keys), collect modes (multiInstance vs monoInstance), and two additional mapping features (conditionalCollection, legacyTextParameters).
Monitor Anatomy
monitors:
<monitorType>: # e.g. fan, disk, battery
keys: [ id ] # optional, default: [ id ]
# removalDelay: 15m # optional, default: 0; only when no inventory exists (see removalDelay)
metrics: {} # optional, monitor-level metric metadata overrides
simple: {} # EITHER a simple job...
discovery: {} # ...OR a discovery job
collect: {} # plus a collect job
| Property | Required | Default | Description |
|---|---|---|---|
keys |
No | [ id ] |
Attributes that uniquely identify each instance. See Instance Identity. |
removalDelay |
No | 0 |
How long an instance is kept after the last run that returned it: a duration (30s, 15m, 1h, 1d) or a number of seconds. See removalDelay. |
metrics |
No | None | Metric metadata (unit, description, type) overriding connector-level definitions for this monitor. |
simple |
One of | None | Single job executed in both the discovery and collect phases. |
discovery / collect |
One of | None | Two-phase model: inventory runs less often, metric collection runs on every cycle. |
Choosing a Job Model
| Situation | Use |
|---|---|
| One request returns attributes and metrics together (typical REST APIs, SQL queries) | simple |
| Inventory is expensive (full SNMP table walk, WMI enumeration) but metrics are cheap to fetch separately | discovery + collect |
Collection requires running a command per instance (e.g. smartctl -H /dev/sda) |
discovery + collect with type: monoInstance |
| The source that returns the metrics omits some instances (idle, disconnected, not reporting) | discovery (inventory) + collect (metrics): a row missing from a collect job never removes the instance |
Prefer simple unless you have a concrete reason to split. A simple job is easier to read, and MetricsHub still runs it at both discovery and collect frequencies.
The simple Job
One pipeline produces both attributes and metrics; each mapped row creates (or refreshes) one instance and collects its metrics at the same time.
Because the job runs on every collect, an instance that disappears is removed at the next collect, when a trusted run no longer returns it. See When Instances Are Removed.
monitors:
fan:
simple:
sources:
fans:
type: http
path: /api/fans
computes:
- type: json2Csv
entryKey: /records
properties: /id;/name;/status;/rpm
mapping:
source: ${source::fans}
attributes:
id: $2
name: $3
metrics:
hw.status{hw.type="fan"}: $4
hw.fan.speed: $5
The discovery Job
Discovery inventories the components: it identifies instances and sets their attributes (identity, model, topology). It runs less often than collect.
An instance that disappears is removed at the next discovery cycle (every 30 collects by default, about 1 hour), when a trusted discovery run no longer returns it. The collect job never removes instances. See When Instances Are Removed.
monitors:
enclosure:
discovery:
sources:
source(1):
type: snmpTable
oid: 1.3.6.1.2.1.33.1.1
selectColumns: "1,2,3,5"
mapping:
source: ${source::monitors.enclosure.discovery.sources.source(1)}
attributes:
id: $4
vendor: $1
model: $2
type: UPS
name: "${awk::sprintf(\"Enclosure: (%s %s)\", $1, $2)}"
Conventions observed across the connector library:
idis the stable technical identifier (serial number, device index, WWN) — it must not change between cycles.nameis the human-facing label, often built with${awk::sprintf(...)}.hw.parent.type(andhw.parent.idwhen needed) attach the instance to its parent in the hardware topology; omit for top-level monitors likeenclosure.- Attributes starting with
__(e.g.__display_id,__device_type_option) are internal: they are available to later jobs via${attribute::...}but are not exported. - A discovery
mappingmay also set slowly-changing metrics (e.g.hw.temperature.limitthresholds) that do not need collection on every cycle.
The collect Job
Collect gathers the frequently-changing metrics for the instances that discovery created. Its mode is selected with type:
collect:
type: multiInstance # or monoInstance
sources: {}
mapping: {}
When type is omitted, the engine defaults to monoInstance. Always declare it explicitly: an accidental monoInstance on a multi-instance monitor silently changes how (and how often) the job runs.
multiInstance: one run for all instances
The job executes once; its final table must contain one row per instance. The mapping re-maps the key attributes (by default id) so the engine can match each row to the right discovered instance:
monitors:
logical_disk:
discovery:
# ... maps attributes including id: $1
collect:
type: multiInstance
sources:
source(1):
type: commandLine
commandLine: CHCP 437&&DISKPART.EXE /S ${file::listVolume.txt}
computes:
- type: awk
script: ${file::diskPart.awk}
keep: ^MSHW;
separators: ;
selectColumns: 2,8
- type: duplicateColumn
column: 2
- type: translate
column: 2
translationTable: ${translation::LogicalDiskTranslationTable}
mapping:
source: ${source::monitors.logical_disk.collect.sources.source(1)}
attributes:
id: $1 # matches rows to instances discovered with the same id
metrics:
hw.status{hw.type="logical_disk"}: $2
legacyTextParameters:
StatusInformation: $3
Rows whose key values match no discovered instance are ignored; instances with no matching row simply get no new metric values this cycle.
monoInstance: one run per instance
The job executes once for each discovered instance. Sources can inject that instance's attributes with ${attribute::<name>} — this is the mode to use when collection requires a per-device command or request:
monitors:
physical_disk:
discovery:
# ... maps attributes including id (e.g. /dev/sda) and __device_type_option
collect:
type: monoInstance
sources:
source(1):
type: commandLine
commandLine: "/usr/sbin/smartctl -H ${attribute::id} ${attribute::__device_type_option}"
computes:
- type: awk
script: "${file::physical_disk_smart_health.awk}"
keep: ^MSHW;
separators: ;
selectColumns: 2
- type: translate
column: 1
translationTable: "${translation::PhysicalDiskPredictedFailureTranslationTable}"
mapping:
source: ${source::monitors.physical_disk.collect.sources.source(1)}
metrics:
hw.status{hw.type="physical_disk", state="predicted_failure"}: $1
Because the run is already bound to one instance, the mapping does not need to re-map id.
monoInstance multiplies the number of requests by the number of instances. Keep the per-instance sources minimal, and prefer multiInstance whenever a single bulk query can return all instances at once.
Instance Identity: keys
keys (monitor level, default [ id ]) defines which attributes uniquely identify an instance. The engine uses it to:
- match
multiInstancecollect rows to discovered instances - recognize the same instance across discovery cycles (instead of creating duplicates)
Use several keys when one attribute is not unique by itself:
monitors:
db_session:
keys: [ db.user.name, db.server.application_name ]
Every attribute listed in keys must be mapped in the discovery (or simple) mapping.attributes, and re-mapped in a multiInstance collect mapping.attributes.
When Instances Are Removed
From MetricsHub Community 3.9.08, a discovery or simple job removes the instances it no longer returns. At the end of each trusted run, the engine removes the instances of the job's monitor type, created by this connector, that the run did not create or refresh:
| Job | An instance that disappears is removed |
|---|---|
simple |
at the next collect |
discovery |
at the next discovery cycle (every 30 collects by default, about 1 hour) |
collect |
never: a missing row only leaves the instance without new values |
A run is trusted only when all of these hold:
- Every source answered. A source that had data and comes back empty (most failures return an empty result too) is retried once; if it is still empty, it has not answered for this run. This only covers the run in which the source first comes back empty: the next run finds no previous data, does not retry, and counts the empty result as answered. Sources read through
${source::...}references (frombeforeAllor another job) count too: one that did not answer the last time it ran makes the job untrusted. - At least one instance was mapped. A job whose table is empty, or whose rows all lack the
keysattributes, removes nothing. - The connector status is OK.
- The run finished in time: no job timeout (
jobTimeout), no interruption, and every job of the parallel pool completed. - No source is incremental: a job that reads an
eventLog[1] source or afile[2] source inLOGmode, directly, through its mapping source, or through source references, never removes its instances.
Otherwise the run only creates and refreshes instances, as before: a source that has just stopped answering, a job that maps nothing, a failing or stopped connector, or a timed-out run never removes anything. Neither does a monitor type excluded from the resource's monitors, since its job does not run.
A source that stays empty or keeps failing is only covered for one run. From the next run, it counts as answered: if the job still maps other rows (from another source of a tableUnion[3], for example), the instances built from the rows of that source are removed. When a source can stay empty for long, keep the rows of the instances with a defaultRightLine[4], or set removalDelay.
These instances are never removed:
- the endpoint host
- the
connectormonitors - on connectors tagged
hardware, the types that take part in missing-device detection:battery,blade,cpu,disk_controller,enclosure,fan,gpu,memory,network,other_device,physical_disk,power_supply,robotics,tape_drive. A vanished instance of these types is kept and reported missing (hw.status{hw.type="<type>", state="present"}=0). The other types of ahardwareconnector (temperature,voltage,led,lun,logical_disk,vm, …) are removed like any other.
What this means for connector authors:
- The rules above only catch whole failures. A row dropped for any other reason looks exactly like an entity that disappeared, and its instance is removed: a filter on a state or a value, an inner
tableJoinagainst a source that misses some entities, a failed fan-out entry. Build the rows that create the instances from an inventory: see Build Instances from an Inventory[5]. - An unstable
idno longer only duplicates instances: the old instance is removed and a new one is created at every run, which restarts its counters andrate()state. - A removed instance is no longer exported; its series ends without a final data point. If the entity comes back, it is created again with fresh counters.
removalDelay
removalDelay (monitor level, next to keys) keeps an instance for a given time after the last run that returned it:
monitors:
replication:
keys: [ id ]
removalDelay: 15m
simple:
# ...
- The value is a duration (
30s,15m,1h,1d, or combined such as1h30m) or a number of seconds. The default0removes the instance at the first trusted run that misses it. A negative value is rejected when the connector is loaded. - A trusted run removes the instance once the time since the last run that returned it exceeds the delay. The delay is wall-clock time, checked only at the end of a trusted run: a
discoveryjob runs about once an hour by default, so a delay shorter than one discovery cycle keeps nothing. Use two discovery cycles or more (for example2h) to survive one missed cycle. - It is inherited and can be overridden through
extends[6], and it is also available for the custom monitors ofmetricshub.yaml. - MetricsHub versions older than 3.9.08 ignore it, so declaring it does not break a connector on an older agent.
Use removalDelay only when no inventory exists: entities defined by their activity (sessions, connected replicas, running queries), or as a stopgap until the connector reads an inventory.
Job Properties Reference
| Property | Jobs | Required | Description |
|---|---|---|---|
sources |
all | Yes | Named sources forming the job's table pipeline. See Sources[7]. |
mapping |
all | Yes | Turns the final table into instances, attributes, and metrics. See Mapping, Metrics, and Semconv[8]. |
type |
simple, collect |
No | multiInstance or monoInstance. When omitted on a collect job, defaults to monoInstance — declare it explicitly in new connectors. |
executionOrder |
all | No | Array of source names forcing a specific order. By default the engine runs a job's sources sequentially, ordered by their ${source::...} dependencies; use executionOrder only when a dependency is invisible to the engine, and list every source of the job. |
Additional Mapping Features
conditionalCollection
Only collect a metric when the referenced value is non-empty — typical when a device reports some sensors or counters only under certain conditions.
Declare it in the discovery (or simple) mapping: the availability check happens while discovering the instance, so the engine knows which metrics to collect for it. The metric's actual value stays in the collect mapping:
discovery:
mapping:
source: ${source::monitors.physical_disk.discovery.sources.source(1)}
attributes:
id: $1
metrics:
hw.physical_disk.size: $6
conditionalCollection:
hw.errors{hw.type="physical_disk"}: $9 # collect hw.errors only for disks reporting an error counter
collect:
type: multiInstance
mapping:
source: ${source::monitors.physical_disk.collect.sources.source(1)}
attributes:
id: $1
metrics:
hw.errors{hw.type="physical_disk"}: $9
See WinStorageSpaces[9] for a real two-phase example, or Ipmi-header[10] (the monitors shared by LinuxIpmiTool and WmiIpmi) for the simple-job form.
legacyTextParameters
Free-text values attached to the instance, inherited from the legacy hardware connector model — most commonly StatusInformation, the human-readable explanation of the current status. Declared in the collect (or simple) mapping, alongside the metrics it explains:
mapping:
metrics:
hw.status{hw.type="battery"}: $2
legacyTextParameters:
StatusInformation: $3
Use it for status detail text only; anything numeric or enumerable belongs in metrics or attributes.
Common Mistakes
- Omitting
typeon thecollectjob of a multi-instance monitor — the engine then defaults tomonoInstance, silently changing the execution model. StatemultiInstanceormonoInstanceexplicitly. - Forgetting to re-map
id(or the otherkeys) in amultiInstancecollect mapping, so no row matches any instance and no metric is ever collected. - Mapping an unstable value (rotating index, display label) as
id, which creates a new instance at every discovery cycle (from MetricsHub Community 3.9.08, the previous one is removed, so its counters restart every time). - Using
monoInstancewith expensive sources on monitors that can have dozens of instances. - Putting heavy inventory work in
collectinstead ofdiscovery— collect runs on every cycle. - Filtering on a state or a value (
Enabledonly,uponly, non-zero readings only,virsh listwithout--all) in adiscoveryorsimplejob. The instance is removed as soon as its state changes, which hides the very failure the state should report. Keep every row and map the state. - Joining the inventory with an inner
tableJoin(nodefaultRightLine) against a per-entity or optional source: an entity missing from the right table is removed. Keep the inventory on the left and add adefaultRightLine. - Building the instances from a fan-out (
executeForEachEntryOf) whose failed entry drops the row: the source still answers, the run is trusted, and the instances of that entity are removed. Join the result back onto the inventory.
Community Examples
- GenericUPS[11] — two-phase monitors with
multiInstancecollect, translations, andlegacyTextParameters - DiskPart[12] —
multiInstancecollect re-mappingid - SmartMonLinux[13] —
monoInstancecollect with${attribute::...}per-instance commands - PostgreSQL[14] — multi-attribute
keys
- [1] sources/event-log.html
- [2] sources/file.html
- [3] sources/table-union.html
- [4] sources/table-join.html#writing-defaultrightline
- [5] design-principles.html#a9-build-instances-from-an-inventory
- [6] reuse-and-configuration.html
- [7] sources/index.html
- [8] mapping-metrics-semconv.html
- [9] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/hardware/WinStorageSpaces/WinStorageSpaces.yaml
- [10] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/hardware/ipmi/Ipmi-header.yaml
- [11] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/hardware/GenericUPS/GenericUPS.yaml
- [12] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/hardware/DiskPart/DiskPart.yaml
- [13] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/hardware/SmartMonLinux/SmartMonLinux.yaml
- [14] https://github.com/metricshub/community-connectors/blob/main/src/main/connector/database/PostgreSQL/PostgreSQL.yaml
