A green security dashboard can coexist with a connected fleet that accepts revoked credentials or silently stops reporting. I would judge security and compliance by repeatable production tests tied to an explicit device population, not by the number of controls marked complete. That position is inconvenient for operations: it makes missing telemetry a security finding and forces teams to prove enforcement while the system is running.
A control is not working if you cannot name the devices it protects
The denominator comes first. A chart showing few authentication failures says little if half the deployed devices have disappeared from the chart. Maintain a fleet inventory with a stable device ID, owner, environment, expected reporting schedule, credential identifier, and lifecycle state. Define which states count as expected in production: deployed and commissioned might qualify, while retired and explicitly quarantined devices might not. Record every state change so a shrinking denominator cannot improve the score without an explanation.
For each expected device, measure the age of its last authenticated receipt at the server. A device-side “sent successfully” event is weaker evidence because the message may never have reached the broker or passed authorization. If devices are scheduled to report every five minutes, use 15 minutes as an initial alert window to tune against observed network jitter and maintenance windows. In an illustrative fleet export, 24 of 800 expected devices outside that window would produce an observed stale fraction of 3%; the count and denominator matter more than the percentage alone.
This small Python 3 check runs against a fleet.json file containing an array of objects with id, boolean expected, and Unix-seconds last_authenticated_at fields. Change the 900-second window to match the reporting contract.
python3 - <<'PY'
import json
import time
from pathlib import Path
rows = json.loads(Path("fleet.json").read_text())
expected = [r for r in rows if r["expected"]]
if not expected:
raise SystemExit("ERROR: no expected devices in inventory")
stale = [r for r in expected if time.time() - r["last_authenticated_at"] > 900]
print(f"expected={len(expected)} stale={len(stale)} fraction={len(stale)/len(expected):.3f}")
for device in stale:
print(device["id"])
PY
Run that calculation against an inventory snapshot and a receipt table from the same time interval. Otherwise, a newly commissioned device can appear stale before its first scheduled report, or a retired device can linger in the denominator. Keep separate counts for devices with no receipt, devices whose last request failed authentication, and devices whose authenticated request was denied authorization. Those states imply different repairs.
I treat Security and Compliance for Connected Systems as a set of claims to test rather than proof of deployment health. A current inventory joined to server-side events is the first test, because no access-control metric can describe devices the monitoring system has forgotten.
Successful connections do not prove that bad connections fail
Measure both sides of an enforcement boundary. For TLS 1.3, specified in RFC 8446, successful handshakes show that valid clients can connect; they do not show that an expired, untrusted, or revoked certificate will be refused. For an MQTT 5.0 deployment, a successful publish to an allowed topic does not establish that the broker rejects a publish to another tenant’s topic. Create controlled negative tests with dedicated identities and topics, and record the decision returned by the same production endpoint that handles ordinary traffic.
Start with three probes: a valid credential requesting an allowed action, a valid credential requesting a forbidden action, and a credential that has been revoked. Give each probe a unique correlation ID. Store the request time, device or test identity, policy version, broker or gateway instance, decision, and response time. The valid probe guards against a system that “passes” every denial test by denying everything. The forbidden-action probe tests policy enforcement. The revoked-credential probe tests whether a credential change has propagated to the point that accepts connections.
Revocation needs its own clock. Measure elapsed time from the recorded revocation event to the first verified rejection at every relevant ingress point. In an illustrative drill, 19 of 20 revoked test clients rejected within 60 seconds would be an observed 95% for that drill, not a general guarantee: the remaining client may identify a stale broker, an open session that was not terminated, or a gap in test coverage. If policy is evaluated through Open Policy Agent, log the policy bundle revision with each decision; otherwise, a denial cannot be tied confidently to the rule set that produced it.
Do not infer that certificate revocation works merely because a certificate has been marked revoked in a registry. That registry records intent, while the production handshake and session behavior reveal enforcement. Test an already-connected client as well as a new connection, because rejecting new handshakes does not necessarily close existing MQTT sessions. Also separate a network timeout from an explicit rejection: a timeout proves the probe could not complete, not that the security control made the right decision.
Keep test identities isolated from customer devices and label their events so alerting does not mistake a planned denial for an attack. Run probes after credential or policy changes and on a schedule that matches the risk of an undetected failure. A passing nightly probe cannot establish what happened during the preceding day; its honest claim is that enforcement worked at the tested endpoint and time.
Audit evidence expires when it cannot be joined to production events
A compliance artifact should let an engineer reproduce a conclusion. For a selected time window, capture the inventory version, the policy or broker configuration revision, the query used to produce the result, the event source, and the exceptions approved during that window. NIST SP 800-53 Rev. 5 can supply control language, but a control identifier is not evidence that the corresponding production path behaved correctly. The join between a named population and its observed decisions supplies that evidence.
Instrument the path with OpenTelemetry where it can carry correlation IDs through gateways and services, and export counters and latency histograms to Prometheus. Keep low-cardinality labels such as environment, region, decision, and policy revision on metrics; put device IDs in bounded logs or traces instead. A Prometheus series per device becomes expensive as a fleet grows, and it still does not replace an inventory join. Prometheus documents a default scrape_interval of one minute; that published default is a collection interval, not proof that an event was observed within a minute, because export failures and query delays add time.
Check the evidence pipeline itself. Measure the fraction of ingress decisions that produce a searchable decision event, the delay between decision and searchable record, and the age of the latest record from each gateway. Compare gateway request counters with received decision-event counts over the same window, accounting for retries and batching. If the gateway reports traffic while the audit stream is silent, mark enforcement status unknown rather than green. A system can keep enforcing policy during a logging outage, but operations cannot demonstrate that fact from missing records.
Retain raw events according to the applicable retention policy, and keep derived reports with their query text and execution timestamp. Record clock synchronization status too: a revocation timeline built from machines with drifting clocks can show a rejection before the revocation request. Security and Compliance for Connected Systems can inform an audit conversation, but a replayable query and its source records are what let the production team defend a particular result.
Page on broken guarantees, not every suspicious-looking counter
Alerting should reflect the promise the service makes. Page when negative probes show an unauthorized action succeeding, when a revoked test identity remains accepted beyond its agreed limit, or when evidence collection is too incomplete to determine either outcome. Route a single stale device to a lower-severity queue unless that device carries a documented higher-risk role; otherwise, routine connectivity interruptions will bury an enforcement failure. Set a proposed 99.5% expected-device receipt target as a value to tune against the device reporting contract, rather than adopting it as a universal security threshold.
There are two reasonable monitoring choices. AWS IoT Device Defender Detect wins when the fleet and its relevant behavior live in AWS IoT Core, because its managed rules reduce the infrastructure an on-call team must operate; it costs service charges and leaves the team responsible for defining useful behaviors and handling findings. Prometheus with the OpenTelemetry Collector wins when decisions cross multiple brokers or self-hosted gateways, because the team controls the instrumentation and queries; it costs collector capacity, storage, cardinality management, and ownership of the alert pipeline. Neither option proves a denied action without an actual denial test.
Use Alertmanager to route actionable failures and Grafana to display the underlying population, probe result, and evidence freshness together. Put the inventory count beside the stale fraction so an apparent improvement caused by missing devices is visible. Put the last successful negative probe time beside authorization-denial rates so low denial traffic is not mistaken for a healthy policy. During an incident, an engineer should be able to answer which guarantee failed, when it last passed, which endpoints were tested, and whether the evidence stream was complete.
I would not page on raw failed-login volume alone, because retries, misconfigured devices, and deliberate probes can all raise it without showing that an unauthorized request succeeded. Investigate that volume with context, but reserve the highest urgency for demonstrated enforcement failures and blind spots. That distinction makes the pager more likely to fire for the event the team actually needs to fix.
The first useful change is a failed-action probe
Pick one production ingress endpoint and one disposable test identity this week. Record an allowed request, a forbidden request, and a request after revocation, each with a correlation ID and server-side decision. Then verify that the inventory, decision log, and alert all describe the same test. If any join fails, fix that gap before adding another dashboard.
