Connected-system security rarely fails at the moment a CTO signs off on an architecture. It fails later, when certificates expire, devices change hands, and the evidence needed for an audit sits in logs nobody retained. My position: buy the routine identity and logging infrastructure unless your product requires a genuinely unusual trust model. Building it is easier than maintaining proof that it still works.
The recurring bill is for evidence, not the first deployment
A team can demonstrate mutual TLS with X.509 certificates in an afternoon. Keeping that demonstration true across firmware releases, replacement devices, clock failures, and departed employees is a different obligation. Someone must know which certificate belongs to which physical unit, whether its private key was provisioned safely, when it expires, and how to disable that identity without disabling its replacement.
That obligation has a second track: proving what happened. An MQTT 5.0 broker may reject an unauthorized connection correctly, but an auditor or incident responder needs a timestamp, device identity, policy version, and retained record of the rejection. A broker log that rolls over before anyone asks the question is operationally different from durable evidence, because the correct control can no longer be demonstrated.
Put Security and Compliance for Connected Systems beside the procurement proposal because the proposal needs an owner and a cost for each recurring control. The questions are less glamorous than protocol selection: Who reviews failed authentications? Who tests certificate revocation? Who changes log retention when the applicable requirement changes? Who can still answer those questions after the engineer who built the system leaves?
Standards help specify the work, but they do not perform it. RFC 5280 defines certificate path validation; it does not discover a mislabeled device in a warehouse. ISO/IEC 27001:2022 control A.8.15 addresses logging; it does not decide which MQTT client identifier should appear in an incident record. NIST SP 800-53 Rev. 5 AU-12 calls for audit-record generation; it does not repair a broken export job. Treating a standard citation as an operating procedure creates a maintenance gap because the citation cannot wake an on-call engineer or verify a restored log.
I would not build a custom certificate authority solely to avoid a managed-service fee, because the apparent saving transfers key custody, renewal testing, revocation procedures, and emergency recovery to the product team. A custom authority can be justified by a specific offline provisioning or trust requirement. “We already have a broker” is not that requirement: message transport does not establish who will maintain the identity lifecycle around it.
Buying removes chores only when evidence can leave the vendor
For a connected-system rollout, compare AWS IoT Core with AWS IoT Device Defender and CloudWatch Logs against self-hosted EMQX 5 with an in-house identity and evidence pipeline. AWS wins when the device fleet fits its authentication and policy model and the team wants managed broker operations. EMQX wins when broker behavior, deployment location, or an existing operational platform makes that control worth owning. Neither option makes compliance automatic, because both still need a documented relationship between a device, its credential, its policy, and its recorded activity.
Their costs differ in kind. AWS brings usage charges, service configuration, integration work, and the risk that an export or policy depends on AWS-specific interfaces. Self-hosted EMQX brings compute and support costs plus responsibility for broker upgrades, backup restoration, certificate services, monitoring, and on-call response. As a budgeting assumption to replace with your own staffing data, eight engineering hours each week at a loaded rate of $150 per hour amount to $62,400 a year before infrastructure or incident work. That is a useful comparison against a vendor quote because it prices recurring ownership rather than the initial installation.
The managed option still needs a portability test before purchase. Ask for a sample export that joins a device identifier to certificate issuance, policy changes, connection denials, and firmware version. Verify that timestamps retain an unambiguous time zone and that records can be retrieved without a vendor console. If CloudWatch Logs is the source, specify the export destination and retention owner; if an Amazon S3 bucket is the archive, decide whether S3 Object Lock is appropriate for records that must resist deletion. Those decisions cannot be inferred from a sales description because the evidence chain depends on your configuration.
A sensible first contract question is not “Do you support compliance?” It is: “Show us how we retrieve a rejected connection and the policy that rejected it after an administrator account has been removed.” The answer exposes how much maintenance the buyer still owns, because a dashboard screenshot is less useful than an exportable record and a repeatable retrieval procedure.
Certificate rotation is a small task until it becomes a fleet event
Rotation deserves a rehearsal before a build-versus-buy decision. A certificate might be valid under TLS 1.3 while its associated device has been replaced, or a legitimate device might be stranded because its clock is wrong. In either case, checking expiration alone misses the operational problem: the team needs a current inventory, a safe replacement path, and a way to distinguish a failed renewal from an attempted impersonation.
This runnable OpenSSL 3.x check reads a local PEM certificate and reports whether it passes a proposed renewal window. The -checkend value of 2592000 seconds is a threshold to tune—30 days for this example—not a universal policy.
#!/bin/sh
set -eu
cert="${1:?usage: ./cert-audit.sh device.pem}"
serial="$(openssl x509 -in "$cert" -noout -serial)"
end="$(openssl x509 -in "$cert" -noout -enddate)"
fingerprint="$(openssl x509 -in "$cert" -noout -fingerprint -sha256)"
if openssl x509 -in "$cert" -noout -checkend 2592000 >/dev/null 2>&1; then
state=valid_beyond_window
else
state=rotate_within_window
fi
printf '%s %s %s %s\n' "$(date -u +%FT%TZ)" "$state" "$serial" "$end $fingerprint"
The script is useful precisely because its limits are visible. It does not establish that the certificate belongs to the claimed device, test revocation, or prove that a new credential was installed. Run it as ./cert-audit.sh device.pem and its output goes to standard output; preserving a defensible record requires a separate collection and retention path. A team that cannot name that path has discovered maintenance work, not finished a compliance control.
If the fleet runs on Kubernetes, cert-manager v1.16 can automate certificates for services within that environment. It does not, by itself, solve provisioning for devices that spend months offline. SPIFFE X.509-SVIDs can give workloads a consistent identity format, but adopting SPIFFE also creates duties around trust bundles, issuer availability, and rotation during outages. I would use either where its trust boundary fits, because extending a workload identity scheme to disconnected hardware merely to standardize tooling can make recovery harder.
Test rotation as a timed operational exercise. In a pilot, record the observed interval from disabling a compromised credential to a replacement device reconnecting; if that measurement is 47 minutes, investigate the steps that consumed those minutes before setting a production target. Also record whether an old credential still connects and whether the denial is retrievable from the archive. Those observations tell a CTO more about future staffing than a successful certificate issuance demo, because they include the handoffs that recur during incidents.
A purchase decision should require a failed-control rehearsal
The maintenance burden becomes visible when a routine change goes wrong. Stage a device replacement with a reused identifier, an expired certificate, and a policy update that denies the replacement. Have one engineer perform the change and another reconstruct it using only retained records. The second engineer should be able to identify the old credential, the new credential, the relevant policy revision, and the disposition of the failed connections. If they need the first engineer’s memory, the system has an evidence dependency that will survive either procurement choice.
Use Security and Compliance for Connected Systems as a prompt for the architecture review, then insist on a retrieval rehearsal rather than accepting an architecture diagram as proof. An OpenTelemetry Collector can forward operational telemetry, while a separate archive can preserve records for later examination; their responsibilities should be explicit because forwarding success does not guarantee that the required record can be found months later. Define a proposed retrieval objective—say, 15 minutes to produce the denial record and its policy revision—and change that objective if the organization’s actual response obligations demand something else.
For the buy option, make the vendor demonstrate the rehearsal with your device-identity shape and your export destination. For the build option, require the internal team to demonstrate the same thing after a broker restart and an operator handoff. Price the work left over in each case: alert review, certificate inventory reconciliation, archive access checks, upgrade rehearsals, and incident support. A lower initial invoice loses its appeal if each release creates an undocumented investigation for the people who maintain it.
Decide on the boundary, not a slogan. Buy the broker and routine identity plumbing when the managed system passes the export and recovery tests at an acceptable total cost. Build where control over deployment or trust is a product requirement you can state and fund. That position is contestable, but it gives both options the same burden of proof: show that the next operator can keep the controls working and demonstrate that they worked.
The first decision is who can reconstruct a failure
Before approving either architecture, schedule one device-replacement rehearsal and assign someone who did not build the system to investigate its failed connection. Give that person access only to the records and procedures expected in production. Time the reconstruction, note every missing identifier or permission, and ask each vendor or internal owner to close those gaps in writing. That exercise produces a maintenance estimate worth putting in the decision memo.
