The expensive failure in connected products is rarely the radio module; it is the unmanaged gap between “device connects” and “fleet behaves.” My position is blunt: a product manager should scope connectivity and device management before feature UI, because every shortcut becomes a field-cost multiplier once devices leave the bench.
Teams under-scope identity because the first demo works without it
Cheap IoT rollouts break at 20 devices – what to fund instead is right that tiny pilots hide real costs, but the breakage usually starts earlier: the team lets serial numbers, shared passwords, or “temporary” API keys stand in for device identity because the demo broker accepts them.
The cost is not philosophical. Without per-device identity, every support action becomes a manual investigation because the backend cannot prove whether a message came from the device, a test rig, a cloned firmware image, or a bored contractor with the staging credentials. X.509 client certificates, TLS 1.3, TPM 2.0, ATECC608A secure elements, and AWS IoT Core policies are not enterprise decoration; they are the cheapest way to avoid treating the whole fleet as one compromised user.
A product manager should budget identity as a product requirement, not as a security hardening task, because it affects onboarding, returns, warranty swaps, analytics, and customer support. If the device ships with one shared MQTT username, revoking a single stolen credential can disconnect every installed unit, which is an unacceptable operational blast radius because support will have no safe partial rollback.
Use concrete acceptance criteria. Each production device should have a unique credential generated outside the application firmware, certificate rotation should be possible without physical access, and the cloud registry should bind device ID, hardware revision, firmware version, customer tenant, and activation state. A practical first target is a certificate validity period of 365 days; treat that as a planning value to tune, not a universal rule, because short certificates increase renewal traffic while long certificates extend exposure after theft.
The mistake costs schedule as well as risk. Teams that skip identity often add it after beta, and retrofitting secure enrollment touches bootloader storage, mobile onboarding, manufacturing scripts, support tooling, and database schema because identity is part of the product’s lifecycle. A “two-day” shortcut can become a six-week rework because the team must migrate already-provisioned devices without bricking them.
Teams choose a protocol before they know the failure mode they can afford
MQTT 3.1.1, MQTT 5.0, CoAP over DTLS from RFC 7252, LwM2M 1.1, HTTPS, WebSocket, LoRaWAN 1.0.4, LTE-M, NB-IoT, and Matter 1.3 are not interchangeable transport labels. They encode different assumptions about power, latency, payload size, addressability, broker state, and operator control. The wrong protocol is expensive because the team has to compensate elsewhere: more battery, more backend state, more retries, or more field visits.
MQTT wins for many always-reachable products because retained messages, QoS 1, persistent sessions, and topic routing reduce backend complexity. It loses on sleepy constrained devices if keepalive traffic wakes the modem too often, because radio attach time consumes more energy than the payload. CoAP wins for request-response telemetry over unreliable links because confirmable messages and small headers fit constrained devices, but it costs more application design effort because developers must model idempotency and retry behavior explicitly. LwM2M wins when remote configuration, firmware update, and object models matter because OMA SpecWorks already defined resources for device management; it costs training time because teams used to REST endpoints must learn objects, instances, resources, and observe operations.
Do not approve “MQTT by default” or “HTTP because everyone knows it” as a scope decision. Approve a failure budget. For example, a tunable starting point for MQTT keepalive is 60 seconds on powered devices, while a battery product may need minutes or hours because each cellular wakeup burns energy and may reset carrier NAT state. AWS publishes an IoT Core MQTT payload limit of 128 KB, so large logs or images must use S3, presigned URLs, or a chunking protocol because the broker is not a file transfer service. LoRaWAN payload capacity can drop to roughly 51 bytes at EU868 SF12 under common regional parameters, so JSON status blobs are wasteful because headers can exceed useful data.
The explicit comparison I would put in a scope document is this: AWS IoT Core wins when the team needs managed certificates, rules engine integration, Device Shadow, and low operations headcount; it costs cloud coupling, policy design, and per-message pricing discipline. EMQX 5 wins when the company needs broker control, private deployment, MQTT 5 features, custom authentication, or data residency; it costs SRE ownership, capacity testing, patching, and incident response. Neither is “best” without the operating model, because the cheaper option is the one whose failure mode the team can actually run.
Teams treat OTA as a feature, then pay for it as an insurance policy
The most expensive device-management mistake is assuming over-the-air update can wait until after launch. OTA is not a convenience feature; it is the only way to repair a wrong radio timer, expired root CA, memory leak, modem firmware bug, or regulatory logging issue without physically touching devices. Delaying it is costly because field service becomes the update mechanism.
I would not ship a connected product without a rollback-capable update path, even for a paid pilot, because pilots create real devices in real environments and the team will learn something that requires firmware change. The minimum viable OTA scope should include signed artifacts, staged rollout, health check, rollback, version reporting, and a way to pause deployment. Mender 3.x, RAUC 1.11, SWUpdate, hawkBit, Zephyr 3.7 LTS MCUboot, and Android A/B updates solve different parts of that stack, but the product requirement is the same: failed update must not equal dead device.
The cost of weak OTA is measurable in operations. If a truck roll costs $150 as an internal finance assumption, then a bug affecting 400 devices becomes a $60,000 avoidable service event before customer credits, overtime, or churn. If the team claims field visits are rare, ask for the measured return rate from the closest hardware program, because optimism is not a deployment plan.
Scope update telemetry early. Track OTA start count, download failure rate, install failure rate, rollback count, post-update boot success, firmware adoption percentage, and p95 download duration. Prometheus, Grafana, OpenTelemetry 1.30, CloudWatch Metrics, Azure Monitor, and Datadog can all show these numbers, but the important product decision is who is allowed to stop a rollout at 2% adoption when errors spike. Treat 2% as a guardrail to tune during beta, because a small canary limits damage while still exercising real networks.
A small broker configuration can express several product decisions that otherwise disappear into vague stories:
listener 8883 cafile /etc/mosquitto/ca.crt certfile /etc/mosquitto/server.crt keyfile /etc/mosquitto/server.key require_certificate true use_identity_as_username true max_inflight_messages 20 max_queued_messages 1000 persistent_client_expiration 14d autosave_interval 30
This Mosquitto 2.0 configuration is not a full platform, but it forces the conversation a PM needs: client certificates, queue limits, offline behavior, and persistence windows. A 14-day persistent session expiry is a product value to tune because it protects intermittently connected devices while preventing infinite broker storage growth.
Teams budget cloud messages and ignore support messages
Product managers often count telemetry messages but miss the support workload created by bad observability. That is a scoping error because every “device offline” ticket becomes human debugging unless the system captures enough context to separate firmware failure, weak signal, carrier outage, certificate expiry, and backend rejection.
Device observability should include RSRP, RSRQ, SINR, RSSI, modem registration state, APN, SIM ICCID, eUICC profile state under GSMA SGP.32 where relevant, firmware version, boot reason, heap watermark, queue depth, last successful publish time, and last cloud command result. Quectel BG95, Nordic nRF9160, Sierra Wireless HL series modules, and u-blox SARA-R5 expose much of this through AT commands, but firmware must sample and report it deliberately because the cloud cannot infer radio conditions after the device disappears.
IoT Connectivity and Device Management Best Practices gives teams a useful checklist, but I would turn it into release gates rather than advice because advice does not stop a shipment. A release gate can say: p95 reconnect time must stay under 30 seconds in the lab’s measured LTE-M outage test, OTA rollback must succeed in 99 out of 100 scripted attempts, and the support console must show the last failure reason without querying an engineer.
The cost of poor observability is slow triage. If support spends 18 minutes per offline ticket as a measured help-desk average and a bad firmware build generates 700 tickets, the team burns 210 staff-hours before fixing the root cause. That cost is worse than the cloud bill because it delays diagnosis while customers are already affected.
Do not let the team hide behind “we have logs.” Logs that require SSH, UART, a JTAG probe, or physical retrieval are postmortem artifacts, not operating telemetry, because they arrive after the customer has already lost service. A realistic scope includes remote log levels, bounded log upload, privacy review, and redaction rules. OpenTelemetry semantic conventions are stronger for services than for microcontrollers, but using trace IDs or correlation IDs across device, broker, rule processor, and API still shortens incidents because engineers can follow one command through the system.
The first scoping move should be a costed failure table
Start with a one-page failure table before approving build scope. List certificate expiry, carrier outage, bad firmware, duplicate device ID, broker queue overflow, dead battery, tenant misrouting, and partial OTA failure. For each row, name the detection signal, owner, customer impact, rollback path, and estimated cost. That document will expose whether the product is ready to ship connected devices or only ready to demo them.
