ZeuZ

IoT Software Testing: What to Cover Around the Device

Key Takeaways

  • Test Studio covers the software around a connected device: web and mobile web dashboards, desktop tools, APIs, and databases.
  • REST, SOAP, and GraphQL checks run in the same suite as UI checks.
  • Database checks confirm the device's readings were written to the right tables.
  • Parallel runs execute on your own nodes.
  • Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one.
Blog

Share with your community!

IoT Software Testing: What to Cover Around the Device


IoT software testing checks everything around a connected device that people and systems depend on: the messaging between the device and the cloud, the APIs and database that receive and store its data, the dashboards and apps that display it, and the update and reconnect paths that keep a fleet running.

Hardware gets the attention in most IoT projects, but most defects customers notice live in the software around it: a reading that arrives twice, a dashboard that shows stale data after a reconnect, or an API that accepts a value the sensor could never produce. IoT testing as a whole spans the hardware and the software. This guide covers the software layer, with a test you can run on a laptop and no device in sight.

We built a small ingest service and ran the MQTT test in this guide with Python, paho-mqtt 2.2, and a local broker. Protocol and cloud behavior come from the MQTT standard and from AWS's IoT documentation, linked where they are used.

What does IoT software testing cover?

IoT software testing covers five layers around the device, and each one fails in its own way. Mapping them first keeps the test plan from collapsing into "test the app".

Layer

What to test

Typical defect

Messaging

Delivery guarantees, duplicates, ordering, topic permissions, and offline behavior

The same reading stored twice, or a command applied out of order

Ingest and APIs

Validation, schemas, error responses, and authentication for device and app calls

An impossible value accepted, or a malformed payload that crashes the service

Data

That readings land in the right tables, with the right device, unit, and time

Readings written under the wrong device, or time zones mixed up

Dashboards and apps

That the web dashboard, mobile web view, and any desktop tool show current, correct state

Stale values after a reconnect, or an alert that never fires

Updates and lifecycle

Provisioning, firmware rollout, rollback, and decommissioning

A failed update that leaves a device unreachable


The firmware and the physical device are a separate discipline with their own test benches. The software layer is where most of the automation pays off, because it can run on every change without a lab.

How is IoT software testing different from IoT device testing?

IoT software testing checks the systems that talk to a device, while IoT device testing checks the device itself: its firmware, sensors, radio, battery, and physical behavior. The two meet at the protocol, so a good plan agrees where one stops and the other starts.

Device testing needs hardware, test benches, and often environmental chambers. Software testing can use simulated devices, which publish the same messages a real one would, so it runs in CI on every change. Most teams need both. If your team owns hardware, a checklist for IoT device testing covers the bench side, and this guide covers everything from the broker outward.

The useful handoff is a contract: the message format, topics, quality-of-service level, and the values each field may hold. The device team tests that the device sends it, and the software team tests that the cloud handles it, including when it arrives late, twice, or not at all.


How do you test device-to-cloud messaging?

iot-messaging-delivery-guarantees-testing

Test messaging against the guarantees your protocol actually makes, not the ones you wish it made. Many IoT systems use MQTT, and the MQTT 5.0 standard defines three delivery levels: QoS 0 delivers at most once, QoS 1 delivers at least once and allows duplicates, and QoS 2 delivers exactly once.

Each level implies tests:

  • QoS 0. Messages can be lost. Test that the system tolerates gaps, such as a missing reading in a chart, without raising false alarms.

  • QoS 1. Messages can arrive twice. Test that the ingest service is idempotent, so a duplicate does not create a second record or trigger a second action.

  • QoS 2. Delivery is exactly once, at a cost in round trips. Test the latency it adds under load.

  • Will messages. The standard lets a client register a Will Message, which the server publishes after the network connection closes. Test that your system marks the device offline when it arrives.

  • Ordering. Do not assume order. AWS's Device Shadow documentation says there is no guarantee that messages arrive at the device in any specific order and that a device can discard messages with a version older than the one it is tracking. Test that stale updates are ignored.

Topic permissions belong in the same suite. Test that one device cannot publish to another device's topic, using ordinary functional tests that expect the broker to refuse.

How do you test IoT applications without real devices?

Use simulated devices: small scripts that connect to the broker and publish the same messages a real device would. They let you test duplicates, bad values, and bursts on demand, which real hardware rarely produces when you need it.

Here is a test we ran against a minimal ingest service that subscribes to devices/+/telemetry and writes each reading to a database. The first test sends the same reading twice, as QoS 1 allows, and checks that only one row is stored. The second sends a value outside the sensor's rated range and checks that it is rejected while the next good reading is kept.

PYTHON

import json, sqlite3, time

import paho.mqtt.client as mqtt

 

def publish(device_id, reading):

    client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)

    client.connect("localhost", 1883)

    client.loop_start()

    topic = f"devices/{device_id}/telemetry"

    client.publish(topic, json.dumps(reading), qos=1).wait_for_publish()

    client.loop_stop()

    client.disconnect()

 

def rows(db, device_id):

    with sqlite3.connect(db) as conn:

        return conn.execute(

            "SELECT seq, temp_c FROM readings WHERE device_id = ? ORDER BY seq",

            (device_id,),

        ).fetchall()

 

def wait_for(check, timeout=5):

    deadline = time.time() + timeout

    while time.time() < deadline:

        if check():

            return True

        time.sleep(0.1)

    return False

 

def test_reading_is_stored_once_even_if_delivered_twice(ingest_db):

    reading = {"seq": 1, "temp_c": 21.5, "ts": 1700000000}

    publish("sensor-42", reading)

    publish("sensor-42", reading)  # QoS 1 allows duplicates, so send it twice

    assert wait_for(lambda: rows(ingest_db, "sensor-42") == [(1, 21.5)])

    time.sleep(0.5)

    assert rows(ingest_db, "sensor-42") == [(1, 21.5)]

 

def test_out_of_range_reading_is_rejected(ingest_db):

    publish("sensor-43", {"seq": 1, "temp_c": 900.0, "ts": 1700000000})

    publish("sensor-43", {"seq": 2, "temp_c": 22.0, "ts": 1700000005})

    assert wait_for(lambda: rows(ingest_db, "sensor-43") == [(2, 22.0)])

Both tests passed. To make sure the first one could fail, we removed the duplicate check from the ingest service and ran it again: the test failed, because the reading was stored twice. A test that cannot fail does not protect anything, so break the code on purpose once for every new test.

The ingest_db fixture starts a local broker and the ingest service before the tests run. In CI, point the same tests at a test broker and a test database, never at production devices.


How do you test the APIs and database behind a connected device?

simulated-iot-devices-test-ingest-service

Test the APIs and database as the record of truth, because that is where a reading becomes something people act on. IoT application testing often stops at the dashboard, but the dashboard only shows what the API returns and the database holds.

  • Validation. Send impossible values, missing fields, wrong units, and oversized payloads, and check the API rejects each one with a clear error.

  • Identity. Check every reading lands under the device that sent it, and that a device cannot write another device's data.

  • Time. Check timestamps are stored in one time zone and that late readings sort correctly.

  • Contracts. Check REST, SOAP, or GraphQL responses keep their shape when a new field or firmware version appears.

  • State after a flow. After a UI action, such as changing a setpoint, check the API and the database both show the new value, not just the screen.

Database checks catch a class of defect that no UI test sees: the screen looks right, but the value stored under the device is wrong, and every report built on it is wrong too.

How do you test IoT dashboards and companion apps?

Test dashboards and companion apps against live and replayed device data, and check what they show when data is late, missing, or contradictory. Most IoT dashboards are web applications, so the same automated UI testing you use elsewhere applies, with a few IoT-specific checks:

  • Freshness. When a device goes quiet, the dashboard should say so, not keep showing the last value as if it were current.

  • Alerts. Push a reading over a threshold and check the alert appears, once, for the right device.

  • Commands. Send a command from the dashboard and check the device's reported state changes, not only the button.

  • Scale. Load the dashboard with a realistic number of devices and check tables and maps stay usable.

AWS's Device Shadow model is a useful way to think about commands: an app writes the desired state, the device writes its reported state, and the service reports the difference in a delta. Test that the dashboard shows the reported state, and flags any delta that does not clear.

What should IoT performance testing measure?

IoT performance testing should measure whether the system keeps up with your whole fleet at peak, not how fast one device responds. Simulate devices at the scale you expect, then push past it.

Measure

What it tells you

Concurrent connections

Whether the broker and services accept the whole fleet at once

Message throughput

Whether ingest keeps pace with the combined publish rate

End-to-end latency

Time from a device publishing to the reading appearing in the API and dashboard

Reconnect storm recovery

How long the system takes to recover when many devices reconnect at once, such as after a network outage

Backlog after recovery

Whether queued messages are processed without loss or duplicates

Database write rate

Whether storage keeps up, and whether queries slow as data grows


The reconnect storm is the test teams skip and then meet in production. When a site loses power or connectivity, every device comes back at the same moment, and the system has to absorb the burst and the backlog together.

How do you test offline, reconnect, and update scenarios?

Test the moments when a device drops off, comes back, or changes software, because those transitions are where state goes wrong.

  1. Disconnect a device mid-session. Check that it is marked offline, that its Will Message is handled, and that the dashboard says so.

  2. Change state while it is offline. Write a new desired state, reconnect the device, and check it receives and applies the change.

  3. Deliver updates out of order. Send an older version after a newer one, and check the older one is discarded.

  4. Interrupt a firmware update. Cut the connection partway through and check the device recovers or rolls back, and that the cloud records the right version.

  5. Decommission a device. Remove it, then check its credentials no longer work and its data is kept or deleted according to your policy.

Steps 01 to 03 run against simulated devices. Steps 04 and 05 usually need a device or a faithful emulator, which is where the software and device teams share a bench.


Where do desktop tools fit in IoT software testing?

Many connected products ship a desktop tool for configuration, calibration, or local diagnostics, often on Windows, and it needs testing like any other client. Desktop application testing covers that layer: the tool's screens, its connection to the device or gateway, and what it writes back.

Web-only frameworks stop at the browser. If your team asks "can Selenium test desktop applications?", the short answer is not on its own, and a guide to desktop automation tools covers the options for Windows and macOS. Keep desktop cases in the same suite as the API and dashboard cases where you can, so one run shows whether the whole product works.

What should you look for in IoT testing services?

Look for IoT testing services that cover the layers around the device in one suite, run against simulated devices in CI, and keep the suite with you. Ask each provider:

  • Layers. Do they test messaging, APIs, database state, dashboards, and desktop tools, or only the mobile app?

  • Steps. Which of test design, test case creation, test automation, test execution, QA reporting and analysis, and test maintenance and optimization do they run, and which stay with you?

  • Simulation. Can they simulate duplicates, gaps, out-of-order messages, and reconnect storms on demand?

  • Evidence. Does every failure carry logs, screenshots, and the change that caused it?

  • Environment. Can they run inside your network, including on-premises or air-gapped sites, when device data cannot leave it?

  • Ownership. Do the tests, data, and licenses stay in your accounts when the engagement ends?

Enterprise platforms such as Tricentis Tosca cover broad packaged-application landscapes. For connected products, check how any tool or service handles the protocol and data layers, not only the screens.

How does Test Studio cover the software around a connected device?

Test Studio, our AI test automation platform, covers the software around a connected device in one suite: web and mobile web dashboards, desktop tools, APIs, and databases.

  • APIs in the same suite. REST, SOAP, and GraphQL checks run in the same suite as UI checks.

  • Database checks. Database checks confirm the device's readings were written to the right tables.

  • Desktop tools. Test Studio automates Windows and macOS desktop applications alongside web and API.

  • Parallel runs. Parallel runs execute on your own nodes, and ZeuZ deploys on-premises or air-gapped when the application cannot leave the network.

  • Scope. Test Studio covers mobile web, not native mobile apps, and it is not a device test bench.

See Test Studio check a dashboard, an API, and a database in one run. Book a demo to try it on your product, or download the solution brief for the detail.

Which option is right for your team?

iot-fleet-reconnect-storm-load-testing

Which option is right for your team depends on who owns the software around the device and how much time they have:

  • You have engineers who own the suite. Start with simulated devices and the messaging tests above, then add API, database, and dashboard checks.

  • You have tooling and a mandate but no bandwidth, or a hard environment. Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one. Talk to the team.

  • You want the function run for you. Our managed QA runs all six steps while the suite and sign-off stay with you. It is the service behind outsourced QA at ZeuZ.

Key takeaways

  • Test Studio covers the software around a connected device: web and mobile web dashboards, desktop tools, APIs, and databases.

  • REST, SOAP, and GraphQL checks run in the same suite as UI checks.

  • Database checks confirm the device's readings were written to the right tables.

  • Parallel runs execute on your own nodes.

  • Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one.

Frequently asked questions

What tools are used for IoT software testing?

Most teams combine an MQTT or HTTP client to simulate devices, a test runner such as pytest to write checks, an API testing tool, a browser automation tool for dashboards, and a load tool to simulate the fleet. A test broker and test database keep runs away from production. Platforms can bring the API, UI, and database checks into one suite.

Book a demo to see API, UI, and database checks in one Test Studio suite →

Can you automate IoT testing end to end?

You can automate most of it. Simulated devices let you test messaging, APIs, data, and dashboards on every change. Physical behavior, radio conditions, and some firmware updates still need hardware or a faithful emulator, so most teams automate the software layers fully and run a smaller set of bench tests on real devices before release.

Talk to the team about automating the software around your device →

How do you test IoT data accuracy?

Send known readings from a simulated device and check the stored values, units, device IDs, and timestamps match exactly, then check every report and dashboard built on them. Include edge values, impossible values, and late readings. Database checks matter most here, because a screen can look right while the stored value underneath it is wrong.

Book a demo to see database checks confirm stored readings →

How do you test what happens when an IoT device goes offline?

Disconnect a simulated device and check it is marked offline, that the dashboard stops presenting its last reading as current, and that alerts fire if they should. Then change its desired state while it is away, reconnect it, and check it applies the change and that older, out-of-order updates are ignored.

Download the solution brief to see how Test Studio runs these checks →

How many devices should you simulate for IoT performance testing?

Simulate at least the number of devices you expect at peak, publishing at their real rates, then increase the load until something slows or fails, so you know your margin. Include a reconnect test, where many simulated devices connect at once after an outage, because that burst is often harder on the system than steady traffic.

Talk to the team about load testing your device fleet →

Who should own IoT software testing, the hardware team or the software team?

The software team should own testing from the broker outward, and the hardware team should own the device and its firmware, with a shared contract between them: message format, topics, delivery level, and allowed values. Each side tests its half of the contract, and a small set of joint tests on real devices confirms they still agree before each release.

Talk to the team about splitting ownership of your test suite →


Related Blogs