How I destroyed a 3000 $ IoT Device, and what it taught me about testing

For years now, I've tested software for industrial IoT devices: primarily controllers and gateways that run in automotive and other industrial sectors. These aren't gadgets available for normal people. They sit on factory shop floors, bolted to heavy machinery and wired into live production lines. They control machines that make real products. Each unit costs between US$1,000 and 3,000, depending on features. 

A couple of years back, one such unit died on my test bench. I destroyed it, and I did it with an idea so simple you'll wonder why nobody had tried it before.

This is the story of a heuristic that I stumbled upon while testing, and the bigger lesson underneath it. Every system you work with hides a risk nobody has poked yet.

Just one more test?

A couple of years back, I was testing a web application that loaded firmware and applications onto these devices over the network. Think of it like your network router's web interface, but for industrial equipment.

While doing my exploratory testing, I triggered the firmware upload for one device. The upload process takes about 1 to 3 minutes, depending on network bandwidth. That was the free time when I thought about another test idea. That gave me one to three minutes of nothing to do.

I could grab a coffee. Check my phone. Stand up and stretch. But I'm a tester. So I did what testers do. I sat there and wondered what else this thing could handle. One idle thought turned into a test idea. You know that itch. When something you've never tried before pops into your head.

What will happen if I access this device in a second tab and trigger another firmware load on it from that tab? How will it behave?

So I opened the same web interface in a second tab. I was able to access the device's software interface. There were no warnings or locks. This tab felt like the device was idle. I decided to load the firmware again, and I pressed the upload button.

What happened next?

Just moments after I pressed the upload button, the device stopped responding mid-flash. All the LEDs present on the device stopped blinking. I had never seen this behaviour.

I power-cycled the device a couple of times. Nothing changed. The device was destroyed!

I went through denial first (maybe it just needs a reconnection to the power supply or a rewiring). Followed by a slow realization that it's not coming back. Then I was in a panic mode, as I realized that I had destroyed a 3000$ device with an idea that was not part of any approved test cases or strategy. The device was sitting dead on my test bed.

It took me some time to gather the courage to talk about it with my team members. However, the senior engineers on my team treated this as a routine event and were happy that we could identify such a loophole in our internal systems. For folks who've spent years in industrial IoT, destroying a device during testing is actually a good job. For me, as someone new to this space back then, it felt like I'd just done some damage.

I ran this test on both a packaged (sealed) unit and an open-board unit on my test bed. The packaged device was now gone. No way to recover it without cracking the enclosure open, which would defeat the whole purpose of a sealed industrial product.

I was able to recover the open-board unit using a JTAG debugger. It’s a hardware-level tool that lets you flash the device directly.

Root cause analysis (RCA) findings

After discovering this issue, we ran a root cause analysis (RCA). We found that the device was being destroyed, not because of a single bug, but because of three open bugs (or loopholes) in our system. 

It died because three separate safeguards were all missing at the same time. Think of it like three locked doors between the user and the flash memory. Any one of them, if in place, would have stopped the issue. In our case, all three were open. 

Figure: Three bugs leading to device failing 

Bug 1: No session lock during firmware load. The web application allowed two concurrent sessions on the same device during a firmware operation. This was the issue that was clearly visible from the direct steps that I had taken. A session lock would have shut the second tab out. 

Bug 2: No operation guard on write or execute-level actions. Ideally, the application should have blocked any new write or execute-level operation while one was already running. Nor was the user notified in the first tab that no further operations are allowed on the device at this time. That guard could have caught the risk even if the session lock failed. 

Bug 3: No firmware-level protection lock inside the device. The device firmware had no safeguard against parallel writes to its flash memory. Two simultaneous writes corrupted the flash. The device also had no fallback partition or mechanism to recover from it. It could have been the last line of defense, and it wasn't there either. 

Risks and issues mitigated

Our devices and controllers run in real automotive factories. They've been deployed with paying customers for years, sitting on live production lines and machinery. 

Now, if any of these devices gets destroyed mid-shift. The line it controls stops. Workers stand idle. Every minute of that downtime burns money, and in automotive plants, a stalled line can cost thousands of dollars a minute. Someone has to drive out, swap the failed unit, and reflash it before production can start again, extending an already long delay. 

Our team then shipped an update that locked sessions per device and blocked the entire interface layer while firmware was loading. And we didn't stop at this one device type. We applied the fix across all similar devices in our ecosystem. We had to make sure no customer or support engineer could accidentally repeat what I'd done.

Industrial IoT is regulated for a reason. Software bugs in this space don't stay just in the software world. They cascade directly into our physical world. A missed session check can destroy even the smartest devices, leaving no way to recover.

What stayed with me?

The heuristic that led me here is simple: Interact with something while it's mid-operation.

If a resource is being created, updated, or deleted, what happens when you poke it during that transition? In a typical web app, you probably get a 500 error or stale data. In an IoT system that controls physical hardware, that means you destroy something on the customer's factory floor.

I could never have found this from the "happy path." One browser, one session, and the firmware upload worked every time. The moment I opened a second tab, the whole thing fell apart. Every system you work with has a risk sitting inside it. Nobody has poked it yet. That's the only reason it looks safe.

And this isn't just an IoT thing. If you're testing a web app that talks to anything outside your control (user geography, localization, device type, and so on), look at it from a different angle. You'll often find it behaves nothing like you assumed. That exposes previously unknown risks. 

Some of those differences might be trivial, but others might be serious problems, like the one I found. Tools like WonderProxy help you see how your application behaves from different locations around the world. What works from your desk might not work the same way from your customer's location in another country, or even a different timezone. 

So poke the transition. Open the second tab. Try the thing nobody tried.

If you test to discover, you are testing to win.