IoT development platforms promise to make the complicated coding process easier, but they’re not doing much for testers. While AI is coming on board to boost test automation--Gartner estimates artificial intelligence will be used in 30 percent of test tools by 2021--it’s not mature enough to help test the often device-heavy IoT apps being created today. And a lot of those apps are in the works. In fact, Bain & Company predicts the IoT market will hit $520 billion by 2021, more than double what it was in 2017.
So what’s the answer for software testers working in the IoT space?
Go Outside
“Testing is a lot more challenging when it comes to IoT,” said Jeffrey Hammond, vice president and principal analyst with Forrester Research and author of IoT Delivery Best Practice: Perform Real-World Testing. Hammond wants testers to actually experience how the real world can impact an app. “One vendor put a beacon near a bridge, but every time the bridge went up that interfered with the transmission,” he said. “That’s not something you can test in the lab.”
The real world brings myriad challenges a tester might miss sitting in a controlled environment, he said. Two examples clients shared with him recently: “Smart” door locks actually break if you try to fix them remotely, while an automotive security API using a car’s VIN number exposed the car’s identity and allowed a hacker to take control of the car’s climate system. “Devs just weren’t thinking about these real world issues,” he said.
Ask Questions
To create better apps that actually function in the real world, Hammond suggests that testers change how they think about the app. Stop testing to see if the code breaks and instead actually see if the device can be used by real people. To do that, he recommends asking lots of questions. Can the device be used one-handed? Can a non-technical store employee install an update? What if it’s loud? What if there is RF interference?
And don’t stop there. “Be skeptical about the use cases,” Hammond stressed. “You want to make sure you’re providing real value.” He pointed to the Juicero debacle, where a $400 machine didn’t turn out to be necessary to actually squeeze the packets of ingredients and the company went out of business.
Get Ahead
Start real-world testing as soon as possible and testers can provide usability feedback before it’s too late to make hardware changes, Hammond said. It can also increase everyone’s confidence when it comes time to do an IoT install or an update. Clients told him they spend a lot of extra “soak time” on factory floors because each one is completely different, and small things from temperature to settings can mean problems for an IoT application.
While field testing, QA teams will also get to experience, in person, what can happen to a network when an IoT application is operating. A crashed network can really help inform design decisions. Not everything has to or should go to the cloud, and some of it really should be dealt with on the “edge,” Hammond said. Testers can play a key role in helping decide what should go where.
The Price Tag
Even though real-world testing will save time, heartache and potentially disastrous problems in the long run, Hammond acknowledges it’s not without its challenges. Even the most thorough test team won’t catch every single potential problem in the field, he said, and physically going to the field takes more time and potentially a good bit more money.
It’s a balancing act, but worthwhile if testers keep three things in mind, he said. “How much compute power do I have? Where do I have it? And what is the real business value of the case?” he said. “That’s what testers can bring.”
