Safer Harbor
Helping companies navigate the treacherous waters surrounding IT security and secure product development
The Boulevard of Broken Things – IoT Buyers Beware
In my last blog post, I talked about the reasons why people fail to upgrade software on Internet-connected devices. But what is the situation like when you’ve actually gotten “a round 2it” and adopted the “keep-my-devices-up-to-date” philosophy? Is everything sunshine-and-roses, or is there still more you need to worry about? Are you, in fact, headed for The Boulevard of Broken Things? (Apologies to Green Day and Al Dublin)
How many times have you wrestled with a software update for one of the myriad devices that you or your company own? If you’re like most people, somewhere around 10% of the time your upgrade will fail, for reasons that may be obvious or indecipherable. If you’re not as lucky, your upgrades might fail up to 50% of the time! This is a big problem for Enterprise IT and even for all of us in our night/weekend job as home IT technician. But if that’s the state-of-the-art in the mature enterprise and commercial software space, what should we expect in the emerging world of the Internet of Things? With 12 billion “things” on the Internet today, that means we’re facing the potential of 1.2 BILLION failed upgrades.
Are these new Things being designed with upgradability in-mind? In the consumer portion of the Things market, the answer is No, as Bruce Schneier details in Wired and his blog. Devices like home routers, modems and switches are cheap enough that the incentive to maintain the software is low. It’s cheaper, simpler and often better to just throw the old device out and buy a new one.
In the Industrial space, however, it is not as clear whether design-for-upgradability is being done, but the odds suggest that things are probably not much better.
It’s important to be an informed consumer, especially if you’re about to deploy “Things” out into your enterprise – or your home. Here are some questions to ask your vendor, along with some context about why you should be asking the question. When I use the term software, I am including any software that would be installed on the device including application software, operating system software, device firmware, BIOS, bootloaders, etc. Basically any software that was loaded into the device by the device manufacturer or the manufacturer of any components installed/built into the device. Any of it may need to be updated in the future.
- Can the software on this device be upgraded?
As mentioned above, many devices are not even designed to be upgraded. They have a software image installed on them at the factory and that’s it. If there’s a bug, you have to replace the device. - Can the software be upgraded remotely via an automated process (i.e. can it be done unattended)?
As a hypothetical example, if you work for an electric utility and you need to install a monitoring device on every power pole in your territory, you don’t want to have to send a truck out to every device if an update is required. The power utility in my area – PG&E – has over 2.2 million power poles and has a scheduled visit to each pole once every 10 years. They clearly don’t have time to visit each pole every time a software update is needed. Even if there were only devices on 1% of their poles, that would be over 22,000 trips.
No matter what business you’re in, think about the cost of sending people to every device each time an update is required, versus the ability of one person (or a small team) to perform the upgrades over the network from a central location. That’s much more cost-effective. - How long does the vendor commit to providing software updates for the device?
Some devices will be supported for years by their vendors. Others may only get 1 or 2 software updates ever. If that’s the case, what will you do if a bug like Heartbleed is discovered 4 years after you buy the device?
Be sure you calibrate your expectations, though. The device you paid $50 for probably won’t receive software updates for 5 years. One you paid $50,000 for might. It costs money for vendors to provide updates for devices. Just be sure you know what to expect and that those expectations match your needs. - What is required to obtain those software updates (support contract, direct Internet access for the device, an update server on your network, …)?
It costs money for a vendor to produce software updates for a device. You may be entitled to updates for a period of time as part of the warranty or pre-paid support agreement. But at some point you may have to pay to continue to get access to updates.
You also need to understand how your device receives updates. Does it expect to download them directly from the Internet? Do you want to allow that? I don’t think I want a power plant turbine to “phone home” and update its software whenever it feels like it. Perhaps you need to install an update server on your network that provides the update files to the devices. This may be better, but do you have a way to find out about new updates and quickly copy them onto your update server? Is there special hardware or software required on the update server to provide these updates?
It’s important to know what you need in order to keep your devices up-to-date. - Are the software updates digitally signed to show evidence of tampering?
If a hacker can slip a modified update onto your devices, he owns you and your network. Even a single compromised device may be enough to put your enterprise data at-risk. Using digitally signed images makes it much harder for the bad guys to try to sneak a modified software image onto your devices. But the vendor has to provide signed images as the first step in the process. That leads us to the next require step to prevent someone from tampering with a software update image… - Does the device check the digital signature of the image prior to installing it?
Having digitally signed images is not useful if the device doesn’t check the image to ensure that it was signed by the manufacturer. The device needs to check the signature of every image it downloads prior to ever running any upgrade code. - Does the device handle software upgrade failures in a predictable, appropriate manner?
As mentioned above, some software upgrades are bound to fail. Sometimes we can’t figure out why. Sometimes trying the upgrade again works fine. But if your device is going to download code and upgrade itself, it needs to handle those failures in a predictable, non-catastrophic manner. The best result is that the device continues to run the previously-installed version with the configuration that it had prior to the upgrade attempt. A device should never be left in an undefined or unprotected state after an upgrade. Ask your vendor to tell you what the device does when an upgrade fails. - How can the administrator know if the software was updated successfully? Can you easily tell what software version is running on a device?
Just because you told a device to update itself doesn’t mean it did so successfully. It’s important to be able to find out what version of software is running on the device at any time. It’s especially important to check that after an upgrade has been attempted. Knowing the installed software version on all of your devices will help you know how to react the next time you hear about a new vulnerability, because you’ll be able to tell quickly whether your devices are affected or not.
These questions and their answers are important whether you’re installing a home router, a global network of sensing devices, or a new power plant. All are Things, and if we’re going to stay off of the Boulevard of Broken Things, we need to be sure our devices were designed and built to be resilient.
The Computer board picture is from Shutterstock. The Champs-Élysées photo is my own.
This post originally appeared on LinkedIn on May 27, 2014.
Steve DeJarnett is Managing Director of Safer Harbor, a consulting firm that helps companies navigate the treacherous waters surrounding IT security and secure product development. We believe that the notion of a completely “safe harbor” is illusory and that companies must always maintain vigilance to protect themselves and their products.
Safer Harbor 