IoT Security: a Distributed Product Failure for the Ages

A Curious Case of Internet of Things

Last year millions of IoT (Internet of Things) devices were compromised and turned into zombies to launch massive DDoS attacks that brought down a huge chunk of the Internet. Those were  not isolated cases; every week there is a new breach, a new security failure that poses a serious threat to our infrastructure, our economy and even our lives.

These ‘insecure’ IoT devices are only going to grow in number. According to Gartner, by the end of 2017, there will be over 8.4 billion connected devices in the world. Take a moment and think about it. What kind of damage could 8.4 billion insecure IoT devices cause?

I’ve spent the past two weeks talking to more than a dozen experts from the IoT world to get a better grip of the situation and understand how real these threats are, what are the causes, and what can be done to mitigate them, if it’s even possible to be mitigated.

I would first like to thank the following industry experts who helped me with this story: Mark Shuttleworth, CEO of Canonical; Philip DesAutels, PhD, Senior Director of IoT at EdgeX Foundry; John Reno, Technology product marketing at Cisco; Nadir Izrael, CTO and co-founder, Armis; Marc Blackmer, Manager, Product Marketing, Industry Solutions Security Business Group, Cisco; Mark Thacker, Security Strategist at Red Hat; Eliav Gnessin, Founder and CTO of Cloud of Things; Noah Harlan, Founder of Two Bulls; Aaron Lint, VP of Research, Arxan; Ken Carroll (Vice President/Principal IoT Architect ) at Cloud Technology Partners; Robert Kusters, Director of Product Marketing, Inpixon; Bruce Schneier, an American cryptographer, computer security professional, privacy specialist and writer; Thibaut Rouffineau, Head of Marketing for Devices and IoT at Canonical; Thomas Pfeiffer, board of directors of KDE e.V….and many more.

What is “Internet of Things”?

Internet of Things or IoT is a marketing term for connected devices, just the way retina display is a marketing term for HiDPI display. The term was coined by Kevin Ashton of Procter & Gamble, in 1999. “Any edge device that connects to either a central system or makes itself available via the internet can plausibly be viewed as IoT,” said Noah Harlan, Founder of Two Bulls.

IoT is a very broad category of diverse devices that can be categorized on the basis of usage. In general, there are three category of IoT devices:

● Industrial IoT: Devices that control infrastructure or robots that build other objects. These have been connected to the internet to report on status, be controlled remotely and keep a system-wide log.
● Consumer IoT: This includes devices like cameras, thermostats, bulbs, smart TVs and more. Every consumer device with an IP address is a Consumer IoT device.
● Enterprise IoT: These sit between the two previous categories and are similar to consumer devices, but with the same expectations and functionalities as Industrial IoT devices, such as performance, connectivity and security.

Ken Carroll, Vice President and Principal IoT Architect at Cloud Technology Partners says these categories can be further broken down into different industries such as: Industrial & Manufacturing, Energy Logistics & Transportation, Retail, Building and City Systems, Healthcare, Agriculture Automotive, Oil & Gas, and Consumer.

According to IDC estimates, Manufacturing saw the largest IoT investment over $178 billion, followed by Transportation ($78 billion), and Utilities ($69 billion). Consumer IoT purchases was the fourth largest market segment in 2016.

If you are curious why we need these categorizations, just look at the largest market and compare with the most vulnerable one. That will help understand what makes IoT devices so insecure. Despite being the smallest of the four markets, Consumer IoT is the most vulnerable category. When you hear horror stories, they are almost always about consumer IoT, the smallest pie of the IoT market. “The devices primarily giving a bad name to the security characteristic of IoT devices are ones which are very cheaply mass manufactured. These tend to be targeted for mass sales into the consumer market, such as cameras,” said Carroll.

A quick peek into the evolution of IoT devices and how Consumer IoT came into existence will help provide clarity.

The Evolution of IoT

IoT devices have evolved from big, expensive and complex machines. “Early on, companies like Burlington Northern Railroad realized that it was very expensive to keep railroad traction motors failing in the field. If you can monitor and predict when certain types of failure will happen then you can fix them even before they happen and save a lot of money,” said Philip DesAutels, PhD, Senior Director of IoT at EdgeX Foundry.

In the early days it was a very expensive and complex feat with satellite communication and hardened computers to run inside of hot trains. Over time, new technologies emerged and companies like Wi-Tronix started offering solutions to the railroad company.

As machine 2 machine (M2M) technologies became simpler, easier and cheaper, IoT moved on from traction motors to heavy equipment to trailer trucks to every car made in the world. It’s now in our cameras, door-bells and even light bulbs.

That’s consumer IoT.

As IoT trickled down, while the basic concept remained the same, it served different purposes

While Industrial or enterprise use case was more about efficient business solutions, consumer IoT was all about products and features at attractive pricing.

DesAutels provided the  example of a waste oil management solution. Traditionally a waste oil management company would roll out trucks once a week to check if the customer needs to pump out waste oil. It’s a very inefficient way as trucks would be driving around even if there were no oil to be pumped. With IoT solutions you can monitor remotely and if pumping is required,  you can optimize the delivery of vehicle in real time, reducing the number of trucks and the number of trips that the vendor make.

In this example, IoT was not a product or feature. It helped a company cut costs and become more efficient. On the consumer side, you will find things like Ring doorbells that provide a sense of security. Google Nest offers the convenience of controlling the thermostat from the comfort one’s bed or even outside the home.

Consumer IoT is no different than any other consumer product, however, the security implications of your personal laptop are totally different from the security of a corporate laptop. Your company provided laptop has better security to protect the confidential company business, it’s guided by very strict regulatory and compliance policies where as your personal laptop has none.  In industrial IoT,  security means the availability of overall system – keeping the factory running, maintaining the reliability and safety of the grid.

“In the enterprise IoT, the investment in securities is being made to protect the confidentiality of business data and privacy of personally identifiable information,” said John Reno, Technology product marketing at Cisco.

In the consumer IoT landscape, neither the vendor nor the user are concerned about the fact that someone may hack into their baby monitors or VTech toys to steal data or compromise the network.

Why are Consumer IoT devices so insecure?

Economy. Consumer IoT is is a very thin margin business where vendors try to cut costs in every possible way to meet a price point that’s attractive to customers. The business model that allows you to purchase, for example, a high quality IP video camera for <$100 does not support lifetime updates of software, it doesn’t allow built in security features.

“Most consumer IoT vendors live off thin margins. As a result, market pressures can drive some manufacturers to strip out security functionality either to get to the market in time to be competitive, or to save on cost for commodity devices where margins are thin, said Marc Blackmer, Manager, Product Marketing, Industry Solutions Security Business Group, Cisco.

A majority of IoT vendors are hardware manufacturers who monetize from selling more hardware devices. They follow the old-fashioned approach to distribution where they develop a product, sell it and move to the next iteration of the products with ‘new’ features, leaving behind a long trail of unmaintained products.

“If economics are there people will create a system. There is no economic incentive for a light bulb manufacturer to produce something that is patchable. Even Microsoft doesn’t patch windows XP, as  there are no economical incentives.” said Bruce Schneier, an American cryptographer, computer security professional, privacy specialist and writer. “A technology solution doesn’t matter and I don’t care.”

However, the economy itself is not to be fully blamed. Lack of legal liabilities, regulatory or compliance restrictions to build security into those products is also one of the major reasons behind unmaintained products. If you purchase a super expensive smart TV or Smart refrigerator from any of the leading manufacturers like Samsung or LG, you will not find any information on their guarantee of support page about the duration that these  $2000+ devices will receive software updates. A typical fridge has a 10-16 year life,  but these vendors may stop pushing updates after one or two years leaving them exposed to attacks for the rest of it’s existence.

In April 2017, security researchers found 40 critical bugs in Tizen OS that runs on millions of devices. In this case we are not talking about$50 IP cameras made by an unknown Chinese ODM, we are talking about premium devices from the world’s second largest electronics company, after Apple.

Not all vendors have malicious intentions to not keep things patched and leave their users vulnerable. Some of these IoT vendors are new to this field and don’t have experience creating IP-enabled devices.

“I’m sure many people would have wondered why they’d need to secure a connected surveillance camera. After all, there’s no valuable data on a camera. Then Mirai botnet hit, proving the value is in how a compromised device can be used,” said Marc Blackmer, Manager, Product Marketing, Industry Solutions, Security Business Group, Cisco.

They ignore the basic concepts of security. “It’s as if we’ve forgotten 20 years’ worth of accumulated security best practices like not using hard coded default passwords for root accounts,” said Nadir Izrael, CTO and co-founder, Armis.

Robert Kusters, Director of Product Marketing, Inpixon said that security risks are created when IoT devices are designed to replace existing devices, offering more convenience and intelligence in order to manage remotely or save on costs.

“These devices often follow the same operating model that they had when they were not connected and security was an afterthought,“said Kusters. “Once a device is connected to a company network or the internet, it becomes a node that any hacker can see, and potentially exploit, either to discover other connected devices, such as company servers or to use as a bot for use in attacks like a DDOS.”

Irrespective of whether the above reason is applicable in the particular use-case, the bottom line is that IoT devices are insecure by design. Economy is certainly a primary factor, but it’s not the only factor. There are technological reasons too. One of the biggest challenges is that these devices are not updatable.

“When devices are deployed which are connected but can’t be updated or rely on hard-to-update common default security credentials, you are asking for a problem,” said Noah Harlan, Founder of Two Bulls.

You might be surprised to hear, but there is a genuine reason why many IoT vendors avoid software updates of their devices. “Software updates can easily cause problems – and the easiest way to avoid problems caused by software updates is to avoid software updates,” said Mark Thacker, security strategist at Red Hat.

Let alone inexpensive IoT devices, software updates have bricked iPhones and iPads. As a result, IoT vendors embrace the ‘release and forget’ model where they never bother to update software on working devices, which leaves these connected devices vulnerable.

That’s a technological problem.

Is there any way that smooth software updates can be guaranteed? Since IoT vendors never planned to update these devices, many such devices lack an interface for users to update them. And if there is a hardware interface for updates, like a USB port, what if an organization has 10,000 of those devices? “The likelihood that they’ll get patched is slim to none,” said Blackmer.

Those vendors who do want to be able to update their devices, still face the dilemma of updates breaking their devices.

“Automatic updates are the way forward for the IoT. Users shouldn’t even have to think about those updates, and should be able to rest assured that any security updates have already been pushed to the device within moments of being certified,” said Thibaut Rouffineau Head of Marketing for Devices and IoT at Canonical.

Canonical has created a lightweight operating system called Ubuntu Core, that’s designed for IoT devices and can provide IoT players with a secure OS that enables them to not only push out updates over the air, automatically, but to have an operating system that rolls back to a previous state if those updates don’t work as designed.

There are some caveats. “There are environments, like industrial control networks and medical device networks, where automatic updates may introduce serious, if not dangerous, problems. Even if the odds are 1 in 1,000, the impact of, say, a bricked device can be catastrophic,” said Thacker.

You don’t want to deploy automatic update mechanisms in those cases. In such cases the update mechanism should be in the hands of the operators to allow for testing ahead of deployment.

Even in solutions like Ubuntu Core, there is a roadblock. “One big question is who should be responsible for updates – the IoT device vendor, the IoT infrastructure vendor, the IoT application vendor or systems integrator, or the customer?” asked Aaron Lint, VP of Research at Arxan.

That’s just one of the many hurdles; the IoT landscape is very diverse. It’s a chipset and processor nightmare. “There are so many different board types and everyone is rolling their own, so it would be a very high cost to earn market share by adding support for new chips and catching up to existing ones,” opines Lint.

In addition, not every device out there is capable of running Ubuntu Core or can support such auto update mechanism. Each solution is very specific to the device, its system resources, its purpose, and many other things.

Whether a device runs Ubuntu or not, Blackmer believes that an ideal operating system would include at least some minimal security capabilities, if even just requiring authentication and forcing the user to change the default credentials. In addition, it must be upgradable.

The new OS approach will make edge devices safer to a degree, but they’re just one part of an overall IoT solution. Most IoT devices run on Linux that already has all the needed capabilities; all that is needed is proper implementation. “Vulnerabilities arise mostly from poor operational practices rather from devices – the devices tend to be secure,” said Carroll.

When we look at the whole problem, especially those related to devices, we realize it’s actually not an IoT problem. In fact, in most cases the security issues that we hear about are not IoT issues, but rather product life cycle issues and are applicable to any product. Even the high-profiled D-link camera vulnerability was a product problem.

“If we take those cameras and those dishwashers 15-20 years ago, we knew how to solve both those security problems . As soon as we got a web server, we learned how to patch it. As soon as we got a remote machine we figured how to do firmware management,” said DesAutels.

DesAutels opined that we should not blame these companies as they are making the same mistakes that their predecessors made in other domains. He stressed that these are not IoT problems, these are classic examples of product life cycle management.

“You need to know SDLC (systems development life cycle), you need to know pen-testing. We know how to build secure products, we do it everyday in the enterprise world where people work across corporate networks securely. We do it in the mobile world. All we need to do is apply the same practices to the IoT world,” said DesAutels.

At the same time it’s unfair to put all the blame on the edge devices alone, though they are big culprits. The entire IoT stack is comprised of three components: the edge device, the backend server and the IoT gateway that sits between the two and plays a critical role in security. Any of these three components can be compromised, as happened in the case of Miele dishwashers where the backend server was compromised that gave away access to connected devices.

An architectural approach is what is needed to build security into the various layers and events involved in an IoT deployment. IoT devices should treat the network as simply a transport to connect to their specific server, not something to interact with. IoT devices should, by design, be restricted from talking to anything but their target server and preventing anyone else on the network from talking to the IoT device.

IoT gateway can play a very critical role in mitigating attacks, especially in the enterprise use case. A hardened IoT gateway can act as a shield for enterprise servers, data assets, and business applications

“Users can enforce security policies such as access controls, validate identities, manage certificates, perform encryption/decryption as needed, implement a VPN for connection to backend systems, etc.,” said Thacker. “Using gateways in this way is part of a multi-layer approach to IoT security that is a must when considering how to protect a distributed computing system, which all IoT implementations are.”

Companies operating in the industrial/enterprise IoT space are already doing it. Cisco offers industrial switches and routers that form the basis of IoT networking infrastructure. These gateway devices connect robots, programmable logic controllers and a variety of sensors that have been in the factory before but were either not connected or connected through proprietary networks.

These solutions allow companies to integrate security within the IoT infrastructure. As a result, not only the modern, but also the legacy infrastructure benefits from the rich capabilities of access control, encryption, authorization… all within the routers and switches. The IoT network becomes the censor and enforcer and allows companies to enforce security within the IoT network infrastructure, explained Reno.

“If you don’t worry about all of it, end to end, then you’re not thinking about security seriously,” said Noah Harlan, Founder of Two Bulls.

There is increasing collaboration between IoT companies that will further make these devices more secure. As the IoT market is maturing we will also see standardization around protocols and transports. It’s a well-established fact that open standard, open technologies are more secure as compared to closed, even if widely used technologies.

 

Industry wide efforts to make IoT more secure

A lot of efforts are already underway in that direction. MUD aka manufacturers usage description is an standard which is backed by Cisco. It’s emerging as a universally standard way of describing device capabilities that will form the foundation of being able to authenticate and authorize devices on the network.

There are industry wide efforts like SDP (security defined perimeters), where the companies are working on implementing additional technologies in devices that might have inherent risks in them to limit their exposure surface.

EdgeX Foundry is working on a project that will provide customers with a way to plug in security that can control components and exposure surface of components. Instead of mandating or guaranteeing security, they are enabling customers to use their own plugins and managers.

“OMA has created some standards, such as DM and LWM2M, and there are a few others too. Some vendors try to establish standards, such as Intel with EPID. The Industrial Internet Consortium has been attempting to construct a standard. Perhaps the best bet are the vendors, like Intel or those in the device management domain (Device Authority, Movana…),” said Carroll.

All of these technologies and measures fail if companies don’t want to use them. At the moment the main cause of lack of IoT security is lack of accountability and security best practices. OS or update mechanisms are not silver bullets that magically solve all problems, it’s ultimately up to the vendors, organizations and consumers to secure their own networks and devices.

We are left with  the root cause as potentially financial in nature. The cost of creating securable devices in the first place is high, and when you throw security into the mix it becomes even more costly. We need to create some incentives for IoT vendors so that they can adopt the best practices and technologies that are applicable in their cases.

IoT needs new business models

At the same time we can’t expect a hardware manufacturer to transform into security experts. They excel at making good devices, let them do it. That creates a unique opportunity for new business model. I call it ‘Remotely Managed IoT’.

We have already seen such a model in the cloud space where start-ups focus on writing exciting applications, without having to worry about their cloud infrastructure. It’s companies like Rackspace or Mirantis that offer managed cloud solutions.

The same model can be replicated in the IoT world where vendors can offer services to manage and keep IoT devices safe and secure. There are companies like Redbend (now part of Harman) that offer complete management of IoT devices, including firmware update management. There is a member of the EdgeX Foundry, Cloud of Things that offer a unique connected device management solution for OEM product manufacturers and systems integrators (SIs) enabling them to connect any device to any cloud, making the device IoT-ready within minutes.

Regulations are the last resort

When no technology, best practice, business model or economic incentive can save IoT, it’s time to look up at regulations. IoT has already caused way too much damage to our economy and looking at the scope of IoT devices, we have no idea what kind of havoc it will cause. The only way to make them secure is by forcing through law. Regulations can in fact help the industry as inspires new business models.

“Lack of regulation and standardization, as well as the lack of accountability for manufacturers means there is no financial or legal incentive to do this right.” said Izrael.

“It’s very easy to create incentives in our society. They are called laws. That’s how we create incentives to pay your taxes to not murder your neighbor. For companies to not put sugar water in baby foods and not to make PJs catch on fire and that’s care of safe, air planes. Laws and regulations are how we do this and when economic incentives are not there that’s when laws step in that’s how society works. If you want to fix this, pass laws.,” said Schneier.

But laws cost money. It makes things expensive. “It’s more expensive for you to buy a ticket on a plane that won’t crash. It’s more expensive to buy a car with safety features,” said Schneier. “These are expenses and we force companies to spend the money and pass the cost onto the consumer because as a society we think that’s a good idea. There is no other way.”

Mark Shuttleworth, CEO of Canonical believes that customer awareness will also play a big role in improving situation with IoT. According to him, it will be a mix of three components. The first is customer awareness where people will be more educated about investing in IoT devices and purchase the ones that are healthy and safe, second would be the cost associated with failure when companies don’t take responsibilities, and the third is regulation.

Commenting on ‘carrot and stick’ or economic incentives and regulatory restrictions, Shuttleworth said, “There are a lot of ways to use commercial leverage to fix that kind of problem but in the end you also want some structural regulation. I am sure different countries will take different approaches, but at the end of the day it’s a combination of market and regulation that will change things.

A lot of work is already underway. Harlan has worked with members of Congress on their first steps looking at IoT, in particular Cory Booker who is co-sponsoring the DIGIT Act which is making its way through Congress. Harlan advised them to work to avoid a patchwork of security rules (one set for health, another for automotive, another for consumer, etc…) as that would lead to conflicting rules and stifle innovation. Regulators should stay out of the minutia of defining how to comply and simply state what compliance means. “This frees the industry to innovate and gives them a bar to measure against,” said Harlan.

DesAutels believes that firmware management and necessity to manage firmware is going to be so critical to product sales that it’s just a matter of time that companies will have to get a firmware management certificate from some regulatory body in the US and in the Europe. He is a strong supporter of firmware management as a requirement in IoT devices.

All of it sounds good, but there is one large, genuine problem that’s still remains unanswered. There are many cases where companies that built those IoT devices cease to exist. Regulation won’t help in such cases. There will be millions of insecure devices waiting to be turned into zombies. DesAutels advocates for building kill switches into devices so companies or government agencies can turn those devices off. It may sound like a good idea, but it can be abused. It’s also  injustice to customers who bought those devices.

Instead of building kill switches, companies should be compelled to allow users to use their own firmware on these devices. Thomas Pfeiffer, board of directors of KDE e.V., said that open source communities can create custom firmware for devices and keep them secure and alive. However, in order for the community to write firmware, they need to be provided with drivers or at least be able and allowed to reverse-engineer them.

“IoT vendors can be legally required to publish the specifications needed to write a driver and copyright and patent law could be modified so that reverse-engineering a driver would not be illegal.”  said Pfeiffer. “What we would like to see is a mindset within IoT manufacturers of the open-source community as friends who can keep their hardware useful even after it is not economically viable anymore for the manufacturer to support them, not as a threat they have to lock out,”

Conclusion

My takeaway from this discussion is that ‘IoT’ itself is not a security risk. It’s not LZ 129 Hindenburg that’s going to kill us all. Different use-cases, different industries create different incentives for IoT companies to adopt different approaches to the market. What consumer IoT needs is more incentives for companies to build security features into their devices.

There are new business models like ‘managed IoT’ that can create a very lucrative market. In the end, whether it’s customer awareness, regulations or new business model a smarter approach to security is healthy, necessary even, for the growth of the IoT ecosystem. Regulations also mean news ways to monetize from it. Consumer awareness means people willing to pay extra for services that can secure these devices.

Unfortunately, there is still some ways to go before we have adequate security best practices adopted on a large scale. Governmental agencies or vendor-led organizations might need to implement stricter regulations in order to enable business models that prioritize better security practices. Vendor consortiums could provide one way to resolve the current security crisis, but I suspect that tighter regulations will provide the requisite “teeth” to firmly push the industry in the right direction. In the meantime, I look forward to seeing vendor-supplied solutions that are available to the general public. Let’s hope better awareness prompts some consumers to implement better security.