In 2013 I wrote a blog Telecoms as a spying tool, in which I mentioned that those who use the internet to spy indiscriminately will have to face the reality that such activities will only start a cat-and-mouse game—the technology will always be able to stay one step ahead of those who are using the internet for criminal purposes.

Since that time some very significant developments have taken place that have confirmed our prediction.

When I wrote that article encryption technologies had already been around for some time, but they were not being used by the majority of internet users. Some websites, particularly those from financial institutions and at least some of the government sites, used encryption technologies, but even at a business level security still had rather low priority. Data breaches that are still hitting the news, show that this continues to be a problem for many of the organisations and people using the internet.

Beyond that, encryption was being used by those who wanted to avoid detection, such as criminals and terrorists. However, in reality people who carry out illegal activities often make simple mistakes, and so in many cases those offenders were eventually caught, with or without the assistance of technology. This is still the case—mass surveillance usually contributes little or nothing to the capture of criminals or terrorists. More targeted approaches that have been used over the years, including the use of technology, have delivered far better results.

Obviously most people were fully aware that the police and security agencies were using technology to try and catch those criminals and there was little or no outcry about it, as people obviously understood the benefits of having vigilant security services. They trusted that those agencies would use the appropriate warrants to conduct their investigations, or that secret actions were at least overseen by their national parliaments.

However, the revelations of Edward Snowden showed that these tools were not only used to catch criminals. They were also used for mass surveillance purposes, and this revelation shocked the entire world. It became clear that very personal information from friendly government leaders was spied upon, and also that computers from companies such as Google were hacked into in order to obtain information on people without any warrant attached.

This immediately brought about a chain reaction (as in the cat-and-mouse example). Internet companies do have a legal obligation to protect the integrity of their ICT infrastructure, the security of their data and the privacy of their customers; and so this was a clear wake-up call for them to massively step up their security. If NSA can tap into these networks and services, others with criminal intentions will also find ways to do so. So encryption technologies were used, this time to introduce a far more general level of protection both at national government levels and in relation to consumer-based services from Google, Facebook and others. Furthermore massive amounts of money were pumped into this market to create even better encryption solutions.

With this explosion in encryption, consumer devices, having data in them, are now also being encrypted. The reality is that encryption is rapidly becoming the norm.

In this cat-and-mouse game law enforcement agencies are complaining that the techniques they used before are not working as well as they did in the past—in fact, for them it is only getting worse. We are already hearing political murmurings, in the UK and Australia for example, that encryption technologies should be banned. This is a clear indication that those politicians have very little idea about technological developments—that they are largely unaware that encryption has already become a very critical part of internet services. Banning encryption is not only counter-productive in an economic sense; it is also too late, since large amounts of information have already been encrypted over the last two years.

The encryption explosion has other impacts too. The cellular networks that used to use deep packet inspection (DPI) to do traffic management are now unable to manage traffic in the same way. Microwave links that use compression to increase capacity are no longer able to carry the same amount of traffic. Content filters and firewalls that used to be used to control access to information are unable to perform these functions anymore without explicit intervention by the client to load compromised security certificates, etc.

However these technical issues will no doubt be resolved over time.

Nevertheless the whole spying affair has totally backfired on those spy agencies that used these tools in ‘illegal’ ways, and as a consequence the whole web is going dark. It is predicted that by the end of the year the majority of US internet will be encrypted, and even email traffic is now being encrypted on a broad basis.

So within a few short years the internet world has changed. So what needs to be done in order to assist police and security agencies in doing their job? Law enforcement agencies still of course have the ability to investigate websites and to get content with warrants. But the reality is that this is becoming a much harder task and will take many more resources than it used to. Governments are trying to insist on access to key materials to allow them to decrypt data in flight as they did before. But this is a very hard thing to enforce without creating holes in the internet and the networking economy that can be exploited by others as well. The internet has become far too important in an economic sense to allow for undermining the communication security of everyone simply in order to enhance the ability to intercept traffic. And it is also Snowden that reminded us that the government can’t keep secrets that well either.

There is no good solution here, only tradeoffs. The risk to security by trying to embed holes for law enforcement is unwise and unworkable. This is an international problem and needs international cooperation—and how will it be possible to get all of the countries to play along? If the certification authorities are deliberately compromised in the cat-and-mouse game new ones will simply spring up that will create an alternate system, maybe run by companies in countries that are less democratic and have even less oversight.

On the other side, technology companies have developed tools to detect network compromises, and it will be more difficult (but of course not impossible) for criminals and terrorists to exploit these holes than it used to be.

Totally predictable within a complex, dynamic technological environment, the natural result of government overreach in this spying affair, is the encryption explosion—especially as it has become clear that adequate supervision is not being applied. The draconian political reaction that we see in the ‘Five Eyes’ countries clearly shows that some of these lessons have not been internalised by policymakers. In other less democratic countries this might also be the case but we obviously hear very little about that from them.

Technology usually doesn’t operate with the degrees of freedom that policymakers want, and engineers are almost always faster than policymakers in adapting to new rules and laws. It’s a losing battle to try and contain that technology.

What this means is that there is a need for governments to be more open and transparent, and to work with the industry in a legitimate way to address some of the problems we all face (terrorism, cybercrime, child pornography, and so on). Governments who are simply playing the cat-and-mouse game are not likely to achieve outcomes that benefit both their people and the institutions that are needed to keep us safe.