Home / Blogs

Behind the Curtain: Making IPv6 Work

Wouldn’t it be nice if turning on IPv6 really was ‘press one button and the rest is magic’ easy?

For some things, it is. If you’re talking about client-side, enabling an IPv4-only home service on DSL or fibre really can be this simple, because all the heavy lifting is being done inside your ISP: you’re not enabling IPv6 in the network, you’re turning on the last mile. It was knocking at your door and you just had to let it in.1

And, if you are a user of compiled systems programs, it’s usually easy: you just run code which says it’s already IPv6-aware, or ask your vendor to modify it. After all, “they all work nowadays, don’t they? Don’t they?” Really?2

But spare a thought for two different places with work to do to enable IPv6: your ISP, and software developers. Those assumptions I stated above about ‘easy-as’ don’t apply for the people behind the curtain, making it work—as discussed on Reddit last week.

ISPs have two things to do when they enable IPv6

The simple thing to do is to enable it inside their own network. It’s simple in as much as it’s understood what has to be done. But it’s not necessarily easy, and the distinction is important.

This really is a well-understood problem these days with several well-understood, nation-scale examples to choose from: Equador, Comcast / T-Mobile / AT&T / Time Warner, Sky, SKTelecom can all show this can be done at scale. It’s not exactly one-click, but it’s no longer a large amount of unknowns and a small amount of known issues: it has become a large amount of known processes, and a small amount of corner cases. But some corner cases (like Skype) are not easy to fix (only Microsoft can fix Skype).

The harder task for ISPs is changing the Operations Support Systems and Business Support Systems—or OSS and BSS. These are the software systems which provide services and, importantly, manage billing and charging.

OSS and BSS challenges are easy to identify, but hard to resolve. The OSS and BSS may be the primary value that the ISP has as their own investment in running the network. If not purely in-house developed, this is very unlikely to be an off-the-shelf solution bought unchanged from an external supplier: it’s much more likely to be a very large, complex, and ageing suite of mutually incompatible systems interacting in different ways.

I’ve seen this in at least two ISPs, both former national monopoly telcos, and in one case, they continue to use IBM 3720 Terminal emulation to drive mainframe applications. In the other, they had at least two different web-based systems to reconcile two distinct software suites behind the scene, each providing part of the solution to commission a new customer.

It’s not uncommon to have three, four or more different provisioning systems live at the same time (each acquired sub-company may still run inside the main brand) and to have to re-enter data into each, for instance.

For example: consider if your billing mechanism is designed to count IPv4 packets using SNMP interface statistics, and you now have two reporting lines in your SNMP, one for IPv4, and one for IPv6? It’s possible that you have to do significant work to capture the IPv6 count. Or, perhaps you have a DHCP IPv4 pool, and therefore do no assignment to specific customer endpoints. But your architects decide to allocate a static IPv6 /56 (this is a common design goal), which now has to be handed out, per customer, from the point of account creation all the way down to the customer’s service going live. How does that work? Do you retrofit a new function into your OSS?

And if the client wants services to run end-to-end to the IPv6 addresses, it’s not impossible that, as the ISP, you will also now need to manage reverse DNS delegations. Where previously you had a simpler model for IPv4 which left reverse DNS as a defined, static list of 255 names per /24. In IPv6 you really don’t want to try and pre-populate the entire reverse DNS map! It’s much more likely that you are going to have to engineer a reverse DNS delegation mechanism, or else do some kind of on-the-fly DNS creation.

Let’s now consider software developers

Imagine you are a software developer and you have a large, mature product which is targeting the online world, and you have written it to use the MySQL database back-end, believing that IP addresses are short strings of at most 15 characters (4 sets of 0 through 255, and three separating full stops) and IP prefixes are at most 18 (an additional “/” marker and a length of up to 2 digits size, encoding 0 to 32).

But now, you have to cope with syntactically detecting IPv6 addresses (new format, new characters “:”) and they’re now 39 bytes long, or 43 if you have /128 addresses. Worse yet, while there are primitives built into MySQL to handle IPv4, they don’t handle IPv6. So a bunch of SQL logic you wrote a good few years ago now has to be re-visited.

This isn’t a hard problem but it’s not exactly easy or simple. It’s easy to say “find all the places you assumed an IPv4 address and change them to be either an IPv4 or an IPv6 address” but it turns out that in many applications, this means quite a bit more work. For example, inputs supplied by users are often type-checked: simple syntactic review of the contents, to determine if it’s legally of the right type. If you only fix the database, you may leave the code unchanged. This would reject an IPv6 address being entered, because the type-checking on values for an IPv4 address only has to recognize full stops and the digits 0 to 9, where the IPv6 code has to recognize additionally the letters ‘a to f’ in upper and lower case, and the colon ‘:’ character. So, even the user interface-code has to change if you do any kind of input type-checking.

Or, maybe your code works directly into the TCP/IP and Sockets() interface. If you are operating down at this level, you will already know about RFC3493 but for those who don’t usually code at this level, you will be taking code written to use older constructs, and calls. Your code very probably doesn’t call the getaddrinfo() or inet_pton() and inet_ntop() functions, but uses older calls and structures which fatally assume IPv4 only.

There is a lot of documentation online to re-write this code for C. There are lots of examples. But, you may dread re-opening the door to this legacy code, because one of the definitions of “legacy” often quoted by software developers, is “the code you are too scared to work on, in case it breaks and you cannot fix it”. And it’s certainly likely the systems which you depend on, and have done these years gone by, pre-date a need to operate with IPv6.

All this is well known. There is nothing new here, we’ve been telling people for a long time about the problem, and how to deal with it. But now, it’s real.

So spare a thought for those working behind the curtain—those who now need to work their magic so everyday users can simply ‘press one button’.

1 Actually, my own recent experience moving from an ADSL2+ dualstack service to the Australian NBN fibre dualstack service was a little fraught: it turns out that sometimes you need open source software, and I had to replace my factory image with a free OpenWRT instance to get it to work.

2 Again, this is more mostly-true. There are going to be corner case applications which don’t work well on IPv6 yet (Skype for instance) but in the main, most modern applications on either Windows, OSX or iOS/Android are going to be ok. The ones which aren’t ok, are very quickly coming to the fore, because we now have significant IPv6-preferencing networks in the USA which drives a lot of market decisions in applications development.

By George Michaelson, Senior Research and Development Scientist at APNIC

Filed Under

Comments

Behind the other curtain... Srinath Beldons  –  Feb 4, 2016 4:48 AM

There are other issues which relate to contracts that Telecom and Service providers have to deal with.

They outsource most of the work and in some cases it is about revenue sharing. All this is governed by the contract terms. For example a problem a service provider I know has sub-contracted the entire network infrastructure setup and operations in a revenue model. Anything such as IPv6 if was not in the contract terms is affecting them as it involves costs and no one wants to pay for that. So if the contract term is 5 years, nothing changes for 5 years.

Some customers don’t want to go for IPv6 on simple grounds, we will use NAT. A few mobile providers I know are doing this.

The provider I use in India right now has IPv6 addresses, but is more interested in offering only to enterprises and not retail customers. BTW they are growing their infrastructure and adding customers on a day to day basis. Reason for not offering IPv6 has no answers. Simple there is no strategy in place.

A very good observation Srinath. As I George Michaelson  –  Feb 4, 2016 5:40 AM

A very good observation Srinath. As I understand it, the contract is an outsource, bundling the acquisition of specific hard- and software configurations. Both could have included IPv6, but it was expressed as an option alongside other feature sets. The configuration option set for the equipment was set at the beginning, and varying the set of choices from the capabilities invokes the contract renegotiate clauses. So we have a vendor of hardware which is IPv6 compliant, and systems in the software which are IPv6 compliant, but because the contract optioned them out, they won't be enabled in the life of the contract. Its plausible that the NMS/OMS costs were also part of that problem.

Organization challenges Srinath Beldons  –  Feb 4, 2016 7:00 AM

A few more non technical but important point that we need to note are

1. Senior management doesn’t see this as a problem so far.

2. Silos within the organization NMS,Customer care,Billing teams lead teams which themselves don’t recognize IPv4 address issues.

3. CGN NAT 444 solutions need to be compared from a cost perspectives in relation to migration to IPV6. It is high time since Finance people within the organization understand more of this issue. Good planning, with incremental purchases of IPv6 enabled platforms and migration is the only way out.

A big mobile provider went in for the purchase of CGN solutions that put most of its India subscribers behind CGN that costed almost 8-10 M US$. Most of the other providers also followed suite. The law enforcement requirements for storing the logs of sessions for 1 whole year.

The main point I would like to which decision of doing migration in an incremental manner vs a big bang purchase helps out is very effective and needs to take into consideration by organizations.

The problems holding up deployment Ross Chandler  –  Feb 5, 2016 8:44 PM

Here's a few problems and solutions I've encountered over the last decade plus. Core routers not forwarding IPv6 reliably and not doing 6PE over link aggregation groups. Solved by waiting for IPv4 traffic growth driving the purchase of newer core routers. No support for business case to update provisioning system to assign static delegated prefixes to broadband users. Solved by assigning prefixes from pools on BNG and minor update to RADIUS profiles. BNGs not relaying already relayed DHCPv6 packets. Wait for software upgrade driven by rules keeping release within few releases of current release. Broadband useage reports. Solved by updating system to fix a separate IPv4 problem in a way that also solved it for IPv6. Modems, including IPv6 in RFPs for many years. Vendors are finally beginning to release software that works well. Mobile billing mediation for IPv6 records. Solved by truncating prefix down to 32-bits as the systems ignore the addresses. Getting funding for some IPv6 work by a couple of siloes. Solved by basing business case on avoiding future costs that another silo was spending on buying IP4.

nobody told them... Dean Durant  –  Feb 5, 2016 3:24 PM

When the decision was made to roll out an incompatible version of IP, do you think they thought about issues like this?  Or is it the fault and or responsibility of ISPs to deal with it?  Otherwise why hasn’t IPv6 taken off like it was supposed to?
There is a Top 10 ISP in the United States, and I guess one would say they were not informed that these business issues would have to be planned for and budgeted for.  Their pricing is highly competitive with the bigger providers.  But apparently they have financial issues, and sought a buy-out.  They are nowhere on IPv6, and have no plan, no expertise, and no money for it.  This is the first I heard of any of these issues.  And I’ll bet the people who wrote the RFCs weren’t thinking about these issues either.

Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

VINTON CERF
Co-designer of the TCP/IP Protocols & the Architecture of the Internet

Related

Topics

Threat Intelligence

Sponsored byWhoisXML API

Brand Protection

Sponsored byCSC

IPv4 Markets

Sponsored byIPv4.Global

DNS

Sponsored byDNIB.com

New TLDs

Sponsored byRadix

Cybersecurity

Sponsored byVerisign

Domain Names

Sponsored byVerisign