Home / Blogs

The Future of Interdomain Interconnection and Traffic Control

Today, we are excited to announce, in partnership with the Open Networking Foundation and Open Source SDN, the official launch of iSDX—open-source controller software for an industrial-scale Software-Defined Internet exchange point. iSDX allows independently operated networks to interconnect and exchange traffic in completely new ways. This software, which we’ve been developing for nearly three years, now finally operates at the scale of the world’s largest IXPs and interoperates with SDN-capable hardware switches, opening up new possibilities for interdomain business relationships and traffic exchange.

For the past three years, a team of researchers at Princeton, led by Arpit Gupta, has been developing mechanisms that could fundamentally change how independently operated networks interconnect and exchange traffic. It is well-known that independently operated networks on the Internet—often called Autonomous Systems (ASes)—are interconnecting and exchanging traffic in vastly different ways than they did even ten years ago. The first formal observation of this evolution was Labovitz’s Internet traffic report in mid-2010, which observed that the structure of interconnection relationships which once formed a relatively strict hierarchy has become decidedly less structured, as large “hypergiant” content providers and content distribution networks gain increasing leverage in peering relationships. The figures below, from the BITAG report on Internet interconnection from 2015, show how this hierarchy has evolved.

The hierarchical structure of interdomain relationships in the 1990s. (Source: BITAG / Click to Enlarge)

The flatter interconnection structure of today’s Internet. (Source: BITAG / Click to Enlarge)

Unfortunately, while Internet interconnection has undergone rapid and dramatic restructuring, the underlying protocols that support interconnection have not fundamentally changed since the 1990s—the last substantial update to the BGP standard, RFC 4271, was effectively to standardize the long-ossified route selection process. And, while there have been some changes to the protocol (e.g., the introduction of route reflectors, 32-bit AS numbers, etc.), none of these things have fundamentally changed the way that ASes might be able to exchange traffic or establish business relationships.

Imagine if instead networks that interconnected could make decisions on how traffic is exchanged based on the application type; we could have more complicated business relationships, such as “application-specific peering”. Or, suppose that networks could change their selection of routes to neighboring networks dynamically, based on (say) whether a certain link was experiencing high traffic volumes, a denial-of-service attack, or a traffic ratio imbalance. Or, suppose that the network switches could dynamically enforce more complex interconnection agreements that could meter and shape specific application flows (e.g., streaming video), without affecting how other traffic flows are prioritized or routed.

We believe Software-Defined Networking (SDN) may hold the keys to some of these new capabilities, and over the past three years, we have been working to build a platform, an industrial-grade software-defined Internet exchange point (iSDX), that can both support these pilot use cases and explore new ones. This project augments a standard Internet exchange point (IXP) architecture, whereby participant networks exchange routing information through a shared route server, to avoid pairwise BGP sessions. We have augmented the “stock” route server to incorporate an SDN control platform based on Ryu that allows participants to (1) use BGP routing as the mechanism to exchange topology; (2) incorporate specific SDN policies that allow participants to override the default forwarding behavior that BGP would otherwise provide.

An SDX controller allows each participating network to both (1) exchange BGP routing information, as in conventional route servers; and (2) express specific traffic exchange directives using SDN programs that the controller composes into a single coherent set of forwarding rules. The controller installs the rules into the switch.

Although we initially presented this idea in Summer 2014, it has taken several years to develop the SDX codebase so that it can operate in industrial-scale settings. One of the major challenges was scale: accepting full BGP routing tables from each participating AS will result in hundreds of thousands of routes that must be translated into forwarding rules in a hardware switch. Unfortunately, most commodity SDN switches cannot support anywhere close to a forwarding table of that size. Over the past year, we have developed ways to compress these tables by several orders of magnitude, which now brings all of the ideas that we envisioned in the initial SDX design much closer to reality.

The paper that Arpit Gupta is presenting today at the USENIX Symposium for Networked Systems Design and Implementation (NSDI) which is a joint work with Laurent Vanbever at ETH Zürich and Marco Canini at UC Louvain, explains all of these optimizations in detail. Today, we are excited to announce the launch of the iSDX open-source project, in collaboration with the Open Networking Foundation‘s Open Source SDN project. The long-running open-source project has been undergoing steady development over the past few years. Over the past year, we have worked closely with partner enterprise networks to harden the code and test it on hardware switches, including the low-cost Quanta LY2. Now, we are excited to welcome the broader community to our open-source effort, through a partnership with the Open Networking Foundation. The Open Source SDN community site has launched, as has a new mailing list for the iSDX open-source community.

Although today’s announcement represents the culmination of several years of work to-date, we believe that this partnership with the Open Networking Foundation will enable an exciting set of new opportunities. We look forward to supporting iSDX and exploring a much broader range of applications, modes, and business relationships for Internet interconnection, ranging from better traffic management and control, better support for interdomain security on the Internet, a much wider range of possible interconnection agreements and business relationships, and many other applications and use cases that we haven’t even yet imagined.

By Nick Feamster, Professor at Princeton University

Filed Under


Prof. Feamster,Several of the pilot use cases Frank Bulk  –  Mar 16, 2016 11:54 PM

Prof. Feamster,

Several of the pilot use cases you listed would seem to violate the FCC’s Net Neutrality rules, as they are application specific.  Is that your understanding, too?

Differentiated Treatment Nick Feamster  –  Mar 17, 2016 12:20 AM

Frank, Good question. I think the short answer is that "it depends". Here's a longer writeup explaining various use cases where differentiated treatment of traffic constitutes "reasonable network management", which the OIO allows for. http://www.bitag.org/report-differentiated-treatment-of-internet-traffic.php That's the quickest answer I could give to a question that concerns a 400+ page document! -Nick

Thanks for replying. I scanned through Frank Bulk  –  Mar 17, 2016 1:15 AM

Thanks for replying. I scanned through that report and I felt a more cautious recommendation. While the "reasonable network management" is allowed, choosing which applications get preferenced (or de-preferenced) is not. No Throttling: broadband providers may not impair or degrade lawful Internet traffic on the basis of content, applications, services, or non-harmful devices. https://www.fcc.gov/general/open-internet

[oops] replied below Nick Feamster  –  Mar 17, 2016 1:51 AM

[oops] replied below

Differential != Degraded Nick Feamster  –  Mar 17, 2016 1:50 AM

Great question. Reasonable network management can sometimes entail differentiated treatment of traffic. Quoting the BITAG report, “When differentiated treatment is applied with an awareness of the requirements for different types of traffic, it becomes possible to create a benefit without an offsetting loss. For example, some
differentiation techniques improve the performance or quality of experience (QoE) for particular applications or classes of applications without negatively impacting the QoE for other applications or classes of applications.” There are many examples of this, largely because different applications have different latency, throughput, and loss requirements.  Happy to discuss more offline.

Working with IXPs? John Souter  –  Mar 17, 2016 8:15 AM

Which Internet exchanges have you engaged with in doing this research?

Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Co-designer of the TCP/IP Protocols & the Architecture of the Internet



IPv4 Markets

Sponsored byIPv4.Global

New TLDs

Sponsored byRadix

Domain Names

Sponsored byVerisign


Sponsored byDNIB.com

Threat Intelligence

Sponsored byWhoisXML API

Brand Protection

Sponsored byCSC


Sponsored byVerisign