Home / Blogs

To our readers: Does your company offer DNS or DNS Security services? CircleID has an opening for an exclusive sponsor for our DNS topic. Gain unparalleled results with our deep market integration. Get in touch: [email protected]

A Clear Case for ISP Regulation: IP Address Logging

Over on the Network Neutrality Squad yesterday, I noted, without comment, the following quote from the new Time Warner Cable privacy policy bill insert:

“Operator’s system, in delivering and routing the ISP Services, and the systems of Operator’s Affiliated ISPs, may automatically log information concerning Internet addresses you contact, and the duration of your visits to such addresses.”

Today I will comment, and explain why such logging by ISPs creates a clear case for regulatory intervention, on both privacy and competition grounds.

ISPs—the providers of “last mile” Internet access—are in a unique position vis-a-vis any other provider of Internet-based services. While any individual Internet service—e.g., a Web site—can log a variety of information about their individual users, ISPs have the ability to log access information relating to virtually all internal and external services that their subscribers visit.

There are some technical limitations. Without using Deep Packet Inspection (DPI), an ISP would normally be unable to differentiate which external virtual server a user was accessing on a single shared IP address, and technologies such as proxies and VPNs also can obscure addressing info.

But from an ISP standpoint, IP address usage information alone could be a veritable treasure trove, particularly from a competitive standpoint.

In the case of Time Warner, their statement regarding IP address logging is buried in a very long privacy policy comprised of very tiny print. It is confusing in some ways. It appears to conflate IP address logging with gathering of personally-identifiable information, and doesn’t seem to explicitly address how long logged IP address data, per se, will be retained. However, it does state that personally-identifiable data will be retained for “as long as it is necessary for business purposes” (“as long as you are a subscriber and up to 15 additional years”).

The privacy concerns related to one entity having a log of virtually every site that you visit on the Internet, and how long you visit those sites, are fairly obvious. As I noted, this capability goes far, far beyond the IP address logging possible by any given non-ISP Internet service.

But perhaps much less obvious is the manner in which such ISP IP address logging capabilities could be abused in anticompetitive manners of direct concern to us all.

If ISPs were just providers of “dumb Internet pipes”—as most were until fairly recently—related anticompetitive concerns would be largely moot. But for many ISPs these days, especially all of the vastly dominant U.S. ISPs, the big money isn’t in providing Internet access, it’s in providing content—especially video content.

The inexorable move of video to the Internet is now driving many of the most contentious Internet-related issues, including battles over pricing and bandwidth caps. In such an environment, knowing as much as possible about how your users partake of the competition is invaluable.

Logged IP address data could provide ISPs with a window directly into how their Internet video competitors and other competitors operate, in a manner only possible by virtue of being ISPs with direct access to the virtually complete data flow of subscribers to and from all sites.

ISPs have access to information in a comprehensive manner unlike any of their competitors: How often are subscribers visiting Google? How much time are they spending on YouTube, and during what parts of the day? Are subscribers sometimes using Hulu more, as opposed to YouTube? How about visits to government sites? Or pay movie sites? Porn sites? What sorts of usage patterns can be derived from all of this accessible usage data? How can we use this information to our competitive advantage as a content-providing ISP who wants to encourage the uptake of our content vs. that of outside services?

In the case of Time Warner, their privacy policy notes that logged IP address data will not be disclosed or used for “marketing, advertising, or similar purposes.” It says nothing about competitive product development and deployment.

To be clear, I’m not accusing Time Warner—or any other ISP—of abusing IP address data in these ways. Frankly, given the current lack of a mandated regulatory disclosure framework, there’s no formal, systematic mechanism to keep the public informed about the presence or absence such activities, now or in the future.

Nor does the capability to collect and log IP address data (functions present in much pro-grade networking hardware for engineering purposes) necessarily indicate that this is actually being done in manners that would negatively impact on privacy and competitive concerns (but the associated lack of clarity on these issues and in regards to data retention policies are discouraging in any case).

Still, it’s readily apparent that ISPs’ unique abilities to comprehensively log IP addresses associated with virtually the entire scope of their subscribers’ external Internet activities, easily triggers significant concerns relating to potential anticompetitive behaviors and potential privacy abuses.

I would assert that regulations prohibiting the use of IP address logging by ISPs in such manners, and mandating routine public disclosures to help ensure that such abuses are not taking place, are immediately called for at the national level.

Filed Under

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Co-designer of the TCP/IP Protocols & the Architecture of the Internet


Can you say, "Paranoid conspiracy theory?" I knew you could. Brett Glass  –  Jun 2, 2009 11:34 PM

Conspiracy theorist Lauren Weinstein apparently forgets that Federal law—CALEA, specifically—requires ISPs to have the ability to gather these statistics about users.

Mr. Weinstein also conveniently fails to note that, ever since the advent of HTTP 1.1, the IP address which a user accesses does not even identify the Web site he or she has visited, since literally thousands of sites can share the same IP.

What’s more, he forgets that telephone companies have long logged every telephone number you call—even on flat rate or unlimited plans. (Take a look at your cell phone bill, for example. Even when you haven’t gone over your allocated number of minutes, every call is recorded.)

Our own ISP gathers access statistics—not as an anticompetitive measure or to spy on our users, but rather so that we can gauge demand for access to certain parts of the net, measure response times, and improve our service.

Lauren’s argument is typical of those of the “network neutrality” loony fringe, which has been persistently lobbying for intentisive regulation of the Net. Their excuse for doing so: that ISPs might one day do something bad. Even though they never have and are unlikely ever to do so—because the market would be sufficient to punish them if they did.

There's calea. And then there's 47 USC 230 which allows ISPs to filter on behalf of their users Suresh Ramasubramanian  –  Jun 3, 2009 2:07 AM

Well, good. I see that a lot of the public policy people have started to advocate what the ISP security crowd has known about for quite a while.


“It’s unreasonable for us to assume no action will occur (on Net neutrality),” said Tom Tauke, executive vice president of public affairs, policy and communications for Verizon. “What we do have to do is make our case.”

Instead of using free market ideology, Tauke told Politico providers will now highlight the consumer choice argument, stressing that many customers want providers to be able to filter out objectionable material for them.

.... Jmil1  –  Jun 3, 2009 8:40 PM

I understand that CALEA requires providers to have the ability, but does this mean IP address logging should be happening all of the time? I’ve been under the impression that a warrant is required, which made me believe there were some limitations as to when it could be done.

@Jmil1: Actually, to ask for the addresses Brett Glass  –  Jun 3, 2009 9:13 PM

@Jmil1: Actually, to ask for the addresses contacted, but not the content, is more akin to a “trap and trace” order, which may not require a warrant.

seems to me there are others who can collect similarly valuable information Bruce Van Nice  –  Jun 3, 2009 11:58 PM

What about Google, and every other search provider for that matter?  Although the kind of information they collect is different, it is, as Google has so amply demonstrated, extraordinarily valuable. 

They maintain their facade of “do no evil” but who really knows what goes on? They have an incredible PR machine and take a look where Eric Schmidt sits when he is in Washington DC.  Isn’t anyone a little nervous about this?  What is Google doing to be sure that the discussion is held on terms that are favorable to them, let alone guiding actual policy? 

If there is going to be talk about how information that is gathered in a network should be handled then everyone, inclduing Google and their peers, minor web sites (which the author has just dismissed as inconsequential) and others need to be a part of it and regulated in the same way.  And we need to be sure that NONE of the discussions are held behind closed doors and that all of the correspondence between Schmidt and his team (and others) is completely transparent. 

That said someone help me out and explain what is wrong if service providers use information they gather in their network to make money?  Google makes money from information they gather at their web site why shouldn’t a service provider be permitted to do the same thing? Providers that step over the line do so at their peril - the Internet has proven to be a very effective policeman!  Maybe we don’t even need regulation at all - when was the last time the government got anything right, especially anything that has to do with technology?

Google's Eric Schmidt even has his own Brett Glass  –  Jun 5, 2009 2:34 AM

Google's Eric Schmidt even has his own astroturf lobbying group in DC: the "New America Foundation." He funnels millions of Google dollars to it every year and is chairman of the group. Its employees, which include Sascha Meinrath and Michael Calabreze, promote views which -- surprise, surprise! -- just happen to be 100% consistent with Google's corporate agenda. Lauren Weinstein, on the other hand, seems to be more of a freelance lobbyist for Google. He, too, claims to be a consumer advocate but mysteriously fails to notice anti-consumer, privacy-invating, or monopolistic behavior by Google. On the other hand, he does publish paranoid fantasies characterizing ISPs as Darth Vader and himself as Luke Skywalker -- see http://www.youtube.com/watch?v=V6aMAm7e1i4

Bruce, Google actually gathers MORE useful and Brett Glass  –  Jun 4, 2009 4:44 AM

Bruce, Google actually gathers MORE useful and potentially damaging information than an ISP. It scans every GMail user’s mail and compiles a dossier that is used to target advertising. It can then enhance this dossier with information gathered via DoubleClick tracking cookies and “Google Analytics” scripts.

Maybe Google is funding “network neutrality” lobbyists like Lauren Weinstein because they are concerned that ISPs might compete with them in doing evil. (Not that they need to worry much. Unlike Google, most of the ISPs I know are ethical.)

First, I am not a lobbyist. Lauren Weinstein  –  Jun 4, 2009 5:31 AM

First, I am not a lobbyist. Unlike some people, I do not call or run around meeting with legislators and pushing my point of view at FCC meetings and the like. I merely write commentaries and papers expressing my personal point of view on these issues and make them publicly available. Secondly, I am not and have never been on Google's payroll relating to network neutrality issues nor anything else. In fact, I've never been paid by anyone for anything relating to my network neutrality or related comments in any manner, other than occasional invited magazine articles, op-eds, and the like. And no paid writing gigs at all recently. I say what I say on these issues because that's what I personally believe. Period. --Lauren-- Lauren Weinstein

So why does the discussion seem to be weighted so heavily to regulating the service providers? Bruce Van Nice  –  Jun 4, 2009 4:40 PM

I agree with you Brett - there is no question that the information Google has is incredibly valuable - they are the richest business on the planet! I find it fascinating that Google has so masterfully guided the discussion thus far - thus my concerns about what is really going on. It is kind of ironic that there is a parallel groundswell around "big (bad) media" these days. But then there is Google quietly (masterfully) working the new media system. To some extent you have to give them credit. But bottom line none of us are going to be better off if the government gets in the middle of this.

Capability required Dan Campbell  –  Jun 4, 2009 2:10 PM

From what I’ve seen in ISPs, CALEA only requires that you have the ability to provide a traffic feed and sampling if requested to do so by law enforcement.  It doesn’t need to be running all the time and you don’t need to provide the historical reports.

Incidentally, there’s a big article in the Washington Post today on Eric Schmidt and his role in the new administration.

Keep in mind that Google is a different animal.  But ISPs from what I’ve seen typically don’t have the time, people or storage to do much with tracking and logging where everyone is going.  I’ve had to jump in a few times during the whole Comcast/BitTorrent/FCC saga when people where claiming that the Comcast “software” or “server” was potentially doing this.  It wasn’t.  It was a third party product that monitored traffic in real time for certain application signatures but didn’t get into the game of storing the data and evaluating it any further.

Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.



Brand Protection

Sponsored byCSC

Threat Intelligence

Sponsored byWhoisXML API


Sponsored byVerisign

IPv4 Markets

Sponsored byIPv4.Global

Domain Names

Sponsored byVerisign