IANA: A Tale of Two Fails

The IANA—Internet Assigned Numbers Authority—is, functionally, the boiler room of the Internet. Every protocol in use to shovel data from Tallahassee to Timbuktu? Listed there. IP addresses? They are the root from which all addresses flow. Domain names? They are the Source. The entire operation is chock-full of magic numbers, numbers that form and fuel the digital world we use daily.

But there are other, lesser-known numbers. AS (autonomous system) numbers, for example. More people have never heard of an AS number than have heard of Jesus. They’re not that interesting, unless you, you know, care about where your traffic goes on the Internet. Then, they’re kind of important. There are other important numbers, too, such as PENs (“Private Enterprise Numbers”).

It is of PENs that I write today.

PART I: A Failure To Communicate

A PEN is a IANA-issued globally-unique number identifying a subtree of OIDs. An IANA OID is a string of digits—its prefix is 1.3.6.1.4.1—used to identify objects in LDAP, SNMP, various databases, and elsewhere. A PEN is the address of a little forest on the Internet, a little forest that you control the whole of. Anybody can get their own little forest, too. All they have to do is request one from the IANA.

They can modify it, too. This is an important function, as contact information changes over time. The IANA now has a Web form to assist in this task; however, this form has some very disturbing features.

Earlier today, I decided to update the contact information for three PENs for which I was listed as a contact. One is mine; one I hold for an open-source project; a third was for a former employer. So I use the Web page—evidently based on a Perl script—to begin changing my contact information. Yet, it worketh not; I entered the number, and this popped out:

“java.sql.SQLException: Access denied for user ‘root’@‘localhost’ (using password: YES)”

Now, I have some issues with the implication of this error message:

(a) The database backing the Web server is sitting on the same host as the Web server. This can be done, but it is not best practice.
(b) The Perl script is accessing the database—a MySQL database, from the look of it—as the “root” user, which is very seriously not best practice.
(c) There is an embedded password in this script somewhere, which is usually not a good idea if you can avoid it. (Which you can’t, sometimes.)

But examining the URL I was redirected to, I found this:

http://pen.iana.org/pen/ModifyPen,$Form.sdirect

Hmm. What appears to be a Perl variable, in a URL, a value presumably fed into something else. Now, I do not know that this is a problem, but I am not particularly sanguine about its innocence. I have visions of nastiness erupting from this script like the Alien as funny things are fed into that variable. I have other visions of other variables being
fished out of the innards of the script, where they were meant to remain in quiet anonymity. I had thought that we had had sufficient experience of unintended consequences on the Internet to identify when we were doing things that might cause us problems.

Something here, it seems, is not right.

PART II: Just When You Thought It Was Safe…

So I try it again a little later. Their Web form, having hiccupped, has now unhiccupped. I enter the first PEN, change the contact info, get the confirmation mail for the change. Because it’s important to confirm changes, right? That they are made by the correct party, and not, e.g., some random hacker on the Intarwebz somewhere? I am certain I have read this somewhere.

And lo! I did get an email with the following link in it [edited]:

http://pen.iana.org/pen/app?page=ConfirmModificationDetail&service=external&sp=S1000&sp=Srequester

I click. I confirm. Life is good.

So I proceed to edit the second one, because hey, that’s why I’m here. Click, change, confirm. And I get another mail, one with the following link in it:

http://pen.iana.org/pen/app?page=ConfirmModificationDetail&service=external&sp=S1001&sp=Srequester

At this point, something begins to bother me very, very deeply. You will notice that the only difference in the links is that the number incremented by *one*.

So, in the spirit of scientific inquiry, and with a sinking feeling of dread in my stomach, I move to change the contact info for the third PEN. Now, this one’s a little tricky, because I don’t actually work there anymore and had forgotten that I still held the conch shell for it, and they’ve captured my email address, so I won’t get that email. Right? And that’s important, because without that email, I wouldn’t be able to confirm the change, and so the attempt to change it would fail. Right? It is not, then, a PEN that I should be able to influence; the very best I can do is send it off and let someone else confirm it. So I change the contact info. And I click. And since I know I’m not going to get this email, I wait a moment, then try the following link:

http://pen.iana.org/pen/app?page=ConfirmModificationDetail&service=external&sp=S1002&sp=Srequester

Doesn’t work. Populated with strange values (looks like hardcoded script defaults). Maybe it’s okay, then, after all. But then, just for fun, I try the next one in the sequence:

http://pen.iana.org/pen/app?page=ConfirmModificationDetail&service=external&sp=S1003&sp=Srequester

*DING*DING*DING*DING*DING* J A C K P O T *DING*DING*DING*DING*DING*

Yup. See the confirmation. Click the confirmation. Wince. Repeat.

I didn’t try a replay of those links, or past links (i.e. by walking the number backwards). I suspect I’d be disappointed.

EPILOGUE: Disappointment

The implications of this should be obvious. I was moving contact info back to its rightful owner, i.e. the organization I used to work for. Nothing, however, prevented me from changing it to whatsoever I wished. There appear to be attack possibilities in the management script; perhaps they are not exploitable, but that the question even arises is disquieting.

Now granted, this is a list that serves chiefly as a convenient collection of addresses for spammers to harvest. However, I simply cannot help but think that some of the traffic those addresses collect could, in some cases, be important. Granted, OIDs are not necessarily immediately important; nobody’s software will instantly stop working if there’s bum contact info for one of them, so nearly all failure modes are recoverable. And granted, there appears to be some manual validation of this list, which would presumably catch something like a DDOS or attempted mass capture of this list. (I hope. The messages I got made it sound like somebody was actually going to look at it before changing it. But who knows?) But what happens when you’re not paying attention? Human error is a constant of the universe; eventually, somebody’s gonna slip up and authorize one or more of these.

Most of my shock comes from the notion that the IANA has showcased a clearly substandard mechanism for maintaining mundane but important information. Setting up reliable and tougher confirmation is easy; any modern mailing-list package has this basic functionality. Tolerably functional and attack-resistant software for the purpose of user interaction via the Web can be obtained and configured by anybody with rudimentary computer skills. It isn’t difficult, and it is that sort of attention to detail that is simply the only acceptable performance level of the IANA. A mistake of this type suggests the possibility of deeper flaws elsewhere, a disquieting possibility in the boiler-room of the Internet. It looks bad. It’s embarrassing. It’s disappointing. I know you have more skill than that, and even the little numbers are important, guys. How about taking care of them?

By Theron Bair, 'Net Worker

Filed Under

Comments

A bit over the top David Conrad – Mar 21, 2009 2:54 AM

Having had some small role with IANA in the past, let me try to provide some factual information regarding the issues you raise.

The IANA now has a Web form to assist in this task;

Actually, IANA has had a web form to allow folks to request and modify PENs for something like ten years.

So I use the Web page evidently based on a Perl script to begin changing my contact information. Yet, it worketh not; I entered the number, and this popped out:

“java.sql.SQLException: Access denied for user ‘root’@‘localhost’ (using password: YES)”

The leading bit of the error message might give you a clue as to what language the PEN application is written in. Hint: it isn’t Perl.

Due to some hardware issues, we had to do an emergency migration of the PEN application software from one machine to another. During that migration and prior to configuring the database server, there was a period in which anyone attempting to create or modify a PEN would have gotten the above error message. IANA (or more accurately, ICANN IT) was remiss from not taking down the application during the migration. As Vice President of IT, I personally apologize for this. It is poor form and can result in people making rash assumptions.

But examining the URL I was redirected to, I found this:

http://pen.iana.org/pen/ModifyPen,$Form.sdirect

Hmm. What appears to be a Perl variable, in a URL, a value presumably fed into something else.

Appearances can be deceiving. Actually, this is how Tapestry (the Java framework used to build the PEN app) constructs short URLs. It isn’t a variable.

Now to the real issue:

Because it’s important to confirm changes, right? That they are made by the correct party, and not, e.g., some random hacker on the Intarwebz somewhere?

When someone requests a change to a PEN, a confirmation message is sent out to both the requester of the change as well as the contact listed for that PEN in a database. Both the requester and the contact must approve the change request. As you identified, there is a known weakness in how the confirmation URLs are built. This problem was noticed (by IANA staff) some time ago and has been on the (long) queue of things to fix, however it hasn’t had very high priority. Why? Well a few reasons:

* If the contact listed in the database receives an unsolicited change request, they’ll generally contact IANA and say “huh?”. IANA staff can then go and undo any theft attempts that might have occurred.

* IANA maintains logs of all changes that have occurred. If an inappropriate change was made, IANA staff can go back through the logs and figure out what happened when and make things right.

* There is indeed a human in the loop for approving changes, so a flood of changes would likely be a bit distracting from the IANA staff charged with processing PEN change requests.

* The PEN registry (derived from the database the PEN application uses) is published and replicated all over the Internet. If someone were to try to steal a PEN by hacking the database, they have to be quite the delayed gratification aficionado.

* To date, as far as I’m aware, and despite IANA having allocated over 33,000 PENs, no one has tried to exploit the weakness you describe.

* PENs are an infinite number space so there has (to date) been little incentive for anyone to bother trying to steal one.

So, yes, we’re aware that there is a risk someone can inappropriately approve a PEN change. However as you note:

Indeed.

Summarizing, in my opinion, you have identified two issues worthy of comment:

- ICANN IT should ensure we take down applications prior to migrations, even in emergencies when the application won’t be down very long. Mea culpa.

- There is a weakness in the way PEN changes are confirmed that should be addressed. We’re actually in the process of rewriting the entire PEN application for reasons unrelated to this issue, however as mentioned, the prioritization on this is relatively low compared to the other developmental tasks related to IANA that are being undertaken.

Thanks for your concern.

Regards,
-drc

P.S. Of course, I’m left wondering why you’d raise this issue publicly instead of first raising with people within IANA. At the very least, you could have been enlightened about your incorrect assumptions about the language used for the PEN application.

# 1 Reply | Link | Report Problems

Over the top? ... nonsense! Theron Bair – Mar 23, 2009 5:33 AM

“java.sql.SQLException: Access denied for user ‘root’@‘localhost’ (using password: YES)”

The leading bit of the error message might give you a clue as to what language the PEN application is written in. Hint: it isn’t Perl.

My apologies, this part was poorly written. At some point, some URL with an ending of “.pl” blinked past my eyes, on the strength of which I developed “Perl Blindness”. I agree, it is highly unusual for a Perl script to throw Java errors, except perhaps as part of a clever scheme to misdirect evildoers. :) I apologize for that confusion, then.

http://pen.iana.org/pen/ModifyPen,$Form.sdirect

Hmm. What appears to be a Perl variable, in a URL, a value presumably fed into something else.

Appearances can be deceiving. Actually, this is how Tapestry (the Java framework used to build the PEN app) constructs short URLs. It isn’t a variable.

If this is not a variable, then so much the better. Do forgive the reflex - when I see dollar signs embedded in strings, and I smell something Perlish somewhere in the chain (erroneously or not), I start to think that there was a mistake made somewhere that somebody will probably pay dearly for.

Summarizing, in my opinion, you have identified two issues worthy of comment:

- ICANN IT should ensure we take down applications prior to migrations, even in emergencies when the application won’t be down very long. Mea culpa.

Indeed. If, as you are suggesting, there was a default configuration in place during the migration forced by the hardware failure, then that prompts a question about how good an idea it is have an default-configured application pointing at the public Internet. I don’t think I’ve heard of one of those working out well in the long run. I assume that it is no longer configured with defaults, but the problem is that if you get sloppy and let a default-configured box get out the door at all, then you’ll eventually either forget about it or fail to configure it quickly enough to avoid problems.

While I am certain that you do know this, it’s good to be reminded from time to time. A real jerk just keeps his peace and lets it happen.

I find myself wondering why this is the case at all. As noted, the effort for such a fix is minimal, so whatever its prioritization, it should be done practically by default, and certainly as a matter of good design. And again, it does raise questions, since design flaws of this type are unlikely to be restricted to one domain.

Frankly, my philosophy is that if a flaw is publicly visible, so are its sequelae. I prefer public correction, since Error hidden is Error cherished. (I am grateful for the boot to my head about Java, clearing out the stuck folly in which I believed erroneously that Perl was somehow involved.) I was directed to this place to post my thoughts, since here it would receive attention from those most inclined to fix it, while still being at least marginally obscure. I must say that I am not, from your response, convinced that a quiet email would have gone anywhere but the bitbucket, since you have already stated that you consider it too low-priority to fix.

But that’s all right. Duty done. Not trying to be a jerk or anything. Carry on.

... When are my PEN entries going to change? :)

# 2 Reply | Link | Report Problems

The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.