NordVPN Promotion

Home / Blogs

Email in the World’s Languages - Part II

Protect your privacy:  Get NordVPN  [ Deal: 73% off 2-year plans + 3 extra months ]
10 facts about NordVPN that aren't commonly known
  • Meshnet Feature for Personal Encrypted Networks: NordVPN offers a unique feature called Meshnet, which allows users to connect their devices directly and securely over the internet. This means you can create your own private, encrypted network for activities like gaming, file sharing, or remote access to your home devices from anywhere in the world.
  • RAM-Only Servers for Enhanced Security: Unlike many VPN providers, NordVPN uses RAM-only (diskless) servers. Since these servers run entirely on volatile memory, all data is wiped with every reboot. This ensures that no user data is stored long-term, significantly reducing the risk of data breaches and enhancing overall security.
  • Servers in a Former Military Bunker: Some of NordVPN's servers are housed in a former military bunker located deep underground. This unique location provides an extra layer of physical security against natural disasters and unauthorized access, ensuring that the servers are protected in all circumstances.
  • NordLynx Protocol with Double NAT Technology: NordVPN developed its own VPN protocol called NordLynx, built around the ultra-fast WireGuard protocol. What sets NordLynx apart is its implementation of a double Network Address Translation (NAT) system, which enhances user privacy without sacrificing speed. This innovative approach solves the potential privacy issues inherent in the standard WireGuard protocol.
  • Dark Web Monitor Feature: NordVPN includes a feature known as Dark Web Monitor. This tool actively scans dark web sites and forums for credentials associated with your email address. If it detects that your information has been compromised or appears in any data breaches, it promptly alerts you so you can take necessary actions to protect your accounts.

In our last installment we discussed MIME, Unicode and UTF-8, and IDNA, three things that have brought the Internet and e-mail out of the ASCII and English only era and closer to fully handling all languages. Today we’ll look at the surprisingly difficult problems involved in fixing the last bit, internationalized e-mail addresses.

All of the extensions described up to this point have been backward compatible with the existing mail delivery infrastructure. MIME encodes all of the extended characters and files as ASCII text, so although you need a MIME-aware mail program on your PC (or web mail or whatever), the mail systems through which the mail passes don’t have to know anything about MIME. Similarly, non-ASCII IDNA domains are encoded as funny looking ASCII A-labels starting with xn— so, again, although applications have to know how to turn UTF-8 into A-labels, the underlying DNS software just sees the ASCII.

Although SMTP changes very slowly, it does have a process to add new optional features, with a way for clients and servers to tell each other which options they support and are using. In 1993, a new feature called 8BITMIME added a simple but surprisingly useful option to send mail with 8-bit characters. It didn’t change any of the other rules, but it did have the effect of allowing any character set that uses the ASCII codes for carriage return, line feed, and period, (as do nearly all but EBCDIC), in mail message bodies. On computers using eight-bit bytes internally (all of them, these days) the code to support 8BITMIME is simple enough that all popular mail systems support it. So if we want to send eight-bit character code in messages, SMTP can handle it.

The main remaining bit of international e-mail that SMTP can’t yet handle is the actual e-mail addresses, both the ones in the message headers such as To: and From: and the ones in the SMTP session. If a mail system is going to be fully international, it also needs to allow non-ASCII text in prompts, error messages, and anywhere else that strings are passed around for presentation to users. This turns out to affect pretty much every piece of software in the e-mail ecosystem, the MUAs (user programs), submission agents that inject mail from user programs, MTAs that move mail from one site to another, and POP and IMAP servers that let MUAs pick up incoming mail. The changes to POP and IMAP and user programs to be language independent are relatively straightforward so I’m going to concentrate on the tricky parts: the format of mail messages, and the SMTP delivery process.

For quite a while people attempted to invent ways to do non-ASCII addresses that were more or less backward compatible with legacy mail. Early work started in China and Japan, where ASCII-only mail was and is particularly unsatisfactory, and initially used simple approaches like allowing various eight-bit extended character sets in mail headers and addresses. Those character sets, such as ISO-2022-x, use sequences of control characters to switch among various “pages” of the character set, so that the meaning of any particular character depends on what shifts preceded it. These didn’t work well for a variety of reasons: the shifts mean that a string’s meaning depends on what shift state it’s in, something that was often implicit or ambiguous, the same set of characters can be represented in many different ways by adding or deleting shifts, and the various experiments tended not to pay sufficient attention to what happened if mail was sent to a mail system that didn’t have their extensions and didn’t understand their extended character set.

UTF-8 doesn’t have the shift problem, but it still uses characters outside the ASCII set that won’t work with legacy mail systems. The next approach was to allow UTF-8 addresses, but encode them as ASCII. For domains, that’s already done with A-labels and punycode; every non-ASCII domain has an ASCII version that starts with xn—. Unfortunately, that doesn’t work with mailboxes (the part of the address before the @.) For one thing, any special characters one might use to mark encoded UTF-8 names have already been used somewhere to mean something else. Hyphens and plus signs are used to mark sub-addresses, exclamation points for uucp addresses, percent signs for source routing, slashes for X.400 compatible addresses, equal signs for VERP bounce addresses, and so on. Even if you could find a sequence of marker characters so arcane that nobody had ever used it, mailboxes are limited to 64 ASCII characters, and by the time you encoded UTF-8 as something along the lines of punycode, the length of the UTF-8 address would be limited to about 18 characters, which is rather short, particularly since people with non-ASCII addresses will want to use all of the same tricks that we ASCII users use with sub-addresses and other address extensions. So that doesn’t work, either.

The most completely worked out experiment, documented in RFCs 5335 and 5336 added a new SMTP option called UTF8SMTP, in which email addresses can be almost arbitrary UTF-8, subject to the 64 byte limit on the mailbox, and the domain has to be one that can be turned into A-labels. Messages are sent using 8BITMIME, and UTF-8 can occur nearly anywhere in the message that ASCII can. Everyone with a UTF-8 address can also have an ASCII address, and one can address mail to both of them, with syntax like this:

To: <utf8mailbox@utf8domain <asciimailbox@asciidomain>>

When sending mail to a UTF8SMTP server, a client sends mail to the UTF-8 address, but provides the ASCII address as well. If the mail has to go to a server that does not handle UTF8SMTP, a complex set of downgrade rules turns any headers with UTF-8 into Downgraded-whatever: headers with MIME encoded versions of the UTF-8, replaces any UTF-8 addresses in the To: and From: lines with comments, and sends it along to the ASCII address.

While it was a magnificent piece of kludgery, after a while it became clear that this dual address system didn’t work either. One reason was that it was very hard to keep the UTF-8 and ASCII addresses straight, so UTF-8 mail tended to leak into ordinary SMTP traffic, and downgraded mail into UTF-8 traffic. Furthermore, even though the dual addresses were intended as a temporary transition feature, on the Internet, no transition feature ever goes away, and Japanese and Chinese users quite reasonably did not want a system that would require that they have an ASCII address forever.

So the EAI working group threw away most of the kludges and is nearly done defining the permanent internationalized mail system for the Internet. In our next installment, we’ll see how your future mail will work.

By John Levine, Author, Consultant & Speaker

Filed Under

Comments

Comment Title:

  Notify me of follow-up comments

We encourage you to post comments and engage in discussions that advance this post through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can report it using the link at the end of each comment. Views expressed in the comments do not represent those of CircleID. For more information on our comment policy, see Codes of Conduct.

CircleID Newsletter The Weekly Wrap

More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

Related

Topics

Cybersecurity

Sponsored byVerisign

Threat Intelligence

Sponsored byWhoisXML API

Domain Names

Sponsored byVerisign

New TLDs

Sponsored byRadix

DNS

Sponsored byDNIB.com

IPv4 Markets

Sponsored byIPv4.Global

Brand Protection

Sponsored byCSC

NordVPN Promotion