In the new global economy, companies have the opportunity to market their wares to billions of customers who don't speak a word of English. Java was designed from the ground-up to help programmers deploy internationalized software. In this article, I show how Java makes your woes in the area of character sets and encoding melt like cotton candy on your tongue.
Dipping My Toes in the Global Pool
For the past year I've been spending a steadily increasing amount of time at work dealing with internationalization of our products. My division of Cisco makes an IP-based phone system that interfaces with users and administrators at dozens of different points. Users have alphanumeric displays on their phones, perform personal and system administration on web pages, and listen to voice prompts when collecting messages. From the user interface perspective there is a lot going on.
Like most new products made in a skunk-works atmosphere, Cisco CallManager was developed with little or no thought towards our international customers. The focus was on quickly developing a stable product with as many features as possible. Of course, our success at this strategy led to immediate discontent from our business partners in Europe and Asia. It turns out that telephone users in France really want to have their instruction manual written in French.
The Four Problems of Text Internationalization
On the surface, modifying your product for users of another language seems simple enough: translate everything and distribute the results. Unfortunately it just isn't that easy. Translation is only the first of four big problems. Translating written material might present some logistical problems, but these are usually more budgetary than technical.
The other three problems are more technical in nature, and Java provides tools to deal with all three. In order, these problems are:
- Managing user-seen content once it has been translated into multiple languages.
- Selecting an appropriate character set and rendering text that uses it.
- Properly encoding text in a given character set so that it can be stored and transmitted in a world of 8-bit bytes.
Java helps you deal effectively with all three of these problems. The first, management of translated content, is handled using Java Resource Bundles. In this article, I examine the next two -- character sets and encodings.
Character Sets
Cisco's business-class IP Phones have a nice LCD screen that presents call status information to users in a fairly friendly way. One of the first problems we ran into when internationalizing the entire phone system was that this display only supported the 7-bit ASCII character set. This had the unfortunate effect of changing the name of Señor Nuñez to "Senor Nunez" when stored as a speed dial. That's because the character set we were using lacked the letters commonly seen in other countries using the Roman alphabet.
A short-sighted solution to this problem might be to expand the character set to a full eight bits, using the upper 128 character positions for the commonly missed characters. And in fact, there is a standard character set called ISO-8859-1 that does just that. Figure 1 shows how ISO-8859-1 populates the upper half of the character space with 112 characters used in Western Europe. (The first 32 positions in the upper half of the space are not used.)
We quickly modified our phone to accept a new font, and soon found that we could properly render names of people in France, Germany, Italy, the Netherlands, and so on. As long as we confined our sales efforts to our friends in NATO all was well. Figure 2 shows our phone happily rendering most of the ISO-8859-1 character set, ready to march onto desktops anywhere the Euro is honored.
Even in today's global economy, many manufacturers of software and other products find this state of affairs to be just dandy. But as soon as you try to sell a phone in Greece or the Russian Federation, you're out of luck. Languages such Russian and Greek just don't have enough characters in common with Western Europe to fit into the ISO-8859-1 character set. Not to fear, because ISO-8859-5 and ISO-8859-7 were created just to deal with this problem.
If you take a look at Figure 3, you can see the seeds of a problem beginning. Adding a new character set to the phone is a manageable problem -- one that operating systems like Windows already manage quite effectively. We simply have to make it possible for our telephone to download one font in France, and a different one in Greece.
But now the phone finds itself in a situation that can seem a bit baffling to mossback programmers such as yours truly. If a multinational company has a Señor Nuñez listed in its corporate directory, things will work just fine when a user in France or Italy looks up his name. But unfortunately, a user in Greece will see the name rendered as of Seρor Nuρez. We now have a fundamental problem: a given numeric value is rendered differently depending on the character set in use. A lowercase "n" with a tilde over it in ISO-8859-1 transforms itself into a lowercase Greek rho when we switch to ISO-8859-7.
This unpleasant situation causes a huge paradigm shift for programmers working on internationalization. There's no longer any such thing as "plain text." When we store a user's name in the database, we now have to also store the name of the character set that properly renders it. The same thing holds true for error messages, speed number labels, soft-keys, you name it.
Or does it?
