Globalization issues in ASP and ASP.NET

Character sets and codepages

We all know that the characters on our computer screen are just a series of bytes. The byte series can be created and interpreted in any number of ways. If the interpretation uses an encoding that's different than the encoding that the byte array was created with, the interpretation will display as garbage. Character sets (charsets) are encoding formats that are usually used by browsers. The Codepage property, which is more applicable for server-side conversions, is just a conversion table that specifies how characters are encoded.

Browsers encode the form post data according to the current character set. If the current character set is "windows-1256," then the byte transmission to the server is also encoded as "windows-1256."

When the ASP is being interpreted, the Form and Querystring collections are not built until they are referenced in code. When they are being built, the string data is transformed to Unicode according to the current codepage. (By default, both ASP and ASP.NET process content by using Unicode format). It is very important that you set the correct codepage before referencing the collections; otherwise, the Unicode representation in memory won't be correct.

To set a codepage, use Session.Codepage or Response.Codepage. The Response.Codepage is only available in Microsoft Internet Information Services (IIS) 5.1 or later versions. For information about the integer values (which correspond to the character set) that we would set these properties to, visit the following Microsoft Web site:

Character set recognition
http://msdn2.microsoft.com/en-us/library/Aa752010.aspxFor example, to set the codepage for the Arabic language, use the following code:

Session.Codepage = 1256

Response.Codepage will only affect the current response. However, Session.Codepage will affect all the responses made by the current user. When the codepage is set by using one of these properties and the Form and Querystring collections are built, this change in the current codepage causes the Response.Write method to transform the Unicode in memory to the current codepage. For more information about this topic, visit the following MSDN Web site:

Setting the Code Page for String Conversions (ASP)http://msdn2.microsoft.com/en-us/library/ms525789.aspxThe bottom-line when it comes to the issues related to charsets and codepages is that client charset and server codepage should match.

Accept languages

If an ASP developer wants to know which languages a user has set in his browser, the developer can use the Request.ServerVariables ("HTTP_ACCEPT_LANGUAGE") variable to find the list of languages that the user would like to read the response in, (such as English, German, or Indian) and the order of preference that the user would like to see these languages in. In ASP.NET, similar information is present in the Request.UserLanguages property as an array.
For more information about how to use this information in ASP code, click the following article number to view the article in the Microsoft Knowledge Base:

229690 How to set the ASP Locale ID per the browser's language settings
 

Displaying multi-byte character sets in Internet Explorer

The only encoding format that can show a multi-byte character set is Unicode (UTF-8). With UTF-8, we can display Cyrillic, Indian, and Japanese all on the same page. If we do not use UTF-8, we can only show one of these languages at a time. To set the charset of the browser, use the Response.CharSet property.

Static multi-byte characters on a page

To display multi-byte characters stored directly in the page, we must first save the page with specific encoding. UTF-8 will be best, but a specific codepage (matched to the codepage of the characters) will work as well.

Saving an ASP file using Microsoft Visual InterDev doesn't help here, since Visual InterDev can only save in ANSI English or Unicode. Any ASP page saved as Unicode is not supported by ASP.

In Microsoft Visual Studio .NET, you can save a file in any encoding. There are two ways to do this. The default way is to save the file by using the current codepage for the user. An additional way to save a file with an encoding is as follows:
On the File menu, click Save File As. In the
Save File As dialog box, click the drop-down arrow on the
Save button. When you click the arrow, the options are
Save and Save with Encoding. When you click
Save with Encoding, the Advanced Save Options dialog box appears where you can select the type of encoding that you want to apply from a list of the codepages that are installed on the computer.


Note This changes the encoding for the save operation, but is for one time only. The next save will be set back to the default.

To change the default codepage, click Advanced Save Options on the
File menu. In the Advanced Save Options dialog box, you can set the default encoding for save operations to the codepage of your choice.

These methods are related to how the file is saved on disk. However, to control the output for ASP, as already discussed, we need to set the Session.CodePage and the Response.CharSet properties. With IIS 5.1 and later versions, we can also use the Response.CodePage property.

Default CODEPAGE on server

The default locale and the default codepage for the page depend on the registry settings for the .DEFAULT user. We can find the international key at registry hive HKEY_USERS\.DEFAULT\Control Panel\International. We can also change the behavior of the locale that is chosen by IIS.

If the logged on user has the same locale set as the above key or the system default, the user setting takes precedence.

Example: Default locale has date format set as 11.1.2004, while the logged on user (with the same locale set) has the date format as 11/1/2004. The 11/1/2004 setting will take effect for ASP.

(For ASP.NET, this can vary. In some installations, the ASPNET user will have its own profile that will show up under HKEY_USERS when it is loaded. In others, it will use the .DEFAULT profile. We can also use the codepage attribute in the <%@ %> declaration. This should be used when the file is saved with a different encoding then the default, such as codepage 932 (Japanese)).

Codepage issues versus font conversion issues: which is which?

At times, you may see a question mark (?) character or a box where a character is supposed to appear.

Runtime encoding changes

In the Application_BeginRequest event, we can modify the value of requestEncoding and have it take effect before the request is processed. For the response, the Page_PreRender event is the last chance to modify the encoding of the output. Also note that Response.Write will put characters into this buffer as soon as we call it, so be sure to have the right encoding set before using Response.Write.

Original data is non Unicode: How to still make Internet Explorer interpret multi-byte charsets?

We can also make ASP.NET behave like ASP if we need to. To make this occur, we need to set the responseEncoding and requestEncoding to windows-1252 (a more complete encoding than iso-8859-1), and use the Response.Charset property to display the text correctly. This works because windows-1252 is a single byte encoding scheme, and does not modify any bytes that are added to the buffer. Thus, double-byte characters are sent as a series of single bytes. We can then tell Internet Explorer how to interpret the bytes by using the Response.Charset property. This scenario may be necessary if the original data is not stored as Unicode or UTF-8, such as a return value from a COM object, or if the data is stored in Microsoft SQL Server in a non N field (such as varchar).

SQL Server and ASP.NET globalization issues

INSERT INTO MusicAlbum (Album_ID, [Year], Name, Artist_ID, Company_ID) VALUES (12345, 2005, N'Abida', 4653, 403)
Or:
Dim t As String = "INSERT INTO MusicAlbum(Album_ID, [Year], Name, Artist_ID, Company_ID) VALUES (12345, 2005, N'" & TextBox1.Text & "', 4653, 403)"

System.Globalization namespace

This namespace is the core of globalization and localization in the .NET Framework. The main class used in this namespace is the CultureInfo class. It holds culture-specific information, such as the date/time format, number formats, comparison information, and text information. For more information about the CultureInfo class, visit the following MSDN Web site:

http://msdn2.microsoft.com/en-us/library/system.globalization.cultureinfo(vs.71).aspxCultureInfo

Neutral cultures vs. specific cultures

A neutral culture is a culture that is associated with a language, but not a specific country/region. A specific culture is associated both with a language and a specific country/region.

An example: "DE" (neutral culture) is for the German language, but "de-AT" (specific culture) is for the German language as it's spoken in Austria. Neutral cultures cannot be used for formatting.

Current thread and culture awareness of .NET Framework classes

All classes and methods in the .NET Framework library where we would expect the output to be culture-dependent have two built-in behaviors:

• They let us specify the culture code while supplying the arguments so that the output is based on the culture specified. This is optional.

• If this is missed (usually it is), the classes are intelligent enough to keep a check on the Thread.CurrentThread.CurrentCulture property and work according to that.

We can modify this property's value with code that is similar to the following:

    Dim ci As CultureInfo
        ci = New CultureInfo("de-AT")
        Thread.CurrentThread.CurrentCulture = ci

In this code example, "de" represents the German language, and "AT" represents Austria. So, in this instance, the DateTime.Now().ToString method would return the date and time in a format corresponding the way the date and time are expressed in the German language in Austria.

The framework ensures (as follows) that the CurrentCulture property is always initialized:

1. Whatever it is set to programmatically.

2. In case it is not explicitly set by the programmer, the property is picked from the configuration files (<globalization> tag).

3. If the property is missing there, it is the culture on which the Web server is running. This is usually the neutral culture that corresponds to the language of the operating system.

Resource files

All .resx, .resource files, and files that have the Build Action attribute set to Embedded Resource that are added to an ASP.NET project in Visual Studio .NET, are automatically compiled and embedded within application assembly as part of its manifest. This can even be done manually by using the Resource File Generator (RESGEN) utility via a Visual Studio .NET command prompt. For more information, visit the following MSDN Web site:

http://msdn2.microsoft.com/en-us/library/ccec7sz1(vs.71).aspxThis is a general concept that is applicable whenever we need to manage application resources that are unrelated to globalization. However, when we are implementing globalization, we should use satellite assemblies.

Satellite assemblies

Satellite assemblies can be used in an ASP.NET project when you make sure the following are true:

1. All the user-interface elements in all aspx files need to be equipped with id and runat=server attributes.

2. We create separate .resx files. Each one must correspond to each culture we want our application to support.

3. We must decide a common first name for all these files for ex. 'Strings'.

4. We name the separate .resx files with the following naming convention commonfirstname. languagecode-regioncode.resx (for example: Strings.de-AT.resx, Strings.en-GB.resx ).

5. We should have the resource file
   commonfirstname.resx (Strings.resx) that has all the strings as we want displayed in the default case.

6. Write code to detect user's culture and set the Thread.CurrentThread.CurrentUICulture property to match to it.

7. Write code to load the resources by using the ResourceManager class.

8. Write code to extract strings from the loaded object, and assign them to user interface elements.

When you have performed these steps, Visual Studio.NET will compile Strings.resx and embed it into the application assembly (MyGlobalizationTestProjectName.dll). However, for all other .resx files, it will generate separate dll files which do not have executable code but only resource data. These actually are called satellite assemblies. Also, Visual Studio .NET places these in folder structure similar to the following:MyGlobalizationTestProjectName
|------- bin
|------en-US
MyGlobalizationTestProjectName.resources.dll
|------ja-JP
MyGlobalizationTestProjectName.resources.dll
|------de-AT
MyGlobalizationTestProjectName.resources.dll

Difference between CurrentCulture and CurrentUICulture

While the methods of classes in the System.Globalization namespace depend on the Thread.CurrentThread.CurrentCulture property to give their output, the ResourceManager class that loads the resource assembly depends on the Thread.CurrentThread.CurrentUICulture property to load the appropriate satellite assembly. The following is an example of C# code:

using System.Globalization;
using System.Threading;
using System.Resources;

//Load resources. 
protected ResourceManager gStrings = new ResourceManager("MyGlobalizationTestProjectName.strings", typeof(MyTestWebFormName).Assembly);

// Get the user's preferred language.
string sLang = Request.UserLanguages[0];
// Set the thread's culture for formatting, comparisons, etc.   
Thread.CurrentThread.CurrentCulture =  CultureInfo.CreateSpecificCulture(sLang);
// Set the thread's UICulture to load resources
// from satellite assembly.
Thread.CurrentThread.CurrentUICulture = new CultureInfo(sLang);

private void Page_Load(object sender, System.EventArgs e) 

{ 

 if (!IsPostBack)  

 {      
// Get strings from resource file and assign to UI elements.
head1.InnerHtml = gStrings.GetString("satellite.head1");
p1.InnerHtml = gStrings.GetString("satellite.p1");
sp1.InnerHtml = gStrings.GetString("satellite.sp1");
sp2.InnerHtml = gStrings.GetString("satellite.sp2");
butOK.Text = gStrings.GetString("satellite.butOK");
butCancel.Value = gStrings.GetString("satellite.butCancel");
   }

 }

Order in which ASP.NET selects satellite assemblies:

When you have set the thread's CurrentUICulture, ASP.NET automatically selects the resources that match, in the following order:

• If a satellite assembly is found with a matching culture, the resources from that assembly are used.

• If a satellite assembly is found with a neutral culture that matches the CurrentUICulture, resources from that assembly are used.

• If a match is not found for the CurrentUICulture, the fallback resources stored in the executable assembly are used.

Note This is based on the more general Resource Fallback Process. For more information, visit the following MSDN Web site:

http://msdn2.microsoft.com/en-us/library/sb6a8618(vs.71).aspx

Manually creating satellite assemblies:

This use of satellite assemblies is where Visual Studio .NET creates the assemblies itself. Visual Studio .NET doesn't strong name satellite assemblies by default, however. If you want to change these options, you would need to create satellite assemblies manually. For more information, visit the following MSDN Web site:

http://msdn2.microsoft.com/en-us/library/21a15yht(vs.71).aspx .

Glossary

ANSI Stands for American National Standards Institute. In this context, it represents a specific codepage for a specific language/character set. Most often refers to the English codepage (windows-1252).

ASCII A 1 byte (or 7 bit) encoding scheme. Only the characters in the range 0-127 are standardized. The range 128-255 is extensions to ASCII and not part of the standard. An example of this is the difference between the upper range of the OEM ASCII chart and the VB ASCII chart.

CharSet Setting used mostly for Internet Explorer and browsers that tells the browser how to interpret the character data. Example: Response.charSet = "iso-8859-1."

Codepage A conversion table that specifies how characters are encoded (usually used for servers).

Globalization Globalization is a process of designing and creating an application so that the unique requirements of a culture, region, or national/regional and linguistic needs can be met. In other words designing an application in a way that it can be localized later is globalization.

Locale/Culture Language and region specific formats/preferences including, date and calendar formats, time formats, currency formats, casing, sorting and string comparison, address formats, phone number formats, paper sizes, unit of measure, writing direction, etc.

LocaleID (LCID) A DWORD value that specifies the language identifier and sorting ID. It can be used to specify the specific region formats for ex date/time etc should be formatted according to.

Localizability Ability of an application to present content for the demanded language/locale.

Localization Localization is the process of translating a user interface into specific languages and/or locales.

Multibyte character set A character set in which the characters are composed of two or more bytes, such as Japanese. UTF-8 also falls under this category. (Unicode technically is in this category, but in Windows, it has its own category.)

Unicode A 2-byte encoding scheme. Windows uses Unicode internally. Any APIs specifically for Unicode are signified by a "W" on the end of the function name. Also known as wide char; cannot be directly used from web applications.

UTF-8 A character encoding where a character can be represented by 1-6 bytes. In Windows, the range is 1-3 bytes. Not supported under NT4 for web applications.



Wide character set An alias for Unicode. Also known as DBCS (double byte character set), UCS-2, UTF-16.

As always, feel free to submit ideas on topics you want addressed in future columns or in the Knowledge Base by using the Ask For It form.