How to Test Programs for Compatibility with Windows

One of the primary tasks in preparing for testing is to write a test plan. In the test plan, you specify the scope and objectives for the testing and describe the methodology you are going to use.

When you develop a test plan for testing application compatibility with Windows, include the following:

• Scope: What priority levels you address during testing?
• Methodology: Who does the testing involve?
• Requirements: What hardware, software, personnel, training, and tools you need to perform the testing?
• Criteria for pass-fail: What determines if an application passes or fails?
• Schedule: How do you plan to complete the testing by the scheduled date?

Establishing the Testing Scope

If your organization uses many applications, you may not have time to test all of them as thoroughly as you would like. Test the highest priority and the most frequently or widely used applications first.

Test both server-based and client-based applications. Client-based applications are usually the most difficult and time-consuming to test because of the amount of applications.

Defining the Testing Methodology

When you plan the methodology, consider the following:

• Where will the testing take place?
• Who will perform the tests?
• How will you communicate with and involve participants?
• How will you schedule the testing?
• How will you manage application problems?

If your organization has a group of application testers, we recommend that you use them. If you do not have such a group, look for ways to use a variety of resources to achieve the best results in a reasonable amount of time.

For example, you can use a few experienced testers to develop a battery of test cases, which they can train others to run. Alternatively, you might have the experienced testers perform a core set of tests and then coordinate with business units to have their experts come to the lab to perform the functions they use in their work.

Devise a process for scheduling test days and communicating with the testers. For example, you might set up a Web site on your intranet where anyone can view test dates, status reports, contact names, and other relevant documents.

Identifying Resource Requirements

As you plan for application compatibility testing, keep in mind the future state of your computing environment. Are you planning to upgrade some of your software to versions that fully use new Windows features? Are you planning to implement new standard desktop configurations or use Terminal Services?

Issues such as these determine the resources that you require and the applications that you are going to test as a suite.

If you plan to deploy new applications with Windows during the rollout, test these applications with the current applications.

You can facilitate testing by setting up a lab where testers can conduct their tests. In such a lab, you can have the necessary tools and equipment available at all times.

In the lab, set up the test computers for dual or triple boot so that testers can quickly access the mode they need to install and test their applications. For example, you might need Windows NT 4.0 and Windows 2000 to test the applications through the upgrade path. To make it easy for testers to restore the computers to their prior state, make disk images of the drives with the base operating systems.

Defining Pass-Fail Criteria

Define a procedure for testers to know when and where they are to log application problems and issues that you want to resolve.

To define the criteria for pass and fail, consider issues such as the following:

• How significant is the problem? Does it affect a critical function or a peripheral one?
• How likely is someone to encounter the problem?
• Is there a way to circumvent the problem?

Your testing schedule depends on many conditions, including:

• How many testers participate.
• Whether the testers are on this project full-time or need to be scheduled.
• The testers' experience levels.
• The number and complexity of the applications.

Testing Applications

Many commercial applications have already been tested to determine how well they support Windows 2000 and later. Microsoft provides a directory of applications for Windows 2000 where you can look up the status of the applications you use. The directory uses the following designations:

• Certified - Indicating that the application was tested by VeriTest and that it takes advantage of new Windows features.
• Ready - Indicating that, according to the vendor, the application was tested for compatibility with and is supported on Windows 2000. The application does not necessarily take advantage of new Windows features.
• Planned - Indicating that the intent is for the application to meet either the Certified or the Ready criteria when it is fully tested.

Testing Strategies

The goal of your application testing is to verify that everything that works on your current platform also works on your current version of Windows. If an application was written for an earlier version of Windows, it does not necessarily use new Windows features, but its functionality should work in Windows 2000 as it does on your current platform.

Commercial Applications

For commercial applications, the first step is to run Setup in check-upgrade-only mode to check for potential incompatibilities. When you run Setup in this mode, Windows checks the installed software against a list of applications known to be incompatible and logs any that it finds. The command line format for check-upgrade-only mode is: Although this tool can alert you to potential compatibility problems, it addresses only a small percentage of your applications and only the applications installed on the computer you are checking.

The next step is to check the directory of Windows applications to determine the compatibility of the applications you use.

Even if you find that some of your applications have already been tested by others, you should test them in your environment. In this case, focus your testing on the way your organization uses the applications. For example, test the following:

• Configurations your organization uses.
• Features most frequently used.
• Combinations of applications you use together.

Remember to test your antivirus software. Many of these applications need to be upgraded because of their use of file system filters. Many Windows NT 4.0 files system filters might not work on Windows 2000 or later due to changes in the NTFS file system.

Custom Applications

If you use custom third-party products or develop applications internally, you need to develop a more extensive testing strategy than for pre-tested commercial applications.

Even if you are testing an application which you did not develop, the Windows 2000 Application Specification can provide insight into testing. The MSDN Web site at http://msdn.microsoft.com (http://msdn.microsoft.com) includes a downloadable version of the specification. The MSDN Web site also contains other important information about testing, such as white papers about exploratory testing and the method that independent testing organizations use to test the functionality of applications vendors submit for certification.

NOTE: The testing suggestions in this section are not comprehensive and do not apply to all situations. They are provided to help you start thinking about how to test.

Test Deployment Scenarios

Test installing and running your applications using the scenarios you plan to use during deployment. For example, you might plan to deploy by installing on clean computers or by upgrading from Windows 95 or Windows 98 or an earlier version of Windows NT. If you plan to upgrade, you might keep the applications on the computer during the upgrade, or you might uninstall them and reinstall them after the upgrade.

Because of differences between Windows 95 or Windows 98 and Windows 2000, some application installations work differently depending on which operating system you use for the installation. For example, if you install an application on a computer running Windows 95 or Windows 98, and then you upgrade the computer to Windows 2000, the application might not work the same way as it would have if you had installed it in Windows 2000. In this case, you might need to uninstall the application and reinstall it after you upgrade or obtain a migration dynamic link library (DLL).

A migration DLL allows an application that was originally installed on Windows 95 or Windows 98 to function correctly after the computer is upgraded to Windows 2000. Migration DLLs can resolve application problems by performing the following actions:

• Replacing or upgrading Windows 95-specific or Windows 98-specific files with Windows 2000-compatible files.
• Mapping Windows 95-specific or Windows 98-specific registry keys to the appropriate Windows 2000 locations.

Upgrade Scenario

If you are planning to upgrade your computers:

1. Install Windows 95, Windows 98, or Windows NT 3.51 or later.
2. Install the application you want to test.
3. Upgrade the computer to Windows 2000.
4. Test the application.

Clean Installation Scenario

If you are planning to install on reformatted computers:

1. Install Windows 2000.
2. Install the application.
3. Test the application.

Test Installing and Uninstalling

Test application installation in a variety of ways, such as the following:

1. Terminate the installation before it is complete.
2. Try all the installation options used in your environment.
3. If your organization allows users to install applications, test the installation both as an administrator and as a power user; then test the application functionality.
4. Try to uninstall the applications.
5. Verify that an application can be installed by an administrator and uninstalled by a user. When logged on as user, the uninstall should be either complete or disallowed.

Test applications using the features, configurations, and application suites you use to accomplish business tasks.

Access Data

Try to access data in a variety of ways, such as the following:

• Access data on a server running the current version of Windows, as well as on a server running Windows 2000.
• Test concurrent use of a database, including simultaneous access and update of a record.
• Perform complex queries.

Test Printing

Print a variety of document types with a variety of printers, such as the following:

• Print documents with embedded files from several source applications.
• Print to printers with long file names.

Common Compatibility Issues

Applications developed for previous versions of Windows might not take full advantage of new features, such as Active Directory or IntelliMirror. This section does not address these new features.

• Windows File Protection: Earlier versions of Windows allowed applications to replace shared system files during installation. When such changes occurred, users frequently encountered problems that ranged from program errors to an unstable operating system.
  
  Windows File Protection is a new feature that prevents applications from replacing system files. This feature verifies that protected system files are the correct Microsoft version. If a file was replaced with an incorrect version, Windows restores the correct version.
• Robust heap checking: Windows includes several performance enhancements in the heap manager. Applications that did not use heap management correctly before might now have their memory management problems exposed. Common problems include using memory after it has been freed and assuming that a memory does not move when it is reallocated to a smaller size.
• Enumeration of hardware devices: Changes in the list of supported hardware devices might cause problems for applications that use devices that are no longer supported.
• Enumeration of fonts: The list of fonts has changed. Because registry keys have been added to support internationalization, some applications might see multiple displays of fonts.
• Changed registry keys: Some registry keys have been moved or deleted. Applications that write to the application programming interface (API) should not experience problems, but they can have problems if they write directly to the registry.
• Version checking: Application installation programs that check versions incorrectly can have problems. Check for the version your application requires or later, unless your application is dependent on a specific operating system or version.
• Windows Messaging Service: Applications that expect Windows Messaging Service (WMS) to be provided by the operating system will not find it.
• File input/output security: Windows has tightened security for file input and output. Applications that use file filters, such as anti-virus programs, may lose significant functionality in Windows 2000 or later.

Resolving Application Incompatibilities

When you encounter application compatibility problems, you need to prioritize them and then assign someone to resolve them. You should have a plan for how to assign problems.

Assigning the appropriate personnel to research and resolve problems is critical to the success of your application testing. Problem resolution might encompass a wide variety of activities, such as the following:

• Researching Web sites for known problems and solutions.
• Contacting vendors for patches, Setup programs, or migration DLLs.
• Contacting Microsoft support.
• Debugging internally-developed applications.

As you research the cause of a problem, consider various approaches to determine the most effective solution. For example, you might choose to:

• Fix the problem if you developed the application.
• Ask the vendor fix the problem if you purchased the application.
• Replace the application with a new version or application.
• Ignore the failure if you have a way to work around the problem.

Always be sure that a problem does not occur on your current platform before researching it as a Windows 2000 compatibility issue. Some of the available resources for researching Windows 2000 compatibility issues are:

• Windows 2000 Application Specification, which you can download from the MSDN Library at http://msdn.microsoft.com (http://msdn.microsoft.com) . Appendix E provides the specific location where can you obtain the specification.
• Windows 2000 Compatibility Guide, which you can find in the MSDN Library at http://msdn.microsoft.com (http://msdn.microsoft.com) . This guide include valuable information about diagnosing compatibility problems.
• Microsoft TechNet at http://www.microsoft.com/technet (http://www.microsoft.com/technet) , which contains updates, white papers, and other technical information
• Directory of Windows 2000 applications, which includes support information and links to vendor Web sites.