You cannot decrypt data on Windows XP SP1 or later versions

View products that this article applies to.

SUMMARY

When you try to decrypt data on a computer that is running the Microsoft Windows XP operating system, and the data was encrypted by using the RC2 algorithm on a computer that is running Microsoft Windows 2000, you may not be able to decrypt the data successfully.

To be able to decrypt the data successfully, you must make an explicit call to the CryptSetKeyParam function. The CryptSetKeyParam function is defined in the Advapi32.dll file to adjust the key length for the RC2 algorithm explicitly.

This article contains the links to download the sample code for both the encryption program and the decryption program. It also contains information about how the problem can be reproduced.

Back to the top

SYMPTOMS

If data has been encrypted on a computer that is running Windows 2000, you may not be able to decrypt the data on a computer that is running Window XP and that has Window XP Service Pack 1 (SP1) installed.

Back to the top

CAUSE

The encryption program and the decryption program use the Microsoft Enhanced Cryptographic Service Provider V1.0 as the Cryptographic Service Provider (CSP). This behavior occurs because a change has been implemented in the RC2 cipher text algorithm.

In the earlier implementation of the CryptSetKeyParam function in the version of Microsoft Enhanced Cryptographic Provider that is included with Windows 2000, the RC2 session keys are 128 bits long. However, the effective key length that is used to expand keys in the key table is 40 bits. This is the default behavior.

For security reasons, Microsoft changed this default behavior beginning with the version of Microsoft Enhanced Cryptographic Provider that is included with Microsoft Windows XP SP1. By default, the effective key length is now 128 bits for a 128-bit session key in the RC2 algorithm.

Back to the top

WORKAROUND

To work around this behavior, you must make an explicit call to the CryptSetKeyParam function in your program to set the correct session key length.

For more information, and for instructions to download and to run the sample encryption program and the sample decryption program, see the "More information" section.

Back to the top

STATUS

This behavior is by design.

Back to the top

MORE INFORMATION

Sample programs

In the following downloadable DecryptSample program, the CryptSetKeyParam function is called to set the effective key length to 40 bits.

The following files are available for download from the Microsoft Download Center:

Collapse this imageExpand this image

Download the EncryptSample package now. (http://download.microsoft.com/download/0/9/0/090ef467-95e9-45b1-8893-8f3dd1eae7b6/encryptsample.exe)

Collapse this imageExpand this image

Download the DecryptSample package now. (http://download.microsoft.com/download/0/9/0/090ef467-95e9-45b1-8893-8f3dd1eae7b6/decryptsample.exe)

For more information about how to download Microsoft Support files, click the following article number to view the article in the Microsoft Knowledge Base:

119591 (http://support.microsoft.com/kb/119591/ ) How to obtain Microsoft support files from online services

Microsoft scanned this file for viruses. Microsoft used the most current virus-detection software that was available on the date that the file was posted. The file is stored on security-enhanced servers that help prevent any unauthorized changes to the file.

Instructions to run the EncryptSample program

On your computer that is running Microsoft Windows 2000, download the Encrypt.exe file. Save the file to the C folder.
Unzip the file to the C:\EncryptSample folder.
Locate the C:\EncryptSample\EncryptInVB6 folder, and then double-click the EncryptGroup.vbg file. The EncryptGroup project group opens in Microsoft Visual Basic 6.0.
On the Run menu, click Start With Full Compile. A Windows Form that is named Form1 opens. This form is the user interface (UI) of the EncryptSample program.
In the Enter full path\filename for a file to encrypt box, type C:\EncryptSample\input.txt, and then press TAB. The name of the file where the encrypted string will be stored appears in the Name and location of encrypted file box.

Note The path of the encrypted file appears as c:\CryptographySample\input.enc.
Click Encrypt. The Encrypt Files message box appears.
Click OK.
Click Exit to quit the program. The program puts the encrypted string in the C:\EncryptSample\Input.enc file.
Copy the C:\EncryptSample\Input.enc file to the C:\DecryptSample folder on your computer that is running Windows XP SP1.

Note If the C:\DecryptSample folder does not exist on your computer, create this folder, and then copy the C:\EncryptSample\Input.enc file to it.

Instructions to run the DecryptSample program

On your computer that is running Microsoft Windows XP SP1, download the Decrypt.exe file. Save the file to the C folder.
Unzip the file to the C:\DecryptSample folder.
Locate the C:\DecryptSample\DecryptInVB6 folder, and then double-click the DecryptGroup.vbg file. The DecryptGroup project group opens in Visual Basic 6.0.
On the Run menu, click Start With Full Compile. A Windows Form that is named Form1 opens. This form is the UI of the DecryptSample program.
In the Enter full path\filename for a file to decrypt box, type C:\DecryptSample\input.enc, and then press TAB. The name of the file where the decrypted data will be stored appears in the Name and location of decrypted file box.

Note The path of the decrypted file appears as c:\DecryptSample\input.dec.
Click Decrypt. The Decrypt Files message box appears.
Click OK.
Click Exit to quit the program. The program puts the decrypted data in the C:\DecryptSample\Input.dec file.

Back to the top

Steps to reproduce the behavior

To reproduce this behavior, comment the following lines of code in the CryptoDecrypt method of the clsCryptoAPI.cls class of the CryptWrap.vbp project in the DecryptGroup.vbg program group.

Note The DecryptGroup.vbg is the same program group that is described in the "Instructions to run the DecryptSample program" section.

    If Not CBool(CryptSetKeyParam(lngkey, ByVal KP_EFFECTIVE_KEYLEN, _
                 lngKeyLen, ByVal 0)) Then

        MsgBox "Error: " & CStr(GetLastError) & " during CryptSetKeyParam!", _
               vbExclamation Or vbOKOnly, "Encryption Errors"
        GoTo CleanUp
    End If

After you had commented the code, run the DecryptSample program to decrypt the file that you encrypted in the "Instructions to run the EncryptSample program" section.

You may not experience this behavior when you create an encryption program or a decryption program in Microsoft Visual Studio .NET by using the Microsoft Enhanced Cryptographic Service Provider. The Microsoft .NET Framework handles this difference in key length internally by changing the key length depending on the operating system. Therefore, this difference in key length becomes transparent when you are developing encryption programs and decryption programs in Visual Studio .NET.

Back to the top

REFERENCES

For more information, visit the following Microsoft Developer Network (MSDN) Web sites:

CryptSetKeyParam
http://msdn2.microsoft.com/en-us/library/ms938335.aspx (http://msdn2.microsoft.com/en-us/library/ms938335.aspx)

The Cryptography API, or how to keep a secret
http://msdn2.microsoft.com/en-us/library/ms867086.aspx (http://msdn2.microsoft.com/en-us/library/ms867086.aspx)

CopyMemory
http://msdn2.microsoft.com/en-us/library/aa366535.aspx (http://msdn2.microsoft.com/en-us/library/aa366535.aspx)

GetLastError
http://msdn2.microsoft.com/en-us/library/ms679360.aspx (http://msdn2.microsoft.com/en-us/library/ms679360.aspx)

CryptCreateHash
http://msdn2.microsoft.com/en-us/library/ms935996.aspx (http://msdn2.microsoft.com/en-us/library/ms935996.aspx)

CryptAcquireContext
http://msdn2.microsoft.com/en-us/library/ms935987.aspx (http://msdn2.microsoft.com/en-us/library/ms935987.aspx)

CryptDestroyHash
http://msdn2.microsoft.com/en-us/library/ms936018.aspx (http://msdn2.microsoft.com/en-us/library/ms936018.aspx)

CryptEncrypt
http://msdn2.microsoft.com/en-us/library/ms936039.aspx (http://msdn2.microsoft.com/en-us/library/ms936039.aspx)

CryptDecrypt
http://msdn2.microsoft.com/en-us/library/ms936011.aspx (http://msdn2.microsoft.com/en-us/library/ms936011.aspx)

CryptReleaseContext
http://msdn2.microsoft.com/en-us/library/ms936171.aspx (http://msdn2.microsoft.com/en-us/library/ms936171.aspx)

CryptDeriveKey
http://msdn2.microsoft.com/en-us/library/ms936014.aspx (http://msdn2.microsoft.com/en-us/library/ms936014.aspx)

CryptDestroyKey
http://msdn2.microsoft.com/en-us/library/ms936021.aspx (http://msdn2.microsoft.com/en-us/library/ms936021.aspx)

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

277786 (http://support.microsoft.com/kb/277786/ ) Encrypting/decrypting data across systems

Back to the top