VB "Bad DLL Calling Convention" Means Stack Frame Mismatch

This article was previously published under Q85108
This article has been archived. It is offered "as is" and will no longer be updated.
SUMMARY
When you call a dynamic link library (DLL) function from Visual Basicfor Windows, the "Bad DLL Calling Convention" error is often caused byincorrectly omitting or including the ByVal keyword from the Declarestatement or the Call statement. The ByVal keyword affects the size ofdata placed on the stack. Visual Basic for Windows checks the changein the position of the stack pointer to detect this error.

When Visual Basic for Windows generates the run time error "Bad DLLCalling Convention," the most common cause when calling API functionsis omitting the ByVal keyword from the Declaration of the externalfunction or from the call itself. It can also occur due to includingthe ByVal keyword when the function is expecting a 4 byte pointer tothe parameter instead of the value itself. This changes the size(number of bytes) of the values placed on the stack, and upon returnfrom the DLL, Visual Basic for Windows detects the change in theposition of the stack frame and generates the error.
MORE INFORMATION
There are two calling conventions, or inter-language protocols: thePascal/Basic/FORTRAN calling convention, and the C calling convention.Visual Basic for Windows uses the Pascal calling convention, as do theMicrosoft Window API functions and other Microsoft Basic languageproducts. Under the Pascal convention, it is the responsibility of thecalled procedure to adjust or clean the stack. (In addition, parametersare pushed onto the stack in order from the leftmost parameter to therightmost.) Because the DLL function is responsible for adjusting thestack based on the type and number of parameters it expects, VisualBasic for Windows checks the position of the stack pointer upon returnfrom the function. If the called routine has adjusted the stack to anunexpected position, then Visual Basic for Windows generates a "BadDLL Calling Convention" error. Visual Basic for Windows assumes astack position discrepancy because the DLL function uses the C callingconvention. With the C calling convention, the calling program isresponsible for adjusting the stack immediately after the calledroutine returns control.

Steps to Reproduce Behavior

Create a simple DLL using Microsoft Quick C for Windows or anycompiler capable of creating Windows DLLs. The following example is inC and written for Quick C for Windows:

STACKING.C

#include <windows.h>long far pascal typecheck (long a, float b, short far *c, char far *buff){short retcode;a = a * 3;retcode = MessageBox(NULL, "I am in the DLL", "BOX", MB_OK);return (a);}				

STACKING.DEF

LIBRARY      STACKINGEXETYPE     WINDOWSSTUB        'winstub.exe'STACKSIZE   5120HEAPSIZE    1024DATA PRELOAD MOVEABLE SINGLE        ; ADD THESE TWO LINESCODE PRELOAD MOVEABLE DISCARDABLE   ; TO AVOID WARNINGS.EXPORTS    typecheck   @1     WEP           @2				

Add the following code to the general Declarations module in a VisualBasic for Windows form:
Declare Function typecheck Lib "d\stacking.dll" (ByVal a As Long,      ByVal b As Single, c As Integer, ByVal s As String) As Long				

NOTE: The above declaration must be placed on one line.

In the Form_Click event:
Sub Form_Click ()Dim a As Long   ' Explicitly type the variables.Dim b As SingleDim c As IntegerDim s As Stringa = 3                      ' Initialize the variables.b = 4.5c = 6s = "Hello there! We've been waiting for you!"Print typecheck(a, b, c, s)End Sub				

Running the program as written above will not generate the error. Nowadd the ByVal keyword before the variable named c in the Visual Basicfor Windows Declaration. Run the program. Note that the MessageBoxfunction pops a box first, and then the error box pops up indicatingthat Visual Basic for Windows checks the stack upon return to see ifit has been correctly adjusted. Because the DLL expected a 4-bytepointer and received a 2-byte value, the stack has not adjusted backto the initial frame.

As another test, first remove the ByVal keyword before the variable'c' that you added in the previous test. Declare the parameter 'a AsAny' instead of As Long. Change the type of the variable 'a' in theForm_Click to Integer. Run the program again. Using As Any turns offtype checking by Visual Basic for Windows. Because the program passedan integer ByVal instead of the long that the DLL expected, the stackframe is off and the error is generated.
REFERENCES
"Microsoft BASIC 7.0: Programmer's Guide" for versions 7.0 and 7.1,pages 423-426
2.00 3.00
Properties

Article ID: 85108 - Last Review: 12/04/2015 09:15:09 - Revision: 2.0

Microsoft Visual Basic 2.0 Standard Edition, Microsoft Visual Basic 3.0 Professional Edition, Microsoft Visual Basic 2.0 Professional Edition, Microsoft Visual Basic 3.0 Professional Edition, Microsoft Visual Basic 1.0 Standard Edition

  • kbnosurvey kbarchive KB85108
Feedback