Monday, May 17, 2010

VB.Net to interop COM dll call failed for variant/object

 I converted one ASP application to VB.Net using Microsoft ASP migration assistant utility. It was interesting
to see that the utility created all the required interop dlls for my unmanaged COM components. so I saved my time in
running tlbimp on each COM/EXE/DLL(unmanaged) or by using Visual studio. Cool Eh!!
I found one very interesting and quite unexplored issue (atleast i didnot get much on googling Smile ) that was                                                                                           my call from managed environment to unmanaged got failed for no obvious reason. Prima facie, I got feeling
that it happened due to some wrong marshalling of data which eventually proved true.
Let me staright jump into the issue by one example and explain its fix.
my backend (C++) has one API implementation something like this
                        /*[in]*/   VARIANT*                     vSomeInVaiant,
                        /* [out] */ VARIANT*             pvSomeOutVariant,
                        /* [in] */ BOOL bSomeBoolean,
                         BOOL*                         bRetVal )

my default VB.Net code (after conversion) was something like this,
blnRetVal = CFoo.CreateMyList(vSomeInVaiant, vSomeOutVariant, bSomeBoolean)
this call was ending up in some unhandled exception resulting a crash in application. To dig more I disassembled
my interop. I found the intermediate signature in my interop dll something like this
.method public hidebysig newslot abstract virtual
        instance int32  CreateMyList([in] object  marshal( struct) vSomeInVaiant,
                                            [out] object&  marshal( struct) pvSomeOutVariant,
                                             [in] int32 bSomeBoolean,
                                             ) runtime managed internalcall
  .custom instance void [mscorlib]System.Runtime.InteropServices.DispIdAttribute::.ctor(int32) = ( 01 00 03 00 00 00 00 00 )
} // end of method IFoo:::CreateMyList
 when I hardcoded vSomeInVaiant in backend, everything worked fine. that meant the runtime marshaller is not able to do
 his job properly for in variant. I donot owe to whatever fix I am going to tell you now.

 I read it somewhere that by default VB.Net passes everything byref so I needed a way to pass it byVal. I found a workaround.
 you enclose your variant (I'm sorry Object in managed env.) inside parantheses. that will make your runtime to evaluate it
 as an expression first and then it will be forced to pass byVal.

 the next thing i did was

 blnRetVal = CFoo.CreateMyList((vSomeInVaiant), vSomeOutVariant, bSomeBoolean)

 believe me the trick worked seamlessly. I applied this trick everywhere in my project and it worked.

 To summarize it,

 pass in variant enclosed in parantheses
 pass out variant as it is
 pass in/out variant ( never tried so far Frown ), you need to let me know.

 I am open to more discussion on this. I will be obliged for any comment.

No comments:

Post a Comment