Saturday, November 29, 2008

Using MSTest With F#

Before creating this blog entry, I had always written tests against my F# projects in C#.  I prefer this approach because it helps ensure optimal interoperability between the F# libraries and other .NET languages.  However, it was always assumed that the tests could just as easily have been written in F#.  While this assumption is accurate for some of the available unit testing frameworks, it is not correct for the MSTest framework.

This post will focus on my attempt to identify a convenient method of creating F# test fixtures using the MSTest unit testing framework.  The following links provide examples using other unit testing frameworks (NUnit, FsTest  library for xUnit).

Creating My First F# MSTest Test Fixture:

To create the MSTest test fixture, a new F# library project was created and a reference was added to Microsoft.VisualStudio.QualityTools.UnitTestFramework.  Figure 1 shows the very simple MSTest test fixture. 

Figure 1

#light
namespace FSharpTests
open Microsoft.VisualStudio.TestTools.UnitTesting

[<TestClass>]
type FSharpTestsInFSharp = class
    [<testmethod>]
    member this.FSharpTests_CanGetCustomerByIdFromDB =
       Assert.IsTrue(1=1)
end 

This code compiled without issue and all seemed well until Ctrl+R,A was used to run all of the tests in the solution.  The Test Results screen appeared and all of the C# TestFixtures were represented (the new F# project was added to an existing solution, which contained a C# test project); however, my new fixture was no where to be seen.

Off to the Command Line:


This was very puzzling, so I decided to dig a little deeper by launching the test from the command line.  Figure 2 displays the basic syntax used (note: The actual location of the test project has been replaced with <project location>).  This test resulted in a message stating "No tests to execute".

Figure 2:

MSTest.exe /testcontainer:"<project location="">\bin\Debug\FSharpSpecifications.dll"

Now to MSIL:

To get a better idea of what was going on, I typed ildasm, opened FSharpSpecifications.dll, and dumped all of the IL code to an ANSI file named FSharpSpecifications.il.  After several minutes of review, it was discovered that though the TestMethodAttribute is defined in the property instance, it is not defined in the method instance.  Figure 3 displays the original IL along with the new line that allows MSTest to locate the test (note: The actual location of the test project has been replaced with <project location>).

Figure 3

//  Microsoft (R) .NET Framework IL Disassembler.  Version 3.5.30729.1
//  Copyright (c) Microsoft Corporation.  All rights reserved.
// Metadata version: v2.0.50727
.assembly extern /*23000001*/ mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 2:0:0:0
}
.assembly extern /*23000002*/ FSharp.Core
{
  .publickeytoken = (A1 90 89 B1 C7 4D 08 09 )                         // .....M..
  .ver 1:9:6:2
}
.assembly extern /*23000003*/ Microsoft.VisualStudio.QualityTools.UnitTestFramework
{
  .publickeytoken = (B0 3F 5F 7F 11 D5 0A 3A )                         // .?_....:
  .ver 9:0:0:0
}
.assembly /*20000001*/ FSharpSpecifications
{
  .custom /*0C000003:0A000001*/ instance void [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.FSharpInterfaceDataVersionAttribute/*01000002*/::.ctor(int32,
                                                                                                                                                    int32,
                                                                                                                                                    int32) /* 0A000001 */ = ( 01 00 01 00 00 00 09 00 00 00 06 00 00 00 00 00 ) 

  // --- The following custom attribute is added automatically, do not uncomment -------
  //  .custom /*0C000004:0A000002*/ instance void [mscorlib/*23000001*/]System.Diagnostics.DebuggableAttribute/*01000003*/::.ctor(valuetype [mscorlib/*23000001*/]System.Diagnostics.DebuggableAttribute/*01000003*//DebuggingModes/*01000004*/) /* 0A000002 */ = ( 01 00 01 01 00 00 00 00 ) 

  .hash algorithm 0x00008004
  .ver 0:0:0:0
}
.mresource /*28000001*/ public FSharpSignatureData.FSharpSpecifications
{
  // Offset: 0x00000000 Length: 0x000003E2
  // WARNING: managed resource file FSharpSignatureData.FSharpSpecifications created
}
.mresource /*28000002*/ public FSharpOptimizationData.FSharpSpecifications
{
  // Offset: 0x000003E8 Length: 0x000000CE
  // WARNING: managed resource file FSharpOptimizationData.FSharpSpecifications created
}
.module 'F#-Module-FSharpSpecifications'
// MVID: {492AF409-0E8B-9CF0-A745-038309F42A49}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x01140000

// =============== CLASS MEMBERS DECLARATION ===================

.class /*02000002*/ public auto ansi serializable beforefieldinit FSharpTests.FSharpTestsInFSharp
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
  .custom /*0C000005:0A000006*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute/*01000008*/::.ctor() /* 0A000006 */ = ( 01 00 00 00 ) 
  .custom /*0C000006:0A000007*/ instance void [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.CompilationMappingAttribute/*01000009*/::.ctor(valuetype [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.SourceConstructFlags/*0100000A*/) /* 0A000007 */ = ( 01 00 03 00 00 00 00 00 ) 
  .method /*06000001*/ public specialname rtspecialname 
          instance void  .ctor() cil managed
  // SIG: 20 00 01
  {
    // Method begins at RVA 0x2050
    // Code size       10 (0xa)
    .maxstack  3
    .language '{AF046CD3-D0E1-11D2-977C-00A0C9B4D50C}', '{994B45C4-E6E9-11D2-903F-00C04FA302A1}', '{5A869D0B-6611-11D3-BD2A-0000F80849BD}'
// Source File '<project location>\CustomerDaoTests.fs' 
    .line 31,31 : 13,15 '<project location>\CustomerDaoTests.fs'
//000031:     new() = {}
    IL_0000:  /* 02   |                  */ ldarg.0
    IL_0001:  /* 28   | (0A)000004       */ call       instance void [mscorlib/*23000001*/]System.Object/*01000001*/::.ctor() /* 0A000004 */
    IL_0006:  /* 02   |                  */ ldarg.0
    IL_0007:  /* 26   |                  */ pop
    IL_0008:  /* 00   |                  */ nop
    IL_0009:  /* 2A   |                  */ ret
  } // end of method FSharpTestsInFSharp::.ctor

  .method /*06000002*/ public instance void 
          get_FSharpTests_CanGetCustomerByIdFromDB() cil managed
  // SIG: 20 00 01
  {
    // Method begins at RVA 0x2068
    // Code size       13 (0xd)
    // This is the new line
    .custom /*0C000002:0A000003*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute/*01000006*/::.ctor() /* 0A000003 */ = ( 01 00 00 00 ) 
    .maxstack  4
    .line 39,39 : 9,27 ''
//000032:     [<TestMethod>]
//000033:     member this.FSharpTests_CanGetCustomerByIdFromDB = 
//000034: //        let customerDao = new CustomerDao() :> ICustomerDao
//000035: //        let customer = customerDao.GetByIdFromDB(2)
//000036: //        Assert.IsTrue(customer.Id = int(2))
//000037: //        Assert.IsTrue(customer.Name = "AABB, Inc")
//000038: //        Assert.IsTrue(customer.Balance = 29M)
//000039:         Assert.IsTrue(1=1)
    IL_0000:  /* 00   |                  */ nop
    IL_0001:  /* 17   |                  */ ldc.i4.1
    IL_0002:  /* 17   |                  */ ldc.i4.1
    IL_0003:  /* FE01 |                  */ ceq
    IL_0005:  /* FE14 |                  */ tail.
    IL_0007:  /* 28   | (0A)000005       */ call       void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.Assert/*01000007*/::IsTrue(bool) /* 0A000005 */
    IL_000c:  /* 2A   |                  */ ret
  } // end of method FSharpTestsInFSharp::get_FSharpTests_CanGetCustomerByIdFromDB

  .property /*17000001*/ instance class [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.Unit/*01000005*/
          FSharpTests_CanGetCustomerByIdFromDB()
  {
    .custom /*0C000002:0A000003*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute/*01000006*/::.ctor() /* 0A000003 */ = ( 01 00 00 00 ) 
    .get instance void FSharpTests.FSharpTestsInFSharp/*02000002*/::get_FSharpTests_CanGetCustomerByIdFromDB() /* 06000002 */
  } // end of property FSharpTestsInFSharp::FSharpTests_CanGetCustomerByIdFromDB
} // end of class FSharpTests.FSharpTestsInFSharp

.class /*02000003*/ private abstract auto ansi sealed beforefieldinit '<StartupCode$FSharpSpecifications>'.$CustomerDaoTests
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
  .field /*04000001*/ static assembly native int _init
  .custom /*0C000001:0A000008*/ instance void [mscorlib/*23000001*/]System.Runtime.CompilerServices.CompilerGeneratedAttribute/*0100000B*/::.ctor() /* 0A000008 */ = ( 01 00 00 00 ) 
} // end of class '<StartupCode$FSharpSpecifications>'.$CustomerDaoTests

.class /*02000004*/ private abstract auto ansi sealed beforefieldinit '<PrivateImplementationDetails$FSharpSpecifications>'
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
} // end of class '<PrivateImplementationDetails$FSharpSpecifications>'


// =============================================================

// *********** DISASSEMBLY COMPLETE ***********************


Running the Tests From the Altered DLL:

With the new line added to the IL file, I created a new dll using the command specificed in Figure 4 (note: The actual location of the test project has been replaced with <project location>), then ran the test using the command specified in Figure 2.  This time the test was identified and successfully run.

Figure 4

ilasm "<project location>\FSharpSpecifications.il" /DLL

Conclusion:

The point of this little experiment was to find a convenient way to implement test fixtures in F# using the MSTest unit testing framework.  Unfortunately, this goal was not reached, though a certain level of success was achieved.  It is likely that full support will be provided when F# is officially shipped with VS, but until that time we can always continue to create F# test fixtures in NUnit/xUnit or use my prefered approach and write the tests in C#.

4 comments:

  1. Not sure if the source is right, but your member is defined as a property, not a method. Adding a unit parameter () to the property might fix it. I.e.

    member x.Foo = 1 // property
    member x.Foo() = 1 // method

    ReplyDelete
  2. Thanks Michael! Changing the member from a property to a method did allow the test to be found when running from the command-line (though the test did not pass from some reason). In addition, the test still doesn't show up in the Test Results screen. This seems very strange. Thanks again for providing guidance on the property to method correction.

    ReplyDelete
  3. I've been looking into this myself and found the following in the VS 2010 RTM:

    You need to specify a unit parameter for the class (otherwise the class will have no public constructor, which is probably why your test failed) and as Michael mentioned you also need to make sure your test is exposed as a method (not a property).

    Unfortunately even with these changes you won't see the test appear in the test view in Visual Studio. This is because it only looks at projects that have been marked as containing unit tests (you'll see the same thing if you create a normal C# class library, reference the unit testing assembly and try to view the tests). However, unlike a C# project it's not simply a case of adding the appropriate project type guid to the project file (adding it actually stopped the project from loading).

    Of course you can still run the tests from the command line (and probably also Team Build).

    Here's an example of a simple MSTest in F#:

    open Microsoft.VisualStudio.TestTools.UnitTesting

    [<TestClass>]
    type BasicTests() =
        [<TestMethod>]
            member v.FirstTest() =
                Assert.AreEqual(1, 1)

    ReplyDelete
  4. Not sure if the source is right, but your member is defined as a property, not a method. Adding a unit parameter () to the property might fix it. I.e.

    member x.Foo = 1 // property
    member x.Foo() = 1 // method

    ReplyDelete