Using MSTest With F#

Before creating this blog entry, I had always written tests against my F# projects in C#.  I prefer this approach because it helps ensure optimal interoperability between the F# libraries and other .NET languages.  However, it was always assumed that the tests could just as easily have been written in F#.  While this assumption is accurate for some of the available unit testing frameworks, it is not correct for the MSTest framework.

This post will focus on my attempt to identify a convenient method of creating F# test fixtures using the MSTest unit testing framework.  The following links provide examples using other unit testing frameworks (NUnit, FsTest  library for xUnit).

Creating My First F# MSTest Test Fixture:

To create the MSTest test fixture, a new F# library project was created and a reference was added to Microsoft.VisualStudio.QualityTools.UnitTestFramework.  Figure 1 shows the very simple MSTest test fixture. 

Figure 1

#light
namespace FSharpTests
open Microsoft.VisualStudio.TestTools.UnitTesting

[<TestClass>]
type FSharpTestsInFSharp = class
    [<testmethod>]
    member this.FSharpTests_CanGetCustomerByIdFromDB =
       Assert.IsTrue(1=1)
end 

This code compiled without issue and all seemed well until Ctrl+R,A was used to run all of the tests in the solution.  The Test Results screen appeared and all of the C# TestFixtures were represented (the new F# project was added to an existing solution, which contained a C# test project); however, my new fixture was no where to be seen.

Off to the Command Line:


This was very puzzling, so I decided to dig a little deeper by launching the test from the command line.  Figure 2 displays the basic syntax used (note: The actual location of the test project has been replaced with <project location>).  This test resulted in a message stating "No tests to execute".

Figure 2:

MSTest.exe /testcontainer:"<project location="">\bin\Debug\FSharpSpecifications.dll"

Now to MSIL:

To get a better idea of what was going on, I typed ildasm, opened FSharpSpecifications.dll, and dumped all of the IL code to an ANSI file named FSharpSpecifications.il.  After several minutes of review, it was discovered that though the TestMethodAttribute is defined in the property instance, it is not defined in the method instance.  Figure 3 displays the original IL along with the new line that allows MSTest to locate the test (note: The actual location of the test project has been replaced with <project location>).

Figure 3

//  Microsoft (R) .NET Framework IL Disassembler.  Version 3.5.30729.1
//  Copyright (c) Microsoft Corporation.  All rights reserved.
// Metadata version: v2.0.50727
.assembly extern /*23000001*/ mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 2:0:0:0
}
.assembly extern /*23000002*/ FSharp.Core
{
  .publickeytoken = (A1 90 89 B1 C7 4D 08 09 )                         // .....M..
  .ver 1:9:6:2
}
.assembly extern /*23000003*/ Microsoft.VisualStudio.QualityTools.UnitTestFramework
{
  .publickeytoken = (B0 3F 5F 7F 11 D5 0A 3A )                         // .?_....:
  .ver 9:0:0:0
}
.assembly /*20000001*/ FSharpSpecifications
{
  .custom /*0C000003:0A000001*/ instance void [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.FSharpInterfaceDataVersionAttribute/*01000002*/::.ctor(int32,
                                                                                                                                                    int32,
                                                                                                                                                    int32) /* 0A000001 */ = ( 01 00 01 00 00 00 09 00 00 00 06 00 00 00 00 00 ) 

  // --- The following custom attribute is added automatically, do not uncomment -------
  //  .custom /*0C000004:0A000002*/ instance void [mscorlib/*23000001*/]System.Diagnostics.DebuggableAttribute/*01000003*/::.ctor(valuetype [mscorlib/*23000001*/]System.Diagnostics.DebuggableAttribute/*01000003*//DebuggingModes/*01000004*/) /* 0A000002 */ = ( 01 00 01 01 00 00 00 00 ) 

  .hash algorithm 0x00008004
  .ver 0:0:0:0
}
.mresource /*28000001*/ public FSharpSignatureData.FSharpSpecifications
{
  // Offset: 0x00000000 Length: 0x000003E2
  // WARNING: managed resource file FSharpSignatureData.FSharpSpecifications created
}
.mresource /*28000002*/ public FSharpOptimizationData.FSharpSpecifications
{
  // Offset: 0x000003E8 Length: 0x000000CE
  // WARNING: managed resource file FSharpOptimizationData.FSharpSpecifications created
}
.module 'F#-Module-FSharpSpecifications'
// MVID: {492AF409-0E8B-9CF0-A745-038309F42A49}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x01140000

// =============== CLASS MEMBERS DECLARATION ===================

.class /*02000002*/ public auto ansi serializable beforefieldinit FSharpTests.FSharpTestsInFSharp
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
  .custom /*0C000005:0A000006*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute/*01000008*/::.ctor() /* 0A000006 */ = ( 01 00 00 00 ) 
  .custom /*0C000006:0A000007*/ instance void [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.CompilationMappingAttribute/*01000009*/::.ctor(valuetype [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.SourceConstructFlags/*0100000A*/) /* 0A000007 */ = ( 01 00 03 00 00 00 00 00 ) 
  .method /*06000001*/ public specialname rtspecialname 
          instance void  .ctor() cil managed
  // SIG: 20 00 01
  {
    // Method begins at RVA 0x2050
    // Code size       10 (0xa)
    .maxstack  3
    .language '{AF046CD3-D0E1-11D2-977C-00A0C9B4D50C}', '{994B45C4-E6E9-11D2-903F-00C04FA302A1}', '{5A869D0B-6611-11D3-BD2A-0000F80849BD}'
// Source File '<project location>\CustomerDaoTests.fs' 
    .line 31,31 : 13,15 '<project location>\CustomerDaoTests.fs'
//000031:     new() = {}
    IL_0000:  /* 02   |                  */ ldarg.0
    IL_0001:  /* 28   | (0A)000004       */ call       instance void [mscorlib/*23000001*/]System.Object/*01000001*/::.ctor() /* 0A000004 */
    IL_0006:  /* 02   |                  */ ldarg.0
    IL_0007:  /* 26   |                  */ pop
    IL_0008:  /* 00   |                  */ nop
    IL_0009:  /* 2A   |                  */ ret
  } // end of method FSharpTestsInFSharp::.ctor

  .method /*06000002*/ public instance void 
          get_FSharpTests_CanGetCustomerByIdFromDB() cil managed
  // SIG: 20 00 01
  {
    // Method begins at RVA 0x2068
    // Code size       13 (0xd)
    // This is the new line
    .custom /*0C000002:0A000003*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute/*01000006*/::.ctor() /* 0A000003 */ = ( 01 00 00 00 ) 
    .maxstack  4
    .line 39,39 : 9,27 ''
//000032:     [<TestMethod>]
//000033:     member this.FSharpTests_CanGetCustomerByIdFromDB = 
//000034: //        let customerDao = new CustomerDao() :> ICustomerDao
//000035: //        let customer = customerDao.GetByIdFromDB(2)
//000036: //        Assert.IsTrue(customer.Id = int(2))
//000037: //        Assert.IsTrue(customer.Name = "AABB, Inc")
//000038: //        Assert.IsTrue(customer.Balance = 29M)
//000039:         Assert.IsTrue(1=1)
    IL_0000:  /* 00   |                  */ nop
    IL_0001:  /* 17   |                  */ ldc.i4.1
    IL_0002:  /* 17   |                  */ ldc.i4.1
    IL_0003:  /* FE01 |                  */ ceq
    IL_0005:  /* FE14 |                  */ tail.
    IL_0007:  /* 28   | (0A)000005       */ call       void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.Assert/*01000007*/::IsTrue(bool) /* 0A000005 */
    IL_000c:  /* 2A   |                  */ ret
  } // end of method FSharpTestsInFSharp::get_FSharpTests_CanGetCustomerByIdFromDB

  .property /*17000001*/ instance class [FSharp.Core/*23000002*/]Microsoft.FSharp.Core.Unit/*01000005*/
          FSharpTests_CanGetCustomerByIdFromDB()
  {
    .custom /*0C000002:0A000003*/ instance void [Microsoft.VisualStudio.QualityTools.UnitTestFramework/*23000003*/]Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute/*01000006*/::.ctor() /* 0A000003 */ = ( 01 00 00 00 ) 
    .get instance void FSharpTests.FSharpTestsInFSharp/*02000002*/::get_FSharpTests_CanGetCustomerByIdFromDB() /* 06000002 */
  } // end of property FSharpTestsInFSharp::FSharpTests_CanGetCustomerByIdFromDB
} // end of class FSharpTests.FSharpTestsInFSharp

.class /*02000003*/ private abstract auto ansi sealed beforefieldinit '<StartupCode$FSharpSpecifications>'.$CustomerDaoTests
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
  .field /*04000001*/ static assembly native int _init
  .custom /*0C000001:0A000008*/ instance void [mscorlib/*23000001*/]System.Runtime.CompilerServices.CompilerGeneratedAttribute/*0100000B*/::.ctor() /* 0A000008 */ = ( 01 00 00 00 ) 
} // end of class '<StartupCode$FSharpSpecifications>'.$CustomerDaoTests

.class /*02000004*/ private abstract auto ansi sealed beforefieldinit '<PrivateImplementationDetails$FSharpSpecifications>'
       extends [mscorlib/*23000001*/]System.Object/*01000001*/
{
} // end of class '<PrivateImplementationDetails$FSharpSpecifications>'


// =============================================================

// *********** DISASSEMBLY COMPLETE ***********************

Running the Tests From the Altered DLL:

With the new line added to the IL file, I created a new dll using the command specificed in Figure 4 (note: The actual location of the test project has been replaced with <project location>), then ran the test using the command specified in Figure 2.  This time the test was identified and successfully run.

Figure 4

ilasm "<project location>\FSharpSpecifications.il" /DLL

Conclusion:

The point of this little experiment was to find a convenient way to implement test fixtures in F# using the MSTest unit testing framework.  Unfortunately, this goal was not reached, though a certain level of success was achieved.  It is likely that full support will be provided when F# is officially shipped with VS, but until that time we can always continue to create F# test fixtures in NUnit/xUnit or use my prefered approach and write the tests in C#.