F# introduces Record Types and Descriminated Unions to the .NET universe. Although I knew that these types eventually boiled down to reference types, I wanted to make sure that nothing funny was going on behind the scenes when they got instantiated. The results weren’t terribly exciting, but I learned some important things about performance testing along the way.
First Attempt
The first thing I did was create a few basic types:
[<Struct>]
type StructWithImplicitConstructor =
val x : int
val y : int
[<Struct>]
type StructWithExplicitConstructor =
val x : int
val y : int
new (one, two) = {x = one; y = two}
[<Struct>]
type StructWithObjectConstructor(x:int, y:int) =
member this.X = x
member this.Y = y
type ObjectWithImplicitConstructor =
val x : int
val y : int
type ObjectWithExplicitConstructor =
val x : int
val y : int
new (one, two) = {x = one; y = two}
type ObjectWithObjectConstructor(x:int, y:int) =
member this.X = x
member this.Y = y
type RecordType =
{ x:int;
y:int }
type UnionType =
| X of int
| Y of int
Then, I broke out a System.Diagnostics.Stopwatch and started testing how long it took to new up these objects. My first results varied wildly. JITer optimizations and garbage collection behaviors yielded bizarre and inconsistent results. After talking this over with coworkers Bill Wagner and Jay Wren, I made a second pass.
Getting Smarter
This time, I started with a more robust performance testing module:
module PerformanceTesting =
let Time func =
let stopwatch = new Stopwatch()
stopwatch.Start()
func()
stopwatch.Stop()
stopwatch.Elapsed.TotalMilliseconds
let GetAverageTime timesToRun func =
Seq.init_infinite (fun _ -> (Time func))
|> Seq.take timesToRun
|> Seq.average
let TimeOperation timesToRun =
GC.Collect()
GetAverageTime timesToRun
let TimeOperations funcsWithName =
let randomizer = new Random(int DateTime.Now.Ticks)
funcsWithName
|> Seq.sort_by (fun _ -> randomizer.Next())
|> Seq.map (fun (name, func) -> name, (TimeOperation 100000 func))
let TimeOperationsAFewTimes funcsWithName =
Seq.init_infinite (fun _ -> (TimeOperations funcsWithName))
|> Seq.take 50
|> Seq.concat
|> Seq.group_by fst
|> Seq.map (fun (name, individualResults) -> name, (individualResults |> Seq.map snd |> Seq.average))
The above code makes it easy to time operations with greater accuracy. The TimeOperationsAFewTimes function takes a sequence of tuples containing the names of operations to test and the operations themselves. It runs the operations a few times and returns a sequence of tuples that contains the operation names and the average run time for each operation.
There are two important lessons for performance testing to take away from the example. First, the garbage collector is run between tests to make sure that each operation has a clean memory slate. Second, the order that the operations run in is varied between runs in order to help negate the effect of JITer optimizations in our tests.
(As an aside, the TimeOperationsAFewTimes function was really fun code to write. It was one of those moments that made me stop and take notice at the power of the F# language. I realize that the code might be a little difficult to read at first glance, though.)
Results
With the above functions, performance testing became a breeze:
printfn "Starting Tests..."
printfn ""
let tmpRec = {x = 0; y = 50}
PerformanceTesting.TimeOperationsAFewTimes
[ "RecordType", (fun () -> {x = 1; y = 50}|> ignore)
"StructWithImplicitConstructor", (fun () -> new StructWithImplicitConstructor() |> ignore)
"ObjectWithObjectConstructor", (fun () -> new ObjectWithObjectConstructor(1, 50) |> ignore)
"ObjectWithExplicitConstructor", (fun () -> new ObjectWithExplicitConstructor(1, 50) |> ignore)
"StructWithObjectConstructor", (fun () -> new StructWithObjectConstructor(1, 50) |> ignore)
"StructWithExplicitConstructor", (fun () -> new StructWithExplicitConstructor(1, 50) |> ignore)
"CreatingUsingWith", (fun () -> { tmpRec with x = 1 } |> ignore)
"UnionType", (fun () -> X 14 |> ignore)]
|> Seq.sort_by (fun (name, time) -> time)
|> Seq.iter (fun (name, time) -> printfn "%s: %f" name time)
printfn ""
printfn "Press Any Key To Continue..."
let tmp = System.Console.ReadKey()
The above code passes a list of tupled names and lamdas to the performance testing functions, sorts them from least to most expensive and prints the results.
Here are the results of a few runs:
Starting Tests...
CreatingUsingWith: 0.001973
StructWithExplicitConstructor: 0.001988
StructWithImplicitConstructor: 0.001992
UnionType: 0.002030
RecordType: 0.002064
ObjectWithExplicitConstructor: 0.002066
ObjectWithObjectConstructor: 0.002081
StructWithObjectConstructor: 0.002092
Press Any Key To Continue...
Starting Tests...
StructWithImplicitConstructor: 0.001407
CreatingUsingWith: 0.001408
RecordType: 0.001423
StructWithExplicitConstructor: 0.001436
ObjectWithObjectConstructor: 0.001438
UnionType: 0.001438
StructWithObjectConstructor: 0.001449
ObjectWithExplicitConstructor: 0.001455
Press Any Key To Continue...
Starting Tests...
RecordType: 0.001365
UnionType: 0.001374
StructWithImplicitConstructor: 0.001380
CreatingUsingWith: 0.001381
StructWithObjectConstructor: 0.001385
StructWithExplicitConstructor: 0.001386
ObjectWithExplicitConstructor: 0.001397
ObjectWithObjectConstructor: 0.001404
Press Any Key To Continue...
The Moral of The Story
The bottom line was that there were not any meaningful differences in the amount of time required to instantiate an object using any of the methods that I tried.
At first, my C and C++ roots were dumbfounded that value types didn’t beat the pants off of everything else. After Thinking critically, I realized that this makes a lot of sense in the managed world because the cost of creating reference types doesn’t include a hefty price for heap allocation.
The other thing that I proved to myself was that Record Types and Discriminated Unions are just as inexpensive to create as any other native .NET type.