Please support our sponsor:
DirectX4VB.Com - All You Need For Multimedia Visual Basic Programming

Main Site Links Resources Tutorials
News VB Gaming Code Downloads DirectX 7
Contact Webmaster VB Programming Product Reviews DirectX 8
  General Multimedia Articles DirectX 9
      Miscellaneous

 

VB.Net versus VB6 performance tests
Author: Jack Hoxley
Written: 9th July 2002
Contact: [EMail]
Download: Complete Archive (54kb)


Contents of this lesson
1. Introduction
2. Maths Performance
3. Memory Performance
4. Further Performance Tests


1. Introduction

As many of you will be aware already - in February 2002 I wrote a review of Visual Studio .Net for this website (you can read it here). In this review I discussed the performance of the new compiler, and came to the following conclusion:

"I will be happy to hold up my hand and admit my fault should my values be proven to be wrong - I still find it surprising that I got those results, and would quite like to be proven wrong! .Net isn't universally slower than the previous versions, as I have found a few cases where it is a bit faster, but that doesn't hold much in it."

At the time of writing that review I'd only been using the .Net languages for a couple of weeks - not really enough time to become particularly skilled with it. However, in the months since then I've done a little more work and a little more research (thanks to Patrice Scribe for getting the ball rolling lately!) and found that there were flaws in the code I used and that the times quoted were therefore misleading.

The main problem with my previous quoted times was that I was using VB.Net to program in a VB6 way - I've used VB6 for 4 years now, and because of obvious parallels I continued writing my code in this way. This is not a good strategy for using .Net; to really get the most (performance and features) out of the .Net frameworks you can't always program in the same way as you would have done before.

So, it's come to this. I decided to write this article to set the record straight - I got it wrong first time around, and hopefully I won't get it wrong this time around. There is no way to conclusively prove and/or test all aspects of performance, but I shall instead cover a few examples that I think are representative of multimedia programming.


2. Maths Performance

Maths is critical to how a multimedia program will perform, the reliance on maths is huge - and therefore even minor speed differences in this area can have a significant impact over all. This section is divided into two sections, the first analyzing the core maths functions (those built into the standard libraries), and the second being more complex algorithms using these functions / performing more complex functions.

2.1 Core Math Functions:
The core math functions to be tested are shown in the following listing:

'//VB.Net functions:
System.Math.Sqrt()   
'Square Root
System.Math.Sin()     
'Sine
System.Math.Acos()   
'Inverse Cosine
System.Math.Atan()   
'Inverse Tangent
System.Math.Asin()   
'Inverse Sine
System.Math.Log()    
'Natural Logarithm
System.Math.Exp()    
'Exponential (ex)
System.Math.Log10() 
'Logarithm to base 10

'//VB6 functions:
'all are equivalent of the above list, but some require addition derivation

Sqr()
Sin()
ArcCos()
Atan()
ArcSin()
Log()
Exp()
Log10()

'the following 3 functions had to be derived:
Private Function ArcCos(A As Double) As Double
    ArcCos = Atn(-A / Sqr(-A * A + 1)) + 2 * Atn(1)
End Function

Private Function ArcSin(X As Double) As Double
    ArcSin = Atn(X / Sqr(-X * X + 1))
End Function

Private Function Log10(X As Double) As Double
    Log10 = Log(X) / LogOf10
'made Log(10) a constant. faster!
End Function

Each of these functions was taken in turn, and put into a simple framework looking like that in the following listing. Each test was iterated 1 million times, and timed using the high performance counter. This overall time was then divided by 1 million to get an average time for each call - in the case of maths functions these are quoted in nanoseconds. Input variables for the maths functions were the loop variables, using a constant would have given the compiler too much opportunity to optimize away the entire test, and using random numbers proved to be misleading (the times represented the performance of the random number generator AND the math function).

'//VB.Net framework:
'//Note: QueryPerformanceCounter() isn't in the .Net framework. The closest
'//possible is System.Environment.TickCount(), but after extensive testing I
'//decided I couldn't trust its accuracy.

bRet = QueryPerformanceCounter(i64Start)
For I = 0 To nTests - 1
        rVal = Asin(I / nTests)
Next

bRet = QueryPerformanceCounter(i64End)
bRet = QueryPerformanceFrequency(i64Frq)
rVal = ((i64End - i64Start) / i64Frq) / nTests


'//VB6 framework:
lRet = QueryPerformanceCounter(iStart)
For I = 0 To nTests - 1
        rVal = ArcSin(I / nTests)
Next
lRet = QueryPerformanceCounter(iEnd)
lRet = QueryPerformanceFrequency(iFrq)
rVal = ((iEnd - iStart) / iFrq) / nTests

Both programs were then compiled using the highest level of optimization possible (optimize for speed), and the system was restarted (to eliminate any overheads). Each program was then run 10 times to generate a list of times for each function. For reference, the test system used was a 700mhz AMD Athlon Thunderbird with 288mb PC100 RAM and Windows XP Professional.

The times recorded (and averaged over the 10 tests) are shown in the following table:

Test Name Visual Basic 6 Time (ns) Visual Basic .Net Time (ns)
Square Root 46.7791 34.4608
Sine 160.4986 162.5435
Inverse Cosine 608.3858 375.6694
Inverse Tangent 259.7525 286.1751
Inverse Sine 405.3237 401.6973
Natural Logarithm 194.9694 226.1549
Exponential 121.0084 193.8514
Logarithm to base 10 209.4735 230.0629

As you can see straight away its a fairly mixed set of results, I've highlighted the tests in red where .Net is slower, and in green where .Net is faster. Currently it's 4 against and 4 for - tiebreak. Three of the tests where .Net wins can be explained quite easily - they are the three function that had to be derived in VB6. This means that there is the function-calling overhead to consider, which will generate some slowdown. However, if you were to perform some high-level optimization of the VB6 code, you can change the times to look like this:

Test Name Visual Basic 6 Time (ns) Visual Basic .Net Time (ns)
Inverse Cosine 435.8120 375.6694
Inverse Sine 427.40923 401.6973
Logarithm to base 10 203.6077 230.0629

In doing some rather trivial optimizations on the code (making them inline, adding constants where applicable) you can reduce the times quite significantly (although Inverse Sine is an exception). However, whilst inverse cosine (for example) got 1.4x faster, it was not enough to catch up with VB.Net's initial performance value.

2.2 Applied Mathematics
This next section of maths related statistics covers applied maths - that is, helper functions / algorithms that are highly mathematically orientated. Anyone who's done any 3D graphics programming (Using Direct3D for example) will know that in every frame alone there can be a huge number of complicated equations and algorithms to process. These aren't necessarily all related to using the core maths functions shown above, but they will feature.

The reason for including this section is that while the core maths functions are regularly used, it's these real world examples that are more common, and any speed difference here is likely to have a far bigger impact on real-time performance than anything else.

Creating a 4x4 transformation matrix
Essential to all 3D graphics is the transformation matrix. This is typically constructed using a series of helper functions (exposed through the D3DX libraries in DirectX8). A while back (in this tutorial) I proposed a faster function for generating a complete transformation matrix. Basically, the source code takes in rotation, scaling and translation parameters and returns a properly generated 4x4 matrix. The source code is fairly boring, so I didn't bother inserting it in the main article text, but you can find it in the downloadable archive.

The times for creating a complete matrix:

Test Name VB6 Times (ns) VB.Net Times (ns)
CreateMatrix( ) 1265.5043 1214.1316

The times are still biased in VB.Net's favor, but the speed difference is much less than that shown in earlier tests. However, the difference shown here of 51.3727 nanoseconds would translate into VB.Net being able to process the function 33,435 times more per-second than VB6 could. Small difference, big result.

Finding the inverse of a 4x4 transformation matrix
Inverse matrices aren't used as heavily as normal transformation matrices, but they are the only method for doing certain very useful techniques (such as 'picking' 3D objects based on a 2D mouse coordinate). Inverse Matrices by their very nature are quite slow; so this speed test is crucial.

As with the CreateMatrix() test, the source code is not presented here for clarity, you can find it in the downloadable archive. Thanks to Eric Coleman for letting me use his optimized inverse matrix function.

Test Name VB6 Times (ns) VB.Net Times (ns)
InverseMatrix( ) 843.4019 1009.7311

The times shown here are very much in VB6's favor. This particular function uses only *, /, + and - operations which would eliminate any of the performance issues with core maths functions shown above.

Normalizing a Vector
This is a rather trivial piece of code - but is often essential to many other vector-based calculations. This test represents the normalization of a 3D vector generated using random numbers in the interval [-5,+5] for each component (x,y,z). The list of vectors were generated separately so as not to introduce any additional variables into the final, timed, test. The second time represents an inline'd version of the code - so as to remove any function-calling overheads.

Test Name VB6 Times (ns) VB.Net Times (ns)
Normalize Vector (function) 194.8496 165.7895
Normalize Vector (inline) 183.5613 209.2288

What is most odd about the tabulated results above is that VB.Net produces a slower time for the inline version of the code. It is traditionally assumed that inline functions (of a similar, small, size) will run faster - as is shown by VB6. Overall, VB.Net has produced the fastest possible time - 165ns is considerably faster than anything VB6 can offer.

Ray-Triangle Intersection Tests
This is a very useful test to perform - essentially this can be considered as a ray-tracing algorithm for triangulated/tessellated geometry. It can also be used quite effectively for collision detection between two 3D meshes. Ray-traced graphics often produce the best lighting/rendering but cannot as yet be used in real-time purposes - instead people often use ray-traced light maps in games.

Test Name VB6 Times (ns) VB.Net Times (ns)
Ray Triangle Intersection 610.4232 733.7162

The code involved in this algorithm is very complicated - and makes use of several nested-functions. It would appear though, that VB6 has the upper hand again.

2.3 Summary

This next table is a final summary of all the times collected for the math functions, this time including a percentage difference to compare the two compilers.

Test Name VB6 Times (ns) VB.Net Times (ns) VB.Net is...
Square Root 46.7791 34.4608 1.36x faster
Sine 160.4986 162.5435 1.01x slower
Inverse Cosine 435.8120 375.6694 1.16x faster
Inverse Tangent 259.7525 286.1751 1.10x slower
Inverse Sine 405.3237 401.6973 1.01x faster
Natural Logarithm 194.9694 226.1549 1.16x slower
Exponential 121.0084 193.8514 1.60x slower
Logarithm to base 10 203.6077 230.0629 1.13x slower
CreateMatrix( ) 1265.5043 1214.1316 1.04x faster
InverseMatrix( ) 843.4019 1009.7311 1.20x slower
Norm. Vec. Function 194.8496 165.7895 1.18x faster
Norm. Vec. Inline 183.5613 209.2288 1.14x slower
Ray-Triangle Intersect 610.4232 733.7162 1.20x slower

There are the final results - VB.Net wins on 5 counts, VB6 wins on 8 counts. Whilst that does make VB6 the "winner" in most senses, there isn't a huge amount in it - VB.Net is on average 1.034x slower (it performs at 96.7% of VB6's speed). As you'll find later on, you can get the source code used here for your own inspection should you not believe the data presented.

Whilst this is not the focus of this article, I've included a quick summary of the above data but for C++; in particular VC++6 and VC++.Net compilers.

Test Name VC++6 (ns) VC++.Net (ns) VC++.Net is...
Square Root 98.8819 36.5847 2.70x faster
Sine 246.7142 176.8138 1.40x faster
Inverse Cosine 179.9197 222.6877 1.24x slower
Inverse Tangent 96.3011 79.1565 1.22x faster
Inverse Sine 166.4979 162.1950 1.03x faster
Natural Logarithm 204.7946 190.4992 1.08x faster
Exponential 171.0973 150.2244 1.14x faster
Logarithm (base 10) 204.6446 190.3695 1.07x faster
CreateMatrix( )  1059.0915 1048.2232 1.01x faster
InverseMatrix( ) 843.6558 826.6082 1.02x faster
Norm. Vec. Func. 124.7684 105.0628 1.19x faster
Ray-Triangle Test 4728.2339 2084.6925 2.27x faster

Bare in mind that the C++ language is an international standard - therefore the source code used for each compiler was identical. The timing differences here really should be only down to the differences in compilers. Given these results it would imply that the VisualC++.Net compiler is capable of producing marginally faster code. The above results show, that on average, the .Net compiler generates code 1.33x faster.


3. Memory Performance

Memory allocation and processing.
Lets take an arbitrary example whereby a function must allocate a significant amount of memory and then perform a process on it. Similar in many respects to directly accessing the memory behind a surface or texture in Direct3D/Draw. The test code looks like this:

'//DESCRIPTION: This sample simulates generating a simple point-light based lightmap. The source code is highly optimised (thanks to Eric Coleman for pointing a few things out!). For example, the use of square roots (for distance calculations) have been optimized away - giving a significant speed boost for both languages.

'//VB.Net Source Code

Private
Sub GenerateLightMap(ByVal LightR As Integer, ByVal LightG As Integer, _
                     
ByVal LightB As Integer, ByVal cX As Integer, ByVal cY As Integer
)

'we take this array as being the equivalent of a 640x480x32 surface
Dim Data(640, 480) As Integer
Dim
X As Integer, Y As Integer
Dim
Dist As Integer, Attenuation As Double
Dim
pxR As Integer, pxG As Integer, pxB As
Integer

For X = 0 To 639
   
For Y = 0 To
479

        '1. work out the distance from this pixel to the light.
        Dist = ((X - cX) * (X - cX)) + ((Y - cX) * (Y - cX))

        'if the distance is within 200px then we consider it to be lit.
        If Dist <= 40000 Then
            '2. We now need to work out the attenuation constant
            '(I'm just gonna use standard linear interpolation)

            Attenuation = 1 - (Dist * 0.000025)

            '3. We now need to work out what colour this pixel will be.
            pxR = LightR * Attenuation
            pxG = LightG * Attenuation
            pxB = LightB * Attenuation

            '4. we now need to pack the colours into a 32bit ARGB
            Data(X, Y) = (pxR * 65536) Or (pxG * 256)
Or
pxB

        End If
    Next Y
Next X

End Sub

'//VB6 Source Code
Private Sub GenerateLightMap(LightR As Long, LightG As Long, _
                     LightB
As Long, cX As Long, cY As Long)

'we take this array as being the equivelent of a 640x480x32 surface
Dim Data(0 To 639, 0 To 479) As Long
Dim X As Long, Y As Long
Dim Dist As Integer, Attenuation As Double
Dim pxR As Long, pxG As Long, pxB As Long

For X = 0 To 639
   
For Y = 0 To 479
       
'1. work out the distance from this pixel to the light.
        Dist = ((X - cX) * (X - cX)) + ((Y - cY) * (Y - cY)))

       
'if the distance is within 200px then we consider it to be lit.
       
If Dist <= 40000 Then
           
'2. We now need to work out the attenuation constant
            '(I'm just gonna use standard linear interpolation)

            Attenuation = 1 - (Dist * 0.000025)

           
'3. We now need to work out what colour this pixel will be.
            pxR = LightR * Attenuation
            pxG = LightG * Attenuation
            pxB = LightB * Attenuation

           
'4. we now need to pack the colours into a 32bit ARGB
            Data(X, Y) = (pxR * 65536)
Or (pxG * 256) Or pxB
       
End If
   
Next Y
Next X

End Sub


The times for the memory allocation and processing tests:

Test Name VB6 Times (ms) VB.Net Times (ms)
Memory Allocation 39.7021 17.6280

VB.Net's version of the code is 2.33x faster than VB6. This would indicate that VB.Net, the CLR and/or the .Net frameworks have a far superior memory manager.

In real-world terms, should you wish to use this process/technique to generate lighting in your games the two times listed above correspond to ~24fps and ~58fps (maximum possible). The former being VB6's time, which while interactive would prove to be quite jerky; the latter being VB.Net's which would be very smooth and pose no real problem to interactivity.

The next table shows the C++ compiler performance for the same code.

Test Name VC++6 Times (ms) VC++.Net Times (ms)
Memory Allocation 28.9813 21.7283

 


4. Further Performance Tests

The tests shown in this article can only represent a small percentage of real-world applications. For multimedia work, the best possible test would be to compare the performance of DirectX through .Net and VB6 - real frame rates and per-frame timing can then be extracted, and actually mean something.

However, this test is not going to happen any time soon. DirectX 9 (at time of writing) is far from being ready for release, but does include a managed-code core, such that it can be properly tested using the .Net languages. BUT, DirectX 9 will not support VB6, so it is impossible to directly compare VB6 and .Net's DirectX performance. The closest we can get is to compare DirectX 8 and DirectX 9 - but this adds additional variables into the system, the biggest one being that test results will reflect any difference in DirectX 9 and DirectX 8 and not the actual language differences.

Ignoring DirectX for a moment - the difference between both languages would be best judged by comparing a much larger program. One example that I would like to have done is a landscape ray-tracer. This type of algorithm can take upwards of an hour to complete, and relies extremely heavily on maths and memory manipulation. Unfortunately this is a rather complex program that would take quite a while to get working properly and comparably in both languages - something I didn't have time to do when writing this article. Maybe in the future.

As mentioned throughout this article, you can download the source code and examine it in your own spare time - generate results specific to you computer etc... The source code is available here, or from the top of the page.

Finally, if you think you have significantly sized code that you think makes a good comparison (regardless of the overall results) between VB.Net and VB6 please email me about it - Jack.Hoxley@DirectX4VB.Com , even if you don't want to share the code!

DirectX 4 VB © 2000 Jack Hoxley. All rights reserved.
Reproduction of this site and it's contents, in whole or in part, is prohibited,
except where explicitly stated otherwise.
Design by Mateo
Contact Webmaster
This site is hosted by Exhedra Solutions, Inc., the parent company of RentACoder.com and PlanetSourceCode.com