COSBI OpenSourceMark Coding Documentation

How to Add a New Test

Note: Updates to the version number should be coordinated with the COSBI committee.

Add Enumerated Type Values

All quick tests must have an enumerated type value to programatically identify each test. This value is specified in the unit uCOSBI_TTest. The enumerated type is named TQTests.

Decide what you want to call the test using the following convention: qtAaaaa. Add this new value to the next to last entry for TQTests. This will place the new value immediately before qtLastTest which must always be the last value. Also, qtFirstTest must always be the first value for TQTests.

If the test represents a new test type, you will also need to modify TTestType; however, this should rarely be necessary since most test types have already been listed. Precede new test type names with the prefix “tt”.

Add a New Checkbox

On the group box gbSelectTests place a checkbox control for the new test. Please observe the established naming convention as follows: “cbox Aaaa” where Aaaa is descriptive of the test. For example, the checkbox control for the Maze test is named cboxMaze.

At the end of the "write tests" section of procedure TfrmOSMark.ReadIniFile add a line for the new test:

cboxMyNewTest.Checked := Ini.ReadBool( 'Tests', 'MyNewTest', TRUE );

And in the corresponding location in routine procedure TfrmOSMark.WriteIniFile add:

Ini.WriteBool( 'Tests', 'MyNewTest', cboxMyNewTest.Checked );

Next, append a single line of code for the new checkbox at the end of the function TfrmQuickTest.GetSelectedTests. If the checkbox is checked, the code will add the test to the set of selected tests. The format of this line is:

if cboxAaaaa.Checked then result := result + [qtAaaaaa];

For example:

if cboxIThreads.Checked then result := result + [qtIdenticalThreads];

Modify OSMark Test Suite

In the unit TOSMarkTestSuite, the procedure TOSMarkTestSuite.TestSuiteRun will have to be edited to accommodate the new test.

if qtAaaaa in fSelectedTests then begin
if qtAaaaa in fSelectedTests then begin
lTest := TAaaaa.Create;
try
RunTest( lTest, aiRunCount );
finally
FreeAndNil( lTest );
end;
end; // if qtAaaaa

If the test is to become part of the official run, its enumerated type value must be appended to the set type OFFICIAL_RUN_TESTS located in the TOSMarkTestSuite unit.

Writing the Test

All quick tests are descendants of the base class TTest, which contains the basic framework for accurately timing any code called from the procedure RunTest. Features such as the status window will be inherited automatically by all of the base class's descendents.

TTest's interface follows.

TTest = class( TObject )

private

protected

fRunCount : integer;

fTotalTime : double;

fMaxTime : double;

fMinTime : double;

fLastTime : double;

fReferenceTime : double;

fReferenceTime2 : double;

fStopWatch : TStopWatch;

fTestName : string;

fTestDescription : string;

fTestVersion: string;

fTestType : TTestType;

fQTestType : TQTests;

fTestAuthor : string;

ffrmStatus : TfrmStatus;

fShowStatus : Boolean;

fOutputForm : TfrmOutput;

function GetAverageTime : double;

function GetTestName : string; Virtual;

function GetTestDescription : string; Virtual;

procedure CreateStatusFull(ai_PauseMilliSec : integer);

procedure SetRunCount( Value : integer );

public

constructor Create; Overload;

function GetFormattedMinTime : string;

function GetScore(ai_index : integer) : integer;

procedure Run; Virtual;

procedure Clear; Virtual;

procedure BeforeTest; Virtual;

procedure RunTest; Virtual;

procedure AfterTest; Virtual;

property RunCount: integer read fRunCount write SetRunCount;

property TotalTime: double read fTotalTime;

property MaxTime: double read fMaxTime;

property MinTime: double read fMinTime;

property LastTime: double read fLastTime;

property StopWatch: TStopWatch read fStopWatch write fStopWatch;

property AverageTime : double read GetAverageTime;

property TestName : string read GetTestName write fTestName;

property TestDescription : string read GetTestDescription write fTestDescription;

property TestVersion : string read fTestVersion write fTestVersion;

property TestType : TTestType read fTestType write fTestType;

property TestAuthor : string read fTestAuthor write fTestAuthor;

property ShowStatus : Boolean read fShowStatus write fShowStatus;

property QTestType : TQTests read fQTestType write fQTestType;

property OutputForm : TfrmOutput read fOutputForm write fOutputForm;

end; // TTest ................................................

Tests that utilize the output form for graphics are descended from TGraphicsTest which is based on TTest and is contained in the unit uCOSBI_TGraphicsTest. Its interface follows.

TGraphicsTest = class( TTest )

private

protected

fPlotStep : integer;

fPaintBox32 : TPaintBox32;

sx : double;

sy : double;

ox : double;

oy : double;

loop_step : double;

public

procedure AfterTest; Override;

procedure BeforeTest; Override;

procedure Clear; Override;

procedure Plot(x, y: double; PlotColor: TColor); Virtual;

procedure PlotAndBuffer(x, y: double; PlotColor: TColor;

BufferColor: TColor32); Virtual;

procedure PlotScale(x_left, x_right, y_bottom, y_top : double); Virtual;

procedure ClearOutput; Virtual;

property PlotStep: integer read fPlotStep write fPlotStep;

property PaintBox32: TPaintBox32 read fPaintBox32 write fPaintBox32;

end; // TGraphicsTest ..............................................

Tests that use only a small amount of code can be added directly to the unit uTests. More complicated tests should be constructed in their own unit. The uses section for independent files will typically contain the following units:

Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs, ExtCtrls, StdCtrls, Grids, COSBI_Common, GR32_Blend, GR32_Image, GR32, ComCtrls, Gauges, uCOSBI_TTest, uCOSBI_TGraphicsTest, uOutput;

For simple tests that are to be added to uTests, you can use the following template for the interface:

// Class name: TAaaaaTest

// Author: Thomas Jefferson

// Date: January 16, 2004

TAaaaaTest = class( TTest )

private

fFieldVariable1 : integer;

protected

procedure MyProcedure;

function MyFunction(aiValue : integer);

public

constructor Create; Overload;

procedure BeforeTest; Override;

procedure AfterTest; Override;

procedure RunTest; Override;

end; // TAaaaaTest...............................................

Of course, not all tests will need the BeforeTest and AfterTest procedures, but some tests will require more features of TTest than is listed above.

The complexity of your test is up to you, but one of the key things that must be understood is that only the code that is executed in RunTest is timed.

Use BeforeTest to instantiate objects and perform other housekeeping tasks that you don't want timed. Similarly, use AfterTest to free objects and do other cleanup chores that you don't want to be part of the test. Both of these procedures are automatically invoked in the proper order whenever the test is run.

The Clear procedure is useful for placing all of your initialization code and is called automatically when your test is instantiated. Of course, you can also call Clear later in your test.

=================

Adding the new test to the Result Viewer.

In procedure TfrmOSMResults.CreateDataset add the following code prior to the "OSMark" field :

with cdsResults.FieldDefs.AddFieldDef do begin

Name := 'MyNewTest';

DataType := ftInteger;

Required := False;

end;

Of course, replace 'MyNewTest' with the name of the new test.

In procedure TfrmOSMResults.AppendDatasetFromFile add the following code:

ReadLn( lTextFile, lsTemp );

cdsResults.FieldByName('MyNewTest').AsInteger := StrToInt( lsTemp );

This code must be added immedeately above:

ReadLn( lTextFile, lsTemp );

cdsResults.FieldByName('OSMark').AsInteger := StrToInt( lsTemp );

Add the following code in procedure TfrmOSMResults.AddNewResults:

cdsResults.FieldByName('MyNewTest').AsInteger :=

round( AOSMarkTestResults.GetBestScore(qtMyNewTest) );

Add this code above:

cdsResults.FieldByName('OSMark').AsInteger :=

round( AOSMarkTestResults.GetCosbiScore(FALSE) );

You now need to edit the clientdataset xml file. Using Notepad or other text editor, open osmresults.xml located in the osmfiles subdirectory.

Locate the following field definition and duplicate it immediately above the original entry:

<FIELD attrname="OSMark" fieldtype="i4"/>

Change the new entry to reflect the new field:

<FIELD attrname="MyNewTest" fieldtype="i4"/>

Save this file and exit the text editor.

Right-click on cdsResults and choose "Load from MyBase table..." and select osmresults.xml located in the osmfiles subdirectory.

Open the DBChart editor by double-clicking on DBChartResults.

Clone the series immediately above OSMark.

Rename the new series by clicking on the "Title" button.

Move the series immediately above "OSMark" series by clicking the up arrow button.

Double-click the new series.

Change the color and pattern of the new series to a unique combination.

Click on the "Data Source" tab and select the proper source for the "Bar" to MyNewTest.

Save all in Delphi and run OSMark. Open up the Result Viewer by

Since none of the existing result sets are valid after the addition of new tests, delete each row and press "Save" before closing the Result Viewer Window. Reopen the Result Viewer window to make sure that there are now no results in the Result Viewer.

Regenerate reference scores and save them.

Tutorial 1 -- Adding a Non-Graphic Test: the MP3 Encoder Test

For this example, we will add a quick test to time the MP3 encoding benchmark that is part of the popular Japanese program GoGo.

Add a new enumerated type value for the test to TQTests in the unit uCOSBI_TTest:

TQTests = (qtFirstTest, qtFib, qtGrid, qtGridFP, qtNBody, qtPlotTrig, qtPlotTrig2,

qtPlotLines, qtRandomDots, qtCircles, qtMaze, qtFern, qtRichEd, qtPi,

qtDhrystone, qtWhetstone, qtBandwidthBP64, qtMemLatency, qtFernThreads,

qtMazeThreads, qtOrthogonalThreads, qtIdenticalThreads, qtJpgDecode,

qtImageResize, qtImageRotate, qtMP3Encode, qtLastTest);

Drop a checkbox control on the “Select Tests” groupbox of the QuickTests main form. Name the control “cboxMP3Encode” and change the caption to “MP3 Encode”.

In the unit uQuickTest, add the code below to the bottom of the function TfrmQuickTest.GetSelectedTests:

if cboxMp3Encode.Checked then result := result + [qtMP3Encode];

We have to update the test suite so that it will know what to do with the new test. Append the following code into the procedure TquickTestSuite.TestSuiteRun located in the unit uTQuickTestSuite.

if qtMp3Encode in fSelectedTests then begin

lTest := TIMp3EncoderTest.Create;

RunTest( lTest, aiRunCount, fFrmOutput );

end; // if qtMp3Encode

Note: The test object is passed by reference into RunTest which frees the object upon completion.

Taking the template interface above, we only have to modify it very slightly and add it to the bottom of the interface section of uTest:

// Class name: TMp3EncoderTest

// Author: Van Smith

// Date: January 23, 2004

TMp3EncoderTest = class( TTest )

private

// we don't need any private field variables or procedures

protected

// ditto

public

constructor Create; Overload;

procedure RunTest; Override;

end; // TMp3EncoderTest...............................................

Now we've got to actually code the test itself. Scroll to the bottom of uTests (but before “end.”!) and add the code below.

// TMp3EncoderTest..............................................................

constructor TMp3EncoderTest.Create;

begin

inherited;

fTestName := 'MP3 Encoder';

fTestDescription := 'Times the built-in 128kbps MP3 encoding benchmark that is part of GoGo encoder.';

fTestVersion:= '1.0';

fTestType := ttCPU;

fTestAuthor := 'Van Smith';

fReferenceTime := 18;

fQTestType := qtMP3Encode;

end;

procedure TMp3EncoderTest.RunTest;

const

// assumes that gogo is in the same directory as the calling program:

GOGO_FILE_BM_COMMAND = 'gogo.exe -test';

begin

// run the GOGO

StartProgramWait( GOGO_FILE_BM_COMMAND, SW_SHOWDEFAULT );

end; // procedure TMp3EncoderTest.RunTest;

// TMp3EncoderTest ends.........................................................

There are a few things to notice. First, all test information must be specified in the constructor for the test.

The field variable fReferenceTime can only be filled out properly once the test has been run on the reference platform, but an initial value is necessary for the program to run.

The procedure StartProgramWait is a common routine declared in the unit COSBI_Common. It will start a program with command line arguments and wait for it to finish.

That's all there is to it! Save your changes and run the program.

Note: Try to keep the run time for any test well under 30 seconds on the reference platform since we don't want an “official run” to take all day.

Tutorial 2 - Creating a Test That Uses the Output Form: The JPG Decode Test

Creating a test that utilizes the output form is easy. In this tutorial we'll create a valuable test that measures JPG rendering performance.

JPG files are those ending with the “jpg” file suffix. Sometimes this suffix is spelled “jpeg”, but in either case the files are pronounced “J-Peg.” JPG files are the most common graphics file format used on the Internet. Using a “lossy” compression scheme, a JPG file can be much, much more compact than its BMP, RAW or TIFF counterparts, yet can be almost indistinguishable from the uncompressed image as long as the user-defined compression levels are not made too aggressive.

Digital cameras commonly store their pictures in JPG format. While small JPGs render almost instantaneously, large pictures from modern multi-Megapixel digital cameras can take a second or two to appear. This delay can be annoying when browsing through a folder containing dozens of pictures.

As in our first tutorial above, add an enumerated type value for our new test to TQTTests in unit uCOSBI_TTest

TQTests = (qtFirstTest, qtFib, qtGrid, qtGridFP, qtNBody, qtPlotTrig, qtPlotTrig2,

qtPlotLines, qtRandomDots, qtCircles, qtMaze, qtFern, qtRichEd, qtPi,

qtDhrystone, qtWhetstone, qtBandwidthBP64, qtMemLatency, qtFernThreads,

qtMazeThreads, qtOrthogonalThreads, qtIdenticalThreads, qtJpgDecode,

qtLastTest);

We'll need to plop a checkbox on the “Select Tests” groupbox. Caption it “JPG Decode” and name the control cboxJpgDecode.

In the unit uQuickTest, add the code below to the bottom of the function TfrmQuickTest.GetSelectedTests:

if cboxJpgDecode.Checked then result := result + [qtJpgDecode];

Now instruct the test suite what to do with this new test. Append the following code to the procedure TQuickTestSuite.TestSuiteRun located in the unit uTQuickTestSuite.

if qtJpgDecode in fSelectedTests then begin

lTest := TJpgDecodeTest.Create;

RunTest( lTest, aiRunCount, fFrmOutput );

end; // if qtJpgDecode

Since this program has a graphic component and needs to use the output form, we need to make our new test a descendant of TGraphicsTest. Copy the following interface into the bottom of the interface section of uTests:

// Class name: TJpgDecodeTest

// Author: Van Smith

// Date: January 23, 2004

TJpgDecodeTest = class( TGraphicsTest )

private

// we don't need any private field variables or procedures

protected

fImage : TImage; //we will use Delphi's Image component to render the JPG files.

public

constructor Create; Overload;

procedure RunTest; Override;

procedure BeforeTest; Override;

procedure AfterTest; Override;

end; // TJpgDecodeTest....................................................

Notice that we've got the BeforeTest and AfterTest prodedures declared. The reason for this is that we need to instantiate an image control on the output form and maximize it so that it will occupy the entire surface of the form.

To prevent memory leaks, we must free the image component in AfterTest.

Here is the code that goes toward the bottom of the implementation section of uTests:

// TJpgDecodeTest...............................................................

constructor TJpgDecodeTest.Create;

begin

inherited;

fTestName := 'JPG Decode';

fTestDescription := 'Benchmarks JPG image decoding performance,';

fTestVersion:= '1.0';

fTestType := ttCPU;

fTestAuthor := 'Van Smith';

fReferenceTime := 14.407;

fQTestType := qtJpgDecode;

end; // constructor TJpgDecodeTest.Create;

procedure TJpgDecodeTest.BeforeTest;

begin

inherited;

// PaintBox32 does not get along with the image component, so make it invisible:

fPaintBox32.Visible := FALSE;

// create the image componest which will do the JPG decoding

fImage := TImage.Create(fOutputForm);

// setting the parent to the output to ensure the component is freed

fImage.Parent := fOutputForm;

// bring the component to the front of any other controls on the form

fImage.BringToFront;

// make the image take over the entire form

fImage.Align := alClient;

// since images are usually stretched/compressed to fit the screen, when

// users look at them let's set stretch to TRUE

fImage.Stretch := TRUE;

end; // procedure TJpgDecodeTest.BeforeTest

procedure TJpgDecodeTest.RunTest;

const

// assumes both of the pictures are in the same folder as the calling program.

JPG_FILE1 = 'pic1.jpg';

JPG_FILE2 = 'pic2.jpg';

var

i : integer;

begin

for i := 1 to 10 do begin

fImage.Picture.LoadFromFile( JPG_FILE1 );

// allow the screen to update:

application.ProcessMessages;

fImage.Picture.LoadFromFile( JPG_FILE2 );

// allow the screen to update:

application.ProcessMessages;

end; // for

end; // procedure TRichEditTest.RunTest;

procedure TJpgDecodeTest.AfterTest;

begin

inherited;

// dispose of the image component to prevent memory leaks

FreeAndNil( fImage );

// restore visibility:

fPaintBox32.Visible := TRUE;

end; // procedure TJpgDecodeTest.BeforeTest

// TJpgDecodeTest ends..........................................................

And just like that, we have a valuable new test!

Note: Sometimes Delphi components behave strangely when being instantiated on the fly. When this happens, drop a similar component on the output form, set its properties identically and, most often, the problems will immediately go away. You can then delete the visual component and the program will continue to work correctly. The problem usually occurs because the unit for the target form does not have all of the required units in its uses section. As a last resort, place the control on the output form and make it invisible until it is to be used by the test.