Life is a Non-Deterministic Finite State Automata
Automation ? (*pGeekiness)++ : Code /eneration;

January 19, 2011

Dynamically changing Sampler properties during a Jmeter test

Filed under: Performance,Testing — Tags: , — admin @ 7:35 am

I often want to vary the value of a Sampler property such as ‘Retrieve All Embedded Resources from HTML Files’ at runtime. The approach I am about to show will let you change (probably) *ANY* Sampler property at runtime providing you know its name!

Create a simple Test Plan with one HTTP Sampler which here goes to www.bbc.co.uk:

To change the value of ‘Retrieve All Embedded Resources from HTML Files’ at runtime, we need to know its underlying property name. The easiest way to do this is to Enable the property, save the JMX file and then open the JMX file in a text editor. Then Disable the property, save the JMX file and open that one up in a text editor: then spot the difference :-) In the case of a HTTP Sampler, it is:

<boolProp name="HTTPSampler.image_parser">true</boolProp>

To change the value of ‘HTTPSampler.image_parser’ during a test, the easiest way is to add a ‘BSF PreProcessor’ to the HTTP Sampler like so:

Dont forget to set the language! Notice the use of the ‘sampler’ object name in the script. It goes without saying that this is the Sampler Context that the PreProcessor applies to. More information on the methods exposed from ‘sampler’ – such as setProperty – can be found here. You can do a lot more with this than you think :-)

Now when you run the test, you will find all of the Content has been downloaded even though it is disabled in the Test Plan:

You could go one step further and use a Jmeter Property that can be changed live during a running test session to represent the Enabled/Disable state of a Sampler property. See my previous post here.


February 16, 2010

Writing a Macro Recorder: Part 5 – Generating VBScript / JScript code and running it as a macro in your app

Filed under: Programming — Tags: , , , , , — admin @ 8:28 am

Part 1 (C#)- Part 2 (Testing C#)- Part 3 (PowerShell)- Part 4 (C++)

The source code for Part 5 can be downloaded here (Visual Studio 2008). Download and use at your own risk.
NUnit v2.5.0.9122 can be found here.
The reference in the test harness is to PowerShell 2.0 CTP2. Modify the reference – System.Management.Automation – and point it to your PowerShell; remove any lines that won’t compile :)
I also added a reference to stdole: change the reference to the version on your machine

This will wrap up the last of the ‘styles’ of languages I need to test. So far, I’ve covered C# (Managed), PowerShell (Scriptable Managed), C++ (Native) and now we need to cover ‘native scripting’: VBScript. What I write here applies equally well to JScript, Perl and so forth. In fact, it was so easy to add JScript I did :)

A few words about the test harnesses
First up, a few things: It is really the tests that do the ‘clever stuff’ – hosting PowerShell, VBScript, compiling C# code or launching Visual Studio. I haven’t really talked about them much because the work to do this resides in just two methods: Compile and Execute. Each language in the test harness implements those two methods. It’s easy to find and it’s isolated.

When you launch NUnit, you will see this:

All of the tests reside in TestingBase and each language derives from TestingBase. TestingBase uses abstract methods – such as Compile, GetGenerator and Execute – to obtain the correct engine and generator to use from each specialized class. The tests are structured in such a way that the specialized language can do whatever it likes, in any way it likes, providing it returns an IBindingList at the end. This means it is easy to ‘plug in’ new languages and get full coverage of all previous tests with that new language.
Back to it…
I am going to test this by hosting VBScript inside my .Net Test Harness and running the generated code. Like in Part 3: PowerShell, we’ll be generating and running real macros here. The generated code will talk to the host. Of course, you could create a .VBS file from the generated code and run that from the command line to test it (without host context); that works too and it’s what was done in Part 4: C++. At this point, we have a rich toolkit of approaches we can reuse for testing generated code. As this is about macros, though: I’ll test the generated code as a macro by hosting the VBScript Engine and executing code inside my application.

I will stick with the same theme throughout. I have added a new tab to the User Interface and I’ve added a new generator that outputs VBScript Code in a way that will work within my Test Harness host:

Recall from the earlier parts: the GridView is bound to an IBindingList. The user drives the GridView which drives the IBindingList; an IBindingList.ListChanged listener forwards the event to the macro recorder which generates the code.

And the single test I have now in TestHarness/TestingBase.cs should look familiar :-)

            // We've set up the handler that will forward changes to the macro recorder...
            // by driving the list directly, we view Recorder.Instance.Generators[0].Text
            // to see what has been generated so far. 
people.ListChanged += people_ListChanged;

Person woo = new Person();
woo.Name = "WOO";
woo.Age = 33;

Person hoo = new Person();
hoo.Name = "HOO";
hoo.Age = 50;


If you step through that code, by the time you get to the end the VBScriptGenerator.Text property will look like this:

    ' We do not create theBindingList1. It will be passed in from the host. 

    Dim thePerson1
    Set thePerson1 = Manufacture("MacroSample.Person")

    thePerson1.Name = "WOO"

    Dim thePerson2
    Set thePerson2 = Manufacture("MacroSample.Person")

    thePerson2.Name = "HOO"


Unlike C++, PowerShell and C#, there is no template we need to ‘wrap’ the generated code to make it compilable. It will compile and run as straight text.

Fascinating. How do we test it?

First some background
VBScript – and JScript, Perl, Ruby and all the other ‘COM’ scripting languages on Windows – are just ordinary COM Objects with their own ProgIDs. Infact, the scripting engine *IS* usually the ProgID! Surprised? Look at HKEY_CLASSES_ROOT and see for yourself :-) If you want more convincing, do this from your VBScript code:

    Dim j
    Set j = CreateObject("JScript")

You’ve just created the JScript engine from VBScript! Of course, the interfaces on the created object – IActiveScript etc. – do not derive from IDispatch so we can’t run JScript code from here.

Which is a shame. That would have made me look smart. And that was a disgression. So let’s get back to it.

When you see a scriptlet, in a HTML page for example, wrapped with something that says script=”VBScript” or language=”JScript”, the host – perhaps Internet Explorer or our Test Harness in this case – creates an instance of that language engine and makes a few calls on the main engine interface. The interface exposed on all language engines is ‘IActiveScript’. The host needs to make the engine ready to execute some scripts on its behalf. In particular, the engine needs to know which Window handle it should use when a modal dialog box is displayed (ie: when we run MsgBox “WOO” in VBScript); what methods and properties from the host might be exposed to the script; and how to tell the host the script contains errors.

To do this, it uses a simple callback mechanism. We (as the host) implement an interface called IActiveScriptSite which exposes a whole bunch of methods and properties. And which, fortunately, Dr Dobbs had provided the Interop signatures for. We tell the engine which callback interface (‘scripting site’) to use by calling IActiveScript.SetScriptSite on the engine and passing in ourselves as the site. The engine can then call back on the interface at various times to tell us thing’s like it’s state, whether the script has finished, and to ask us for objects referenced in the script.

If our VBScript code contains this:

    MsgBox Woo.Hoo

It is fair to say that ‘Woo’ is not part of the VBScript Specification. The engine will ask the host to return the IDispatch* of Woo. Or, in the case of C#, an object reference whose class is attributed with [ComVisible(true)].

It’s dead, dead easy.

Something that caught me out in the old days: When you run a Javascript in Internet Explorer and use ‘alert’, a dialog box is displayed to annoy the user. But when you run Javascript outside of Internet Explorer and use ‘alert’, you get an error. Why? When Internet Explorer hosts JScript it exposes a method called ‘alert’ that scripts can use. It does this by providing a method called ‘alert’ via its own implementation of IActiveScriptSite. If you want JScript code to work in your app, and you want an alert method exposed, you have to implement it yourself. MsgBox, in VBScript, is part of the language so we don’t have this problem.

We will expose some host-specific methods to VBScript and generate code to use them.

Let’s do it
Now that’s in place I can wander through the code that gets generated in the UI. As I’m generating VBScript code, and I’ll be dealing with the Person object that is known about only within my C# host, I cannot create that in VBScript. ‘Person’ has no ProgID. Now: I *COULD* use the various attributes in System.Runtime.InteropServices so that my Person class is exposed to COM as a ProgId, and is creatable, but I won’t do that. Instead, I will have VBScript ask the host (ie: the test harness) to manufacture objects on my behalf by type name. ie:

    Dim t

    Set t = Manufacture("Some.Class")

In my host – I have provided a shell IActiveScriptSite implementation in the Test Harness that you can use – I implement a method that simply instantiates an object of that type. Everything I expose to script is wrapped up in a single class called ScriptVisibleHostProperties:

// ScriptVisibleHostProperties.cs
// Comments removed. 
    public class ScriptVisibleHostProperties
        // This is the property our macro code expects to be around when it is run. 
        public object theBindingList1
                return m_theBindingList1;
                m_theBindingList1 = value;

        protected object m_theBindingList1;

        public object Manufacture(string typeName)
            // Just create a brand new object of the specified type. 
            // I will look in the Assembly that contains our stuff.
            object t = typeof(MacroSample.Recorder).Assembly.GetType(typeName).InvokeMember("", BindingFlags.CreateInstance, null, null, null);

            return t;

From my VBScript code I can now call ‘Manufacture’ and access a property called ‘theBindingList1′. ie: the same binding list the macro code assumed would be around when it was generated.

When it comes to testing this – see TestHarness/ComScriptingBase.cs – the Compile stage sets up the IActiveScript (for the language we want) ready to execute the code; the Execute method just runs the script by setting the scripting engine to the ‘Connected’ state (bizarre, but there you go). It then returns the object that was returned by the VBScript Engine:

// VBScript.cs::Execute
MyPeopleCollection macroPeople = new MyPeopleCollection();

ScriptVisibleHostProperties props = theHost.NamedItems["MyHostProperties"] as ScriptVisibleHostProperties;
Assert.AreNotEqual(null, props);

props.theBindingList1 = macroPeople;

// Setting it to State 2 (CONNECTED) actually sets the engine running.

return macroPeople.TheRealBindingList;

I’ll explain why I do not pass a BindingList collection directly into the script later and all the nuances in the generated code.

The important thing is: from this method, we return theBindingList1 (the object the macro code expected to be around) after it has been passed into the VBScript engine and the macro code executed. The state of the object returned should be exactly the same as the one we built up in our TestingBase.cs test.

Super. Duper. Cool :)

Quite a few actually. Let’s start at the beginning :-)

When I first started, I got this error:

System.Runtime.InteropServices.COMException : Exception from HRESULT: 0x800A000D

‘Type Mismatch’ . I was exposing my ScriptVisibleHostProperties class to the VBScript engine, but I had not marked it with the ComVisible(true) attribute:

    public class ScriptVisibleHostProperties

Once I got past this point, the ActiveScriptingHost.OnScriptError method was being called when I made a mistake and I could work out what was going wrong.

Like: Object required:

System.Runtime.InteropServices.COMException : Exception from HRESULT: 0x800A01A8

… I needed to mark my Person class with the ComVisible(true) attribute as well. If you get an ‘Object Expected’ error when you are running hosted VBScript, the first thing you should check is that your classes are marked as ComVisible(true). This includes the classes of any nested properties, too. I suppose the easiest way is to make everything in the assembly visible!

Then there was an issue with this:

System.Runtime.InteropServices.COMException : Exception from HRESULT: 0x800A01AE

‘Class does not support Automation: Age’. Hmmm. VBScript can see my class. It can see the Name property. But it can’t see the Age. What’s special about Age?

	public int? Age

It’s a nullable type. I would have expected the default marshaller to convert this into a Variant (VT_NULL) if it was a null value and the appropriate VT_xxx if it was set. But no: ‘Class does not support Automation. ‘

I tried various things, but all the interesting Marshalling attributes weren’t allowed on properties so I went with a hack. I wrapped the property name with explicit Getter/Setter methods and you can see them in the generated VBScript code. Doing this with all of your properties would be a pain; not really an issue with generated code though :) I assume one of the magic attributes in the deep, dark System.Runtime.InteropServices namespace will do what I want…?

And the biggest of all. I passed in a BindingList to VBScript but when it executed the ‘Add’ method I kept getting an error. It could not see ‘Add’. Probably because BindingList does not have the ComVisible(true) attribute set. I tried deriving from this class, setting the attribute on my derived class and passing an instance of that in instead but that didn’t work either. As you would expect. The best I could come up with was a hack: to create a new collection – MyPeopleCollection in the test harness – that exposed two methods: Add, and RemoveAt. They delegated to a ‘real’ BindingList.

I pass an instance of MyPeopleCollection to the Execute method but return the .TheRealBindingList to the test harness.

And that just about sums up all of the language ‘styles’ I’ll need to be generating and regression testing: Managed (C#, VB.Net, Managed C++), Managed Scripting (PowerShell), Native (C++) and COM Scripting (VBScript).

Powered by WordPress