If you’ve ever used the likes of GTEST for C++ unit testing, you will be familiar with the ASSERT and VERIFY semantics:

• ASSERT – bail out of a test as soon as a condition is false (ie: NULL Pointer)
• VERIFY – acknowledge that a condition is false but continue with the test anyway and get as far as you can. The test still technically failed but more information was gleamed.

Verify is ideal for functional / UI / UAT Automation because it lets the test get as far as it can and elicit as much information as possible before the test completes and a summary of failures are reported: it’s more useful for a developer to know that 5 numeric fields on a form are invalid instead of just one. A colleague pointed out recently that various UI Automation tools tend to implement similar semantics using ‘LogFail’ or similar statements – however, as a developer/tester I find the ‘Verify’ semantics more fitting but they are not part of MsTest.

In this rather long post, I will put Verify into MsTest by wrapping Assert.AreEqual with Verify.AreEqual (for all samples here, this is just an ordinary VS2010 Unit Test). I will provide nothing but a bare-bones implementation here (and I’ve just noticed the code prettifier I use has messed up some of the snippets on this post… please see the source code above for the complete code).

When you do this:

	Assert.AreEqual(1,0)

The unit test fails immediately. We want to do this:

	Verify.AreEqual(1, 0)

Where the failure is ‘noted’ but the test continues: but when the test completes, if there were verification failures in the test, we need to throw an exception so that the unit testing framework designates the verification failures as a test failure. How to do this in MsTest? There’s a few hurdles to cross!

Syntax
All Assert methods are static. Like so:

	Assert.AreEqual(...)

Assertions have no state – a failure is propagated to the test host immediately so static methods are a good fit. Verification failures on the other hand will ‘accumulate’ so we need to preserve state. For this post I have chosen to go the ‘instance’ route so here is a simple Verify class:

public class Verify
{
	public Verify()
	{
		Exceptions = new List();
	}
	public void AreEqual(int left, int right)
	{
		...
	}

	public readonly List Exceptions;
}

However: you can implement Verify methods using thread local storage and static methods but I am trying to keep this long post shorter!

The key is the implementation of the Verify methods: all an Assertion does is throw an exception when a condition is false so all we have to do is sink & record that exception by wrapping it with our Verify calls:

public class Verify
{
    ....
    public void AreEqual(int left, int right)
    {
	try
	{
		Assert.AreEqual(left, right);
	}
	catch(UnitTestAssertException ex)
	{
		Exceptions.Add(ex);
	}
    }
}

If the assertion fails; we essentially ‘note’ the failure but continue. Putting it all together, we might have a test like this:

protected Verify Verify;

[TestInitialize]
public void Init()
{
	this.Verify = new Verify();
}

[TestMethod]
public void Pointless()
{
	Verify.AreEqual(1,2);
	Verify.AreEqual(3,3);
	Verify.AreEqual(3,4);
}

NOTE: Even though Verification violations occurred, as far as the Unit Testing framework is concerned the test technically passed – no exceptions were thrown by the Unit Test! So we need to check for verification violations in the Cleanup method and then throw our own Exception if Verifications were logged during the test.

When executing that test, we have two Verification errors. But what to do with them? If we are running Pointless with Associated Automation within MTM, we want the test to fail; MTM has no concept of a Warning or a Partial Failure. The test either passes or fails so from MTM’s perspective unit tests should exhibit the same behavior. The Verifications are only useful for troubleshooting, logging and triage so they need to appear in the final log / TRX. If we are running the tests within Visual Studio as a Unit Test, we still need the test to fail for the same reason as above to integrate with the toolchain. How to do this? The easiest place to look for any logged verification failures is in the [TestCleanup] method. If you throw an exception in TestCleanup, the exception/failure is still associated with the Unit Test that has just run (ie: the method containing the Verify methods):

[TestCleanup]
public void Cleanup()
{
	if(Verify.Exceptions.Count > 0)
	{
		throw Verify.Exceptions.First();
	}
}

The unfortunate side effect of this is that the exception/stack trace in the Test Results / TRX file looks like it came from Cleanup method and not the test itself. Clicking through takes you to the common Cleanup method which is kind of annoying:

But we can fix that.

CHECKPOINT:We can accumulate verification failures during a unit test and throw an exception in TestCleanup if any verification failures occured. The exception we manually throw could contain a description of every verification encountered so far (for this post: I am dealing with only the first exception and I am keeping message formatting as simple as possible).

But what if the Unit Test contains ASSERTIONS *AND* Verifications? Like so:

[TestMethod]
public void Pointless()
{
	Verify.AreEqual(1,2);
	Assert.AreEqual(3,3);
	Verify.AreEqual(3,4);
}

I have decided that the Assertion gets ‘priority’ – it is that exception/assertion we want to propagate ‘out’ to the unit testing framework. We can determine if a ‘real’ Assertion or Exception was thrown in the Unit Test by looking at the CurrentTestOutcome property:

public TestContext TestContext { get; set; }

[TestCleanup]
public void Cleanup()
{
	if(TestContext.CurrentTestOutcome == UnitTestOutcome.Passed)
	{
                // If we only have Verify failures, as far as MsTest is concerned, the test will pass! So we need to spoof a failure...
 		if(Verify.Exceptions.Count() > 0)
		{
			throw Verify.Exceptions.First();
		}
	}
}

Easy! So we can comfortably mix assertions and verifications in a single functional test and it will “just work” as far as the tool chain is concerned; if a real assertion happens, that one gets propagated. In C++/GTEST, an ASSERT is used to validate a pointer (little need to go on if its NULL…!) and VERIFY is then used for individual properties. In a functional test, an ASSERT might look for a key component of a page; and the VERIFY calls for fields for example. It depends if it fits what you are trying to do. Use your judgement. This will not be suitable in all circumstances.

Fixing the Stack
As stated, if we throw an exception from TestCleanup, the stack trace looks like this:

That’s not good enough! It shows the Cleanup method itself, not the actual line of code where the Verify call was made. Thanks to the .Net designers, this is easy to fix though If you examine System.Exception, you can override two key properties: Message and StackTrace (and there’s a section for each in the TRX file) Yes – as you can override the stack trace text, you can ‘inject’ a stack trace into an exception and fool anything that interprets that exception about its source – such as the TRX viewer. And it’s easy to get the stack trace. Just do this:

	string stack = Environment.GetStackTrace();

Trivial! But we will be getting the stack trace in our Verify method… we need to ‘unwind’ a bit. To ‘pop’ a few lines we just do this:

List t = new List(Environment.StackTrace.Split(delims, StringSplitOptions.None));

// 'Pop' a few lines
t.RemoveRange(0, 2);

// Reconstruct
string stack = String.Join("\r\n", t);

With this, we can ‘inject’ a stack trace into our exception. The only way I could find to do this is to create a custom exception class (gives us more flexibility…) and override its virtual StackTrace property. So it makes sense for our new exception to wrap the original exception and delegate every other call to it (where possible):

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace HaveCompTest
{
    // Might as well derive from the unit test exception baseclasses... and you should probably use InnerException : - )
    public class MyVerifyWrapperException : UnitTestAssertException
    {
        // Pass in the original assertion exception
        public MyVerifyWrapperException(UnitTestAssertException utex, string spoofedStackTrace)
        {
            OriginalException = utex;
            SpoofedStackTrace = spoofedStackTrace;
        }

        public override System.Collections.IDictionary Data { get { return this.OriginalException.Data; }}
        public override string Message { get { return OriginalException.Message; }}
        public override string Source { get { return OriginalException.Source; }}

        public override string StackTrace { get { return SpoofedStackTrace; } }

        public readonly System.Exception OriginalException;
        public readonly string SpoofedStackTrace;
    }
}

Getting there! So now when we wrap our original Assert.AreEqual call with our Verify.AreEqual call, we can create our new exception type with the StackTrace we want:

try
{
            Assert.AreEqual(left, right);
}
catch(UnitTestAssertException ex)
{
            string[] delims = new string[] { "\r\n" };
            List t = new List(Environment.StackTrace.Split(delims, StringSplitOptions.None));
            // Choose how many lines to strip...
            t.RemoveRange(0, 3);
            string stack = String.Join("\r\n", t);

            // The stack trace now looks like it was thrown directly within the test method itself instead of here.
            MyVerifyWrapperException e = new MyVerifyWrapperException(ex, stack);
            _Exceptions.Add(e);
}

KOOL! So now when we throw our verify exception in [TestCleanup] like so, in the TRX viewer we see this:

Clicking through takes us straight to the location of the Verify failure.

DONE!

Putting it all together

The source code for a simple skeleton class can be found here (just add it to an ordinary MsTest Unit Test).

Tips

You’ll need to wrap all the Assert.XXX calls… a delegate is your friend when doing this… () => { Assert.AreEquals(…) }

The error message says ‘Assert.’ in TRX. Modify the ErrorMessage in MyVerifyWrapperException to say Verify…

You might want a clickthrough to all Verifications in the Stack Trace view…

If mixed Assertion / Verification failures occur within  a test, you might still want the Verifications to show up in TRX…

Use Thread Local Storage to implement static Verify syntax so they are similar to Assert…

Key features include:

• Programmatically automate the User Account Control [UAC] elevation and credential dialog boxes on the standard or secure desktop on Vista and Windows 7 (x64/x86)
• Requires no tweaking to your environment, no digital certificates, no modifications to your UAC settings and no application manifest modifications. Install and go!
• Can be driven from any programming language or scripting environment – including C#, VBScript, Perl, PowerShell or (if you’re really hard) C++. Extensive samples are provided in C# and VBScript.
• Dialogs can be automated by any user at any privelege level – ideal where elevated applications need to be launched as part of a test.

UacAA might be handy for:

• Launching elevated applications as part of your automated test suite to modify the environment between tests.
• Integration / smoke testing where UI automation is not a primary objective but where User Account Control automation is required.
• Testing your application behaves correctly under Uac!
• Running your integration tests on an isolated P2V-SOE production clone without making any changes to the environment whatsoever.
• Reducing the number of snapshots and virtual machine baselines that need to be managed.
• Automating security-related tests that were previously manual in nature.

How it works:

• Set an expectation that a User Account Control dialog box is to be displayed and indicate what to do when it is – such as enter credentials and Confirm it.
• Do something that requires elevation such as launching an application.
• In the background, UacAA is monitoring for the presence of the UAC dialog – if it appears, the dialog will be automated according to your expectation so that your application can continue.
• Optionally ask UacAA what it has just done and the kind of dialog that was displayed (elevation vs credentials) and the desktop on which it appeared (standard or secure)

Known limitations:

• This version will automate only one dialog per expectation – it is currently not possible to say "enter these credentials and select Confirm whenever UAC dialogs appear from now on"

Microsoft has provided a way of automating User Account Control by leveraging its UI Automation technologies – but this involves UAC modifications, installation of digital certificates and an application with UIAccess rights for UI automation. This level of security is clearly required in production environments where trust (obtained via the certificate) is paramount because otherwise we would back in the days of the “shatter attack” (UAC appears on the Secure Desktop to prevent exactly that kind of attack!).

In test environments – where free text passwords tend to lay around with wanton abandon – you just want the darn dialog to go away but to otherwise retain UAC.

UacAA lets you do this.

Where the Microsoft supported way is convenient, it should be adopted. Where it is inconvenient, or where perhaps integration testing is the objective and no infrastructure or applications are around to facilitate UI automation easily, UacAA might fit the bill. UacAA should only be used on test environments and never installed on server or systems where arbitrary content from the Internet can be executed.

UacAA must absolutely not be used as a hackey-wackey way of executing scripts that require elevation on desktop systems!

Mid-way through the series, Visual Studio Service Pack 1 was released. How amusing! So this is an update to incorporate the Service Pack 1 changes.

See Part 5 for the Source Code and scripts.

DISCLAIMER!

Do not use this code under any circumstances (should just about cover the possibilities!).

I am using an undocumented API in order to construct the Physical Environment in Lab Center and set up the Test Controller Topology. I have tested the registered environments using MTM, use it often and come to no grief. My Lab Center and TFS system appears to be stable. But you use this at your own risk! At the very least, it would be sensible to do a full back up of your TFS Installation and ideally test this prior to production deployment. Use at your own risk

Parts 1, 2, 4, 5 are the same: nothing changes. The only changes you will need to make are down to the installation automation in Part 3.

I will not be providing an updated script to do this but if you have been following the series and want to stick to the same structure, you need to make Service Pack 1 available under the VisualStudioGumpf directory by unpacking your ISO there:

You will probably also need to create a new BAT file to launch “setup.exe /passive” from the Service Pack 1 location. Drop this into your Golden VM at the usual place:

And then write a new function to launch that from PowerShell.

InstallServicePack1 $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $VisualStudioGumpfUnc;

Troubleshooting
If you get problems – try it manually first! The only part where I do anything undocumented is to create the Physical Environment. If you happen to get a situation where you can do this registration process manually, but not automatically, please let me know so that I can fix it

Source Code Changes
I have no idea how many lines of white powder I had up my nose when I wrote this comment:

// I am not going to check this here but only one machine in an environment can be of a given role.
Dictionary agentsToAdd = new Dictionary();
Dictionary machineRoleInfo = new Dictionary();

But it is clearly wrong!

Apart from having to install Service Pack 1, I haven’t had to make any changes: the environment still gets created and all appears normal. You should be able to target your created environment from MTM:

And run your Unit Tests, Integration Tests and CodedUI Tests on it:

Tchau!

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

See Part 2 for Requirements (PowerShell, Vix etc.).

DISCLAIMER!
Do not use this code under any circumstances (should just about cover the possibilities!).

I am using an undocumented API in order to construct the Physical Environment in Lab Center and set up the Test Controller Topology. I have tested the registered environments using MTM, use it often and come to no grief. My Lab Center and TFS system appears to be stable. But you use this at your own risk! At the very least, it would be sensible to do a full back up of your TFS Installation and ideally test this prior to production deployment. In particular: this has not yet been tested with Visual Studio 2010 Server Pack 1 (it has now, and it appears to work as well as pre-SP1) but use at your own risk

In future, if and when Microsoft provides a way to create Phyical environments and construct Test Controller / Test Agent hierarchies programmatically, this is the only stage you will have to change in this series. Everything else you have done as part of this series still applies.

Background
That aside: this is the REALLY AWESOME PART!! Before going on, consider provisioning a brand new virtual machine using the scripts at the end of Part 4.

There are no officially supported APIs to manage the Test Controllers and Test Agents in TFS. This is unfortunate because it stops you creating Physical Environments and constructing your Test Controller / Test Agent hierarchies on the fly. You proceed at your own risk!

The Test Controllers and Test Agents are all written in .Net and Microsoft has a tendency to make all of its classes ‘internal’ so you cannot reuse them willy-nilly. I like this approach: rather than obfuscating their code they are polite enough to let us wander around their work if we need to in order to troubleshoot or find out WTF is going on. It’s a pragmatic solution between opening things up to all and sundry and having to support their firstborn for life.

The code we need to invoke to construct our Test Controller and Test Agent hierarchies is stored in ‘internal’ classes so we need to find a way of instantiating those classes and invoking their methods.

A major help is that IL has no concept of ‘internal’: it is the compiler that enforces that contract opening up all kinds of creative possibilities! But that aside, you can create internal classes from other libraries just by using reflection like so:

System.Reflection.Assembly controllerAssembly = System.Reflection.Assembly.LoadFile(theAssemblyPath);
Type controllerTestEnvironmentType = controllerAssembly.GetType(theFullyQualifiedTypeName);

object testEnvironment = controllerTestEnvironmentType.InvokeMember("", System.Reflection.BindingFlags.CreateInstance, null, controllerTestEnvironmentType, null);

…and you can get hold of properties, invoke private and protected methods and everything else you want using nothing more than reflection. But in .Net 4.0 there is an easier way! The ‘dynamic’ keyword will leave property and method resolution until runtime. Think of it as IDispatchEx from the COM World or expandos in JavaScript. It is very cool because by wrapping an internal type you can treat it like any other object. I use the code from that link verbatim but there’s a lot of extra stuff you can do with that if you put in the time.

The reason I am mentioning it? Because you will see it in my code when I invoke the internal or private Test Controller classes:

            dynamic controllerConnectionManagerWrapper = new AccessPrivateWrapper(controllerConnectionManager);

            controllerConnectionManagerWrapper.UpdateTestEnvironment(testEnvironment);

Manually creating a Physical Environment using a UI
The source code this time includes all of the scripts and some supporting Visual Studio 2010 Solutions that you will have to build yourself:

The first thing we need to do is to make sure you can manually construct Physical Environments using my User Interface because it drives the underlying TFS libraries. If not: not much point going on!

Assuming you have heeded the disclaimer, you will need to add references to several standard TFS API’s including:

The interesting three are:

It is important you set *ALL* of those DLL’s to ‘Copy local: True’ for the Reference. You will need to add them to AgentInteraction and PhysicalEnvironments as well.

Now run the PhysicalEnvironments application and you will see a glorious ‘Graham UI’. I loathe arty-farty work and I loathe writing User Interfaces so much, and have consistently done such a bad job of it, that I never get asked to write them any more (mission accomplished!). Select the TFS Team Project Collection / Team Project and observe what happens:

This UI is listing all of the Test Controllers on the left: that is the same list you see when you create a new environment in Lab Center and are asked to select the Test Controller you want to create the environment on:

When you select one of the Test Controllers (such as Moo1250) you will see the list of Agents available. This list is identical to what you can see in Lab Center as well:

So we have all of the information we need to start constructing Physical Environments. Enter a new name for the Physical Environment you want to create – such as “WooHOO” – tick the single Agent, enter a sensible Role name, and select “Create New Environment”:

Your UI will now gloriously lock up for about a minute (I provide no UI Cues) whilst Lab Center does whatever it needs to do to create your environment. Assuming all went well (you’ll know when focus returns to your window) you should be able to see your environment in Lab Center:

AWESOME! So we have all the bits and pieces.

Before going on, delete the environment in Lab Center and then continue.

Automating the creation of a Physical Environment using PowerShell
The PowerShell script obviously doesn’t use the User Interface but I have created a dirty, hackey-wackey Console application called ‘AgentInteraction.EXE’ that is only moderately unforgiving of command line mistakes but can be used to drive the registration process.

To set this up to work, create a new subdirectory under VisualStudioGumpf called AgentInteraction and drop these DLL’s (AgentHelpers/bin/Debug), AgentInteraction.EXE and anything pulled in as part of the build into it:

Assuming you have removed the Test Controller and everything from Lab Center, you should be able to execute code like this:

# Obtain a list of all Agents known to a Test Controller
\\10.0.0.1\VisualStudioGumpf\AgentInteraction\AgentInteraction.EXE Command=ListAgents TestControllerUri="MOO1251.WOOHOO.COM:6901"

# To create an environment on the Test Controller MOO1251 and with a single Agent called MOO1251 with a Role of 'Desktop Client' you would execute:
\\10.0.0.1\VisualStudioGumpf\AgentInteraction\AgentInteraction.EXE command=RegisterEnvironment testcontrolleruri="MOO1251.WOOHOO.COM:6901" TeamProjectName="TestSample" EnvironmentName="My New Environment" Agents="MOO1251,The Web Server"

… and see your environment created in Lab Center from PowerShell:

Kool!

Automation Summary
Assuming everything is in place, you should now be able to dynamically provision new virtual machines in VmWare, install the Test Controllers and Test Agents, register them with TFS and create a new Physical Environment on a whim. The key differences for this post are:

# ... variables from previous stages

# The Team Project Name we want to register the environment with and the environment name
$TeamProjectName = "TestSample";
$EnvironmentName = "My Brand New Environment";
$MachineRole = "Database Server";

# ...

# STAGE 5: Create the Environment
# It can take a while for the VM to spin up and the Test Controller + Test Agent to come online... so use the RETRYCOUNT here.
Invoke-Expression "$VisualStudioGumpfUnc\AgentInteraction\AgentInteraction.EXE command=RegisterEnvironment testcontrolleruri=$($MachineName).$($DomainName):6901 TeamProjectName=`"$TeamProjectName`" EnvironmentName=`"$EnvironmentName`" Agents=`"$MachineName,$MachineRole`" RetryCount=10";

Agents=`"$MachineName,$MachineRole`" ";

As a final test, set it as a targettable environment in MTM and run your Tests:

Mission Accomplished! However… there is an anomoly! When I went into the Test Controllers view and clicked a Test Controller, I noticed that duplicate Test Controller names appeared for those that were put on Virtual Machines that I had provisioned:

At first I thought it was some SID issue with the cloned virtual machine but it appears to be down to a legacy DNS Issue:

I was able to repeat this 100%. If you remove the legacy DNS Record (ie: for Moo1400 which is a virtual machine I purged) and restart MTM, the problem goes away (I wasn’t able to repeat it after this).

So I guess you need to purge the DNS Record as well as the Computer from your Active Directory when you throw away an environment.

L8r!

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

See Part 2 for Requirements (PowerShell, Vix etc.).

After installing the Test Controller and Test Agents and ensuring the patch level matches our baseline, we need to automate the registration of the Test Controller with TFS and automate the registration of the Test Agent with the Test Controller.

Manually registering the Test Controller
Before going on I will assume you have created a new Virtual Machine by running the scripts created in Stage 2 and installed the Test Controller and Test Agent in Stage 3. We will use that as a baseline. Use the snapshot and restore functionality of VmWare liberally here – that’s what it is for!

Checkpoint 1 – Machine with Test Controller and Test Agent installed but not configured
There are a lot of possiblities and topologies when setting up environments, test controllers and test agents. For this series I am creating a single VmWare machine that will contain both the Test Controller *AND* the Test Agent.

If you’ve set up a Test Controller before you will be familiar with this user interface. Launch it from within your Virtual Machine:

There are so many moving parts and opportunities for failure so large that the best way to identify issues with the script is to do everything manually first. So…

Enter the parameters and make sure the configuration works if you do it manually:

If the operation was successful you will be able to see the Test Controller in Lab Center:

If you cannot see the controller in Lab Center at this point, you need to work out why. There is little point going on. The first thing to do is to disable the Firewall on your provisioned virtual machine or add an exception for the Test Controller application to see if that helps: this is usually the problem (my batch file in the source code package adds an exception using netsh). Otherwise, the possibilities are too many to mention here. Whatever changes you make you will either need to put into the PowerShell script or backpatch into your baseline VmWare image and re-provision.

The User Interface Application used to configure the Test Controller above is called ‘TestControllerConfigUI.EXE’. If you hunt around on your virtual machine disk you will find one called ‘TestControllerConfig.EXE’ that lives in the same directory. ‘TestControllerConfig.EXE’ is what we use to automate setting those parameters.

Now revert to Checkpoint 1 in the VM. To Undo (unregister) the Test Controller from Lab Center, do this first within your Virtual Machine.

We will now automate the Test Controller registration using the command line tool:

I have no idea what can take so long: but it does.

Assuming that works you can confirm the Test Controller has registered itself with TFS by reopening Lab Center and seeing your Test Controller in all its glory:

GOOD! We now have everything in place to register the Test Controller with Lab Center. If you needed to make any changes to your virtual machine to get this working (such as modifying the Firewall settings) you either need to make that part of your Virtual Machine baseline or you need modify the scripts I have written to create Firewall exceptions on the fly.

Finally, to ensure this is going to work from VMRUN.EXE, execute this (after copying across the Guest/Script batch files from the attached source code into the Virtual Machine C:\Scripts directory as usual):

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestController.BAT http://WIN-TSKURFCLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection WOOHOO.COM\TfsTest whatever SomeOtherText

VmWare Workstation:

vmrun.exe -T ws -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestController.BAT http://WIN-TSKURFCLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection WOOHOO.COM\TfsTest whatever SomeOtherText

Note the addition of ‘SomeOtherText’ to the end. For some inexplicable reason, one less parameter than usual was being passed through to the Batch file from VMRUN.EXE. I have no idea why, but adding another one to the end fixed it. Bizarre!

Manually registering the Test Agent
Checkpoint 2 – Test Controller Installed and Registered with Lab Center. Test Agent Installed but not Registered with Test Controller.
Now we just need to do the same thing for the Test Agent. We know the Test Controller works so we just need to register our Test Agent with it so we can add that Agent to new environments in Lab Center. Open up the Test Agent Configuration Tool and set up the parameters you want. I am going to run this as an interactive process (not a service) so I’ll point out the difference later when we automate it:

Notice we are using ‘localhost’ to talk to the Test Controller we installed earlier on the same machine. If you are provisioning multiple machines here, you obviously point it to where your Test Controller is installed

Once again, do that manually to check it works and troubleshoot any issues. To check that we are getting visibility over our Test Controllers and Agents we can now go into Lab Center, select Controllers and observe that our Test Agent is present:

If you can’t see your Test Agent you have to troubleshoot why before you proceed.

Once this works, revert to Checkpoint 2. To automate the Test Agent configuration from the command line, there is a tool called ‘TestAgentConfig.EXE’ that does it for us within your VM:

In the above case, I was running the Test Configuration utility as WOOHOO.COM\Administrator but want the Test Agent to run as Graham (Administrator account on the Virtual Machine). So I need to reboot (and you will see that in my scripts).

When I restarted the Virtual Machine, I was told that QTAgentService.EXE was still blocked by my firewall: so I need to add an explicit exception for that to the Windows Firewall and you will find that call in the ConfigureTestAgent.BAT file which looks like this:

REM Add a Firewall Exception for our Agent Service executable.
netsh firewall add allowedprogram program="C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\QTAgentService.EXE" name="QTAgentService.EXE" mode=ENABLE scope=ALL profile=ALL

For now, Unblock the application using the Firewall Alert (if you are asked) manually so that you can continue. When you run the ConfigureTestAgent.BAT batch file using VmRun.Exe, the exception will be added.

The parameters map one-to-one with what is in the UI. Once your Agent User Interface comes up, you should be able to see your Controller and Agent in Lab Center again:

FINALLY! Revert to Checkpoint 2 again. To ‘Undo’ (Unregister the Test Agent with the Test Controller) just do this from a Command Prompt in your Virtual Machine:

C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\TestAgentConfig.EXE Delete

Now we can see that this is going to work using our VmRun.EXE call with our Batch File:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestAgent.BAT Graham moo localhost:6901 SomeExtraStuffSoTheParameterGetsSentThrough

VmWare Workstation:

vmrun.exe -T ws gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestAgent.BAT Graham moo localhost:6901 SomeExtraStuffSoTheParameterGetsSentThrough

After that, check once again you can see your Controller and Agent in TFS:

Awesome! So that’s all the moving parts tamed.

Automation Summary
Unpack the Source Code for Part 4. With the source code, you should be able to provision a new virtual machine from VmWare; install the Test Controller and Test Agent; and register them with TFS automatically.

This snippet shows the key changes to the VmWareTfsIntegration.ps1 script for this stage:

# ...

# Test Controller Configuration Parameters
$TestControllerTfsTeamProjectCollection = "http://WINTSKURF-CLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection";
$TestControllerTfsServiceUser = "WOOHOO.COM\TfsTest";
$TestControllerTfsServicePassword = "whatever";

# Test Agent Configuration Parameters
# I want to log in the user automatically so I can do UI Testing: these are the credentials under which the Test will be run.
$TestProcessUsername = "Graham";
$TestProcessPassword = "moo";
# The Test Agent needs to know what Test Controller to connect to. As we only have one machine, use localhost.
$TestControllerUri = "localhost:6901";

# ... Provision and Install Test Controller + Test Agent

# STAGE 4: Configure Test Controller + Test Agent
ConfigureTestController $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $TestControllerTfsTeamProjectCollection $TestControllerTfsServiceUser $TestControllerTfsServicePassword
ConfigureTestAgent $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $TestProcessUsername $TestProcessPassword $TestControllerUri

Before going on I recommend constructing a Physical Environment using that Controller and Agent and targetting it via MTM in a simple test. If you can do this, you know the system works end-to-end and it will make Stage 5 much easier

Only one more step! And that is to automate the creation of a Physical Environment in TFS so we can run our tests against them using MTM.

L8r!

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

See Part 2 for Requirements (PowerShell, Vix etc.).

Like Part 2, this does not really belong in this series but has been provided for completeness. What we need to do is to automatically install the Test Controller and the Test Agents and any patches onto our virtual machines so we are ready to register that testing infrastructure with TFS.

Manually installing the Test Controller and Test Agent
The first thing we need to do is to make the Test Controller installation software available SOMEWHERE on the network accessible from your brand new provisioned Virtual Machine in Stage 2. Vix will be used to install the software in the virtual machine so the user we log on to the Virtual Machine as must have access to that network location.

My personal network at home is called WOOHOO.COM (don’t ask) and I have a share called VisualStudioGumpf whose directory structure looks like this:

Because I am not a System Administrator, it is open to EVERYONE as Read and ADMINISTRATORS as Read/Write.

If you lock down the permissions of that share remember that a lot of the work will be done from within the security context of a user in the virtual machine. Whatever user you log into the virtual machine as during this stage must have READ access to that directory.

That directory structure is an unpacked version of the ‘X16-49583VS2010Agents1.iso’ Agents ISO from MSDN. My scripts assume that directory structure. I have an additional complication in that my Test Controllers and Test Agents have since been updated with a Knowledge Base patch that I need to run as well (VS10-KB983578-x86.exe or VS10-KB983578-x64.exe) which reside in the Patches directory:

My scripts are written on the assumption those patches are there: customize the scripts and install whatever is needed to establish your TFS, Test Controller and Test Agent patch level. Either way, in this case, you only need to install the KB Patch once so do it after you have installed the Test Controller and the Test Agent.

The first thing we need to do is to create a Response File for our Test Controller and Test Agents so we can automate their installation. Log onto your newly provisioned VIRTUAL MACHINE as the user you intend to run the installation process as: for now I recommend the Domain Administrator or some other super user to remove security complications [as you can tell I am not a Sysadmin!]. Open up a command prompt and execute the Test Controller installer directly but specify the CreateUnattend File switch for the TestController and TestAgent. This will create a Response File we can later use to automate their installation:

Go through the Wizard as you normally would and select ‘SaveSettings’ at the end. When setting up the Response File for the Test Agent, do not select the Performance Tools for now – do that later if you need to support that.

Now we have a Response File for the Test Controller and Test Agent installation on the network share:

KOOL! Now we have all the bits sorted out and we just need to automate the installation of the Test Controllers and Test Agents using those Response Files. We can do this using Vix and running the executables directly from within the context of a user logged into the virtual machine. Before continuing, add the files from the Guest\Scripts directory in the attached Zip to the C:\Scripts directory in your Golden Image: (if you used the ones from Part 2, replace them: I have made a few modifications):

All of the InstallXXX batch files access one parameter – the UNC of the VisualStudioGumpf directory (ie: \\10.0.0.1\VisualStudioGumpf). From there, they can locate the rest of the things they need and they all pretty much look like this:

REM www.havecomputerwillcode.com
REM %1 - The UNC of the VisualStudioGumpf directory. ie: \\10.0.0.1\VisualStudioGumpf

SET SEE_MASK_NOZONECHECKS=1
CMD.EXE /C %1\Agents\TestController\setup.exe /UnattendFile %1\ResponseFiles\TestController.INI

To make sure everything works, you can run the Test Controller Installer manually:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever  -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestController.BAT \\10.0.0.1\VisualStudioGumpf

VmWare Workstation:

vmrun.exe -T ws -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestController.BAT \\10.0.0.1\VisualStudioGumpf

It seemed to take forever to execute the installation because of all the pre-requisites. To fix this, have a look at the Application Event Log on your Virtual Machine and double click each MsiInstaller reference:

To speed up this process, install as many of the pre-requisites as possible on your Golden Image. You will find the pre-requisite installation packages in the WCU sub-directory of the TestController and TestAgent.

Regardless, you can then do the same with the Test Agent:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever  -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestAgent.BAT \\10.0.0.1\VisualStudioGumpf

VmWare Workstation:

vmrun.exe -T ws -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestAgent.BAT \\10.0.0.1\VisualStudioGumpf

… and then finally the patches (if necessary). The .BAT file installs the 32-bit version so change it if you have a 64-bit VM:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever  -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestPatches.BAT \\10.0.0.1\VisualStudioGumpf

VmWare Workstation:

vmrun.exe -T ws -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\InstallTestPatches.BAT \\10.0.0.1\VisualStudioGumpf

You can validate that everything is working hunkey-dorey by looking at Programs on your Virtual Machine:

KOOL!

Automation Summary
Unpack the Part 3 Source Code above and you should now have enough information to automate the provisioning of a new virtual machine and the installation of the Test Controllers and Test Agents. The key differences in the script are:

# ... The VmWare and Domain Parameters from Part 2

# The share location on the domain network which contains an 'Agents', 'ResponseFiles' and 'Patches' directory.
$VisualStudioGumpfUnc = "\\10.0.0.1\VisualStudioGumpf";

# ... The provisioning code from Part 2

InstallTestController $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $VisualStudioGumpfUnc;
InstallTestAgent $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $VisualStudioGumpfUnc;
# If you want to install the KB Patch, run this   Check that the InstallTestPatches.BAT file is installing the correct Platform for your Virtual Machine (32-bit vs 64-bit)
InstallTestPatches $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $VisualStudioGumpfUnc;

Now all the grunt work has been done we can actually start integrating this with TFS and Lab Center…

L8r!

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

Requirements
To do this you will need:

VmWare (The scripts are written for Workstation / Server using VMRUN.EXE/Vix so you WILL need to customize the scripts if you use PowerCLI)

This series has generated a lot of interest so I would like to clarify what I am doing: I am putting together a PowerShell script, piece by piece, that will provision a new, single VmWare machine (replace with virtualization platform of choice), install and configure all of the necessary Test components from the ground up and then register that machine as a physical environment in TFS/Lab Center so it can be targeted using MTM. Despite all the moving parts, this is the bare minimum you need to be able to do to automate the integration of VmWare with TFS/Lab Center/MTM. After that, I will move onto Build-Deploy-Test and then finally customizing the Build Workflows and UI’s explicitly for VmWare. Given the complexity of this, my assumption is that you will follow this through stage by stage and customize the scripts for your Enterprise.

To John, Brian and the half a dozen others who contacted me about these posts: I will reply shortly. I work as Performance Tester during the day and it has been quite a fortnight…!

I have decided as part of these scripts to provision a machine and join it to my domain because this is the easiest approach from an Administration point of view. At a later date, I will update this to work with Workgroups too (when I deal with Fencing). One thing I have learned about Lab Center and TFS is that it touches every part of your infrastructure – DNS, IPSec, DHCP, Hyper-V, SCVMM, TFS and all of its moving parts such as Test Controllers and Test Agents. So each of these posts will be split up into a ‘Manual’ and an ‘Automated’ section. The Manual section will guide you through what the script is going to do step-by-step with the expectation that you follow it through manually in your Enterprise. There are so many moving parts it is guaranteed you will need to understand what the scripts are doing and you will have to modify them for your Enterprise.

Over to provisioning!

Manually provisioning a new Virtual Machine from VmWare using PowerShell
It goes without saying that the first thing we need to do is to get hold of a new virtual machine. This is by far the hardest part because there are so many possibilities depending on what VmWare installation you have and this stage is included only so we have something that works end to end: for this part, you will have to do most of the heavy lifting yourself even though I have provided a script that kind of works with VmWare server and should work with VmWare Workstation (let me know if it does!). Before continuing, I will assume you have a baseline Virtual Machine image already – a ‘Golden Image’ – that is configured in VmWare. This is just an ordinary Virtual Machine that you want to make a lot of duplicates of:

To simplify the script and any issues that might come up, consider doing the following with your Golden Image:

All we are going to do is to make lots of identical copies of that machine (ideally use Sysprep but there’s too many possibilities to discuss here). Every Enterprise has a way of provisioning Virtual Machines dynamically so this is none-issue and I am only including it here for completeness.

And this is where the fun starts! You see, if you are using VmWare Workstation, you can use the ‘VMRUN.EXE Clone’ command to clone a virtual machine. If you are using VmWare Server, you cannot use the VMRUN.EXE Clone command because its not supported.

Annoying.

It is for this reason I am going to fudge this post. Rather than go through all the possibilities, I am going to leave the provisioning step up to you: it is the only part of the process that differs depending on whether you are using VmWare Workstation, VmWare Server, ESX, PowerCLI, Virtual PC or Virtual Box and you will know how to make this cloning happen using whatever virtualization platform you are using. By the end of this post, you just need to have a Virtual Machine that has been joined to a domain.

My script supports VmWare Workstation and (with a fudge) VmWare Server so I will go through the steps for those. But first a few things:

VmWare Workstation supports full machine cloning and that is included in the script. Unfortunately, I do not have VmWare Workstation to test it so let me know if it works

VmWare server does not support Cloning via Vix. But it does allow a single Snapshot for each virtual machine so I treat that single snapshot as the ‘Clone’ (ie: every time I run the script I will revert to it). This is just so I can run the script repeatedly and there are hackey ways to clone a Virtual Machine in VmWare Server (hunt around) but I won’t elaborate here: do whatever you need to do to clone a VmServer image if you want to use that as part of your Enterprise but I won’t inlude the logic in the script (replace the revertToSnapshot call with your Cloning Logic). To set up the snapshot baseline for VmWare Server, power down your Golden Image and take the one and only snapshot you are allowed.

Any operations performed on VmWare from VmRun.Exe require a ‘Virtual Machine VMX Location’. So the first thing to do is to get this identity for the virtual machines. VmRun.Exe (part of the Vix installation) can be used for this:

VmWare Server:

vmrun -h https://localhost:8333/sdk -T server -u "Administrator" -p "whatever" listRegisteredVM

VmWare Workstation:

vmrun -T ws listRegisteredVM

The output of that will be something like this:

Once you know the original image name, you can clone it by doing this:

VmWare Workstation:

vmrun.exe -T ws  clone "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "[VmWareImages] Your New VM Machine Name/Golden XP Pro Service Pack 3.vmx" full

VmWare Server:

# We cannot Clone on VmWare Server so we just revert to a baseline so I don't need to provide all the script stuff to do the duplication in other ways
vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever revertToSnapshot "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "PassInAnyNameHere"

Once that has been done, we just need to start up the machine:

VmWare Workstation:

vmrun.exe -T ws start "[VmWareImages] Your New VM Machine Name/Golden XP Pro Service Pack 3.vmx"

VmWare Server:

# Note: With VmWare Server we start the ORIGINAL machine name because the API does not support Cloning.
vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever  start "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx"

There are a few ways to rename the machine and join it to our domain. NETDOM.EXE is part of the resource kit and is one solution. But WMI is an alternative and it requires no resource kit so I will use that. This is the post I have based the Rename Machine and Join Domain logic on.

The easiest way to rename your cloned machine and join a domain for this series is to ensure the Configuration scripts are already inside your Golden Virtual Machine image at C:\Scripts like this:

We can then rename our machine like this:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever  -gu Graham -gp moo runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\RenameMachine.BAT Moo222

VmWare Workstation:

vmrun.exe -T ws -gu Graham -gp moo runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\RenameMachine.BAT Moo222

…and then we can join it to our Domain:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever -gu Graham -gp moo runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\JoinDomain.BAT Administrator whatever WOOHOO.COM

VmWare Workstation:

vmrun.exe -T ws -gu Graham -gp moo runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\JoinDomain.BAT Administrator whatever WOOHOO.COM

Kool!

There’s a lot more you might want to do to set up a machine ready for automated testing and Build-Deploy-Test. After Joining the domain, you can connect to it directly using WMI using PowerShell or VBScript from anywhere and managing it is easy. More information can be found here in the ‘Virtualization Visual Studio ALM Rangers Base Image Guide.PDF’ (part of the Full Package). Lots of PowerShell snippets such as disabling the screen saver, disabling the firewall and so forth. I will not include that information here but it is something to bare in mind if you want to scale this out across an Enterprise.

By now the Virtual Machine you have just created should be registered with your Domain Controller:

You now have a cloned machine that has been joined to your Domain. For the uber-aware, you might be wondering about the impact of just cloning Virtual Machines and how the ‘SID Problem’ might manifest itself. If you can: use Sysprep for the virtual machine instantiation step. If not, you might not have anything to worry about anyhow! (unless you use WSUS…).

Automation Summary
Assuming all of the above can be done one step at a time manually, you should be able to provision a brand new virtual machine and join it to a domain. The source code contains a file called VmWareTfsIntegration.PS1 and for VmWare Server provisioning it looks like this (replace ‘server’ with ‘ws’ to run against VmWare Workstation):

# VmWare Authentication Parameters. See VmRun.EXE for more information.
#
$VmWareHostType = "server";         # VmWare Server 2.0 (replace with "ws" if Workstation)
$VmWareHostName = "localhost";      # Only needed for VmWare Server. Ignored by Workstation.
$VmWareHostPort = "8333";			# Only needed for VmWare Server. Ignored by Workstation.
$VmWareUsername = "Administrator";  # Only needed for VmWare Server. Ignored by Workstation.
$VmWarePassword = "whatever";       # Only needed for VmWare Server. Ignored by Workstation. 

$VmWareGuestUsername = "Graham";    # Administrator user name you would log in as interactively.
$VmWareGuestPassword = "moo";       # Password for Administrator account. 

# Parameters required for joining a domain
#
$DomainUsername = "Administrator";
$DomainPassword = "whatever";
$DomainName = "WOOHOO.COM";

# The new machine name
$MachineName = "Moo222";

# ...... See PS1 for more information .... 

# STAGE 2: Provision a new Virtual Machine and join it to a domain
CloneVirtualMachine $VmWareConnectionParameters $VmWareGoldenImageSourceLocation $VmWareGoldenImageTargetLocation;

StartVirtualMachine $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword;

RenameVirtualMachine  $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword $MachineName;
StopVirtualMachine $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword;

StartVirtualMachine $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword;
JoinVirtualMachineToDomain $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword $DomainUsername $DomainPassword $DomainName;
StopVirtualMachine $VmWareConnectionParameters $VmWareClonedImageLocation $VmWareGuestUsername $VmWareGuestPassword;

The output should look something like this:

If you are using some other virtualization toolkit, it doesn’t matter: all you need at the end of this section is a machine joined to a domain.

This provisioning step is all a bit of a faff if you don’t have VmWare Enterprise software and I almost skipped it in this series (I wish I had ). But all that aside: we are now ready to automate the installation of the Test Controller and Test Agent and get started on what this post series is really about.

An excellent introduction on VmRun.Exe can be found here

L8r!

Introduction
There have been a few good posts recently on how to integrate Physical Environments into the Build-Deploy-Test cycle in Visual Studio 2010. This one by Colin Dembovsky (and the Darshan Desai one it references) explains how to target a physical environment out of the box with TFS.

But setting up those physical environments and registering them with Lab Center is still a manual task. What I want to do as part of my Build-Deploy-Test cycle is to provision a brand new virtual machine from VmWare, join it to my domain, install the test controllers and test agents, register those test controllers and agents with TFS and construct a Physical Environment in TFS/Lab Center containing those VmWare machines ready for testing. And I want to be able to do that by calling a web service, running a script or pushing a button. And I want the entire process to be automated. How?

This series of posts will achieve that:

Background
I am often asked by customers whether it is possible to use VmWare machines as part of their development effort in VS2010 and Lab Center. And the answer is Yes.

And No.

And Of Course.

And It Depends

So I’m glad I cleared that up! Whether they can use VmWare is really the wrong question. What they should be asking is Can we use VmWare as part of our Test Automation? and Can we use VmWare as part of our nightly Build-Deploy-Test cycle?. So here goes…!

Can we use VmWare as part of our Test Automation?
Yes. Microsoft Test Manager [MTM] can do this by targeting a ‘physical enviroment’ that contains VmWare machines. Before targetting VmWare from MTM you need to:

That is a typical approach taken by clients that have an investment in VmWare. Once the VmWare machines have been configured, you can target those machines in MTM:

The astute technical term for manually configuring those VmWare machines, test controllers and physical environments is: ‘ballache’.

These posts will show you how to automate the creation of VmWare Environments for use in MTM and Lab Center. By the end of this series of posts, you will have a set of scripts that will automatically construct brand new VmWare Environments from the ground up, install the Test Controllers and Test Agents, register them with TFS and construct a Physical Environment to be ready for use with MTM. All automatically.

We do a build every night and want to install and test our software on VmWare Environments automatically afterwards. Can we do that?
Yes! Absolutely! The problem is that out of the box, the Build-Deploy-Test Workflow is designed to target Virtual Environments managed in Lab Center. Lab Center can only manage virtual environments built on Hyper-V and SCVMM. Lab Center has no idea how to start, stop, provision, clone or manage snapshots on a machine that is not managed by SCVMM and it collectively treats every such machine as a ‘Physical Machine’. So if we want to target VmWare and its features as part of Build-Deploy-Test, we need to write the necessary glue ourselves.

This post sums it up nicely!

Let’s get started…!

Create a simple Test Plan with one HTTP Sampler which here goes to www.bbc.co.uk:

To change the value of ‘Retrieve All Embedded Resources from HTML Files’ at runtime, we need to know its underlying property name. The easiest way to do this is to Enable the property, save the JMX file and then open the JMX file in a text editor. Then Disable the property, save the JMX file and open that one up in a text editor: then spot the difference In the case of a HTTP Sampler, it is:

<boolProp name="HTTPSampler.image_parser">true</boolProp>

To change the value of ‘HTTPSampler.image_parser’ during a test, the easiest way is to add a ‘BSF PreProcessor’ to the HTTP Sampler like so:

Dont forget to set the language! Notice the use of the ‘sampler’ object name in the script. It goes without saying that this is the Sampler Context that the PreProcessor applies to. More information on the methods exposed from ‘sampler’ – such as setProperty – can be found here. You can do a lot more with this than you think

Now when you run the test, you will find all of the Content has been downloaded even though it is disabled in the Test Plan:

You could go one step further and use a Jmeter Property that can be changed live during a running test session to represent the Enabled/Disable state of a Sampler property. See my previous post here.

Kool!