HaveComputerWillCode.Com

PLEASE NOTE: This is for Visual Studio 2010. For the VS2012 version, please click here.

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Part 1 – Introduction.

Part 2 – Provisioning a new VmWare machine using Vix and joining a domain.

Part 3 – Automating the installation of the Test Controller and Test Agent.

Part 4 – Automating the configuration and registration of the Test Controller and Test Agent with TFS.

Part 5 – Automating the creation of a Physical Enviroment in Lab Center.(Source Code) [Updated: Removed comment]

Part 6 – Service Pack 1 Notes

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

See Part 2 for Requirements (PowerShell, Vix etc.).

DISCLAIMER!
Do not use this code under any circumstances (should just about cover the possibilities!).

I am using an undocumented API in order to construct the Physical Environment in Lab Center and set up the Test Controller Topology. I have tested the registered environments using MTM, use it often and come to no grief. My Lab Center and TFS system appears to be stable. But you use this at your own risk! At the very least, it would be sensible to do a full back up of your TFS Installation and ideally test this prior to production deployment. In particular: this has not yet been tested with Visual Studio 2010 Server Pack 1 (it has now, and it appears to work as well as pre-SP1) but use at your own risk

In future, if and when Microsoft provides a way to create Phyical environments and construct Test Controller / Test Agent hierarchies programmatically, this is the only stage you will have to change in this series. Everything else you have done as part of this series still applies.

Background
That aside: this is the REALLY AWESOME PART!! Before going on, consider provisioning a brand new virtual machine using the scripts at the end of Part 4.

There are no officially supported APIs to manage the Test Controllers and Test Agents in TFS. This is unfortunate because it stops you creating Physical Environments and constructing your Test Controller / Test Agent hierarchies on the fly. You proceed at your own risk!

The Test Controllers and Test Agents are all written in .Net and Microsoft has a tendency to make all of its classes ‘internal’ so you cannot reuse them willy-nilly. I like this approach: rather than obfuscating their code they are polite enough to let us wander around their work if we need to in order to troubleshoot or find out WTF is going on. It’s a pragmatic solution between opening things up to all and sundry and having to support their firstborn for life.

The code we need to invoke to construct our Test Controller and Test Agent hierarchies is stored in ‘internal’ classes so we need to find a way of instantiating those classes and invoking their methods.

A major help is that IL has no concept of ‘internal’: it is the compiler that enforces that contract (well… its not THAT simple!) opening up all kinds of creative possibilities! But that aside, you can create internal classes from other libraries just by using reflection like so:

System.Reflection.Assembly controllerAssembly = System.Reflection.Assembly.LoadFile(theAssemblyPath); 
Type controllerTestEnvironmentType = controllerAssembly.GetType(theFullyQualifiedTypeName);

object testEnvironment = controllerTestEnvironmentType.InvokeMember("", System.Reflection.BindingFlags.CreateInstance, null, controllerTestEnvironmentType, null);

…and you can get hold of properties, invoke private and protected methods and everything else you want using nothing more than reflection. But in .Net 4.0 there is an easier way! The ‘dynamic’ keyword will leave property and method resolution until runtime. Think of it as IDispatchEx from the COM World or expandos in JavaScript. It is very cool because by wrapping an internal type you can treat it like any other object. I use the code from that link verbatim but there’s a lot of extra stuff you can do with that if you put in the time.

The reason I am mentioning it? Because you will see it in my code when I invoke the internal or private Test Controller classes:

            dynamic controllerConnectionManagerWrapper = new AccessPrivateWrapper(controllerConnectionManager);

            controllerConnectionManagerWrapper.UpdateTestEnvironment(testEnvironment);

Manually creating a Physical Environment using a UI
The source code this time includes all of the scripts and some supporting Visual Studio 2010 Solutions that you will have to build yourself:

The AgentHelpers library creates physical environments in Lab Center and sets up the Test Controller / Test Agents ready for use in that environment.

A crude UI Application (PhysicalEnvironments) that uses the library to create physical environments in Lab Center

A command line application called AgentInteraction.EXE is used to set up the environments as part of the PowerShell script

The first thing we need to do is to make sure you can manually construct Physical Environments using my User Interface because it drives the underlying TFS libraries. If not: not much point going on!

Assuming you have heeded the disclaimer, you will need to add references to several standard TFS API’s including:

The interesting three are:

Microsoft.TeamFoundation.Lab.Activites – in C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\TeamLab\Microsoft.TeamFoundation.Lab.Activities.dll

Microsoft.VisualStudio.QualityTools.ControllerObject – in C:\Windows\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.ControllerObject\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.ControllerObject.dll

Microsoft.VisualStudio.QualityTools.ExecutionCommon – in C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0\Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll

It is important you set *ALL* of those DLL’s to ‘Copy local: True’ for the Reference. You will need to add them to AgentInteraction and PhysicalEnvironments as well.

Now run the PhysicalEnvironments application and you will see a glorious ‘Graham UI’. I loathe arty-farty work and I loathe writing User Interfaces so much, and have consistently done such a bad job of it, that I never get asked to write them any more (mission accomplished!). Select the TFS Team Project Collection / Team Project and observe what happens:

This UI is listing all of the Test Controllers on the left: that is the same list you see when you create a new environment in Lab Center and are asked to select the Test Controller you want to create the environment on:

When you select one of the Test Controllers (such as Moo1250) you will see the list of Agents available. This list is identical to what you can see in Lab Center as well:

So we have all of the information we need to start constructing Physical Environments. Enter a new name for the Physical Environment you want to create – such as “WooHOO” – tick the single Agent, enter a sensible Role name, and select “Create New Environment”:

Your UI will now gloriously lock up for about a minute (I provide no UI Cues) whilst Lab Center does whatever it needs to do to create your environment. Assuming all went well (you’ll know when focus returns to your window) you should be able to see your environment in Lab Center:

AWESOME! So we have all the bits and pieces.

Before going on, delete the environment in Lab Center and then continue.

Automating the creation of a Physical Environment using PowerShell
The PowerShell script obviously doesn’t use the User Interface but I have created a dirty, hackey-wackey Console application called ‘AgentInteraction.EXE’ that is only moderately unforgiving of command line mistakes but can be used to drive the registration process.

To set this up to work, create a new subdirectory under VisualStudioGumpf called AgentInteraction and drop these DLL’s (AgentHelpers/bin/Debug), AgentInteraction.EXE and anything pulled in as part of the build into it:

Assuming you have removed the Test Controller and everything from Lab Center, you should be able to execute code like this:

# Obtain a list of all Agents known to a Test Controller
\\10.0.0.1\VisualStudioGumpf\AgentInteraction\AgentInteraction.EXE Command=ListAgents TestControllerUri="MOO1251.WOOHOO.COM:6901"

# To create an environment on the Test Controller MOO1251 and with a single Agent called MOO1251 with a Role of 'Desktop Client' you would execute:
\\10.0.0.1\VisualStudioGumpf\AgentInteraction\AgentInteraction.EXE command=RegisterEnvironment testcontrolleruri="MOO1251.WOOHOO.COM:6901" TeamProjectName="TestSample" EnvironmentName="My New Environment" Agents="MOO1251,The Web Server"

… and see your environment created in Lab Center from PowerShell:

Kool!

Automation Summary
Assuming everything is in place, you should now be able to dynamically provision new virtual machines in VmWare, install the Test Controllers and Test Agents, register them with TFS and create a new Physical Environment on a whim. The key differences for this post are:

# ... variables from previous stages

# The Team Project Name we want to register the environment with and the environment name
$TeamProjectName = "TestSample";
$EnvironmentName = "My Brand New Environment";
$MachineRole = "Database Server";

# ...

# STAGE 5: Create the Environment
# It can take a while for the VM to spin up and the Test Controller + Test Agent to come online... so use the RETRYCOUNT here.
Invoke-Expression "$VisualStudioGumpfUnc\AgentInteraction\AgentInteraction.EXE command=RegisterEnvironment testcontrolleruri=$($MachineName).$($DomainName):6901 TeamProjectName=`"$TeamProjectName`" EnvironmentName=`"$EnvironmentName`" Agents=`"$MachineName,$MachineRole`" RetryCount=10";

Agents=`"$MachineName,$MachineRole`" ";

As a final test, set it as a targettable environment in MTM and run your Tests:

Mission Accomplished! However… there is an anomoly! When I went into the Test Controllers view and clicked a Test Controller, I noticed that duplicate Test Controller names appeared for those that were put on Virtual Machines that I had provisioned:

At first I thought it was some SID issue with the cloned virtual machine but it appears to be down to a legacy DNS Issue:

I was able to repeat this 100%. If you remove the legacy DNS Record (ie: for Moo1400 which is a virtual machine I purged) and restart MTM, the problem goes away (I wasn’t able to repeat it after this).

So I guess you need to purge the DNS Record as well as the Computer from your Active Directory when you throw away an environment.

L8r!

PLEASE NOTE: This is for Visual Studio 2010. For the VS2012 version, please click here.

This series of posts will show you how to provision VmWare machines, install Test Controllers and Test Agents, register those with TFS and construct a Physical Environment within Lab Center so that it can be targeted by Microsoft Test Manager. All automatically using PowerShell (and VmRun.EXE):

Part 1 – Introduction.

Part 2 – Provisioning a new VmWare machine using Vix and joining a domain. (Source Code) [See Part 5]

Part 3 – Automating the installation of the Test Controller and Test Agent.(Source Code) [See Part 5]

Part 4 – Automating the configuration and registration of the Test Controller and Test Agent with TFS.

Part 5 – Automating the creation of a Physical Enviroment in Lab Center. (Source Code)

Part 6 – Service Pack 1 Notes

Build-Deploy-Test with VmWare is a superset of this so I will tackle that after this series.

See Part 2 for Requirements (PowerShell, Vix etc.).

After installing the Test Controller and Test Agents and ensuring the patch level matches our baseline, we need to automate the registration of the Test Controller with TFS and automate the registration of the Test Agent with the Test Controller.

Manually registering the Test Controller
Before going on I will assume you have created a new Virtual Machine by running the scripts created in Stage 2 and installed the Test Controller and Test Agent in Stage 3. We will use that as a baseline. Use the snapshot and restore functionality of VmWare liberally here – that’s what it is for!

Checkpoint 1 – Machine with Test Controller and Test Agent installed but not configured
There are a lot of possiblities and topologies when setting up environments, test controllers and test agents. For this series I am creating a single VmWare machine that will contain both the Test Controller *AND* the Test Agent.

If you’ve set up a Test Controller before you will be familiar with this user interface. Launch it from within your Virtual Machine:

There are so many moving parts and opportunities for failure so large that the best way to identify issues with the script is to do everything manually first. So…

Enter the parameters and make sure the configuration works if you do it manually:

If the operation was successful you will be able to see the Test Controller in Lab Center:

If you cannot see the controller in Lab Center at this point, you need to work out why. There is little point going on. The first thing to do is to disable the Firewall on your provisioned virtual machine or add an exception for the Test Controller application to see if that helps: this is usually the problem (my batch file in the source code package adds an exception using netsh). Otherwise, the possibilities are too many to mention here. Whatever changes you make you will either need to put into the PowerShell script or backpatch into your baseline VmWare image and re-provision.

The User Interface Application used to configure the Test Controller above is called ‘TestControllerConfigUI.EXE’. If you hunt around on your virtual machine disk you will find one called ‘TestControllerConfig.EXE’ that lives in the same directory. ‘TestControllerConfig.EXE’ is what we use to automate setting those parameters.

Now revert to Checkpoint 1 in the VM. To Undo (unregister) the Test Controller from Lab Center, do this first within your Virtual Machine.

We will now automate the Test Controller registration using the command line tool:

I have no idea what can take so long: but it does.

Assuming that works you can confirm the Test Controller has registered itself with TFS by reopening Lab Center and seeing your Test Controller in all its glory:

GOOD! We now have everything in place to register the Test Controller with Lab Center. If you needed to make any changes to your virtual machine to get this working (such as modifying the Firewall settings) you either need to make that part of your Virtual Machine baseline or you need modify the scripts I have written to create Firewall exceptions on the fly.

Finally, to ensure this is going to work from VMRUN.EXE, execute this (after copying across the Guest/Script batch files from the attached source code into the Virtual Machine C:\Scripts directory as usual):

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestController.BAT http://WIN-TSKURFCLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection WOOHOO.COM\TfsTest whatever SomeOtherText

VmWare Workstation:

vmrun.exe -T ws -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestController.BAT http://WIN-TSKURFCLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection WOOHOO.COM\TfsTest whatever SomeOtherText

Note the addition of ‘SomeOtherText’ to the end. For some inexplicable reason, one less parameter than usual was being passed through to the Batch file from VMRUN.EXE. I have no idea why, but adding another one to the end fixed it. Bizarre!

Manually registering the Test Agent
Checkpoint 2 – Test Controller Installed and Registered with Lab Center. Test Agent Installed but not Registered with Test Controller.
Now we just need to do the same thing for the Test Agent. We know the Test Controller works so we just need to register our Test Agent with it so we can add that Agent to new environments in Lab Center. Open up the Test Agent Configuration Tool and set up the parameters you want. I am going to run this as an interactive process (not a service) so I’ll point out the difference later when we automate it:

Notice we are using ‘localhost’ to talk to the Test Controller we installed earlier on the same machine. If you are provisioning multiple machines here, you obviously point it to where your Test Controller is installed

Once again, do that manually to check it works and troubleshoot any issues. To check that we are getting visibility over our Test Controllers and Agents we can now go into Lab Center, select Controllers and observe that our Test Agent is present:

If you can’t see your Test Agent you have to troubleshoot why before you proceed.

Once this works, revert to Checkpoint 2. To automate the Test Agent configuration from the command line, there is a tool called ‘TestAgentConfig.EXE’ that does it for us within your VM:

In the above case, I was running the Test Configuration utility as WOOHOO.COM\Administrator but want the Test Agent to run as Graham (Administrator account on the Virtual Machine). So I need to reboot (and you will see that in my scripts).

When I restarted the Virtual Machine, I was told that QTAgentService.EXE was still blocked by my firewall: so I need to add an explicit exception for that to the Windows Firewall and you will find that call in the ConfigureTestAgent.BAT file which looks like this:

REM Add a Firewall Exception for our Agent Service executable. 
netsh firewall add allowedprogram program="C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\QTAgentService.EXE" name="QTAgentService.EXE" mode=ENABLE scope=ALL profile=ALL

For now, Unblock the application using the Firewall Alert (if you are asked) manually so that you can continue. When you run the ConfigureTestAgent.BAT batch file using VmRun.Exe, the exception will be added.

The parameters map one-to-one with what is in the UI. Once your Agent User Interface comes up, you should be able to see your Controller and Agent in Lab Center again:

FINALLY! Revert to Checkpoint 2 again. To ‘Undo’ (Unregister the Test Agent with the Test Controller) just do this from a Command Prompt in your Virtual Machine:

C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\TestAgentConfig.EXE Delete

Now we can see that this is going to work using our VmRun.EXE call with our Batch File:

VmWare Server:

vmrun.exe -T server -h https://localhost:8333/sdk -u Administrator -p whatever -gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestAgent.BAT Graham moo localhost:6901 SomeExtraStuffSoTheParameterGetsSentThrough

VmWare Workstation:

vmrun.exe -T ws gu WOOHOO\Administrator -gp whatever runProgramInGuest "[VmWareImages] Golden XP Pro Service Pack 3/Golden XP Pro Service Pack 3.vmx" "C:\Windows\System32\CMD.EXE" /C C:\scripts\ConfigureTestAgent.BAT Graham moo localhost:6901 SomeExtraStuffSoTheParameterGetsSentThrough

After that, check once again you can see your Controller and Agent in TFS:

Awesome! So that’s all the moving parts tamed.

Automation Summary
Unpack the Source Code for Part 4. With the source code, you should be able to provision a new virtual machine from VmWare; install the Test Controller and Test Agent; and register them with TFS automatically.

This snippet shows the key changes to the VmWareTfsIntegration.ps1 script for this stage:

# ...

# Test Controller Configuration Parameters
$TestControllerTfsTeamProjectCollection = "http://WINTSKURF-CLU7G.WOOHOO.COM:8080/Tfs/DefaultCollection";
$TestControllerTfsServiceUser = "WOOHOO.COM\TfsTest";
$TestControllerTfsServicePassword = "whatever";

# Test Agent Configuration Parameters
# I want to log in the user automatically so I can do UI Testing: these are the credentials under which the Test will be run. 
$TestProcessUsername = "Graham";
$TestProcessPassword = "moo";
# The Test Agent needs to know what Test Controller to connect to. As we only have one machine, use localhost.
$TestControllerUri = "localhost:6901";

# ... Provision and Install Test Controller + Test Agent

# STAGE 4: Configure Test Controller + Test Agent
ConfigureTestController $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $TestControllerTfsTeamProjectCollection $TestControllerTfsServiceUser $TestControllerTfsServicePassword
ConfigureTestAgent $VmWareConnectionParameters $VmWareClonedImageLocation "$DomainName\$DomainUsername" $DomainPassword $TestProcessUsername $TestProcessPassword $TestControllerUri

Before going on I recommend constructing a Physical Environment using that Controller and Agent and targetting it via MTM in a simple test. If you can do this, you know the system works end-to-end and it will make Stage 5 much easier

Only one more step! And that is to automate the creation of a Physical Environment in TFS so we can run our tests against them using MTM.

L8r!