My Biased Opinion of Coded UI Test Framework

In November 2012 I was hired as a QA Engineer to automate the testing of .NET web applications. Prior to this the only exposure I’d had to test automation was in my previous job where I spent my last few months at the company building a Proof of Concept (PoC) solution using Selenium RC/Grid, NUnit3 and Specflow (Gherkin) framework, which looked great and did what we needed it to do. Unfortunately, the company didn’t see the benefit in test automation and that was the breaking point for me, which brings me back to November 2012 and my new QA Engineer role.

When I first started my new role, my main responsibility was to design and develop as many regression tests as I could whilst also working on the other aspects of my job. When it comes to working for a company that predominantly uses Microsoft technology it leaves little choice in frameworks and tools that can be used. It was pretty clear that I would have to endure – and I say this in hindsight – Coded UI Test (CUIT) framework, Microsoft Test Manager (MTM), and Visual Studio 2012 (VS2012). Additionally, Powershell (which I recommend when it comes to automation in Windows OS), MSBUILD, Team Foundation Server 2012 (TFS2012) and Hyper-V had to be used to get this heavy machine working as part of a continuous integration (CI) and deployment (CD) process.

Here’s a quick overview of how and why I think CUIT framework is useful:

– Microsoft has designed a pretty slick working MTM system and once it’s up and running you can do a lot of tasks.

– CUITs are easy to setup and run locally, the project comes with an out of the box Click and Record (Coded UI Test Builder) feature.

– It’s pretty easy to locate the element and create asserts with the Click and Record feature.

– It’s possible to hard code a lot of the test steps.

Ok, now my harsh and honest opinion: Each of the above positive points come with couple of negatives. I’ve worked with CUIT for over 2 years and here’s some of the difficulties and real struggles I found with this technology…

– MTM is complicated and comes with a lot of clutter. It’s great that I can record exploratory testing steps and script them in one place, but it’s sluggish when push comes to shove and has crashed on multiple occasions for me.

– Microsoft Lab Management (MLM) which is accessible from the same interface as MTM. As great as it may sound to have it all in one, in this case it isn’t because MLM comes with a shed load of features too.

– Getting those acceptance tests running on multiple VMs as part of CI build is a spider web of mess and basically just a nightmare, there are way too many restrictions. I had to join all myHyper-V Virtual Machines on to a domain, file and printer sharing enabled and have the correct firewall setting configured. Do you want to run multiple test in parallel as part of a build? Go and learn MSBUILD and how to modify the XAML build definition file – not as simple as it sounds.

– Yes it’s easy to setup and run locally, but trying to get them to run across multiple VMs nightly as part of CI builds wasn’t so easy. You have to jump through fire hoops to get it all running smoothly and once you do, Microsoft will do you a “favor” and push an update of some sort (e.g. IE hotfix) that will cause an encrypted message to be thrown every time a test is kicked off.

– Great, so I can use the Coded UI Test Builder from within Visual Studio 2012 Ultimate to click and record elements, actions and asserts that Selenium can also do and for free. Free advice, don’t use the click and record feature in either Selenium IDE or Coded UI Test Builder, just hard code the damn thing and think about using Page Object Model pattern.

– The problem with the click and record feature in CUIT framework is that overtime it builds out an enormous UIMap with a bunch of page element search and filter properties and it just becomes impossible to open, causing even high spec PC to crash. Don’t get me started on the maintenance of it, the duplicate ambiguous hierarchy for each element that is really “well defined”. Of course I’ve read the best practices, creating multiple UIMaps for different pages, trust me, it doesn’t help!

– As you may have gathered the general theme that Coded UI Test portrays is slowness, sluggishness and infuriating, depressing, hair-clutching moments. I’ve created and compared a simple login test between CUIT and Selenium – it took about a minute for the CUIT login test to run in comparison to the Selenium login test which took 10 seconds.

– Implementing Cross Browser testing with CUIT framework? Forget about it! Not impossible, just pointless.

– Documentation, if you’ve ever worked on anything MS related, you’ll find that there are lots of broken links and misleading information. My suggestion is stackoverflow and comments below MS Posts, you’ll find that a lot of the time the answer you’re looking for is in the comment section.

Fortunately, I’ve managed to grow my team and once again build out PoC using Selenium WebDriver, NUnit3 and Page Object Model pattern moving away from the Coded UI Test framework. The future is bright, and I’ve already started experimenting with Specflow (Gherkin)which looks great and I can already see some potential benefits.

If you’re reading this and are in the same shoes as I was and have to use this technology then good luck, I don’t know what MS has to offer with VS2015 Enterprise in terms of Automated UI Testing, I do like their new vNext Build system and hopefully they have re-vamped their CUIT framework for the better. But the cost of the license and previous experience has left a negative after taste for me. I will be posting more specific instructions about dealing with the problems I’ve listed above, hopefully they will be useful and apply to latest MS technology, so watch this space.

Keep Calm and Carry On Learning…