TestComplete vs python – test automation with commercial tools or open source

Recently I was given a unique opportunity to try in parallel two approaches to test automation. I was automating tests of 2 applications. One was tested using python scripts and open source libraries, one was automated using well known commercial tool (TestComplete).

I decided to share my observations with others. I believe they might be helpful, since open source vs commercial dilemma is still present in test automation.

The context

My current client takes test automation very seriously. The stability of the GUI interface is crucial for their applications. For them test automation is not nice to have – it is a must.

One of their major applications is automatically tested with TestComplete. Let’s call this application CommApp (application tested with commercial tool). Another one is tested with python using open source library pywinauto. Let’s call this application OpenApp (application tested with open source tools).

Why 2 approaches in one company?

Firstly – why TestComplete? When I joined the project, TestComplete was already there. I did not investigate all the reasons of this purchase. For sure, the most important one is that TestComplete supports some GUI controls, which no one else does. Period. In this particular context it was a killer feature.

Here the first general observation – when you are to automate GUI, check carefully what types of controls are you going to automate. Then think to what extent you need to test them. Unless you answer these 2 questions, you risk wasting money on purchasing tools which will be of no use.

Let’s talk about it a little bit more. When you have some uncommon, not so widely supported GUI control, your problems will be about reading it, about checking the state of the control. Writing to control, changing it, is simple – you click it or use a keyboard. Both mouse and keyboard are automated in all GUI automation tools. You might need some additional coordinates math to find out where to click, but it is doable with minor to moderate effort. The problem is about reading what is in the control. This is where implementation details of your control come into play. You can of course try finding those details yourself. However, be warned – thing twice, no, not twice – think 8-10 times before diving into it. Inner workings of GUI controls is a very technically sophisticated area, dependent on many factors, requiring intimate knowledge of the platform on which your software runs.

If reading a control is so difficult, the next question is – do you really need to read it? After all, if it is a 3rd party widget, testing it is not your task. You test the logical flow triggered by some actions on this control, not the control itself. From the other hand, sometimes you might need to read what is there. There is no universal answer here. Just make sure, that you and your team have taken this question into consideration.

If you are on a position of choosing the technology on which your software will be created, keep aforementioned factors in mind. One day you might realize that test automation is really needed. If your application uses well known, widely supported controls, you will have many commercial and free options to do that. You will also have some communities, forums, groups in which you will be able to find support. Otherwise, you will be forced to buy expensive commercial tools. Or to make several workarounds and compromises.

That’s it for now. I hope I’ve whetted your appetite. If so, keep checking my posts ??