Commercial vs open source - part 2
This is the second part in my series on test automation with Test Complete and open source.
You can find the previous episode here.
In my previous post I have explained why we are using TestComplete. One may ask – if you have TestComplete, why are you using open source? It is a reasonable question. Here is the story. When I started to support my current client, they asked me to help them in test automation of an application (I will call it OpenApp) with controls which TestComplete was unable to support. Technology used in OpenApp was similar to the one used in CommApp, but due to some implementation tricks, gui widgets became hard to control even by TestComplete. And my client came to me for help. 2 factors were decisive in choosing the open source – price of TestComplete and my personal preferences. TestComplete is expensive. My client had a pool of licenses, all of them were used, so quite understandably, they were not willing to dedicate such an expensive resource for experiments. Secondly, I personally prefer open source. I also have some concerns about commercial tools (which, by the ways, occurred to be well justified – I will write about it later). Therefore, I was happy to go with pure python scripting, supported by pywinauto library.
Those open source tools also were uncapable to fully support controls in question. However, in my previous post I have written that toughness of automation of a gui control is in reading it. Not writing. That was the case in my task as well. It has occurred, that although reading some grids would help, one can test without it. One can test a lot of scenarios with a very detailed verification of results. In some, single cases, reading the grid was indeed very important. In such a case we used OCR (Tessaract). However, I have to warn you – OCR did its job, but do not think of it as a universal tool for tough GUI controls. It is slow, it does make mistakes. You can teach it, but it does require some knowledge how to do it. If you have a skilled developer, who knows how to use and teach OCR, then, who knows, maybe you will be able to handle any widget ever written. However, without those skills, do not expect from OCR to be plug-and-play, which will correctly read all the texts. Secondly – it is about scanning images, so if the areas you scan are big, it might considerably slow down your tests.
With all those compromises and workarounds I have created a set of robust test cases, with very detailed verification of results, which covered important functional paths in OpenApp. Everything was implemented from scratch, using python. Flexible logging, progress tracing, restarts, resuming from the last failure point, sophisticated reporting of database contents – all this is in the framework. It did require some effort, but not as big as one might initially think.
One may stop here and say – isn’t all this given out of the box in commercial tools like TestComplete? Therefore, isn’t writing a test harness and typical test infrastructure an additional cost of open source? Yes, this effort should be considered in our equation if we want to make a fair comparison. However, is TestComplete’s out of the box infrastructure exactly what you need? Something, which you just take and use? This is a different story, and I will talk about it in my next post.