Problems of TestComplete

This is the forth part in my series on test automation with Test Complete and open source.

You can find the first episode here.

Let’s set the record straight – I am biased towards open source. And it is difficult for a human being to fully control his/her biases. I will do my best. I DO want to make a fair judgement.

In this chapter I will write about problems of TestComplete. However, I don’t want to write a rant on how bad commercial software is. Therefore, with all honesty I can tell you that TestComplete is a good, useful, stable piece of software. It does provide valuable support in test automation. I am not questioning its usefulness.

TestComplete suffers from two reasons:

• It is a commercial software, so bells and whistles are more important than fitting to widely adopted standards
• The population of user is relatively small

I have wrote how does it impact the software in previous post here.

TestComplete is a yet another commercial test automation tool, which tries to convince you, that you can automate tests without programming. Trust me – in most cases you cannot. TestComplete designers are using a notion of a keyword testing, which is expected to deliver the promised land of test automation without programming. However, in my project it has occurred yet another time that you cannot do a flexible, full blown automation without coding. So the analysis of a keyword testing ends here.

TestComplete allows you to write test cases with python, but… When one hears – this tools uses python – one thinks of the whole python ecosystem of tools and libraries which will be available. However, it is not so in TestComplete. It does integrate to some extent with some version of python, but clearly warns in documentation, that if you install some library not originally installed, you are on your own. It is probably not a problem for pure python libraries, but it is a problem for libs which have binaries. For example ODBC. Who knows, maybe it will work. And maybe won’t. When you are on billable project with deadlines, you are not so eager to experiment with extensions, which according to tool’s provider might not work. Secondly you cannot use standard “pip” to install a package, but you have to place the right files in the right location. One might say “no big deal”, but it is annoying and unexpected.

When we are on python, TestComplete clearly suffers from too small population of users. Its python has some surprising annoyances, which are totally unexpected. So far we have found 2:

1.      It takes an extremely long time to get a field from a row retrieved from the database. You make a select, you fetch, you now have the row data in your memory. And you want to get column’s value. It takes extremely long time – 0,1 second. Today, for RAM operations 0,1 second is eternity. You do expect to find bottlenecks in you software, but not there.

After several hours of experimenting we have given up. We have reduced the number of columns we retrieve.

2.      When you create a new file (open, write something and close) you have to wait some time, for the file to be created. If just after “close”, you will try to open the file again, you will get “file does not exist”. It was so surprising, that it took me some time to realize, that that was the actual problem.

TestComplete uses ADO to access the database. Or some python dialect of ADO. Documentation is poor and it redirects you to some other places, which also are not so clearly documented. Sure, if I have spent couple of days I would probably understand ADO better and would know better where to look for help. However, if I was given a chance to install ODBC I would be on the well standardized ground in 30 minutes.

Logging – “tail -f” is a mantra of testers and admins. Once you know where the logs are you have all the power of tail, grep, scripting language like perl, python on your side. Providing your logs are line oriented. TestComplete creates separate xml log for each test case. XML format is of course supported by many tools and there are many useful thing you can do with them. However tail -f does not work, which is annoying.

TestComplete has some internal libraries, some internal constants, which can be used in python code but are not defined in a pythonic way. As a result, when you want to generate a documentation, with Sphinx or pydoc, you cannot do it. I am assuming some of my readers might not know python, so I will give a general explanation why it is not possible. Documentation generators (tools, which parse your code and retrieve specially formatted comments and build document out if it) for python try to compile the module, from which they extract comments. If the file does not compile, then comments are not extracted. Since some TestComplete specific libraries do not follow the pythonic way of defining a library, the modules using those libraries cannot be compiled outside of TestComplete and correctly parsed.

TestComplete provides intellisense. But it does not work on classes, especially user defined.

None of those aforementioned problems is a blocker. You can create a workaround, you can make a reasonable compromise. However, making them takes time.

In my second post I have written something like this:

“With all those compromises and workarounds I have created a set of robust test cases, with very detailed verification of results, which covered important functional paths in OpenApp. Everything was implemented from scratch, using python. Flexible logging, progress tracing, restarts, resuming from the last failure point, sophisticated reporting of database contents – all this is in the framework. It did require some effort, but not as big as one might initially think.

One may stop here and say – isn’t all this given out of the box in commercial tools like TestComplete. Therefore, isn’t writing a test harness and typical test infrastructure an additional cost of open source?”

So here is the answer – the fact that commercial tool promises plug and play out of the box, does not mean that it is actually delivered. It occurs that adopting the out of box functionality to your needs does take time as well. And there are unexpected problems. There is more – if you create a testing infrastructure in pure python, your intellectual investment is more likely to pay back. You’ve learned and developed something, which you will be able to use in other jobs, where python is needed. When you adopt TestComplete to your needs, you develop something which will be used in a job in company, which happens to use TestComplete, which is far less likely than the first case. TestComplete is expensive and it is by no means a de facto standard.

Is TestComplete a waste of money? Absolutely not. It is a stable (it was not in the past, but there is a noticeable improvement), it is useful and it does provide support for test automation. And in my case – it does support some controls, which no one else does.

However, is it worth to buy TestComplete? I believe it is not. Most of features it delivers can be get for free or developed in a manner, which better fits to an open source ecosystem. Moreover, those features, which are delivered with TestComplete are not good enough to justify the price.

TestComplete has a fancy gui actions recording. This is nice, but can be easily replaced by using WinSpy. It will not look so sexy, but will do the job as good as recording.

It is pretty possible, that after reading this post, someone will respond with clean solutions to all the problems described above, with links to according chapters of TestComplete documentation. Would it make my points invalid? No. If the tool fails to deliver commonly accepted solution to a common problem then the tool has a problem. Looking for an "error" string in logs is done this way - “tail -f logfile.log | grep -i error”  from the command line. Period. If tool’s provider forces me to read his documentation to find out, that in order to search the log I need to run a gui tool LogSuperScanner, in which in 4th level of menu I can specify what I want to grep for – then I feel that someone is wasting my time. Someone is forcing me to spend my precious time on learning inferior solution to a problem, which was already solved well.

Summary

My recommendation from the experience documented above is this:

Unless TestComplete handles some type of software which no other tool does, avoid it. This tool is being developed by smart guys, but it suffers from problems typical for specialized tools having limited population of users. It is improving, becomes more and more robust. The problem is – you can get the same or higher level of robustness, usefulness and stability for free.

Going open source requires that you possess development skills. So does TestComplete. The difference is that open source does not make any promise that you can automate without those skills. TestComplete (like all other commercial tools) makes this promise. And fails for keep it.