How effective is your Test Automation Suite?

"No tests will be changed,Only Transformed "

Although automated test cases cannot be compared to manual test , fact is planned test automation solutions saves manual testing effort. Success of Test automation could be gauged on following factors:

• BIG SAVINGS!!!..Benefits gained via test automation in terms of saved manual testing hours regression time of additional tests performed and a number of defects found (including how early in testing phase).
• It's no joke time when automation takes time How much effort and time engaged in automation when compared to manual.Sometimes the effort to test automation is more or less same when executing the test manually.If such is the case it could be discouraging towards the effort for automation.
• Isn't analysis part Automated!! Quite often effort to analyze failures and errors of SUT being tested via test automation take more time than when manually.Although this factor can be curbed with of verbose logging on both system under test and test automation suite.
• Someone said execution takes time .Really?? . To maintain automated tests can be quite challenging and failure to match expectations can lead to failed Automation project.Automating test cases that are fragile to changes requires more maintenance and effort required to keep tests updated can be expressed in hours or percentage of time spent in maintaining.
• The whole sheet is RED!!? A considerable number of tests can fail during execution of tests in automation suite.Analysis of such cases takes significant time to confirm if bug triggered tests failure or if there are more issues than one.Better test design is required if of such observation is incrementing.
• How many tests are they ?the automated ones? Although not a true indicator still counts matters and shows progress test automation project is achieving.
• Not a Joke Fail....Count of automation test passed and failed is another indicator progress in project .Analysis is still required to confirm if failure is caused by system under test defect or via problem in automation solution.
• AAAAArrGH!! is not my expression.One of the major frustrations faced in test automation is false test results.False Fail is the case when a problem is in Test automation Solution and not in the System being tested, kinda of false alarm.False Pass is when test results are all GREEN(my kinda color) because test automation failed to identify a bug in the SUT.Confirm and editing of Test Verification with reviewing of checks is required at this stage.
• What Features are covered in test automation? Number of features automated can be considered as a metric for functionality and feature coverage and hence the effectiveness of test automation.
• How long will the reply take? Some features in SUT are critical to response time.If SUT is not performing in same nature within test defined expected time, it indicates that design considerations for test solutions need reviewing and better definition.

In the end, the cost it takes to collect all the information above should be easy to acquire via test reports ,If not... it's time to automate !!!