Automated testing is actually the only way to handle complex software these days - otherwise you have many individuals who spend their time doing manually the tests that need to be done.

What tools?  There is a whole subset of the software industry that created automated testing tools.  I've used shell scripts that run simple tests and compare output; the 'expect' programming language to automate interactive keyboard testing; WinRunner to automate GUI level testing; and LoadRunner for pushing products with heavy loads.

Many exist in the Open Source community also.  Your mileage may vary depending on the knowledge of the tool and its applicability to the testing you are attempting.

What is the ROI?  Freeing up testers to develop new tests for functions being added to the product, the ability to run testing 24x7 without actually having to pay people to be in the testlab during all of it.

I've worked 3,5 years as a software developer for testautomation. You don't have to use a tool, you can create your own testautomation program. This programm should be manually tested by the software testers, because the most important thing is that to trust your testautomation. It must be clean and as much error-free as possible, so that each time it reports an error, you can be almost certain that the error lies in the tested program and is not a false alarm of testautomation.

Testautomation should test the components of a program that remain (almost) constant over time and usually comprise the basic functionality that you don't want to miss - the things that should work no matter what. For those, you can provide an API and build your testautomation in any language you want, using that API. You usually want the manual testers to spend time testing the new features, exploring the app and not check the same basic things repeatedly at each release - that bores many of them.

What I saw as a big plus was automated data generation (which is not exactly testautomation). Having a program that traverses your application, simulating many user's actions (much faster than any human tester can do), can really find bugs in unusual scenarios (again following paths that human testers don't usually follow, because they don't reflect logical steps) and uncovering memory bottlenecks that a human tester will need to run the app days to uncover them.

Many of your questions are answered in the book : "xUnit Test Patterns: Refactoring Test Code" from Gerard Meszaros.

Thank you for satisfactory answers.

Designing own testing API and development of own test automation tool. It really makes sense to me.

But, It's like all these roads lead "Unit Testing" or a "modified Unit Testing".

Am i wrong?

Thank you for satisfactory answers.

Designing own testing API and development of own test automation tool. It really makes sense to me.

But, It's like all these roads lead "Unit Testing" or a "modified Unit Testing".

Am i wrong?

One needs to differentiate between the automatic generation of tests and the automatic execution of tests.

Unit testing should be implemented as functional testing of components from a contract/user perspective, i.e. that the service/method/operation is nor refused (no exception thrown) if all pre-conditions hold and that all post-conditions hold if the service is provided. The component is tested in a test-bed of mock objects which fulfill their respective components (usually provided by a dependency injection framework) and hence one only tests the functionality of the component being tested - not the lower level components it uses for its own functionality. In integration testing, the mock objects are replaced by actual system objects (once again usually via dependency injection).Note unit and integration tests can be both done across levels of granularity and that they should be the same test code, just that the code is run against the component in a mocking or real environment.

The execution of both, unit and integration tests is usually automated within a build environment like Maven or Gradle. However, the generation of these tests is less often automated. In the case where the component/services contracts are formally or semi-formally specified (e.g.\ using the object constraint language), the generation of these tests can be automated.

Of course there is still non-functional testing like performance, scalability,. security, reliability, ..., testing. Again, the execution of these tests could be automated, whilst the automatic generation of these tests can be automated only to some extend.

In theory generic automated testing (or tools) would not exist. Test automation would be likely to be program specific -- when you design your program you make also consider the design of a program to test your program:-)

I fully agree with Fritz that you need to distinguish between automatic generation and automatic execution.

Years ago, I worked in a project in which automatic generation and execution of tests was essential.  We defined two suites of tests: CST (Confidence Self Test) and MST (Maintenance Self Test).  To test a piece of equipment, we designed a matching test harness and used ATLAS (Automatic Test Language for All Systems, quite an arrogant name!) to program it, so that the all inputs of the equipment under test could be stimulated and all output sampled with the appropriate delays.

A program would then compare the outputs produced by the equipment under test with the expected values.

Once test harness and software were fully functional, we could use them to diagnose problems in faulty equipment retrieved from the field.

Being agile these days requires efficiency in delivering rapid changes while maintain some level of quality.  To do that, it is essential to have a continuous delivery pipeline, with automated tests to ensure some level of quality and speed.  I've worked in software teams for 30 years, but the last 10 or so have seen this aspect of agility offer teams a tremendous speed and quality advantage, but it comes with a price of trade-offs.  Your last two questions reflect that price.

What percentage of your test cases are automated? What is the ROI?

My work in cloud or RESTful API development teams (JEE/Javascript/Python/Ruby) never sees anywhere near what the team would consider adequate test automation. What is that?  It's generally the shortest-distance path from delivering features in sprints that are demonstrable and minimally acceptable.  Much like the software itself, tests are features too, but often unrecognized as such during sprints or continuous development or delivery.  Minimal tests perhaps best describes this phenomenon. So, in many teams, the percentage is typically small compared to what is often believed by the team to be actually needed.  However, the ROI on whatever automated tests (e.g., Jenkins, cucumber, etc.) there are is often perceived to be high.  As the software evolves in features, the tests coevolve with it, so percentage and ROI of automation are very contextual (i.e., not useful to compare across projects).   For example, my last cloud product had no unit tests, and, we thought, too few system (end-to-end) tests.  We favored end-to-end tests over unit tests to enhance or delivery speed, but we sacrificed some quality to do it.   In short, agility, includes automated tests but there are trade-offs during exploring new features and exploiting delivery of those features (Lyytinen & Rose, 2006).  I don't see a lot of automated test generation. Dev/testers often create them them--all hours of the night. ;)  Depending on the APIs and other contextual software packages being used, you can adopt a lot of tests in incorporate them, depending.  Often this does not happen, or is done on an as-needed basis.  I could go on but I believe that addresses your last two questions.

Lyytinen, K., & Rose, G. M. (2006). Information system development agility as organizational learning. European Journal of Information Systems, 15(2), 183–199.

I started writing automated test 12 years ago, and work in different companies where there was no automation, or full automation of tests. The development cycle can not be shorter than the longest of its activiy. Current tools increase the development efficiency and allow reducing the development cycle. Once the development get into faster cycle, you end up with more features that are prone to regression. The analysis and development stays short, but the time to retest become more and more important because you have to retest an important number of features. In a context where you only need a couple of cycle, this is not a big issue.

In a context where you release in a production environment, the cycle keeps adding up. After many cycle, the effort to manually test everything become so high that it will reduce your speed, you will need to wait for the project to be QA. Then you become to increase the cycle, increasing the amount of feature per cycle, which in turn increase the QA testing time. In the first cycles, creating test automation is bigger than doing manually, but with time, it gets easier to simply add new test than retest everything. To avoid facing the challenge of doing test on a big system, you should automate from the start, if this is not done, you will end up with untested feature and a technical debt, having to eventually make these test sooner or later. You should test as much as possible. If not possible, you should do at least basic testing or everything, and make deeper test for section where regression is most likely to happen. There will always be some amount of user validation, like checking field layout and color scheme, but these don't have to be redone every time.

Automated test most often goes with automated execution of test. When pressure pushes the development to deliver, developer tends to run only specific test but not all. To make sure all test are run, they must be run at least as part of the build process. If is not necessary to make a build to develop the application, build will be forgotten from time to time, and code will end up being committed without testing, delaying the testing to later on.

The best time to fix some regression is just after you introduce the error. The way to catch error as soon as possible is to automate test execution. Typical triggers are source repository or daily build. Getting the test done quickly will help you catch the error quickly, and the more quickly you fix it, the less it cost. The quicker the development cycle, the quicker the fix can be deployed if an issue gets in production. This is one place where you get a ROI.

The quicker the development cycle, the quicker the fix can be deployed if an issue gets in production. Most of the time, we develop software for supporting operations, or reselling. Getting shorter development cycle bring the feature faster, so the end user can benefit from it more quickly, thus allowing business to get faster ROI, or the end user to buy more of the software.

That works best when QA department works closely with Dev. It is even more efficient if you collaborate with operations, so you can have software build (with unit test), deployed ion a staging environment and then tested with more complex scenarios.

This brings you toward DevOps, where the tests and deployment are automated as much as possible. If you search on the topic you will find some interesting numbers for the ROI and company profitability.

Dear all,

please visit the following website to see what we are doing about SOA test automation.

http://web.ita.es/midas/

Cheers,

Fabio De Rosa

Hi Deniz,

Automating Test cases on a module is very much recommended if the module is not prone to frequent changes. I have been writing automated tests where the modules are dealing with a lot of data and exhaustive manual testing is very time consuming. Of course there is an overhead of writing these automated test cases, which should be accounted in the overall project planning. Manual testing is not fully replaced however. I have been mixing 70% automation and 30% manual testing as a fairly good approach. If your software is following SOLID principles then writing automated tests cases is beneficial (because new changes will be written in the extended classes and not in the base class, hence the automated test cases dose not needs to be rewritten). Also its easy to cover a wide number of tests by writing simple loop statements in test cases.

If you are using Microsoft platform .NET then Visual studio has in build test projects which can be used in integration with the source code. NUnit (www.nunit.org) is also available on Microsoft platform. NUnit dose not needs source code if the dll are generated from Managed code. you can also try Telerik framework for testing (http://www.telerik.com/teststudio/testing-framework) 

For Web applications there is WatiN (www.watin.org) testing framework. For Java platform there is JUnit (www.junit.org) and HP LoadRunner framework to do unit testing.

Again if you are good with SOLID principles then STAF (http://staf.sourceforge.net) Software Testing Automation Framework is also recommended. Good thing about STAF is its cross platform and covers a variety of software.

Hope this helps!

Automation is a reality. I see it all the time. I can say, only slightly less cynically, that almost all executable tests are automated because a computer is running them. I suspect that the core of the question relates more to test management automation than to test automation. (This dichotomy lies within the second half of the dichotomy between test generation and test execution.)

Its value is highly suspect. On one hand, there is research that supports the value of regression tests, to ensure that you don't regress into earlier bugs. By all means, automate tests that check against regression into fixed bugs and run them 24/7. But great testing is based on ongoing exploration and what James Bach calls "exploratory inference." That means that human intelligence must be in the test loop, always using the dynamics of test execution (not just the results!) as input for creating another test (see his "Lessons Learned" book). Testing entails learning about program dynamics, and automation short-circuits the learning loop. This is a recurring theme of modern testing books, including not only the above reference but other classics like Weinberg's "Perfect Software."

Toyota, which is often cited as a source of lean practices, noted that automation diminishes opportunities for learning. Robots don't learn. So Toyota is in the process of pulling robots off of its assembly line and replacing them with people (http://qz.com/196200/toyota-is-becoming-more-efficient-by-replacing-robots-with-humans/).

Great software projects focus primarily on exploratory testing, which is the essence of "exploratory inference." Great agile projects are grounded in user experience testing — which is an analysis technique that precedes implementation work, and even design work. Neither of these has any substantial test management automation component.

And you were wise to ask about ROI. I didn't see that anyone answered that one. In my recent posts on unit testing (which touch on test automation) I developed some pretty solid models that establish why may common testing approaches contribute to negative ROI. See: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf, http://www.rbcs-us.com/documents/Segue.pdf.

As noted by all other anwsers, it is real. However, since no one mentioned one of the critical problems: Building the test oracle, that is, the part of the system that can tell if the test object behaved as expected or not. For well-defined parts based on formal specifications such as low-level test based on execution paths, data flow etc. where we have well-define test requirements and test criteria, this is typically a question concerning observability and reproducibility (cf. regressions testing) (e.g. Schütz "Fundamental issues in testin distributed real-time systems"). For high level test/black box tests for, for example, user interfaces, etc. the problem become much harder. Finally, the oracle  itself is a complex piece of software that is error-prone and the results can be misunderstood (e.g., the oracle just states pass/fail without giving any explanation).

Another problem of the same complexity in many situations is to build an environment simulator (e.g., testign you control program of the moon lander or nuclear power plant against a simulated environment). However, this is not applicable to all projects. 

Jonas — thanks for pointing out what should be obvious but which always seems to be overlooked. The agile literature in particular is full of advices to automate, but they never advise on how to make good tests. Automated crap is still crap.

You hint at another problem that people ignore, related to the complexity of the test harness. Most tests are code. The product is code. Code is written by programmers. When they write code, programmers insert about three system-affecting bugs per thousand lines of code. If you have more tests than code, then the error population of the tests is higher than that of the code. In such cases testing decreases your quality (i.e., your ROI). Automation makes it work because it removes the consciousness of what the test is actually checking; this is less of a problem in exploratory testing.

We need more good empirical research on this topic. In the same sense that Abrahamsson and Siniaalto's work pointed the way to discrediting TDD, it would be good to have analogous studies that characterise coverage testing, unit testing, and the broader spectrum of fault removal techniques. The data on this are sparse (I know — I've recently been researching this topic and have found few good data) and it's an important area. I hypothesise that it's vastly overrated and that it is the source of immense waste.

Matthieu, If you type "http://www.rbcs-us.com/documents/Segue.pdf" into your browser you will find that the site indeed exists. I guess that ResearchGate needs to do more testing to make sure that their software properly parses text and properly associates links with the URLs it thinks it identifies. Why, they could even automate the tests.

I can't think of any hypothesis that would suggest that testing code should be any more reliable than code delivered to the field. A good test should have complexity equal to that of the code being tested. If there are languages or techniques that we can apply to test creation that would lower the fault rate, it would make better sense to instead apply those to the code than the tests. Developers often push back in a direction analogous to your objection, and I reply by asking them why they take the smart pills when they write their tests, instead of when they write their code.

While I agree that we need more research on this topic, it's not hard to develop some simple and very convincing and powerful models of the interactions between tests and the code under tests. Most of my recent work has been in this vein, using simple statistical analyses and common sense. There are many ideas from lateral thinking and systems thinking which alone can destroy the rationalisations behind many contemporary testing approaches such as unit testing of object methods and TDD; you can find the foundations of such thinking in much of the good contemporary testing literature. This is also why I mentioned the Toyota trend of infusing more thinking into the development process, simply by removing automation. If we do testing research it might focus more on the psychological state of a programmer in green-bar fever than on fault density. Thinking often takes a back seat to doing during automated unit testing. You don't need a researcher to figure that out: just observe a frantic developer going through the paces to meet a deadline.

In the end, it's true that we'll probably need more research to substantiate these arguments, because in the end it's managers who make the decisions about testing policy, and numbers drive them. Mind you, there are still managers out there today who are suspicious of testing because it is the source of so many bugs (it's really true, and there are a few published studies of this). So we need your research and its numbers. Thinking doesn't seem to work. But that's O.K. As Weinberg says: you can turn anything into a number.

In the fields of English and Foreign language education.  There are a variety of computer generated tests online and elsewhere that people can take with randomly ordered questions, etc.

However, most areas of higher language attainment, like writing essays, giving speeches cannot likely be replaced very well by machine or computer generated tests.   In short, for drilling at lower levels, some kinds of tests work.  Others will not work currently.  Moreover, it will be decades before such software will be accepted. 

Reform and change are slow and should be of a cautious nature.

Well I've tried some tools to automate tests following the headlines of Behaviour Driven Development. For Ruby: Cucumber, PHP: Behat/Mink, .NET: SpecFlow. Actually for .NET there's the Team Foundation Server, amazing tool to apply Scrum in the project managment linking it with BDD. For Java there´s a tip, if you´re trying to test a web site there´s Jbehave but I prefer Cucumber for Java, but I get only to work with Maven, maybe there´s a bug with Ant. For a desktop and mobile application I've tried FEST with nice results.

My team is very small, and we practice pair programming form XP, so in the early beggining of development everyone is involved in writing tests code. Later, the roles are dvidided. Finally, there´s a tool that works fine for my team: Jenkins that able us to control the status of project within nightly builds and test automation.

We just have to remember that in software development not everything can be tested. The cost in this case might rise so much that the accuracy falls down dramatically. I think that testing behaviour we're also testing methods. It´s obviously that if you test a click, some on_click() event is callback, so the inside code is tested.

You can find more documentation by googeling speeches of Dan North, Gojko Adzic, Mike Cohn, Kent Beck, Jeff Southerland and many other that had write a lot about this topic.

In realistic SDLC, test case should to be  developed parallely along with application itself. In some cases, where application consist various different modules or functional area such as ERP system generally have HR, Logistic and Finance sub domain etc. In that case functional test and integration test can be done with the help of tool like AnthillPro* which run only after unit test have passed. 

Gopal,

1. Why? That is, what is the rationale for this selection? Does it form a spanning set?

2. How is this better than the alternatives? Do you have models, data, or theory to substantiate your choice?

3. In what contexts does this combination work or not work?

Stefan, testing of anything requires that the right elements are being tested.  How does one know this if one does not know the value of tested variables to the end user and how those variables will be tested?

In any case, I will assume I am off topic.  [On the other hand, I have just given an unintended--ironic--example to you of how language affects our ability to test anything. ;)  ]

We have implemented an automated testing framework and related tool suite for web applications and mobile apps. We leverage Selenium and provide a user readable script for test execution. Anyway, not all scenarioes covered. Please refer to our paper at http://dl.acm.org/citation.cfm?id=2659136&dl=ACM&coll=DL

Thank you for all answers.

You can use Jenkins too, below I leave you a link where explain many things to do with this tool

http://kybeleconsulting.us4.list-manage.com/track/click?u=b054d2f7f9c1c53928fabc63d&id=0fc37bd3db&e=565b891c86

The short answer is: of course, test automation is not only real but also nowadays critical in many industrial settings as manual testing may not be scalable considering the complexity and size of such systems especially where continuous

integration and deployment are applied. That said, manual testing is still necessary and should not overlooked, for instance, in ensuring safety. 

On the other hand, test automation may not always be cost beneficial so a careful cost-benefit and ROI analysis needs to be done. Here is a great article on the topic: "When and what to automate in software

testing? a multi-vocal literature review" by Vahid Garousi and Mika Mäntylä .

Also here is the link to a 3 year European research project called TESTOMAT which we are soon starting on this exact topic: https://www.researchgate.net/project/TESTOMAT-The-NEXT-Level-of-Test-Automation-in-CI-CD-Agile-flow

https://www.researchgate.net/project/TESTOMAT-The-NEXT-Level-of-Test-Automation-in-CI-CD-Agile-flow

https://itea3.org/project/testomat.html

Article When and what to automate in software testing? A multi-vocal...