Dear Muhammad,

Code coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite. If you inject faults to cover the code provided for error handling code means you cover the related code. The sugesstions if a technique is better in case of numbers of discouver faults is in my opinion wrong. Because covering code and detecting faults is not related. The fact you may cover more code by technique B  instaed of A also doesn't mean the technique is better. Because this depends on the implementation of your code. Also the discovered errors in the code doesn't mean the technique is better. Because your implementation may be faulty, caused by the bad day you had as you coded this stuff.

I assume you need a method to evaluate techniques to discover errors in source code. Measuring the coverage is related to the effort which is used for the implementation of the test cases. But i think this should be a second or later step.

First you have to find the errors wich are hard to find and need much more time and effort to find. In the following link a model is described how failures are detected.

https://books.google.de/books?id=EaefcL3pWJYC&pg=PA216&lpg=PA216&dq=%22failure+detection%22+curve+pdf&source=bl&ots=8Dct-WnkaG&sig=lQBzm2K6wL6657HGLp3V2V_3WIs&hl=de&sa=X&ei=N7QvVfGDMo35aJTxgWA&ved=0CFgQ6AEwBjgK#v=onepage&q=%22failure%20detection%22%20curve%20pdf&f=false   

If you compare a technique by the errors hard to find compared with the effort creating a test case. this should be a better metric.

Dear Muhammad 

please read these articles which will help you in solving your problem. 

Khoche A, Sherlekar SD, Venkateshesh G, Venkateswaran R. A

behavioral fault simulator for ideal. IEEE Design Test Comput 1992;

9(4):14–21.

 Levendel YH, Menon PR. Test generation algorithms for computer

hardware description languages. IEEE Trans Comp 1982;C31:577–89.

 Mao W, Gulati R. Improving gate level fault coverage by RTL fault

grading. Proceeding of international test conference 1996 pp. 150–159.

@Alex...You said earlier that covering code and detecting faults is not related. So can i say as for that example is Error recovering rate is higher for B by 20% over A? And thanks for the recommended book.

Dear Muhammad,

I strongly agree with Alex when he says that all faults does not have the same cost and value.

In my opinion, let's say you have 3 techniques for discovering faults namely (A,B and C) and A and B detect 95% of all the faults you could ever detect in your code and C detect only the 5% left. Then if the few faults discover by C have a really high severity (in term of impact and ease to reproduce), then method C should not be neglect only because its coverage is lower just by looking at the numbers.

I would maybe advise you to attach a weight and a probability to your faults.

Regards,

Maxime

Dear Muhammad,

it is not fully clear to me if what you have in mind is related to mutation testing, as Stefan mentioned, or if you are actually trying to test fault handling mechanisms (like storing information redundantly in two places, to recognize bit-flips in memory - which is not easily done without changing the code itself to emulate those faults).

Independent of that, if you want to compare test design methods, achieving a high coverage (by whatever coverage metric ever), you will always want to look at what is the "shared" coverage between two methods. So if the 10 "items" covered by your Method A are included in the 15 of your Method B, you could suggest dropping the use of Method A as not gaining anything (B clearly better than A). If none of the covered items of Method A are  included in the ones from Method B, you would apply both methods and gain 100% coverage. In this case none of the single methods is really better than the other.

And of course, coverage evaluations need to be done on a larger set of systems under test. What works well in one instance might be less than satisfying in the other.

Best

Rupert

The precise metric for measuring code coverage cannot be the test techniques that identify more injected faults. Some of the factors that affect mentioned claim is the way and places in which faults are seeded, the impact (cascading or localized) the seeded fault has on overall system during run-time, whether the seeded fault impacts behavioral or non-behavioral specification of the system, and such other factors.

To compare two different test design methods merely comparing number of seeded faults found is trivial and not accurate. Other considerations here include the number of common seeded faults found, comparison of complexity of seeded faults found, and others.

Lots of things need to be thought through to make it precise.

Muhammad,

Metrics dealing with software is a dangerous playground.

The real question is: "What do you want to know?"  About the software under test?  About the test "Coverage".

I've had considerable success with the following technique:

Parse source code locating all integer values.

Loop for each integer value

Replace integer with 1 + Integer

Compile

Run test suite

Record success/failure to detect injected error

Replace integer with Integer-1

Compile

Run test suite

Record success/failure to detect injected error

End LOOP

Coverage = Successes / 2*

I was embarrassed.  I thought I had a really good test suite.  Detected 35% of the faults injected.  Note that this was only a test for one-off errors. 

Of course this requires a fully automated and efficient test suite.

Coverage?  I don't think that is a practical idea.

Alan A. Jorgensen, Ph.D.

See "Microsoft Calculator Challenge"

http://kaner.com/pdfs/measurement_segue.pdf

Thanks everybody. It really helps.

Hi Muhammad,

Looks a bit like the way virus checkers work or research on people for a certain desease.

You know there are 25 faults because you inserted them yourself but are you sure there were no unintential faults allready in your dataset?

You use two techniques (A,B) to find the faults.

In research you should devide the findings in the techniques into

• true positive (a fault is found and it is indeed a fault)
• false positive (a fault is found whereas it is not a fault)
• true negative (a item is accepted and it is indeed a valid data)
• false negative (a item is accepted whereas in fact it is a fault)

You get 4 values for A and 4 for B.

Suppose they both give only true results (10 truly faults found by A and 15 by B) then when the 10 of A are also included in the 15 of B, then B is better than A. If all 10 of A are outside the 15 of B, then you should use both techniques to cover all 25 faults.

If possible, the nature of the dataset is exactly known and also the valid values of the data-items, i would try to device a technique that checks exactly that. In theory that technique would catch all faults in the dataset.

Like Scott wrote: Good luck in this...