Hi Hussam,

It's absolutely possible, if not necessary, to have simulation software to test/QA your software. There is a new industry in cloud computing of Test-as-a-Service, TaaS where specialized products will go through the source code, parse it, and automate the authoring of the testing scenarios which aim to test code integrity, software bugs, performance (by simulating a high workload), among other aspects of software engineering. 

Hope this helps,

Bash

Hi Bash,

Thank you for your answer. Can you please give examples. 

Hi Hussam,

As the other respondents have indicated there are plenty of examples of simulated data sets being used. Typically we use simulated data when we want to ensure the data for an experiment has known or specific properties. Examples might be I want to :

- assess how robust my algorithms are in the face of noise, so I artificially inject noise into the data set(s)

- test various data imputation algorithms so I artificially remove values from either a synthetic data set or a naturally occurring one to see how well the imputed value compares with the 'true' value

- compare the performance of different classifiers (which could be from your question defect prediction systems) on training data with different characteristics such as the prevalence of the positive case (ie faulty code unit) so one could create data sets with 5%, 10%, ...

How reliable the results are depend upon how heroic your assumptions are but you might consider a ix of both simulated data and naturally occurring data.

Hope this helps

Martin

Dear Hussam

from a more practical point of view there are several ways to analyze data in software engineering, especially with respect to inspection and defect prediction. One technique that is also used in the CMMI context and has shown to be effective in the software context is SixSignma. To have an idea look at the following publications and the included reference list:

- MARIA TERESA BALDASSARRE, NICOLA BOFFOLI, CAIVANO D (2010). Statistical Process Control for Software: Fill the Gap. In: ABDURRAHMAN COSKUN. Quality Management and Six Sigma. ISBN: 978-953-307-130-5, doi: 10.5772/9925

- TERESA BALDASSARRE, NICOLA BOFFOLI, CAIVANO D, GIUSEPPE VISAGGIO (2004). Managing Software Process Improvement (SPI) through Statistical Process Control (SPC). In: F. BOMARIUS AND H. IIDA. Lecture Notes in Computer Science: Product Focused Software Process Improvement. vol. 3009/2004, p. 30-46, Heidelberg:Springer Berlin, ISBN: 978-3-540-21421-2, doi: 10.1007/b96726

N.BOFFOLI, G.BRUNO, CAIVANO D, G. MASTELLONI (2008). Statistical Process Control for Software: a Systematic Approach. In: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement. p. 327-329, New York:ACM, ISBN: 978-1-59593-971-5, Kaiserslautern, Germany, October 2008, doi: 10.1145/1414004.1414070

For what concern reliability of the six sigma analysis consider that the "sigma" element is a robust way to monitor and control data that faces the data noise problem in an excellent way. 

The technique has a broad range of applicability. At the moment I'm using it to analyze Customer Relationship Management Data, Ticketing Issues and Open Data in the public sector.

If you need further information do not hesitate to contact me.

Regards

Danilo

One answer to your question is yes, and as you see, there are a large number of publications using simulation. Another approach is to focus in on the questions. First, it appears that you want to use simulation to establish the effectiveness of various software engineering techniques in the development process. If that were your question, there are a number of significant challenges facing this research.

The first and most significant challenge is acquiring actuate statistical data on the effectiveness of the software developer. What we know from the PSP is that the defect yield from reviews is directly proportional to the training, experience, time to prepare, and programming language. There is also some apocryphal data indicating that culture also has an effect. Therefore, for the simulation to reflect actual development practice it would be necessary to identify the effectiveness and efficiency of each developer for each development activity.

Another challenge is with the characteristics of the algorithms. Predicting the total number of defects is actually simple, predict the number, tell the developers that number, and the developers will find that number of defects and a few more. What is a bit more challenging is to predict the number of failures by criticality and then don’t tell the developers that number. In general, software engineering researchers are not trained in the pitfalls of human subject testing.

Now were come to the fundamental issue, are you are new ground with this research no. Has the research been published, yes. For example, the research conducted and published on McCabe Cycolmatic Complexity. An interesting metric, but the only thing that researcher have been able to use it to predict the time to create white-box test case. Is there need in the literature for more and better research in this area yes.

Dear Hussam

Here is an article that explains one of the possible approaches:

Lj. Lazić, D. Velašević. “Applying Simulation and Design of Experiments to the Embedded Software Testing Process”, SOFTWARE TESTING, VERIFICATION AND RELIABILITY, Volume 14, Number 4, p 257-282, John Willey & Sons, Ltd., 2004

To see a commercial approach and how one vendor delivers what you are asking check out Microsoft's test products and think along those lines for free open source versions, niche players, etc, http://www.visualstudio.com/en-us/explore/testing-tools-vs.aspx