Simulation can be applied on Process Performance Models (PPM)

Research on assessment techniques are difficult to research for a number of reason.

1. CMMi, like the CMM before it, approach the quality of a development organization based on activities performed and not results produced. Meaning that the using a tool or technique badly weights equally with using it well. Take inspections for an example. Many organizations perform inspections then to check a box on their QA process and not to find defects. Yes they find defects, but they occur very late in the process. Sometimes with the software is in system test.

2. Simulation works well when the behavior of each element is known a predicable for a given set of stimuli. Unfortunately, a software development process dependent on humans is unpredictable. Look at poling in American elections, which are frequently wrong for a host of reasons.

3. Most corporation will not provide the type of data that you would need to do either a simulation or empirical research. Corporation do not like to disclose defects or problems with their products. Frequently, test engineers in corporation are required to sign non-disclosure agreements (NDA) to work on a specific project in addition to the general NDA the sign to get the job.

On the positive side you could study the number of organization using an assessment technique compare the number of companies developing software, and why these companies uses that assessment model. I don’t think this would make a good thesis, dissertation or article, because typically the answer would be to get certain government contracts or as a status symbol.

Having a process is very important and CMMi provides an excellent framework for building a process.  However, efficacy is another matter.  It can be shown that the eradication of SW error is intractable for a code body of any significant size.  In this case, the only solution is a negative feedback loop by which one can manage error reduction.  Thus, while CMMi focuses on how one builds a process, for efficacy, the focus must be upon how one controls error and how SW error "works" which will dictate how such controls must be formulated.

The most effective error reduction negative feedback technique is "Errors/KLOC" -- calculating such a figure from current field results, applying that figure to new releases, comparing projections from the current new release to the previous and forcing the projection down as possible.  This is the only effective way to reliably reduce error -- managing the error count downward.

However, while this technique has been shown to be effective, there are non-obvious consequences that arise as error is reduced.  Specifically, it can be shown that there is a probability associated with each and every error.  This is the probability that the error will show itself.  It can also be shown that the underlying function for this probability distribution is (at least) exponential -- specifically that there are "orders of magnitude" differences as this probability shrinks.  This is why the problem is intractable.

This also says that, as efforts are made to reduce error counts that there is a disproportional amount of "easy" error that gets removed first.  (i.e. the high probability errors).  Unfortunately, this leaves the "harder" problems undiscovered and, most likely, to be found by the users of the software. Thus, error counts is not the only factor that must be addressed.  Error impact must also receive attention.

So, while a provider may be successful in removing error over a long period of time, they will be less successful in reducing total impact and will, through the very removal of error, provide users with more time to focus on problems that are more highly impacting. 

The net will be that user satisfaction will not improve simply because counts of errors are improved.  Instead, there will be a shift from a focus on the count of errors to a focus on the impact of errors.  (i.e. No longer will you hear "You have too many errors".  Instead, you will hear "This problem hurt me badly") 

Thus, it is very important to look beyond the simple count of errors to the impact of those errors.  So, two simultaneous efforts are needed:

This, of course, throws wide open questions of SW Architecture: Are some coding techniques more or less capable of preventing both counts of errors and/or minimizing the impact of residual error.  What about languages?  What role do they play? etc. etc.

There has been a lot of work done on the subject of the behavior of SW error and user satisfaction with SW Quality.  I am interested in finding collaborators to further this work.  I would also be willing to share the work that has been done and to have a fuller discussion on this topic with others who are interested.

Smitha,

An interesting question. Actually, I agree to the statements above and like to add that we, among other things, tried to address questions like yours with out paper (see below) by working out where research is available and where are the gaps. If you look at Fig. 4, you see in the column on standards that there are some white spots. This gives you basically two options:

An extra note: we currently work on the update of this study, which we expect ready in a few days. Just let me know if this is interesting...

Best, Marco

Conference Paper Software Process Improvement: Where Is the Evidence?

Since CMM and its various derivatives are process driven,the following broad areas can be looked into:

i) The process integration while moving through the various levels.

ii) The measurement and control of the process

iii) The measurement based feed back mechanism for evaluating the maturity of the process(es).

There are a plethora of CMMi research topics, but there are no topics that can employ simulation. There are three process variables that are not quantifiable.

1. Human developers

2. Software Architecture

3. Type of software IoT, Medical, Avionic, etc.

From Quality Engineering aspects the mapping of CMMI and Agile Practices is also missing from literature