The output of research is a contribution to our knowledge base. If you developed a program that is better, that does not mean necessarily that you have contributed to our knowledge base. You need to prove that your program is indeed much faster or of wide practical application.

@Ian Kennedy, my question is exactly about the difference between "faster" and "practical" terms that is determinant for selection of the research methodology and consequently the evaluation method.

To show that your program is faster, you have to benchmark its speed of execution against existing software.

To show that it is more practical, you have to run a test with real users, asking them to (blind) rate the practical usefulness of your program compared with an existing program.

I am currently writing decision support programs and algorithms and as you have noted most Journal Editors/boards require empirical results of the program being tested for publication. I have found that this approach is totally unsupportive. For instance if you are writing a program that emulates human decision making processes or that simulates human emotional reactions to a situation how exactly are you supposed to define measurements or empirical results other than yes the program did act as expected or no it didn't. Journal Editors/Boards do not like this as they have developed a culture of testing and retesting and having long lists of data that prove or disprove a hypothesis in their articles.

On submission of my manuscripts and research results, in many cases, I am being told that the readership of the journal will not like the results as there are no long lists of test data but when you are basically asking a computer program 'what should I do in this situation?' or 'how do I solve this problem?' it either gives a workable response or nonsense. How am I supposed to present this in a long list of data? Not only this but I can generally present my results in a screenshot of computer program execution, again many Editors do not like this as it depicts the fact that yes the program is working or no it is not. I think this appears too simple to many Editors. Also if you are writing innovative or pioneering algorithms and functions that are several thousand lines long who is really going to read through all of the code?

This approach makes no sense to me and I believe the approach of supplying long lists of test data is ingrained in many scientists psyche. Now we have computers and software that eliminate the need for covering every combination of calculation and that including every calculation would not be feasible in most cases of long computation why some Journals demand these long lists of test data

is beyond me and is actually hampering many software engineers research and dissemination of their ideas.

I am evaluating my SE research these days and following Barbara's DESMET evaluation approach. I think she has mentioned all appropriate ways for evaluation.

I am doing controlled experimentation with 20 subjects for its quantitative and qualitative features. We should know about efficiency, accuracy and performance about our research. You can define criteria and then compare it with others.

I advise you to read the paper "Preliminary Guidelines for Empirical Research

in Software Engineering" by Kitchenham et al. (IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 28 (8), 2002. (This is not the only paper on paper on this issue, but it wil teach you a lot about how to conduct or evaluate research in SE.)

In addition to the above suggestion you might consider:

@article{seaman1999,

title={Qualitative methods in empirical studies of software engineering},

author={Seaman, Carolyn B.},

journal={Software Engineering, IEEE Transactions on},

volume={25},

number={4},

pages={557--572},

year={1999},

publisher={IEEE}

}

@article{runeson2009,

title={Guidelines for conducting and reporting case study research in software engineering},

author={Runeson, Per and H{\"o}st, Martin},

journal={Empirical Software Engineering},

volume={14},

number={2},

pages={131--164},

year={2009},

publisher={Springer}

}

Although they can entail a great deal of work case studies can be very insightful ways of evaluating methods, tools, etc in situ.

For a more reflective piece on strengths and weaknesses of empirical software engineering as a whole check out:

@inproceedings{sjoberg2007,

title={The future of empirical methods in software engineering research},

author={Sj{\o}berg, Dag IK and Dyb{\"a}, Tore and J{\o}rgensen, Magne},

booktitle={Future of Software Engineering, 2007. FOSE'07},

pages={358--378},

year={2007},

organization={IEEE}

}

The big issue in my opinion is how researchers have reported their empirical results. Many of them hide, forget or neglect empirical packages. How can other researchers replicate and confront the state-of-art results with new data? How can we trust on current results with unavailable or incomplete experimental material? Is it science for engineering software products and improve knowledge? Or art of publishing papers? I've seen many software engineering research papers lacking important experimental material even when the research methods are well described and applied. Although it has improved with respect to research made decades ago, still we need to take it seriously and dedicate time and effort to refine and polish our experimental material, make them available as important assets as the results are... And ask our peers for doing that.

Thanks to all answers, Specially to dear Peretz Shoval and Martin Shepperd for their valuable references. I hope I will read all of them. In this area since experience has a major role in publication and demonstrating the research results, the knowledge of senior RGs here is so precious.

@ Simon Bulgacs : Applaus! I fully agree with your observations and complaints. It's a ridiculous, myopic world where standards or crietria for publication are set and used without asking (1) the intellectutal owners of the knowledge (2) whether those standards and criteria are applicable or not.

I may add another line of methodology which is: single case studies. See attached link below.

Thanks to Juan Llorens and Paul Vossen for their references.

It seems to me that there is an important distinction between engineering and scientific research. Software engineering is about creating software for people to use and it results in a product to be sold or distributed. Meanwhile, research results in knowledge to shared via publication. These are not the same thing.

Realistically, the research related to a software product has to happen either before or after the development: either you incorporate some scientifically evaluated algorithm or framework into the product, or you create the product and scientifically evaluate how well it solves the problem that it was intended to solve. In the former case, you are performing engineering, and in the latter you are performing scientific research.

To empirically evaluate the value of a new piece of software, it should either be tested based on the problem it was trying to solve. If the problem was that other, functionally equivalent software is too slow, then the new software should be performance tested on a variety of hardware architectures. If the problem was to fill a gap for which there was no software, then the new software should be evaluated by users for its ability to fill that gap.

Aside from publicizing a software package to academics, what's the point of publishing an academic paper about software if it contains no empirical results? "Look what I made" is not a result, it's an unproven method. On the other hand, "The software I made improves the performance of task X by Y%" is a method with empirical evidence to suggest its worth. While the development of new software is engineering, proving that it's valuable through a controlled empirical study is science. The latter should be published, the former should not.

Hi Mathew,

I understand your point. However there is a scientific field dedicated to research how software is engineered, that is, how software is developed and maintained with cost, quality and time constraints. There are infinity open research questions on how to build and maintain better software. For researchers in that field the issue is not to publish 'a software' or a piece of that, rather, the issue is how to prove that 'a new software engineering method/technique' or 'new software engineering tool' is 'better'. So the most important point in software engineering research is HOW software is being built to reduce cost and improve quality within time/productivity constraints instead of WHAT is built.

Hi Bruno,

Of course you're right, but as you say, it's not the new technique alone that merits publication, it's the study of how that technique affects the field. I didn't mean to restrict my argument to software products alone.

@ Mathew and Bruno : Quite right so! I like to add and emphasize, that there should be much more co-operation and co-ordination between the research side and the development side of software engineering in the sense of a combined Research & Development orientation, as is the case in most engineering fields.

Why is this insight / principle so often ignored by software engineers and so poorly realized in most software engineering projects? It should not be either-or but instead both-and, because it will turn out as a win-win situation.

For many years already David Parnas has given a most relevant and interesting talk about closely related issues, I attach the slides for those who want to compare their opinions with his ...

Thanks for the answers.

Well, are the results published and evaluated based on scientific research are the same with engineering one? The question is maybe we call the methods used for the research results evaluation as empirical but that itself is also different among different research approaches. I would like to know how you discriminate the evaluation methods for a scientific approach or engineering approach for software engineering research.

@Matthew Spencer, of course nobody will consider "Look what I made" as the research result. The logic which is used to demonstrate the functionality and operabality of a software engineering output, either a software product or other types of results is different and it is the important element we are talking about. Even we call it empirical, but this empirical also should be adapted to software engineering research and is definitely different from the empirical methods used in other disciplines in which the contribution is merely obtained from human based experiments.

@ Ali Tizkar. Your asked "Are the results published and evaluated based on scientific research the same with engineering one?" I am not sure whether I understand your question as you intended. Here is my pennyworth of thoughts about it.

Research approaches, methods and techniques will always be different because they should be chosen to accommodate your research goal. If your research goal is to *evaluate* something (software or not), again, the researcher has a lot of options to choose from, depending upon the aspect of that something he/she is interested in, but also upon such trivial things as how many resources he/she has got to do the research. And even then, at the end of this chain, there is usually not a single answer, thus the researcher has to make a more or less subjective choice among the possibilities.

However, although there is not a universal and objective answer to the question what is the best method to use, I don't think there is an *intrinsic* difference between a scientific approach and an engineering approach: in both cases you must be able to answer questions like "what do I want to evaluate?", "what are my evaluation criteria?", "what are my assumptions and hypotheses?", "how will I set up my evaluation study?", "what specific methods and techniques will be adequate to collect my data and to analyse them?", etc.

If you apply the foregoing to the field of software engineering, first thing to ask is: what is the object and target of your evaluation? Did you develop a piece of software that implements a new algorithm to solve a known problem? Then perhaps you are interested in two fundamental question: (1) Does your software indeed solve the problem? (2) Does your software solve the problem better than other existent software? Both are evaluative questions, but quite distinct as you will agree. However, both are not asking for the practical use of your software, rather they are focussing on mathematical and technical aspects of your software. The audience who will be interested in the answers to these questions will probably be fellow engineers and mathematicians with a keen interest in algorithms and their implementations, including issues of complexity theory.

Now if your software is more of the "interactive application software" type, then your questions will be quite different, e.g. "Does my software solve the practical problems of the intended users?", or "How does my software compare to another software product that claims to serve the same intended users?. Again, evaluative questions, but these require a different set of methods and techniques, well-known in the area of business computing and usability engineering, for instance. Your audience will be analysts, researchers and practitioners from the corresponding fields.

The meta-questions now become: "Are the methods and results of my evaluation study significant and innovative enough to be published? and "What are the most appropriate publication channels and forms to use for my evaluation study?". As to the first meta-question: if your evaluation study is e.g. part of an internal project within a company, and only using standard methods and techniques, don't publicize it, because nodoby will be really interested. As to the second meta-question: you have to decide, whether you want to make your study public in electronic form or in paper form; in the latter case, you will have to find the publisher or journal which is particularly interested in and oriented towards the kind of software that you developed and the evaluation study that you conducted. You need to be careful here, and accept and respect the fact, that no single journal or publisher will serve all purposes, or the particular purpose that you have in mind.

I hope this answers some of your questions.

Hi. there is a whole bunch of literature available, where researchers from empirical SE propose methodologies for evaluating SE techniques, methods, tools. The leading point in identifying the "right" evaluation method is the research question, cf. for example Easterbrook, S. M., Singer, J., Storey, M, and Damian, D. Selecting Empirical Methods for Software Engineering Research. Appears in F. Shull and J. Singer (eds) "Guide to Advanced Empirical Software Engineering", Springer, 2007. There you also find other interesting papers. Recently a book on case studies in SE was published by Per Runeson.

@Andreas Jedlitschka, Thanks for your comments and references.

@Paul Vossen , let me give an example to elucidate this matter and what I stated:

Suppose we want to develop an algorithm for POMDP solver. What is defined as a metric to measure an algorithm performance is the algorithm complexity. the current algorithms which are introduced in the literature have the complexity of O(2^n). We introduce a different algorithm with complexity of O(n^2). Here what I want to say, the empirical method for evaluation of this proposition does not need any tangible structure, any hardware or computer system. The results that based on computer science formalism shows and demonstrates that this algorithm has this complexity suffices to prove its effectiveness and efficiency.

Therefore, the method we use for evaluation in software engineering discipline is obviously different from other branches of study and I wanted to highlight this statement.

As a matter of fact, Ali Tizkar, I wouldn't call this "evaluation" in the first place, although you may of course use the term "evaluation" for whatever purpose you have in mind that is somehow related to comparing two or more options / alternatives.

The reason I would avoid the term in this case, is that "evaluation" is already a highly overloaded and overused word, mostly seen in connection with e.g. (software) project evaluation, (software) process evaluation, (software) product evaluation, where the evaluators are human beings as stakeholders using their expert knowledge, and not mathematical principles or procedures or criteria coming from a formal theory. In the latter case most people would use a phrase like "analysis of of according to ".

Thus in your case "effectiveness" is "formal effectiveness" and "efficiency" is "formal efficiency". That are quite different beasts as "empirical effectivess" and "empirical efficiency" as meant e.g. in the international ISO-standard 9241 part 110 devoted to empirical evaluation of software by conducting a test or survey or the like.

In your case I am sure there are several specialized magazines which would be eager to publish your results, though *not* under the label of "evaluation" but of course as a complexity-theoretic analysis. These magazines don't expect any empirical proof of your results, because as you fully rightly states, that would be non-sensical, given the analytical means you have used to arrive at the results.

I advise you to have a look at the following book. I find it very useful:

Guide to advanced empirical software engineering

F Shull, J Singer, DIK Sjøberg - 2007 - Springer

I advise you to have a look at the following book. I find it very useful:

"Guide to advanced empirical software engineering"

By F Shull, J Singer, DIK Sjøberg - 2007 - Springer

I advise you to have a look at the following book. I find it very useful:

"Guide to advanced empirical software engineering"

By F Shull, et al. Springer

I advise you to have a look at the following book. I find it very useful:

"Guide to advanced empirical software engineering"

I advise you to have a look at the following book, I found it very useful.

"Guide to advanced empirical software engineering"