Thanks for the informative replies. It is a general question, by design. Any specific answer would need to be focused on the total software engineering process or a specific component of the software engineering process. I did not want to exclude any specific response by providing a very narrow question.
For example there are model oriented formal software specification languages, like SOFL, VDM-SL and Z (Zed). Software engineering must produce predictable results, therefore some type of formal specification of what to expect seems reasonable.
There is also the NModel approach which provides model based specification, analysis and testing.
Does anyone know of any other formal methods that are in use today in software engineering?
I looked at your conversation and it looks very interesting to know something new from software engineering. Algebraic Graph transformation are something what is accurate for me, because I am mathematician :)
Yes, there is a very tight connection between systems science, systems engineering, software engineering and mathematics.
The "Algebraic Structure Theory of Sequential Machines," Hartmanis and Stearns, 1966, is an early example of this type of connection.
"Structuring Complex System," John N. Warfield, 1974, is another example.
"Systems Analysis and Management," Donald Steward , 1981, is an example that apples to software (equation solving) optimization.
"Model-Based Systems Engineering," A. Wayne Wymore 1993 is an example that proposes a 'system design language' based on the mathematics of sets.
"Abstract State Machines," Borger and Stark, 2002, is an example that presents a method for high-level system design and analysis.
There are others...
The key aspect that I am looking for is practical application of these formal methods to software and systems engineering. To date practical, deployed software engineering methods based on formal methods appear to be very limited.
Thank you very much, Joe. I can see two approaches at our Department, one is based on methodologies of software engineering (structured, OO, using CASE etc.) and approach based on "theory of systems" and I am somewhere in the middle :D I teach the basic principles of knowledge representation and knowledge systems and usually I have a problem what to use for knowledge engineering....thanks once again. Eva
Yes, the area of knowledge engineering is a very diverse and interesting field.
John Sowa developed conceptual graphs to support knowledge representation.
An Ganter and Wille developed "Formal Concept Analysis" as a knowledge representation tool.
Both are graph based approaches.
Warfield developed "model exchange isomorphisms' that suport the development of a clear, structured mental model of a domain. This mental model can then be transformed into a binary matrix system structure. The binary system structure can then be transformed into a directed graph representation of the mental model.
Again graphs are used to specify, document and communicate system structure.
I believe that there may be undeveloped areas in Warfields work that will support the development of practical, formal system and software engineering methods.
I'm confused as to what part of Software Engineering you want to look at formal models and methodologies for? Are you looking at the business area as a whole from a process improvement/ project management perspective? Other answers look to be from an IRR, knowledge representation and software interaction point of view.
My primary goal is the insertion of formal methods into appropriate areas of the software engineering process. I view this as an incremental improvement activity that inserts executable formal methods (rules) in the background of an executing software engineering process.
The software product that I am developing is applied in large-scale systems contexts that include business, project management and other aspects systems production.
In general you might think of this as inserting rules-based formal methods as background execution tasks in an agile software production process.
gotcha thanks for the clarification. Is there a sub- industry target for your product? or are you creating a generic product? I would imagine that one industry's best practices would champion differing SDLCs and PM methods from another industry.
The target domain is systems science and engineering, which can apply to a very wide range of industries. The initial work is focused on structuring ill-defined and/or unknown problem spaces. John N. Warfield did a significant amount of work in this area. This work includes the development of "A Science of Generic Design" and "Interactive Management."
A key aspect of Interactive Management is the process of Interpretive Structural Modeling. Interpretive Structural Modeling has a primary process core, based on mathematics and rules, that is used to support the practice of Interpretive Structural Modeling. The primary process core is called Basic Structural Modelling.
At this time my efforts are focused on developing a standard software set to support the Basic Structural Modeling processes. Structured processes like SDLC and the IEEE software engineering processes are valuable tools. However, my project will be focusing on the area of problem definition (what is the problem) and conceptual solution development.
My current plan is to include logic and rules in the software product that will provide logical tests and process verification. While strong process management is needed, there is also a place for formal systems to validate and verify the products created by the system.
Dear Joseph, I noticed you to be interested in definitions of ”problem” in conceptual level and thereof solving methods. You may find some useful thought provocative of my work in Artificial Generic Intelligence. Good luck on your efforts, they seem to be valuable considering endeavours in the whole mankind.
Hi Joseph, I would like to refer Formal and Emerging Methods in Requirements Analysis, as published by Eric J. Braude: http://bcs.wiley.com/he-bcs/Books?action=mininav&bcsId=5626&itemId=0471692085&assetId=233160&resourceId=22842&newwindow=true
Yes, problem definition is a major factor in my research. However, the problem definition process can go through a number of phases, and there always needs to be an initial phase.
In terms of the scientific method, problem definition can be viewed as hypothesis generation or abduction. "Adbuction is a mental process whereby theories, conjectures, scientific hypotheses, and explanations are produced involving situations where the outcome has not yet been established as a belief by either induction or deduction." Warfield commenting on C.S. Peirce's use of the term abduction.
How is an hypotheses generated? A picture comes to mind of a brilliant person sitting alone thinking pure thoughts and creating an hypothesis. Warfield created The Science of Generic Design and Interactive Management to support 'group abduction' or the activity were 50, 100 or 150 people create an hypothesis as a structured group.
The group abduction process is facilitated using computer based logic code and networked systems. So this process must be reduced to executable code.
The problem solver you referenced appears to be very effectively once a hypothesis is generated and preliminary solutions are developed.
Requirements as well as constraints are important in software development life cycle. In most of the software development processes, these two changes frequently. To address these frequent changes software development follows design patterns (MVC, MVVM etc.) and development processes (agile, scrum etc). Now cloud and mobile infrastructures adding to it. For example mainframe, client server, web apps, and mobile apps have different design approaches. In addition to it development languages and frameworks also enforces best practices, standards, patterns etc. It seems very wide scope. Also to consider whether solution detects the infrastructure or other way round. It is very interesting industry wide problem. It is a major undertaking. And if it can be done will help and expedite the development cycle. Good Luck and Best of wishes.
I tend to create a clear distinction between constraints and requirements,
Constraints are most often dealt with in the planning stage (before requirements analysis) because they have a strong tendency to define the problem. If the constraints change, the problem changes.
Requirements analysis is only one activity done in requirements engineering. I suggest you have a read on all parts of requirements engineering, i.e. requirements elicitation, requirements specification, requirements analysis, and requirements validation.
In software engineering, requirements engineering is the first phase, and it is where the problem is identified, analyzed, and documented. Planning is done after requirements are specified, not before. How do you plan if requirements haven't been elicited, analyzed, specified, and validated?
The referred material is not merely analysis. It is about formal and emerging methods. These can be used in requirements analysis instead of heuristics.
Multiple approaches to support all software engineering activities exist. Best resources, in my experience, are IEEE CSDA and IEEE CSDP:
If you read most of these software engineering books, you will find different authors calling same things different names. Their approaches are presented as standards, but eventually, you need to select from all available approaches every time there are multiple choices, and always select based on specific circumstances. There are no standards for every case universally. Also, I suggest software engineering practitioners are a different community than software engineering researchers.
See the guide to software engineering body of knowledge (SWEBOK) http://www.computer.org/portal/web/swebok/home
and http://www.computer.org/portal/web/swebok/html/contents
This is claimed to be a generally accepted knowledge in software engineering.
Formal methods are mathematically-based techniques for the specification, development, and verification of software. That said, they are not "practical". The SWEBOK covers all knowledge areas, and you will learn how to systematically solve challenging problems, whether it is requirements, engineering management, software design, or other areas.
It is important to bear in mind the SWEBOK tells you all about available methods, and when to use them. It doesn't say we can use the same approaches over and over again for all projects. There is no silver bullet [Glass]. However, now there is the SEMAT kernel by Jacobson et al. [The Essence of Software Engineering, 2013]. He has unified all software engineering activities into one common foundation.
I agree that there are many different types of processes.
The processes you mention are standard approaches.
My comment was directed toward making a distinction between the concept of a requirement and the concept of a constraint.
In my mind a requirement and a constraint are two very different things.
Constraints can not be traded-off by the developer.
Requirements can be traded-off by the developer.
Cost constraints, schedule constraints are usually prepared in the initial system planning sessions. These constraints are imposed by the system customer and con not be changed by the system developer.
There are system design constraints that are imposed by the developer as a side effect of the developer design decisions. Use Python or use a three-tier client server architecture. Design constraints are different from system constraints.
I believe that formal methods can be effectively applied in practical ways.
Rule-based code checking is one such example, natural language translation is another.
We may have different views of the system of interest and the system development process. In any case when the customer plans for system acquisition and deployment that is usually done prior to any detailed software requirements analysis activities.
System planning and analysis have a range of interrelated phases which must be carefully applied to create a successful system.
I like Marek's answer concerning the different activities in software engineering that may be respresented with formalism. I would say that all the activities in software engineering may utilize formal methods. However, many focus on requirements and design, utlizing formal languages to represent the requirments and the design. For example Z language is often mentioned for requirements, and Alloy may be used for design abstraction. Of course, it is assumed that one is trained in logic, at least at the predicate calculus level.
Books by authors such as D.C. Ince or D. Jackson may be a good start.
Sorry, in my previous comment I left out some pretty important, current activities involving JML and Behavioral Interface Specification Languages by people such as Gary Leavens and Rustan Leino.
I replied unrelated to constraints, regarding Joe's ideas on planning and requirements. Joe believes "The step before requirements analysis is called planning. In my mind the problem identification and problem structuring happen in the planning step."
To shed some light on the subject of the interconnectedness between planning and requirements, one may refer to the PMP (Project Management Professional) standards.
methods of requirements elicitation are applied first, before planning can be done. That is to develop the preliminary Project Scope which is later used to estimate the budget, schedule, etc.
That makes requirements engineering start first, and usually simultaneously with planning. Since requirements engineering is an iterative process, only some requirements are initially collected. Let's call these preliminary or high-level requirements. They will be changed and managed throughout the project, including after it ends, in the software maintenance phase.
The suggested "problem identification and problem structuring" therefore happens in requirements engineering which effectively addresses it, and the initial plan is derived from the preliminary requirements.
For completeness, one step back is the project charter:
http://www.pm-primer.com/develop-project-charter/
It contains the justification for the project, high-level goals, objectives, project constraints, etc. These are the basis for some requirements, but not enough to define the Project Scope. Project charter combined with the product of requirements engineering create Project Scope that is the basis for planning.
If there are time/budgetary constraints in advance before the planning, they will be used in feasibility analysis and compared against the preliminary project schedule / budget to determine if the project is feasible.
There are well defined processes, including systems engineering, project management, software production, hardware production and system integration.
Formal methods are scattered in localized areas through the total system development lifecycle.
It seems to me the the software processes have a relatively high number of formal processes.
The are a few formal processes that could apply to the complete system development process or just the software development process or just the hardware development process. The formal processes include:
-- The Algebraic Structure Theory of Sequential Machines
-- Boolean Reasoning (Frank Markham Brown - The logic of Boolean Equations)
-- Interpretive Structural Modelling ( John N. Warfield)
-- Pseudo-Boolean Programming (Peter L. Hammer)
These formal approaches apply to a sequence of steps that have a value and goal set in a given context.
However, some times there is great confusion related to which process is the controlling process.
If one applied formal methods the control of the complete system development process, which has the software engineering process as a sub-component, then the confusion associated with process application and precedence control would be greatly reduced and/or eliminated.
I wanted to add to this discussion Model-based UIs (http://www.w3.org/2005/Incubator/model-based-ui/wiki/Main_Page), it is a Model Driven Development technique focused producing UI that can be easily modified considering different constraints such as the device or the user capabilities.
I also recommend User Centred Design techniques to dont lose the focus from the target users.
I read something interesting about product lines. Similarly to the cars manufacturing, it is possible to create the core of the product and then build the final product adding the specific requirements, such when you buy a car (core product) and you decide tune it with some extras.
recently I have conducted a study and still working on. You can find all information related to Process, approaches, methods, models, techniques, Framework, recommendation etc. You can google it Study: Systematic Mapping of Value-based Software Engineering - A Systematic Review of Value-based Requirements Engineering. Software development and management process and sub-process areas.............................. I hope, it will be helpful
Some very interesting ideas and concepts are presented in the document.
One type of value is the value of the "first mover."
Microsoft did not create a valuable company by using a robust software engineering process.
Microsoft created vast wealth by establishing the value context. Everyone uses Windows. The global context was leveraged to create more value in a number of interesting ways, none of which were associated with a software engineering process.
Microsoft only began to look at software engineering processes when they started to loose substantial market share in certain areas.
It appears to me that the value of software is more strongly related to the software application context than it is to the software engineering processes that create the software.
Before I respond, I would appreciate if you send me one of your papers on "formal methods for software engineering", just to see your logical starting points. In my opinion, a real question is whether formal processes for software engineering have any tangible practical value and how they may be compared with software engineering methodologies that contain formal knowledge representation methods (e.g., Executable UML)
I kind of agree with your statement that “Formal methods are scattered in localized areas through the total system development lifecycle.” There is a reason for this; formal methods, as we know them, do not have expresivness to address all dimensions of software engineering.
All major software development companies are moving to agile processes, more specifically Scrum, and abandoning traditional methodologies. For example, today all Microsoft development tools are built with a 3000 person Scrum. It cuts release time in half and eliminates 90% of their bugs. See Agile Software Development with Visual Studio for a good book on how Microsoft does Scrum.
Microsoft is actually internally releasing a new version of Microsoft developer products every three weeks. This would be impossible by deploying the conventional methodologies mentioned in this thread.
These military-type standards are great for highly-controlled, structured environments.
The necessary controlled context is not as prevalent as it used to be, even in the military.
Security and safety critical systems need formal control.
No one runs safety critical systems with Microsoft software.
I believe there is a range of software engineering processes that are appropriate for the given context.
A major issue in government software acquisition, is the trade-off of cost and value vs software reliability and performance. These are very difficult decisions and some times they are made wrong.
In any case, I am interested in a formal model that would be used to select the formal software engineering process to create the software.
One software engineering process does not fit all cases.
Formal models may help in the production of acceptable software products.
It appears to me that there are a number of formal models and a number of “model-exchange-isomorphisms (MEI)” that are needed to produce acceptable systems.
The first formal model that must be addressed is the system model in the customers mind.
There are a number of methods that can be used to establish the customer model.
The first MEI must transform the customer model into a system concept framework (SCF).
Once the SCF model is established then a set of executable system models can be aligned with the SCF.
SCF compliance testing can be automated and provide feedback to the development team.
There are a number of other formal processes that could be used after the SCF is established, including:
Abstract state machines, Nmodel, Pmodel and others..
It is true that software development , tools, practices and processes are changing.
However, a major problem with software production today is the creation of secure software.
While Microsoft may use Agile processes these processes are encased in the Security Development Lifecycle, which looks a lot like a waterfall process. See:
Take a look on BIP. It's a framework for component based modelling, used for a)designing a model for the system, b) verifying properties on model and c) generating code for execution.
Software Engineering has developed concurrently with programming paradigms , the current programming method is Object Oriented Programming , the most accurate Engineering tool is Unified Modelling Language (UML) that supports software development at all developing phases starting from( Requirement setting , Analysis , Design )
the most up-to-date and ongoing research is Agent Oriented Software Engineering comes as successor of Object Orientation , you can check my ARL theory regarding this matter.
To the extent that things can be predicted in advance, they can be automated. That is what compiler optimization and design tools and editors are about.
In my 30+ years of experience, the further you stray from a process like the above, the closer you get to failure.
Things like PMBOK, PRINCE, ISO9001 are good for billing and complying with mandated requirements. It seems to me that in practice they are used more to justify errors and failure rather than to aid development.
The best report on software is the working software, documentation and a working build system.
Think about it: What actually happened during the development of the successful software we all use?
We can and should automate what makes sense. The only people that need to be intimately familiar with the mechanism are the people building the tools or people studying them to make better tools in the future. To the extent that it is important that people running a project need to know something, they need a guideline as above and the experience to know when to keep their hands off and let people do their work.
If you are interested in the intersection of software with industrial hardware for the purposes of control and automation, you may be interested in some of the work done by Prof Valeriy Vyatkin on the application of formal methods in this domain: http://www.vyatkin.org/
Thanks for the information about the Agent Role Locking Theory. In general it appears to me that this type of diverse, agent-based evaluation of the software solution space is required. The search space is so large, that any practical approach must be very efficient. I will give this work more consideration as I move forward.
Great insights.... The sequential flow of development is bounded by context and value changes.
On one level (usually a high level of abstraction).. The software engineering process is sequential.
At other levels, (lower levels of abstraction).. The cost trade-off between performance of existing packages and building new software is very complex. These are no usually sequential decisions..
A key aspect of any pragmatic formal software engineering process set is the transition from high-level sequential processes to lower level concurrent and multi-threaded processes..
Yes, I would like to understand how to make software production a repeatable, industrialized engineering process..
This goal would apply to all application domains of the software code.
Thanks or the references.. I will need time to work this material and see if I can understand the approach.
It seems like a set of model units could act as agents in a controlled context. Evolutionary programming could then be used to select the best set of model agents for a given task.
Similar to genetic programming, but different in many respects.
A widely formal method used in software engineering is formal verification of software. Formal verification is the use of mathematical techniques to ensure that a design conforms to some precisely expressed notion of functional correctness. It is a complementary approach to software testing in order to minimise the probability of software faults and failures.
If you want to consider formal verification, it means that you need to have
a model of the system and properties to check. In this case the techniques are mainly
model checking (usually finite systems) with techniques such as explicit techniques, symbolic techniques (decision diagrams) or SAT based techniques. There is also
theorem proving that needs to guide a tool to perform a set of proof manipulation such as substitution, equivalence modification, induction .... Theorem proving is more general but difficult to apply, it consists in the construction of a mathematical proof.
Creating the initial system model is always a challenge.....
I have been reading George Boole's "An Investigation Of The Laws Of Thought On Which Are Founded The Mathematical Theories Of Logic and Probabilities."
It appears to me that the original "algebra of logic" may be a good choice in the area of formal systems work.
Do you have any information about applications of Boole's original algebra to the domain of software engineering?
Applying Boole logic in SE seems a little bit like using sticks for counting. There are
extensions of boole algebra such as algebraic specifications that are much more powerful and they can be used in SE. look at http://en.wikipedia.org/wiki/Specification_language or
I find OPM to be very interesting with many aspects that directly connect to the system structure area that I am now exploring.
A key difference between OPM and the ART and/or BSM techniques, that are the current focus of my work, is the fact that OPM seems to use the system concept in the "system design mode." My current focus in on activities that are in the "system discovery" mode. Once an unknown system structure has been discovered, identified and documented, then the system design mode may be entered to create a system to interact with the newly discovered system. You may think of the system discovery mode as "system problem description and/or requirements analysis" type of activities.
The ART and/or BSM work is now focused on describing an previously unknown system structure. So, the bulk of the approach focuses on system structure, not system behavior. However, when it is time to transition to system design mode, OPM is being evaluated for inclusion because of the very tight symmetry in method components and approach.
However, my interest in this area was initiated by reading a book from 1963, "Principles of Logic Design," by John N. Warfield. This book pointed out a number of facts that were new to me. Some of these facts are: Boole used ordinary algebra in his logic. Switching algebra devised by Claude Shannon in 1938 is different than the original algebra that was used by Boole. A more recent book, "Boolean Reasoning, The Logic of Boolean Equations," Frank Markham Brown, 1990, also pointed out these differences. So, I am in the process of reviewing the original work. At this time it appears to me that there may be some valuable applications in the system discovery, design and verification areas.
Large sets of Boolean recursive logic equations may be valuable in the identification of unique system configurations and solution pathways.