I would like to have a comparison of simulated vs analytical results in research related to computer communication networks. Why are analytical research work considered more powerful than simulated research works, though more assumptions are involved in analytical works as compared to simulated ones?
In any system (or model), you have parameters and other conditions (e.g. initial conditions) with make results different. In simulation, values of these effective parameters must be specified and result will be true just for that value of parameter. But analytical result, give a general description of system for any value of parameters. For example, suppose that we want to solve a simple linear ode: dx/dt=a*x, which a is parameter. In numerical result, we choose an initial condition and a value for "a" and find solution, but we can't deduce general result for any initial condition and any value of "a", while analytical result (if can be derived) gives result for any initial condition and any value of "a".
A computer simulation, a computer model, or a computational model is a computer program, run on a single computer, or a network of computers, that attempts to simulate an abstract model of a particular system. Computer simulations have become a useful part of mathematical modeling of many natural systems in physics (computational physics), astrophysics, chemistry and biology, human systems in economics, psychology, social science, and engineering. Simulation of a system is represented as the running of the system's model. It can be used to explore and gain new insights into new technology, and to estimate the performance of systems too complex for analytical solutions.
In any system (or model), you have parameters and other conditions (e.g. initial conditions) with make results different. In simulation, values of these effective parameters must be specified and result will be true just for that value of parameter. But analytical result, give a general description of system for any value of parameters. For example, suppose that we want to solve a simple linear ode: dx/dt=a*x, which a is parameter. In numerical result, we choose an initial condition and a value for "a" and find solution, but we can't deduce general result for any initial condition and any value of "a", while analytical result (if can be derived) gives result for any initial condition and any value of "a".
There's a lot of difference between different simulations with equal parameters, which really shows the challenges of simulation. Like Y. Azizi wrote, a simulation model always abstracts away from the real world. This is a core advantage of simulation (it doesn't cost a lot of money to run a quick simulation), but also means that your results may not transfer to the real world.
Conference Paper Simulation of ad hoc networks: ns-2 compared to JiST/SWANS.
For the performance analysis of wireless sensor networks which model is a best one Analytic model or Simulation ?
Analytical model represents performance in ideal conditions, even simulation shows performance in ideal conditions but with GUI normally. But i recommend to perform real testbed experiments. It has more worth :)
Mathematical analysis provides a perfectly clear relationship between inputs and outputs. Furthermore, it provides a limited number of solutions for a given input and the solution can be optimized. On the other hand, simulation results need to be statistically interpreted, since they often refer to stochastic processes. Furthermore, there are several issues that any simulation needs to cope with: data independence, random number generation, representative outputs, removing transient periods, etc.
The takeaway must be that there is no bad approach. Analysis must be done as long as it is possible, but simulation is a good approach in scenarios where analysis cannot be used, due to complexity or due to the lack of a closest form for the solution.
Even the well settled Netwon Laws of mechanics assume that bodies don't change their mass and therefore are only applicable for objects joggling within the earth's physics; i.e. at very much low speeds compared to the speed of light as mass will change as speed approaches the speed of light. Therefore, as you said Osama, assumptions are involved in analytical works as well. However, Newtons laws remain laws. In other words, you can't infer a law from simulation while analytical works themselves usually give birth to laws.
Infering laws from simulations is never a goal, though. Both analytical and simulated analysis of a system has its flaws- the only reason we do it is to save resources. Besides them we still have other options; proofs or experiments.
Formal or mathematical proofs are the best wey we have, but such proofs are virtually always impractical to create (outside of pure mathematics and pure theoretical computer science).
Analytical results are the next best thing - using a simplification of the system, we can derive a bound or sometimes even a proof. The impact of simplifications should be considered carefully - this is a common weakness in sensor networking papers that take this approach.
Simulated results are easy to design, repeat and statistically analyze; however, they do not provide a representation that is as accurate as analytical models. Simulatilons are usually the right choice for ad hoc networks - where the cost of experimentation is too high. The additional advantage is that you can do rapid prototyping and find initial results quickly, which gives you feedback on the protocol you're designing.
Experimental results are the hardest and most costly to obtain, but they provide insights for those things of which we have no good models. For example - in current research in vehicular networks, we are very much interested in how vehicles move on the different road types.
The best tool I found for visualizing graphs and networks is GraphThing but if your planning to use some library so that you can program things as per your needs than there are two library a gcc based SNAP(Stanford Network Analysis) or you can use the java based JUNG (http://jung.sourceforge.net/) the best open source available in the domain.
I agree with Rens van der Heijden. Just a small addition, in my opinion both analytical and simulation results can be complementary. In some cases, the analytical model provides you a good overview of the general case, but some small details may be not well represented and when you perform simulation you may find some deviations that are not considered in analytical model. This discrepancies can be even bigger when you go to real experimental results, but as Rens well mentioned, in ad hoc networks, the choice for simulations is the one that suites the best due to all reasons that he mentioned.
In summary, what I would recommend is whenever it is possible, do both analytical models and simulations.
Usually a heuristic based analytical argument needs simulation/testbed results to proof accuracy. However, for a mathematical/Quantitative based analytical argument usually numerical analysis results are enough to proof its accuracy.
No, a numerical analysis would not be sufficient, because the most important constraint is missing: assumptions made by the proof.
Let me elaborate on that:
The more important issue is that a mathematical or analytical proof requires abstraction. This abstraction may lead to simplification of the problem, or to leaving out important factors. The advantage of simulation there is that (ideally speaking) the results will provide feedback. Sure, you may have a proof that shows your ad-hoc routing scheme is perfect, but if you don't consider hidden terminals in your proof, then you still haven't proven that it works. Sometimes those results are still valuable, but one must be very careful and explicitly write down the assumptions made by the proof. This is difficult -some assumptions are so inherent to the field that it seems crazy to write them down- but it is one of the things I check for when I review a paper.
@Rens van der Heijden let me rephrase " for a mathematical/Quantitative based analytical argument usually numerical analysis results are enough to proof its accuracy with given assumptions".
In simulation programs we also make assumptions. Additionally for a given problem set different simulation tools gives different results which makes simulation results weaker and needs strong analytical argument. However I agree with you that "The advantage of simulation there is that (ideally speaking) the results will provide feedback."
When I do literature review I group and rank the papers in following categories.
1-The papers which presents mathematical/Quantitative based analytical argument followed by Testbed/Simulation results.
2-The papers which presents mathematical/Quantitative based analytical argument followed by numerical
analysis.
3-The papers which presents Heuristic based analytical argument followed by Simulation results.
In a manner to compare results obtained in two different ways be sure that assumptions are the same. Otherwise you risk mixing oranges and apples.
In your case, both analytical and simulation results can be significant and the extent of significance is measured by reliability. Be sure to find reliability of analytical model you plan to use, e.g confidence interval (CI). Also calculate all relevant reliability parameters of your simulation obtained results. Then you simply say, I compare analytical results with xx% CI and results obtained by simulation of the same phenomenon with yy% CI.
The analytical discussion provides a better clear picture of the approach used in the different scenario. It also focuses on the limitations and overheads generated during the process. On the other hand, every simulator has its own limitations and can not be completely applied everywhere.
Hi
I disagree with most of what was written here. Simulation is not that bad all the time. Math is not that good all of the time. In most of the cases, simulation is used when math models are hard to deploy. If we look to distributed systems and P2P research, most of this work is simulation. It was hard to create math models in most of the research cases. Moon asked about new protocol, If you can extract math model go with it. However, simulation sometimes is more hard that math modeling (NS2, OMNET++, etc..). Do not forget that there is a new direction to use real data in simulation configuration.
@Mohammad Masoud " It was hard to create math models in most of the research cases" your this claim itself suggesting the importance of Mathematics. In order to proof the hardness of some problem you need mathematical proof thn you can give some kind of heuristic solution and simulation results.
Either approach can work, just keep in mind that in most cases, mathematical models do not account for all types of channel conditions, hidden terminals and so on. The whole point of simulation is to run such a model* in a simplified form that makes it computationally feasible to do so.
* usually a simulator is composed of numerous individual models, not just one.
In fact, both have their own advantages and the decision to give the priority over other is depend upon the problem also.
Because of the mathematics involved in the analytical model, people have a way of tracking and understanding from where you have obtained the results and how the end formulas where formulated or derived. However, in simulations it is extremely difficult for any one to really tell what you have implemented and what kind of hidden tricks were done behind the scenes, so normally are more sceptical when it comes to simulations.
Any modelling , the analytic results are something deterministic and don't variate with time(iteration), while as simulation results may variate and can't be taken as a fundamental(Bedrock) for further research.So fundamental research in any domain is to be purely based on analytic findings.
I can understand your point that we try to emulate the real world scenario more closely in simulations and still why do we believe in analytic more. This can be justified by this, "In simulations we try to provide dummy for every factor(parameter) and still aren't sure about the outcome, while as in analytic modelling we try to be true at least to numbers , single(few) factors. "
Regarding Computer Network simulation and modelling, i suggest you that it purely depends on the area of your concern. If you can provide an analytic model better you go for it, else simulations are what is the last hope !
I will followup with Osama's question. "Simulated vs Analytical Results". What about any one of this compared with "Real-world Results". The reason for this question is that, sometimes for the real-word cases you do not have strong mathematical deductions, however, one is able combine Simulation studies and then proceed to implement a real-world testbed. Does it mean that that result might not be as strong compared with Analytical results?
Both are needed because it is a way for verifying analytical and simulation results.
A lot of times, it is a good approach to look into potentials for optimizations/new approaches using a simulation approach. Based on the findings, the limiting assumptions can be relaxed until you reach an implementation...
They are complementary rather than competing. Some problems are better suited to one method or the other but in most cases persuing both can give significant insights of different nature (e.g. generality of the solution vs scalability or performance issues, etc.). Moreover, some of the high profile journals or conferences require work in both aspects. Finally, in my understanding a lot of people have a preference for analytical results because such results are indeed proof of significant work with potentially high impact and citation count (assuming no errors, etc.) while on the simulator side historically only a very small number of open source simulators reached a level of critical mass that attracted significant attention (e.g., OMNET, Simplescalar, etc.) and these can only be produced by the collaboration of relatively large teams.
Analytical modelling is more theoretical while an effective simulation should actually give insights to practical realizations. The main problem in simulations are actually the quality of the simulation tools available. Thus in implementing simulations there is a need for comprehensive attention to details in representations of systems under investigation and the actual choice of the coding language (object orientation over procedural i.e. C++ over any other programming languages for engineering simulations).
http://www.mycplus.com/featured-articles/top-10-applications-written-in-c-cplusplus/
One should try it, comparing simulated results with theory , then if possible with real time experiments. Even different different simulators will produce different results for same problem. Every simulator have some different approach towards assuming certain default values, their ability to use hardware , time and space complexity of algorithms etc.
The real challenge here is level of control and predictability. If we consider the axis analysis - simulation - measurement, analysis is perfectly controllable and once phenomena can be accurately modelled, it advance our state of the art in science (engineering). This is a clear motivation to strive for analysis. In most of the cases, however we are not able to sufficiently accurately model complex communication systems, i.e. we have to make (sometimes rough) assumption that lead to general understanding of system performance. These assumptions will make our perception of the system functionality fuzzy with respect to what may and will happen in realisations of the system. For example, if our analysis tells us that with a certain channel model, modulation, coding, fading, error correction, and packet sizes we are able to just have an average performance giving us a low packet error rate, simulations or measurements may show that this is not the case. Reality imposes randomness and that randomness will work against our system in the example case if more than average amount of bit errors occurs in any specific part of the received packet.
Now simulations and measurements inherently take such randomness into account. The difference is that in simulations one can control the randomness whereas with measurements the randomness (outside anechoic chambers) is related with time of day, temperature, environment, solar radiation, etc. With simulations one can explore the behaviour of the system's component in the presence of random behaviour in a controlled way and gain insight on what parts of the system do not function well in a realistic environment. As said, with measurements the environment is very difficult to control well enough to isolate the problematic areas.
What I typically do is model the system behaviour to the extent reasonable (extending complexity as I go) and build a corresponding simulation environment. Then I find out whether the simulation model and analysis match at the complexity level that analysis can reach and if they do I have reasonable confidence in the simulation model. After that I start exploring the simulation model, increasing complexity and finding out the impact of different components of the system to the overall performance. This method enables providing more insight to the system than pure analysis and while it may not be as tractable as analysis, it may be as important.
Analytical research work proves and focuses on your research algorithm or model usually mathematically and statistically while simulation aims to compare your research work to other similar work under the same simulator's environment with the same parameters in order to test its performance
Comparing the two approaches to research, the most interesting would be the analytic approach one as it gives accurate results. Awkwardly, it becomes very quickly inapplicable, because of the size of the model or due to, for example, its non-Markovian nature (as argued by some researchers), so we need to apply approximation methods to get more results. However, even these approximation methods may become inefficient, in which case the last resort is to use simulation.
The Point I am driving at is to avoid a situation of saying one is better than the other but rather look at the two objectively and use them as tools that can give us results depending on our circumstances, situation and need. Yes we can compare the two but as we may all know in research, at a certain point, you will need one or the other or both to meet you goal.It is a point of trying to strike the balance here.
Simulation may not always produce accurate results, it works on logical manipulation. Whereas analytical results are accurate as these are obtained by proven mathematical manipulation.
Attached intro lecture to my simulation course may help clearly illustrate the differences.
Thanks a lot Dr. Trivedi for your useful note. I used your papers, your beneficial softwares (SPNP & Sharp) for my thesis and also for simulation course as a lecturer in university.
The simulation cannot produce instant results and takes time to be performed. In fact, the simulation requires many independent runs which may take long time to produce accurate results. The input values for the various parameters as well as the actual running of simulation program or software usually have to be done manually, whereas analytical methods can obtain instantaneous results and with mathematical formulas being easily integrated with other algorithms.
Analytical Approach or method Utilizes statistics, algorithms, and sophisticated data search capabilities to discover hidden patterns and relationships in data and offer more efficient prediction of future results.
Whereas, the simulation methods cannot produce an efficient instant results and takes series of processes to be performed
Analytical result is more accurate than simulated result, which often times depends on lots of conditionalities
Both results from analytical and simulated approaches are very important in research outputs. In my own opinion, the analytical approach gives more robust discussion of results than the simulated or forecast ones.
If you would like to give answer on question: "What accuracy and certainty (reliability) modeling results will I get considering stochastic factor?" In this case, better to perform first step based on the use of simulation approach. Next step is considering overall simulation trend and refinement of assessment criteria. After that you can build quite accurate analytical model in accordance with chosen and adjusted criteria. At the same times, my proposals did not purport to be absolute dogmas. In the near future time I will plan to leverage this approach in my researches.
I would state that Simulated and Analytical results are both important , due to the fact that in transitioning from theory to practical realisations, Neither Simulated or Analytical results will give 100% accuracy for proof of Concept.
To achieve practical results both Simulated and analytical models are essentials.
I do not believe that analytical RESULTS are necessarily more powerfull than those obtained in simulations.
I have already submitted work using mathematical modeling and the peers who evaluated it asked me if I would add a simulation to enrich the paper.
If the system is programmed correctly in the simulator and the model is also correct and has the corresponding parameters, it should be possible to obtain quite similar results. I could do a little experiment once to validade my simulation and it worked.
However, in terms of FEASIBILITY you may find analitical modeling more attractive, since in simulators like NS-3, creating a greater topology may take more time, while in modeling you may manipulate some parameters. Everything depends on your problem. Further, it may also depend on the type of conference (and the area of your peers) you are submitting to.
If you decide to use a simulator, avoid creating your own, because you would need to validate it first.
Try to use some well-known simulator. If you decide to model your system,
do not forget to detail and justify all your assumptions, symbols, and equations.
Dear Osama,
The simulated and analytical are solutions to the same problem
one in the form of mathematical formulas and the other is a numerical solution for the same problem.
The effort spent to get the analytical solution is much greater than the effort in simulation based on numerical solutions implemented by the computers.
The analytical solution is transparent but may be approximate
while the simulation based on computes is not transparent ans needs to make many runs to bring transparency in it.
The numerical solutions needs computers to carry out and the analytical solutions needs pencil and paper.
Best wishes
For linear Systems under some conditions the analytical results and simulations are equivelent, but for specific linear system the simulations May be give fake results. But in the case of nonlinear system which always sensitive to initial conditions and may have more than one solution the simulation results always give one solution depending on the initial conditions
Please read my work
American University Laboratories for Electrical Engineering (part 3)
It has a good way to compare theory and simulation.
"Analytical runs when repeated under same conditions hardly give the same response."
They give you similar results if you run a stationary analysis.
There seems to be an automatic assumption among answers that analytic solutions are necessarily in closed form formulas. There is an intermediate possibility where underlying equations are solved numerically using a software package such as SHARPE or SPNP. Simulation produces one sample path per run and hence necessarily requires many runs and even then produces only a confidence interval for the results. Analytic solution by contrast does not need multiple runs.
I am not sure what this comment below refers to:
"Analytical runs when repeated under same conditions hardly give the same response."
That comment applies to simulative solution but not analytic! Perhaps the author is referring to same algorithm implemented by different individuals? Sure numerical errors in solving underlying equations may give different answers but not if you use the same implementation again for the same inputs.
I also don't believe that effort spent deriving analytic-numeric solution is much greater than simulation --assuming you are using the right software package.
The class notes of my course may be useful. I am attaching them together with two papers where both analytic-numeric solution as well as simulation are carried out.
My 2017 (green)book with Bobbio (as well as my queueing book with Bolch) may also be helpful.
Because Analytical Results are based on analytic reasoning as opposed to synthetic
- Badges
- Science topic