Here now the book Au/Wang is discussed. Unfortunately the method championed there SS [sic!] is some sort of rain dance or voodoo ritual. The authors promise you if you perform the ritual/algorithm you will obtain a meaningful estimate for your failure probability. This is demonstrated by a number of examples. No general proof is given. This works only for limit state functions with a simple geometric structure, for more complicated structures it fails in many cases, as the writer has shown (see the papers on Researchgate).

The maths here is the maths of the ancient Egyptians, who proved the theorem of Pythagoras by examples. But after them came the ancient Greeks and they introduced in maths the concept of a proof. In spite of these advances here in SS there is a regression to the Egyptians all is based on simple examples and from there it is extrapolated that it works for complicated cases too, no proofs at all. Reminds a little bit of the civil engineer who wanted to prove that all odd numbers are prime.

The writer has tried to explain to the SS people several times that the whole approach is flawed, but they continue to rearrange the deck chairs without taking notice. The basic misunderstanding of SS and all these fumbling around SS derivates is that they do not understand the problem.

If one wants to calculate failure probabilities, it means to find an integral over a huge domain in the n-dimensional space whose location is not known exactly.

If one wants to avoid to integrate over the whole domain, one must restrict the integration to a subset (pun intended) of the original domain. And then one invariably ends up in neighborhoods of the points in this domain where the PDF is maximal, i.e. the modes (I avoid the expression design point but they are the same), because somewhere here the main probability mass is located. Now --- to counter objections --- in high dimensional spaces the immediate neighborhood is not quite enough, but one has to go a little bit further.

The problem of SS is that basically it is a MC version of FORM/SORM, but claims that geometry is totally irrelevant. Howver, please have a look at all the SS-examples, the data move always to the modes/design points. Why not to start from these points? We know where they are. Is it necessary to find them by lengthy MC procedures?

Anyway, everybody who understands maths, should see that if one wants to find

P(g(X) 0 if the structure of the function g is unknown. This works only if the function g has a simple structure, for example if it is a homogeneous function or a transformation of it, otherwise everything can go south.

When the number of samples is large and involved many variables. A general computer with low CPU.

Dear Xiaocheng,

For such examples, the main problem is the cpu of the computer! :)

The aim of my question is clarifying the statistical problems of Monte Carlo simulation (not the Computational cost/time of simulation)

Monte Carlo simulation has many advantages, however, at least one drawback: the choice of the probability distributions.

If the probability distributions are completely wrong, does not matter how many “run” we have, the results are not reliable.

Hello Roberto,

Monte Carlo simulation is based on the law of large numbers (which has some limitations). Therefore, even if the probability distributions are properly determined, this approach may fail to converge in some cases.

It would be interesting if one present an example problem (i.e. a problem with normal random variables!) that Monte Carlo simulation be disable to solve it.

I'm working on the 'general accuracy' of reliability methods in the probability estimation and I wish to find a researcher with the same interest.

Hi Mohsen,

It is a good question. As we all know, the Monte Carlo simulation (MCS) is based on the Law of Large Numbers (LLN). That is to say, if there is a case that LLn is invalid, MCS will be infeasible. For example, g(X) = X is the limit state function with X being the variable following a Cauchy distribution. Even it seems in reality the limit state funciton is not likely to happen to be a Cauchy distribution, but in theory I don't think MCS can work for this case.

Best regards,

Chao

Hi Chao,

Good Answer. As you mentioned, LLn is not valid for the Cauchy distribution (and some Pareto distributions). Now, let g(X) as the limit state function with the mentioned distribution. We can transform this function to standard normal space, say g(U). So, we have a function in standard normal space that MCS fails to provide a solution for it. So, how you can trust to MCS when you solve a complex problem?

I know there is not a better solution for solving a black-box reliability problem (It should be mentioned that there are some new approaches that are presented for solving heavy tail distribution, but they are not general approaches) but, Here I wish to find such problems.

Hi Mohsen,

The basis of the Monte Carlo simulation has been discussed in a book , which may be helpful for you. I would like to share with you: Au, Siu-Kui, and Yu Wang. Engineering risk assessment with subset simulation. John Wiley & Sons, 2014.

Please see Section 2.5 Direct Monte Carlo

Best wishes,

Chao

Thank you Chao, I will have a look on the book.

Nonetheless, I would be grateful if you write your answers/findings here.

Please let us know what do you think about it.

Dear Chao,

I read the book. It was a good book. However, somethings are overlooked in this reference.

When u1 and u2 are standard normal random variables, then u1/u2 has the standard Cauchy distribution. So, as I mentioned before, we can present a performance function with standard normal random variables in which MCS fails to accurately solve it (i.e g=4-u1/u2). It is regardless of the PDF types of random variables.

The question is quite general. Therefore IMHO it is necessary to philosophize a little bit.

R. Hamming said: "The purpose of scientific computing is insight, not numbers."

The so-called solutions found by Monte Carlo methods are just big heaps of numbers.

Do they give insight? This depends on the structure of the problem. If we have a forward problem, i.e. we have a given structure and we need only one additional property for example the failure probability of a beam collapsing, maybe this is enough. I switch now to the field of structural reliability --- where the questioner

and me are working; usually here this is never the question. One is interested in more, to understand why failure occurs and to wat changes in the parameters it is sensitive. So this can be seen more as an inverse problem. Not only the probability, but what is the combination of factors leading this result. So IMHO things in this context should be seen from a more structuralistic view.

Here MC might help, but it is then only a tool in larger context. MC alone cannot give solutions, it can give only numbers.

For problems one needs some sort of analytic structure, which then can be beefed up with MC approximations.

MC can always produce numbers. But is "42"*) really the solution? This leads to the problem: What is the question?

Without a theoretical superstructure how one can trust such results? Empirical evidence says the engineer. Please remember the story of the civil engineer, who wanted to show that all odd numbers are prime numbers.

If there is no theoretical foundation for a MC result to the theoretical result, why to believe that it means anything. An extremely dangerous thing is that Monte Carlo methods may lead people to claiming to have found solutions

for problems which have no meaning.

Let me give an example:

In SIAMUQ Vol. 3 (pp. 922) in an article the problem of maxima of the solutions of

an elliptic PDE is studied and using the subset simulation method, a variant of MC, a "solution" is found; the solution is a probability, i.e. a number. However this number does not mean anything as the problem is formulated. First, the random process used for the coefficients, has no derivative (this fact can be found in any textbook), but the PDE system involves its derivative; so this is not a PDE system. Now, maybe someone will object, this can be seen as a SPDE system (however not mentioned in the paper).

But now here the second problem pops up. For such systems, the convergence of the functionals of the sample paths does not necessarily follow from the weak

convergence of the paths. Such problems have been studied in probability theory since long and cannot just be ignored. A sequence of maximum probabilities for approximations does not necessarily converge towards the maximum probability of the limit process. So Monte Carlo methods produce here a "solution", which

as it stands, has no meaning at all; the PDE system is no PDE system and the convergence of the "approximations" to the "solution" is uncertain. But to understand this, one has to know a lot about maths, not only how to run MCMC's.

Many people now believe if their MC spits out numbers that they automatically mean something. THIS IS WRONG!

I think that one has to take a more structuralistc approach. This is outlined in my paper "On onions and subsets: lost in outer space" , available on Researchgate.

*) Douglas Adams: The hitchhiker's guide to the galaxy.

Here now the book Au/Wang is discussed. Unfortunately the method championed there SS [sic!] is some sort of rain dance or voodoo ritual. The authors promise you if you perform the ritual/algorithm you will obtain a meaningful estimate for your failure probability. This is demonstrated by a number of examples. No general proof is given. This works only for limit state functions with a simple geometric structure, for more complicated structures it fails in many cases, as the writer has shown (see the papers on Researchgate).

The maths here is the maths of the ancient Egyptians, who proved the theorem of Pythagoras by examples. But after them came the ancient Greeks and they introduced in maths the concept of a proof. In spite of these advances here in SS there is a regression to the Egyptians all is based on simple examples and from there it is extrapolated that it works for complicated cases too, no proofs at all. Reminds a little bit of the civil engineer who wanted to prove that all odd numbers are prime.

The writer has tried to explain to the SS people several times that the whole approach is flawed, but they continue to rearrange the deck chairs without taking notice. The basic misunderstanding of SS and all these fumbling around SS derivates is that they do not understand the problem.

If one wants to calculate failure probabilities, it means to find an integral over a huge domain in the n-dimensional space whose location is not known exactly.

If one wants to avoid to integrate over the whole domain, one must restrict the integration to a subset (pun intended) of the original domain. And then one invariably ends up in neighborhoods of the points in this domain where the PDF is maximal, i.e. the modes (I avoid the expression design point but they are the same), because somewhere here the main probability mass is located. Now --- to counter objections --- in high dimensional spaces the immediate neighborhood is not quite enough, but one has to go a little bit further.

The problem of SS is that basically it is a MC version of FORM/SORM, but claims that geometry is totally irrelevant. Howver, please have a look at all the SS-examples, the data move always to the modes/design points. Why not to start from these points? We know where they are. Is it necessary to find them by lengthy MC procedures?

Anyway, everybody who understands maths, should see that if one wants to find

P(g(X) 0 if the structure of the function g is unknown. This works only if the function g has a simple structure, for example if it is a homogeneous function or a transformation of it, otherwise everything can go south.

Dear Professor Breitung,

Thank you very much for your participation in this discussion and important comments. I have read your papers. In this field, you are a thinker and I really like your viewpoints.

However, noting that I’m agreeing with most of your opinions, I have some concerns:

1. It seems that you are OK with the accuracy of FORM/SORM algorithms for finding design point and providing an approximation of failure probability.

However, I use simulation methods in my studies since I cannot trust to FORM results since the design point may be determined incorrectly. Existing algorithms are not robust for complex problems. Here, if design point is determined properly, importance sampling provides more promising results compare to SORM.

2. FORM is calibrated to find P(R

Dear Professor Karl Breitung ,

Thanks for your contribution, and I would like to focus on your statement:

“MC alone cannot give solutions, it can give only numbers. For problems one needs some sort of analytic structure, which then can be beefed up with MC approximations”.

I am more in the field of risk assessment (human and environment), however, the basic concepts of Monte Carlo simulation are similar and I agree with your statement.

Often, a “tiered approach” may be helpful in deciding whether or not a Monte Carlo analysis can be useful. One begins with a fairly simple screening level model and progresses to more sophisticated and realistic (and usually more complex) models only as warranted by the findings and value added to the decision.

At the beginning of the procedure there is always the definition of the “Assessment Questions”. At this step, risk assessor and risk management have to agree with them, otherwise there will be a mess in all procedure, especially in the risk communication!

Secondly, there will be the development of the Conceptual and Mathematical Models, otherwise “The so-called solutions found by Monte Carlo methods are just big heaps of numbers”

Expert judgment can be included either within the computational analysis by developing distributions using various methods or by using judgments to select and separately analyse alternate, but plausible, scenarios.

When expert judgment is employed, it should be very explicit and transparent.

One may naively think that he can assess reliability of a complex structure given a high-performance computer (eg. a quantum computer). However, generally, it is very difficult to define reliability of a complex structure. Uncertainty propagation is just one aspect of assessing reliability. Before assessing reliability, one must have good knowledge about plasticity, damage, fracture, or even complexity science and system science.

@ Mohsen Rashki

Dear Mohsen,

thanks for your answer. We disagree in some points, but this might be due to the fact that I expressed myself not clear enough.

1) To your first point:

When I came up with the SORM approximation a long time ago, the most important point was that this was based in concepts of asymptotic analysis giving a sound mathematical foundation for the procedure. Seeing failure probability estimation as a problem of asymptotic analysis gave the whole field a structure and also a quality criterion. This criterion is that for increasing \beta (decreasing P_f) the relative error of the estimator should decrease. In the last decades of the last century many people were quite content with SORM. With the increasing dimensions it became clear that the efficiency of SORM is decreasing substantially. Now IMHO the right way would have been to find ways to improve SORM. But with the appearance of SS many people jumped on the bandwagon and abandoned the concepts of asymptotic analysis totally. The fact that a method derived from these concepts does not work well in high dimensional problems do not show that the whole concepts are wrong. So nowadays there is no umbrella concept. Certainly if one would apply the quality criterion of asymptotics to SS, the method looks bad, since the variance of the estimator rises as yeast dough for P_f getting very small.

But I am not a dotard who is moaning that his results from a long time ago were the high water mark of the art of structural reliability maths. If the accuracy of SORM is no more sufficient, one has to improve it. And this I will do. The basic idea is:

1) Calculate a SORM (or FORM) approximation P(F_Q)

2) Estimate the fraction P(F)/P(F_Q) by MCMC as r.

3) Estimate P(F) by P(F_Q)*r.

To your second point:

This is not quite correct. FORM is a linear approximation to the failure probability.

The original LSF g is replaced by its first order Taylor expansion at the design point g_L and then P(g_L

Dear Karl,

Your comments are very interesting and important. I believe that this discussion could be helpful for a structural reliability researcher. Therefore, I’m willing to represent my opinions. I hope they be relevant.

1. Obviously SORM is an important method. It is still in the competition with the advanced simulation methods. For improving the SORM, the control variate technique may be helpful. The idea of control variates is to provide an imprecise (but fast) approximation of the failure probability, and then improve it using small sample size simulation (i.e. using MCMC as you mention in the comment). I have applied it to FORM and achieved very promising results (One of my published paper in this regard is “First order control variates algorithm for reliability analysis of engineering structures” and the others are under review).

2. As I mentioned before, FORM is calibrated for estimating P(R-Q

Dear Mohsen,

many thanks for your reply:

1) I would not overestimate SORM. However I would like to emphasise once more the importance of asymptotics, there might be other methods. I see the failure probability calculation question as an asymptotic problem. One should not only study the performance of an estimator for a problem given by S={g(x)=b}, but consider the asymptotics, i.e. the surfaces S(r)={g(x/r)=b} as r -> oo. How behaves the estimator? A good estimator should not have an increasing variance, its relative error should go to zero. This is my opinion.

2) I do not get your point. In FORM the original LSF g is replaced by its first order Taylor expansion at the design point g_L and then P(g_L

Dear Karl,

Thank you very much for your response and helpful comments.

1. If we know where are the important failure domains (and if limit state function is not Ice-land shape), employing SORM is completely logical. Here, I'm agreeing with you.

I think I found the problem! It seems that you assume that the important failure domains are known (i.e. by using FORM algorithms). However, for a real-world problem with implicit performance function, I cannot trust to FORM algorithms for finding design points and importance failure domains. Existing gradient-based algorithms are weak. So, researchers prefer to use a robust algorithm (i.e. Heuristic) for finding importance failure domains. The performance of SS is same as a Heuristic optimization method (as you mentioned in your papers) and researchers often use this approach in analysis (we know that this approach has many drawbacks).

If you have time, please read: 'Hybrid control variates-based simulation method for structural reliability analysis of some problems with low failure probability'

2. Please solve the mentioned problem. For the proposed example, 120 is the resistance and the random variables are load parameters. Decrease and increase in X1 may lead to failure for the problem. FORM is not calibrated for solving such cases and presents wrong results.

3. Thank you for your consideration. I’ll send you the Code and manuscript.

Dear Dr Rashki

The ideas and how you look at developing and improving reliability methods have always been very interesting to us. I have also commented my opinion in the attached pdf file, according to your descriptions.

with best wishes for you.

Hi Farid,

Nice attempt ;) That performance function was just an imperfect example. We should find better examples.

Thanks. But I haven't added anything to structural reliability theory. Hope for the feature.

Dear Mohsen,

I fear I have to contradict you here.

1) You say there are no reliable methods to find design points without specifying the culprit. I guess you mean the old Hasofer-Lind algorithm; it is know since 45 years that it has convergence problems. But AFAIS these were solved by

Rackwitz and Der Kiureghian 30 years ago. Did you try to use these algorithms?

I am using in the moment in Octave the standard algorithm for constrained optimization sqp and it works like a charm. For a problem with four design points I let it run 20 times from random starting points, make a cluster analyis of the results and out come the four points.

And this is no stochastic optimization!

Now, you again clam that things like SS are useful for finding the regions near design points for complex problems. NO, if you speak with an expert for non linear optimization, he will explain you that such a sequential( continuation) method for

minimization does not work properly if the functions are not of a simple structure.

IN SS the original problem: find the maximum of th PDF under g(x)c_2 ... >c_n=0:

find the maximum of the PDF under g(x)

The Monte Carlo Simulation technique is an accurate method but is one of the means the most costly in computation time, especially when it is necessary to use, an independent tool for the evaluation of the limit state function , such as, finite element calculation codes; requiring multiple calls for evaluation, and the computation time is much more important when the structure presents a nonlinear behavior or that the numerical model is complex. this property makes it ineffective or impossible from a time point of view.

Dear Karl,

1. I know you like to have bet with civil engineers since you have always won them in the past! (last year, I read your discussion with Marco Broccardo). Here, would you like to have a new bet in this regard? ( just kidding)

2. I think you have repeated my statement with different words! FORM is not calibrated for solving such problems.

Many thanks for your reply.

Hello again dear Dr Rashki

Is this the problem?

"The complex numbers" (i.e. calculating sqrt of negative points).

{Please say yes ;) }

for example: g=1-sqrt(x) (in standard normal space).

Pf for this problem is about 0.16, because the indicator I for complex numbers is calculated based on the real part of these numbers. I think there is a bug here.

Hi Farid,

Simply consider an space with the safe and failure domains. Performance function should determine such that for any point (say sample), you have a clear understanding about the state of the system (to know the system is in failure or safe state). Considering this, I think the problem you defined is not a good performance function (and the problem is not related to the statistical problem of MCS).

Thanks for you effort ;)

The problem of MCS is the distributions with heavy tails. For such problems, even for large probabilities, increase in sample size would not led to convergence of MCS. For example, if the failure domain has Cauchy distribution, MCS (and its variants) fails to properly solve the problem. I'm looking for a real world problem with such specification.

Dear Mohsen,

I apologize if I was a little bit rude with my joke about SORM. But now often these things are mixed up and I wanted to clarify it beyond any doubt.

When I came into structural reliability long time ago, it was --- to quote what Margaret Hamilton said about another project--- like the Wild West,

nobody of us knew what he was doing. This evolved with the time and I think and I am still thinking this that asymptotic analysis is the right umbrella structure

for this topic. Unfortunately in the last years I see a re-wildwestization of the field, not much serious mathematical concepts, only examples to prove something. But with examples you can only show what does not work as in the example you presented, but not why something works.

Nowadays where all say that we have to work in in bloody **** high dimensional spaces one needs a mathematical understanding more than ever, because geometry in those spaces is quite different from geometry in two or three dimensions. Intuitive little pictures might be misleading. Not all can be extracted from MC numbers without understanding the structure.

I think now it is a good time to discuss things via internet, because most people are sitting at home. I will now prepare a question about SS. Stay tuned!

Dear Karl,

It is great. I like this challenge.

Not only one question, but al least two. The first is out. It is simply for warming up!

Nice try Karl, I'm ready for the next ;)

This question is actual for the case of the function under integral is unknown. Then Monte Carlo simulation is able to get mathematically precise and reliable solution.

Dear Dmitry S Oleinik, It seems that you are working on Monte Carlo methods (I read your page information). We know many things about the capabilities of Monte Carlo simulation but recently, I have achieved some interesting results. I have used Methodic doubt in reliability analysis and found there is a gap between truth and the current reliability interpretations (For instance, I have proved that the current interpretation of the importance sampling approach is not mathematically correct). In these regards, I will publish some papers that may be interesting for you.

For the case of the general accuracy of Monte Carlo simulation, I put away believes. We know that the law of large numbers has limitations. Here, I just presented a doubt by asking a question!

From point of view of pratice an experimental procedure lead to systematic error. I mean just existence of the procedure... We are not able to estimate this error, because the true truth is unavailable... But I prefer consider a pure mathematics... :)))

I will read your works necessarily later... :)

Generally, I believe that the comments of Karl Breitung are mathematically relevant and close to truth. But unfortunately, they are often overlooked.

I will discuss about this issue once my papers being published.

Thanks for your comments and contribution.

Dear Dmitry S Oleinik,

One of my papers regarding Monte Carlo methods is now available (Entitled: Structural reliability reformulation). You may find that all of existing methods (including advanced Monte Carlo methods) are derived based on some assumption and when the employed assumptions no longer hold for a problem, the proposed approaches may present completely wrong solutions. This is in agreement with the comments of Prof. Karl Breitung about the application of existing advanced Monte Carlo methods in reliability analysis. Many times, authors solve simple problems in their publications and their results misinform other researchers about the robustness and accuracy of these simulation methods.

Hence, for small size problems, I'm agreeing with the Asymptotic idea of Karl (that requires using design points or MPP in analysis ) noting that it is not still the best solution since finding proper design points would be difficult, especially for some real worlds problems with many small island shape failure regions (i.e. dynamical problems) and requires using evolutionary/heuristic approaches with many function calls in analysis. For high dimensional problems, I believe that the concept of reliability index (proposed by Hasofer and Lind) is meaningless. It is the subject of my next paper.

Thank you for the reference... I will study it...