Hi,

I think if you just want to use it to make your things faster without really going into detail and do research into that area then its ok. On the other side I never heard that someone did research in parallel computing based on matlab implementations. I guess C is more what you should use then.

Best,

Michael

Dear Khalil,

I think you asked a very controversial question but of course valuable for everybody who heard about this problem but did not use parallel computing. I am a supporter of Mathematica software but a few years ago I also used to use MATLAB software. Because they possess a lot of common features then I can tell you  how this problem is in Mathematica.

If you want to use parallel computing in this software you have to know perfectly your algorithm of computing. It is dangerous that you will try to compute some unknown variables before they can be able to do. Of course there are a few commands that let you control the process of computing and avoid errors. You should be aware that only part of our numerical algorithms can be completed in this way. If you program your parallel code in the right way you should not obtain any bad results.

I like parallel computing because if I have a few cores I can speed up some part of my calculation also a few times -there are theoretically laws in this subject for example Amdahl’s law and Gustafson's law

http://en.wikipedia.org/wiki/Amdahl's_law

http://en.wikipedia.org/wiki/Gustafson%27s_law

To sum up If you analyze carefully and apply a control command you should just enjoy the advantages  of this kind of computing.

Not sure about Matlab, but one general disadvantage of parallel computing is that you may not get exactly the same answer from 2 different parallel runs.  It's not that one run is right and the other wrong, just that e.g., the contribution to a global sum from differernt parallel threads may come in a different order from run to run, and so roundoff error can accumulate differently.   

E.g., write a loop to sum up (1/float(i)) from i=1 to 10,000, and compare with the answer you get when you sum  (1/float(i)) from i=10,000 to 1.  I got 9.787616 in the first case, and 9.787607 in the second.  Both answers are equally "correct" (algebraically), but also equally "wrong" (numerically). 

Depending on your algorithm, such roundoff errors can accumulate very fast.  In that case, different parallel runs may give quite different answers, but the real problem is not with the paralllelism, but with the underlying algorithm. 

In principle, parallel runs should give you the same answer - or at least, many equally valid ones.  The idea of parallelism is not to change the answer, but to arrive at it faster.  However, in some cases the computational "overhead" of the parallelism can overwhelm the intrinsic parallel speedup, and leave you with little or no performance gain.  That can be another real disadvantage - again, depending on your algorithm.

Dear Khalil,

After reading very interesting Enda's comments you should read the following discussion

http://mathematica.stackexchange.com/questions/28896/why-is-parallel-slower

Mathematica possess the special function to sum in parallel mode (ParallelSum[]). This function gives always the right result (it has proper inner control). But the problem is that parallel summation can be slower than sequential (traditional). If you read this discussion you can learn that after some modifications parallel computing is really faster than  sequential. You can also find about topics like overhead in parallelization and other interesting ones which said  Enda.

Dear Enda,

I agree with you that parallel computing can be a great challenge if we want to write own program using standard compilers with appropriate library. I remember that I was very angry when the results differed from the expected when I programmed own threads or processes in C under Linux. I tried to use for controlling parallel computing different mechanisms but sometimes they did not work “properly”. I recalled one episode connected with “pipes” when I wanted to move some code between processes. After a lot of trials it turned that when I programmed against all the known rules program became to work properly.

But on the other hand if we use commercial software like Mathematica, Matlab etc. we do not have to think about a lot of different control commands. We should use only the recommendations how to use the “parallel” commands.

Very important and useful informations, Thanks all of you.

Dear Khalil,

I think that it depends on the type of problem you are willing to solve. If your problem is highly parallelizable and you do not wish to worry about parallel programming details, then the Parallel Computing Toolbox (PCT) plugin may be very useful in this sense. It hides from the programmer all details about data parallelism exploration. For instance, I have used this plugin to parallelize large and sparse linear systems. As a consequence, I've been capable of demonstrating that the parallelized version has a superior computational performance over the sequential one.

If you wish, feel free to contact me. There are on my profile some papers that deal with that.

Hi,

I think if you just want to use it to make your things faster without really going into detail and do research into that area then its ok. On the other side I never heard that someone did research in parallel computing based on matlab implementations. I guess C is more what you should use then.

Best,

Michael

I think there are two main points to consider here:

1) There is always an overhead associated with parallel processing. The size of this overhead depends on the method of parallelization (granularity), but some examples of the extra cost (i.e. increased execution time) incurred are due to: data transfers, synchronization, communication,  thread creation/destruction, etc. These costs can sometimes be quite large, and may actually exceed the gains due to parallelization, resulting in code that actually runs slower, or not nearly as fast as expected. This result often surprises people. So prepare to be disappointed. 

2) Well written Matlab code, even without any extra toolboxes (=$$)  will actually utilize multiple cores, so if you have a new Intel CPU, with 4 cores, each capable of running two concurrent threads, then you may see a good speedup of up to 8x, relative to a single thread. By well-written, I mean without loops.

And yes, as Michael points out, if performance is important, write your application in C++. I think Matlab makes people lazy (me included).

If you don't mind using Microsoft products, Visual Studio now comes with some nice parallel programming language extensions using PPL (for multicore CPUs) and/or AMP (for GPUs). For an application I was working on, I found the former more gratifying.