I think the approach that most modern operating systems use is reasonable; the scheduler at least makes some attempt not to move a process around between cores too often. The process core switch is expensive and the cache (L1 at a minimum) in the new core may not be loaded with any useful data for that process.

I'm just saying that moderns OS schedulers do take into account some cost associated with moving a process to a different physical core. There are even tweaks for sceduling with Intel's Hyper-Threading, etc.

However, while I think the OSs do a good job for the most part, things can be improved. If you are interested in optimizing a specific application or set of applications, you can manually bind a process to a core or set of cores, and even change the set on the fly. CPU affinity in linux isn't too bad: there is #include with sched_setaffinity() for process affinity and pthreads has pthread_attr_setaffinity_np(). There are also command line tools that can modify already running processes.

So, while I don't know what is best, I think it depends on the application(s), and it could be an area to explore.

I've opened this discussion in order to explore possible system level optimization, instead of application level optimisation. Precisely, I'm not interested in PC operating systems, but in industrial OSs that would provide stable and independent program execution on multi-core and multiprocessor architectures. However, the principle is the same regardless of how we use the computer, so the question was therefore generalized.

You are right that most modern OSs do a good job, but there is always place for advancement and optimization, so this is where my question fits. What are your ideas and thoughts on how these methods, algorithms and procedures can be optimized and made more efficient and resource cost sensible.

If I were to pontificate, I would say that I think it really does depend on the application and some concept simiar to the "no free lunch theorem" applies here: "any two optimization algorithms are equivalent when their performance is averaged across all possible problems".

I'm sorry I cannot help you more, but my approach usually starts with some application (HMMER, Dock6) or class of applications (NCBI's BLAST suite, also phylogenetic tree builders: PhyML and RAXML), and works from there to improve performance or capability. When writing a tool from scratch, it is also practical to manually include API calls to set the appropriate affinites. I really don't know what best to improve in OS methods, algorithms and procedures w.r.t. CPU affinity.

Perhaps "train" a scheduler with a set of "representative" applications chosen to cover many common classes of software behavioral patterns. Then as the scheduler runs, it could try to classify the processes into the appropriate representative behavioral pattern, and thus know the "optimal" affinity policy for the application.

Some of this is already done with identification of CPU-heavy vs. I/O-heavy applications. However, with a machine learning approach, you could use loads of real production software to train a scheduler that would then know about lots of application behavior patterns, perhaps even complex ones.

I don't know, just thinking out loud here.

Also, I've only been talking about the scheduler being appropriate for the software needs. The harware complicates things as well. I think it would be great if modern OSs knew more about the CPU cache and NUMA memory layout than they seem to currently.

As I mentioned before, there are scheduler modifications for some hardware such as Intel's HT, and some modifications for AMD's new chips which have a shared L2 cache between pairs of cores, making a process core switch within a pair of cores less costly than across pairs of cores. NUMA makes things even more complicated.

My thoughts (both are just an application of the NFL philosophy) would be:

1) Have a scheduler that knows about many different behavioural patterns rather than trying to find a single optimal policy for all/most applications. Perhaps apply a machine learning approach and train with real production software applications?

2) Have the scheduler be intimately aware of the specific hardware details such as cache and core heirarchy, and NUMA topology, so the affinity policy can be paird with the specific hardware used.

Sharing memory definitely is a must when switching tasks between cores is required. This way, moving program's stored resources won't take up as much time as it would when all values were to be moved at some point.

I'm more interested in a dedicated approach when it comes to operating systems for embedded and/or industrial solutions (whereas PC architectures where users decide what to run and how many programs to run is a different case in its own right).

Having the scheduler be hardware specification aware sure is a bonus, but would probably cause a big impact on the overall size of the executable and thus impact the affinity of tasks that are being scheduled and thread-maintained.

Yes, for embedded applications, the complexity of the scheduler(s) I describe seems impractical.

This is currently done in a couple of cases:

1. Above the operating system level when the operating system is a hypervisor and the "processes" are OSes themselves.

2. The Apache Mesos project treats everything as a service and applications can access those services. This is again not at the OS level.

There have been OSes written which farm work out to nodes on which the OS is not running.

At the end of the day, compute power, state on the silicon, etc. are increasing at a rate that the things we considered as overheads (process scheduling, etc.) are small overheads today.

The bigger concerns are easy process migration and the ability to move a processor correctly. If we have checkpointing in place, this is easy. Without checkpointing, tricks have to be played.

SUMMARY

For embedded systems, run a single OS on the multicore system until it will not scale any longer. At that point, run multiple copies of the OS and use something like Apache Mesos to create services. The services model will also allow moving applications to different systems which do not share resources with the main multiprocessor.

This model is where supercomputing is going. Look at Unified Parallel C.

What you are describing is called multi-threading. CPU chips with multiple cores provide complex memory management hardware that makes sure that when a thread on one core changes memory that is shared with threads on other cores, that the cache lines for the other cores are set so that a thread on any core that did NOT change the location will have to grab the value from RAM and NOT from the out of date value in cache.

POSIX compliant operating systems include library routines to support multi- threading which ensure that only one thread at a time can access shared memory with write access.

There are also Real Time Executives that support multi-threading. I recommend L4 (see http://en.wikipedia.org/wiki/L4_microkernel_family).

I agree with Rivalino that the choice depends upon the workload. So one can combine both the techniques and take decision based on the load. 

Different processes can be assigned to different processors. Rather than distributing processes among processors, processors are distributed among processes. Still dynamic decision can be if some process is not getting CPU on one processor then it can be moved out to some other processor which is free. But for this transferring L1 cache data is an overhead which has to be taken into account as each processor will have their own cache. Some estimation algorithm can find out the time taken  to do all these with the time taken to wait in same processor queue. whichever is less scheduler can choose that one. Most of the modern operating systems which are optimized for multi-processor architectures do this.

However you can also exploit all the power of multi-core architecture by developing application that can use all the processors at the same time which is called as parallel applications.

Hope it helps :)