I want to perform parallel runs of the same program in LINUX command line. I heard that is possible (and really simple) to perform and I am wondering if one you can help me with that.
For run the program I need some initial files that are distributed in several pastes in one directory. I want to assess that directory, list all the folders, then use each folder path and run the program. I am running a package of PAML called codeml.
Possible pseudo-code I believe:
1. Determine the main directory
2. List all the folders
3. Run the the codeml package for all the folders (8 at time, preferentially )
Do you think is possible to do this?
Do I have to have great programming skills to perform it?
I take it from your explanation that you are looking for a means to start several copies of the same process simultaneously, with each given a folder name as an argument. This is easy to do with a shell command line. For instance, the following example starts several copies of the xterm process with the vi editor running inside, one for each subfolder in the current folder:
for folder in * ; do if [ -d "$folder" ] ; then sh -c "xterm -e vi $folder/x &" ; fi ; done
To explain: the outer 'for' loop iterates through all files/folders in the current folder. The 'if' statement selects from this set only those filenames that are in fact folders. The 'sh' command starts a new copy of the command shell, with the actual command passed to it as a parameter after the -c option. The actual command is written with an ampersand at the end, which allows the command to run in the background without holding up the calling process; so the 'sh' process terminates and the 'for' loop continues, starting the next instance.
The code above will run as many instances of process, as many directories you have.
You may write something like this:
i=0;ls | ( while read d; do if [ -d $d ]; then i=$((i+1)); echo i>.i; cd $d; (DO_YOUR_PROGRAM ; j=`cat .i`; j=$((j-1)); echo $j > .i) &; cd ..; i=`cat .i`; while [ $i -gt 8 ]; do sleep 1; i=`cat .i`; done; fi; done)
Logic - for each dir in current, increase counter, store it in file, run your command in background, decrease counter in file AFTER command terminates. Wait until counter from file goes below limit BEFORE moving to next directory.
Caveats - may be you will need some locking on .i if DO_YOUR_PROGAM terminates fast. Read "man flock" in this case.
But if you don't like scripting, you may use some que management system - torque, for example to do things like this in more general way. It can even email you when job is done :)
To add to what the others have said, be aware that there are many shells such as tcsh, bash, ksh, ... Use the right shell when implementing the commands suggested.
You can send the process to background and don't have to wait for the loop to end before the second, third.. eight starts. If you didn't start in the background (using &) sign, it needs to wait, so it'll be sequential program
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python