Hello Everyone,
I am trying to run very large systems (membrane protein systems) for longer period of times (say=500 ns). Due to very strict wall time constraint the running MD systems may be stopped in between, hence, I need to restart the system again and again as I have to run a very large number systems using HPC facility. Is there any automated tool or shell script which can use the append command (something else) to do it when it stops..I need a shell script which can resolve me issue.
Looking forward for a positive response from your end.
Thanking you
Budheswar
Below is a part of a bash script. It's set stop after 12 hours of wall time using the -maxh flag. It resubmits the script if you haven't exceeded the maximum number of iterations. The -noappend flag creates a new trajectory, checkpoint, etc file. Later you'd have to concatenate all of these using gmx trjcat. The outfiles thing is because my cluster will resubmit a script even if the folder is empty, which is annoying.
topdir=your_start_directory
script=resubmit_example.sh
input=production.tpr
maxiter=####
outname=your_hearts_desire
count=`cat $topdir/count.txt | awk '{print $1}'`
currout=$topdir/part$count
prev=$(($count - 1))
prevout=$topdir/part$prev
next=$(($count + 1))
if [ $count == 1 ] ;
then
echo started$count
cp $topdir/$input $currout/part1.tpr
cd $currout
gmx mdrun -s $currout/part1.tpr -deffnm $currout/$outname$count -cpt 100 -cpo $currout/part$count.cpt -c $currout/$outname$count.pdb -maxh 12
else
echo restart$count
cd $currout
gmx mdrun -s $currout/part1.tpr -deffnm $currout/$outname$count -c $currout/$outname$count.pdb -cpi $prevout/part$prev -cpt 100 -cpo $currout/part$count.cpt -maxh 12 -noappend
fi
echo making files$count
outfiles=`du $topdir/part$count | awk '{print$1}'`
count=$(($count + 1))
if [ $count -gt $maxiter ] ; then
echo "Reached $maxiter iterations. Job Complete"
else [ $outfiles -gt 50000 ] ; then
mkdir -p $topdir/part$count
cd $topdir
echo $count > $topdir/count.txt
echo starting new$count
cp $topdir/$input $topdir/part$count/part1.tpr
sbatch $script
fi
tldr: make a count.txt file with "1" in it. Use a count in bash script. Set max iterations. Use -noappend flag. Have it resubmit the script if you havent exceeded the max iterations.
Dear JD Tamucci,
Thanks a lot for above bash script. I am confused, how can I run it using my existing shell script which I use while submitting the job to the batch HPC cluster. Further you have mentioned to create a file count.txt with 1 in it and use a count in bash script. Set max iterations ..and use of -noappend flag. Could you please fell free to elaborate in detail.
Thanks a lot for your time.
With Warm regards
Budheswar
I included the above script so you could take elements from it and incorporate it into yours.
The count.txt file just keeps track of what iteration your on. You could include that in the script as well with a touch count.txt command.
The number for "max iterations" is set by you. If you 500 ns, and you get 50 ns/day with a maximum wall time of 1 day then you would need 10 days to accomplish your job. You'd set the max iterations to 10 in these scenario.
The -noappend flag makes a new file instead of overwriting your existing files. I like to do this for file security anyway. It's a bit of a pain but if something goes wrong then I don't lose weeks of previous runs.
If you implement these types of changes into your script and include a line at the bottom of your script where you conditionally resubmit the same script and reset the count then you'll be able to continue a job by resubmitting the same script over and over again.
Dear JD Tamucci,
Thanks a lot for your detailed explanation. Anyways I will try this script once in one of my system, how it works. If it works fine in my system, then, I am going to use this in other systems too.
With Regards
Budheswar
- Badges
- Science topic