Whether it be an automated genome annotation or an analysis which involves huge data, we need to know the computational resources needed for the task.
Can anyone please suggest how to estimate it. What are things we need to know in order to estimate it?
I think there are way to many variables involved to make any type of real estimate. Single thread or multiple? Parallel ? Single processor, multi-processor or Cluster? If cluster how busy is internode communication? etc etc etc. I think the answer is run it once then you'll know the next time!
when you going for huge data analysis especially in genome, its gud do it in Linux environment, linux provides complete usage of computational power as windows not much efficient in this terms. when u need high data to analyse, its better to run on GPU than CPU.
Run 1% of the process, discard the startup/loading time, multiply by 100.
(I'm serious)
I think there are way to many variables involved to make any type of real estimate. Single thread or multiple? Parallel ? Single processor, multi-processor or Cluster? If cluster how busy is internode communication? etc etc etc. I think the answer is run it once then you'll know the next time!
"I think the answer is run it once then you'll know the next time! " -- I think this is more or less true... One can always estimate, but you may be surprised how wildly wrong the estimate turns out to be (especially with things like PSI-BLAST, etc). One person above recommended doing 1% of the job and projecting from there. You could try that, but it probably will be too far off. I would say more like 10% to 50% of the job and scale from there...
The problem with the 1% etc is that you have to assume a constant load during the processing. If the program starts by doing massive I/O to populate massive memory arrays then this is not going to reflect processing at all. for example I've just been running some MRI analysis software that starts by loading the dicoms and manipulating the data into it's required structures. This causes a massive peak in memory usage that then subsides followed by a peak in processing as the actual analysis fires up giving a totally false impression of what the actual analysis entails, By trying to start the processes in a simple loop there is a massive surge im memory usage that means the system requires to use. The system then requires to use a lot of swap and the processing grinds to a relativehalt as the processor has to deal with all the swap I/O. However by knowing of the initial surge a delay can be built into the launch and the surge never requires swap memory and runs purely in ram, allowing much faster execution of the whole job.
I thnk the answer is suck it and see and keep an eye on whats happening so you can optimise what you are trying to do.
The real problem is how much data must go through: if you rent cloud resources this can be both a big cost and a big loss of time. It is also important to know when you need the results. I once saw one of the nicest computer graphics film at Siggraph probably 30 years ago, it was done on one of those old Apple II computers, it took about a year of computation, but the result was fantastic.
It also depends on how regular your processing time is for the input "units". For example, I perform PSI-BLAST searches, and the search time for protein sequence A may be 100x that of sequence B, even if both sequences A and B are roughly the same size. Using a small 1% fraction of the input dataset may not be a good representative sample of the real runtime for reasons unrelated to data size. It seems funny, but the above are correct; run it once and then you will know.
It's the same thing as with graphics, one scene is in the desert and no matter how much detail it is mostly ray tracing, other scenes are in the forest and it is radiosity and it takes 100x more.
What really fascinates me thow is what happens after: it's all about how you present the data...
I am little late to this discussion, and I am with David/Aaron/Milko: the question as posed is intractable.
It is equivalent to asking what is the fuel consumption of a car, without specifying the engine, the fuel, the driver, the terrain, the weather, etc.
Otherwise stated: you are asking what the dynamic of a program is without any specification of algorithm, data set, or computer architecture.
Having said that: the question: how much disk/memory/cpu/bandwidth/time will this program take to run is a very valuable question, and is the basis of well defined procurement policies in any supercomputer purchase. In a nutshell, it is a characterization problem where you run your program on a collection of inputs and collect the resource utilization. A previous poster indicated that Linux is preferred over Windows, but I would respectfully disagree: Windows has a sophisticated resource measurement system and is equally capable to answer this question as a Linux environment. The characterization is typically guided by the typical workload configurations of the research population that utilizes the machine. In many parallel machines, there tends to be a point of diminishing returns where increasing the size of the machine, or increasing the parallelism of the algorithm, does not yield a proportional return. Your task is to characterize that curve for the classes of programs that you need to run.
NERSC does this type of workload characterization for current machines and future procurements, and has a wealth of information on this topic. You can get started with these links:
http://www.nersc.gov/systems/systems-overview-table/
http://www.nersc.gov/users/computational-systems/
http://www.nersc.gov/users/computational-systems/hopper/performance-and-optimization/
- Badges
- Science topic
- Similar topics
- Bioinformatics
- Bioinformatics and Computational Biology