There are several scenarios where throttling is going to happen.

The mechanisms of throttling are CPU/GPU/accelerator implementation specific. Although common sense can be used, good chunk of that information is confidential in terms of actual implementation.

Bottom line, you have some power constraints and the more you know about the application the better you will leverage the power headroom. Additionally, there is thermal inertia that you want to factor in the clock speeds. 

Finally, and eventually, compiler/profiler technology will optimize execution path to get the highest performance under a power envelop/budget. That means collecting power traces on top of performance traces and making trade off decisions of performance vs power.  

This is already impacting the developers of highly optimized compute intensive library kernels and I/O (network,storage) drivers where the software tuning needs to factor that the hardware influences their execution based on power credits. That means, if a piece of sw runs out of power credits, it will throttle down that subsystem as needed, which impacts/cascades into other subsystems as well.

You should be able to find quite a bit of information out there on throttling effects while running High Performance Linpack.

K.Goto, developer of highly optimized BLAS libraries has given many headaches to both AMD and Intel due to burning more than expected power and therefore making the CPUs throttle.

My presentation on "Understanding Bulldozer architecture through HPL benchmark," talks a bit about these issues at the end.

http://www.hpcadvisorycouncil.com/events/2012/European-Workshop/Presentations/12_AMD.pdf

Regards,

Joshua Mora.

HPCGuru.

This is probably hard to give a general answer, but I will try. It mainly depends on how well the applications can utilize the processing hardware.

You are looking for applications having a high arithmetic complexity (i.e., applications burning a large number of CPU/GPU cycles per memory access) while being well suited for parallelization (i.e., there are only few dependencies between threads -ideally zero- and a moderate/very large number of threads that can be executed in parallel in the CPU/GPU case).

If this is just for prototyping or a showcase, I would go for convolution operations, as they allow you to nicely adapt the arithmetic complexity while the memory accesses can be prevented automatically by caching. If you do not need scaling, go for the LINPACK benchmark. A nice metric to measure results might be the number of FLOPs per Watt as defined by the Green500 list.

Another metric utilized in many papers that helps in the calculation of aggregate statistics is the JPP metric (jules per pixel) or the JPS (jules per sample) and JPF (jules per frame).

well, the problem is that HPC machines are normally designed to not throttle - if they did throttle, the only interpretation is that the heatsink is inadequate, or the airflow, or the incoming air temperature.  further, current HPC chips (say, e5-2690 v3) have a kind of built-in throttling, in that you can only achieve max clock by using a subset of cores.  to realy take advantage of improved cooling, you'd have to somehow get Intel to let you exceed this programmed TDP-based capping.  and as an HPC person, I'm not sure I'd go for a product like that, since although you might support higher clocks than normal, any chip's power-performance efficiency drops off as you really push it.  (a pitch based on reducing cooling costs would be more successful, I think.  and/or improved cluster density...)

I also note that the range for Intel's "turbo" modulation is about 10%.  that's not a knock-your-socks-off kind of competitive advantage...

Thanks to all for your insights. I think I now see a path for proving the value, and likely that path may be equal to the challenge of making the current invention practical. Having said that, the project remains fun. As to improved cooling costs, I don't plan to worry about that as fundamentally the purpose of HPC is to process data, the quicker the better so lower cooling costs are a second order issue. The major component of the project is being simulated using COMSOL at UW and a few months from proving the project is possible and it will be a few years before all of the life, cost and performance goals are met.

I know almost nothing about your product, but:

• HPC is not about "quicker the better".  no HPC facility is ever oblivious to operating costs - and many consider density to be very important.  which is why people have been doing some installs water-to-the-socket for years.  there has been no strong move in this direction simply because operating costs are still a minor component of overall costs.
• IMO, the real breakthrough would be an cross-vendor standard that included heatpipes within the chassis and cold plates on the sides of the rack.  being able to put different vendors in one rack is important, as is the ability to mess with nodes without plumbing concerns...

Hi Mark,

The business model is to license the inventions to any and all vendors and I agree it requires multiple vendors for any chance of wide acceptance. The major technical challenge is to prove (to the industry) that the life time, for practical purposes, is the same as a block of copper or heat pipe and keep the cost very close to passive heat sinks. Practical acceptable life times means there will be no failures over the expected life of the computer system.  Matching the industry keep out standards /CPU socket with 6X improvement in degrees C/W is easy, but matching the small size of any alternative and with 6X performance without piping is not easy. So there are two options (if we solve the technical issues)- build a small volume cold plate and piping to common radiator or develop standalone heat sinks that directly replace passive alternatives, but that solution would not fit all small form factor systems.

The research concerns a new type of micro pump with very long life and ~$10 cost to build and new ways to create cold plates at a lower cost. Essentially there have been many attempts to replace passive heat sinks with higher performance but none have met the industries requirements for cost, size and RAS (reliability and service) for wide spread acceptance.

And even if technically successful, proving the value of better thermal performance (the discussion above) will be a challenge. But, if does become widely accepted, the potential effect on the industry could be significant.

My background is a retired Intel EE, which is why I sought out the SW advice.

To Robert.

I didn't know until now your end goal. If I understand well, you just want a cpuburn application to stress the system while you design an efficient cooling solution ?

Current cooling solutions with heatsinks have a thermal sensor that measure intake temperature and if getting too close to tcase/tcontrol/tjunction then it feeds back into fan rpms to increase from 3000 to 6000.

Peltier solutions is something that was in use for overclocking.

I have seen some papers on micro air pumps heating directly on the heat sink to improve thermal efficiency.

I used to study heat exchanges as part of my MsC and PhD. I think you need to get into codesign and embed the cooling solution in the chip. The latest thing is micro heat exchanges. In my days of PhD, when I studied convective coefficients, you could go with laminar or turbulent flow. It is studied that when you break the turbulent boundary layer by means of little bumps along the flow,  it needs to develop again the thermal layer, increasing the convective coefficient, hence the heat transfer. micro heat exhangers have different reology on the fluid so it is not well understood, that is, it requires further experimental data to come up with convective coefficients that you can put into your CFD model. Make sure your CFD code handles well the range of 10^-6 since most code working in double precision will run into rounding issues for micro scale unless you do non dimensional analysis.

By having a 3D chip design with micro heat exchanger embedded, in the same way as plate based heat exchanger, you can improve drastically the cooling.

On the power side, there is research on both near threshold voltage and stacked voltage to use less power.

Thanks- your comments on CFD precision, I added that as a check point to the "micro channel" cold plate. Given your background on heat exchangers I really appreciate the expert insights.

On cooling semi conductors directly, from the work of Tuckerman and Pease 1982 and forward folks have been trying to cool the semiconductor directly with fluids. IBM has some success with solar cells but I don't think any mainstream processor has used this due to a low tolerance to hot spots/clogging and the increased complexity in processing the Si. The Professor I am working with at UW developed an outstanding solution using phase change and micro jets (>>10 X performance in terms of degrees C/W) but that solution faltered due to reliability issues . All of these concepts are possible but well beyond my near zero budget.

I've read many papers on micro channel cooling both for cold plates and directly on Si and reviewed the 100's of attempts to produce an improvement in heat sink performance with both liquid cooling and heat pipes. Why have none of the liquid cooled solutions reached a critical mass for wide spread acceptance? The performance is better in a smaller space than heat pipes. My opinion is none have the reliability, price, keep out and ease of installation to become widely accepted.

Sandia's air cooled solution (extremely fast air currents) looks promising from a performance point of view. What is lacking? No published reliability specifications and the cost and size looks like a challenge. Perhaps there is a path to commercial success for this technology and I think some folks are working on it. 

My goal is modest. Produce the same performance achieved by the smallest liquid cooled self contained heat sink but increase the reliability by 100X and reduce the cost by about 70%. Achieve 0.1 degrees C/W or better in a form factor compatible with 80% of the CPU/GPU sockets/computers.

I have an invention for the micro pump that looks like it will meet the requirements. It is a completely different ME design and, to meet the size requirements, new brushless DC motor design. Once we complete the pump research and prove the reliability, it can be licensed for use in the existing cooling market and other applications (even the on Si solutions you suggested above but are out of my league) 

After that, the next costly item is the cold plate. Can this be designed to meet or beat any alternative performance but cost around $2.0? I have some thoughts on this (perhaps mostly wishful thinking) using a different mix of materials and emerging manufacturing processing.  Your points above on micro-channel fluid models not accurately predicting behavior and therefore pointing the way to optimal channel size/shape matches what I've read on the many attempts to optimize a design.  My plan is use the published micro channel (and pins etc) results to guide the options explored. CFM will of course be used but likely this part of the project will be more driven by what is possible in terms of manufacturing process, testing and iterative improvements. Likely is this highest risk area in terms of meeting cost goals.

If you want to see the results of the UW work and perhaps critique the project as it progresses, let me know.

It depends heavily on granularity as well as the overlapping between TLP and DLP of an algorithm. As such, it is necessary to do some experimentation for each case. Furthermore, it depends on the platform.