DEVELOPER DETAILS ADVANTAGES OF HINT BENCHMARK 05.10.96 by Alan Beck, managing editor HPCwire ============================================================================= Ames, Iowa -- John L. Gustafson, computational scientist at the DOE's Ames Laboratory and a developer of the HINT (Hierarchical INTegration) benchmark, will soon be calling more widespread HPC-community attention to the advantages of his metric for evaluating relative performances of increasingly compute-intensive machines. In an exclusive interview with HPCwire, Gustafson detailed HINT's approach to high-performance benchmarking, which is based upon optimization of an integration curve. Following are selected portions of this discussion: HPCwire: Benchmarking is a vital topic to the HPC community; few others are followed as closely... Gustafson: "And few others are filled with more myth and deception." HPCwire: Do you feel HINT is actually a superior measure to SPEC, NAS or LINPACK benchmarks? Gustafson: "There is no question it's a better predictor of actual performance. It has to be. It is a much more complete picture and much more focused on memory bandwidth. There isn't another benchmark in creation that gives you the whole range of performance of all memory regimes, from the registers to the cache to the memory to the disk. "What they (i.e. the other benchmarks) do is take one slice of the machine and say: 'This particular size problem really runs well.' And you have no idea whether doubling the size of the problem would cause the machine to come to its knees or not...HINT doesn't fix the problem size or execution time. We let it go from the microsecond to the ten-second range, and we watch the machine go through all its performance figures, from what's fast -- usually what's close to the central processor -- to more distant things. You can watch the machine slow down as it deals with larger and larger chunks of memory -- just like real applications do." HPCwire: Doesn't HINT work on a rather simple principle -- establishing boundaries for integration? Gustafson: "It does. We worked for three years to boil it down to the simplest benchmark you could create that still captures the complicated aspects of full-sized applications. And I think we did that, by basing it on integration. It captures things like Monte Carlo simulations and integral equation simulations in a way that makes sense, and for which there is no shortcut to the answer. There's no way to precalculate the answer and just spit it out, as there is for some benchmarks. "The key to benchmarking is to make it as simple as possible -- but no simpler. LINPACK can be expressed in about 13 lines of FORTRAN; it's nothing but a multiply and add and memory fetch. And that's way too simple to represent a complete architecture. It doesn't exercise anything in the machine. SPEC exercises all kinds of things in the machine, but you can't do the full range of parallel computers and small computers; it's a very specific size. And it keeps changing too. You never know what SPEC you're talking about, because it changes every few months." HPCwire: Who is using HINT now -- and for what? Gustafson: "It's been used in at least two national laboratories for procurement and has been extremely revealing of false claims by vendors. I won't name names. But in one case the secondary cache turned out not to exist. It turned out to be an auxiliary program cache that had to be loaded separately. In another case, the floating-point performance was revealed to die quickly on a certain range of memory. "We have companies that are using it to organize their entire cadre of workstations so that they know what all the performances are within their shop...That way they know which machines are appropriate for which tasks. We've also used it on the IWAY to assess heterogeneous performance: we can do any kind of parallelism you want. There aren't many benchmarks that can do that -- tell you the power of a heterogeneous collection of workstations." HPCwire: We know there is no perfect benchmark. What are HINT's limitations? Gustafson: "Any benchmark that selects a mix of operations is not going to be the same as any other application. By trying to run through ranges of memory, we hope the HINT performance is correct along some point of the curve. But let's take digital signal processing applications, where all you want to do is fast Fourier transforms -- something very specialized. I'm not going to claim that HINT can tell you how to rank machines. "But I haven't heard of a case yet where the linear ranking predicted by HINT is different from the application ranking by any great degree -- where it really mispredicted things. That happens with LINPACK all the time -- where LINPACK figures are higher, but actual performance on real applications is much lower. So I think we're doing better by emphasizing the memory aspect. "HINT very strongly emphasizes moving data around...It tends to de- emphasize floating-point more than some of the other scientific benchmarks. But if you do have an extremely floating-point intensive thing like matrix multiply -- that's all you want to do with your machine -- then you're probably better off using LINPACK." HPCwire: So why isn't HINT more prominent? Why are SPEC, LINPACK and NAS still reigning supreme? Gustafson: "It takes a while for benchmarks to become known. LINPACK has been around for a long time. SPEC has a company behind it, licensing and aggressively marketing it. But we're about to do the same thing with HINT. You may soon see HINT become the Underwriter's Laboratory of the computer benchmark world. It may be a nonprofit approach. "SPEC is a consortium of workstation vendors and others that have a vested interest in their own performance being bandied about. I don't think that's the right way to have an objective evaluation of computer performance. It's very important to keep it objective and noncommercial but accurately maintained." More information on HINT is available at the HINT Web site http://www.scl.ameslab.gov/Publications/HINT/ComputerPerformance.html. ************************************************************************** H P C w i r e S P O N S O R S Product specifications and company information in this section are available to both subscribers and non-subscribers. 936) Sony 905) Maximum Strategy 937) Digital Equipment 934) Convex Computer Corp. 930) HNSX Supercomputers 932) Portland Group 921) Cray Research Inc. 902) IBM Corp. 915) Genias Software 909) Fujitsu 904) Intel Corp. 935) Silicon Graphics **************************************************************************** Copyright 1996 HPCwire. For a free trial subscription to HPCwire, send e-mail to trial@hpcwire.tgc.com.