DOE Special Programs
Data Archives Internal Projects Hosted Projects Completed Projects Hosted Meetings |
The Effective System Performance (ESP) BenchmarkIt is now generally recognized in the high performance computing community that peak performance does not adequately predict the usefulness of a system for a given set of applications. One of the first benchmarks designed to measure system performance in a real-world operational environment was NERSC's Effective System Performance (ESP) test. NERSC introduced ESP in 1999 with the hope that this test would be of use to system managers and would help to spur the community (both researchers and vendors) to improve system efficiency. The discussion below uses examples from the Cray T3E system that NERSC was operating in 1999.
Click here to download the ESP software. When you have run the ESP suite, please let us know your results. Improved MPP System Efficiency Equals Million-Dollar SavingsIncreasing the effectiveness of NERSC's Cray T3E from 80% to 90% would be equivalent to adding more than $2 million in additional hardware for the following reasons:
In fact, over 18 months from October 1997 to July 1999, NERSC increased T3E utilization from ~55% to ~90% -- a value of $10.25 million (Fig. 1). This is almost equivalent to the improvement in processor/price performance from Moore's Law.
Theoretical Peak Performance Does Not Equal Real-World ResultsPeak operations per second says nothing about how specific scientific codes will perform. The percentage of peak performance achieved on NERSC's Cray T3E varies widely (Table 1 and Fig. 2).
Effective System Performance (ESP): A New MetricESP is designed to evaluate systems for overall effectiveness, independent of processor performance. The ESP test suite simulates "a day in the life of an MPP" by measuring total system utilization. Results take into account both hardware (PE, memory, disk) and system software performance. Developed by NERSC as part of a major system procurement process, ESP is designed to predict the effectiveness of a system before purchase, as well as to evaluate system changes before implementation. The goals of the ESP benchmark include:
Overview of the ESP Testing ProcessThe ESP test employs of a suite of scientific applications. The standard NERSC ESP suite is a set of 82 individual jobs, two of which are full-configuration jobs, namely, calculations that use the entire system. Except for the two full-configuration jobs, these jobs are submitted to the system's job scheduling software in an order given by a particular pseudo-random number generator. At time zero, jobs are submitted until the total number of PEs requested by the jobs in the queue exceeds twice the number of PEs available in the system (Fig. 3). Several minutes after the start of the test, another batch of pseudo-randomly selected jobs is submitted, until the total number of PEs requested by the new jobs in the queue exceeds the number of PEs available. Again several minutes later, the remainder of the jobs in the suite (except for the full-configuration jobs) are submitted.
Twenty-four minutes into the test, the first of the two full-configuration jobs is submitted. This job is to be run immediately upon submission, before any other non-executing code in the queue is run. This may be achieved by one of the following methods:
Immediately after the full-configuration job is completed, the system is entirely shut down and then rebooted. After the system has been rebooted, the job suite is restarted. Three hours after the test has started, a second full-configuration job is submitted. All codes must run to completion, with verified correct results. The system effectiveness ratio, E, is then computed as where
While the ESP as described above is targeted to the NERSC T3E, the test can easily be modified to make it more appropriate for other systems. Click here to download the ESP software. We are extremely interested in knowing the results of running the ESP suite on different systems. When you have run the suite, please let us know your results. References
|
Page last modified: Fri, 09 May 2008 21:21:40 GMT Page URL: http://www.nersc.gov/projects/esp.php Web contact: webmaster@nersc.gov Computing questions: consult@nersc.gov Privacy and Security Notice |