Standard Performance Evaluation Corporation

"SPEC" redirects here. For other uses, see SPEC (disambiguation).
Standard Performance Evaluation Corporation
Formation 1988
Type Not-for-profit
Headquarters Gainesville, Virginia
Membership
Hardware & Software Vendors, Universities, Research Centers
Staff
4
Website http://www.spec.org

The Standard Performance Evaluation Corporation (SPEC) is an American non-profit organization that aims to "produce, establish, maintain and endorse a standardized set" of performance benchmarks for computers.[1]

SPEC was founded in 1988.[2][3] SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website. Results are sometimes informally referred to as "SPECmarks" or just "SPEC".

SPEC evolved into an umbrella organization encompassing four diverse groups; Graphics and Workstation Performance Group (GWPG), the High Performance Group (HPG), the Open Systems Group (OSG) and the newest, the Research Group (RG). More details are on their website; .

Membership

Membership in SPEC is open to any interested company or entity that is willing to commit to SPEC's standards. It allows:

The list of members is available on SPEC's membership page;.

Membership Levels

SPEC Benchmark Suites

The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise).

The SPEC CPU suites test CPU performance by measuring the run time of several programs such as the compiler GCC, the chemistry program gamess, and the weather program WRF. The various tasks are equally weighted; no attempt is made to weight them based on their perceived importance. An overall score is based on a geometric mean.

Portability

SPEC benchmarks are written in a portable programming language (usually C, C#, Java or Fortran), and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks. SPEC has rules that attempt to limit such optimizations.

Licensing

In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model might seem to be in violation of the GPL as the benchmarks include software such as GCC that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.

Benchmarks

Current Benchmarks

Power

Other SPEC benchmarks incorporating power measurement

◦SPEC ACCEL

◦SPEC OMP2012

◦SPECvirt_sc2013

Network File System

Graphics and Workstation Performance

•SPECviewperf® 12

•SPECviewperf® 11, performance of an OpenGL 3D graphics system, tested with various rendering tasks from real applications

•SPECwpc

•SPECapcSM for 3ds Max™ 2015

•SPECapcSM for Maya® 2012

•SPECapcSM for PTC Creo 2.0

•SPECapcSM for Siemens NX 8.5

•SPECapcSM for SolidWorks 2013

•Previous versions of SPECapc and SPECviewperf benchmarks

Virtualization. SPEC's updated benchmark addressing performance evaluation of datacenter servers used in virtualized server consolidation. SPECvirt_sc2013 measures the end-to-end performance of all system components including the hardware, virtualization platform, and the virtualized guest operating system and application software. The benchmark supports hardware virtualization, operating system virtualization, and hardware partitioning schemes.

High Performance Computing, OpenMP, MPI, OpenACC, OpenCL

•SPEC ACCEL, SPEC ACCEL tests performance with a suite of computationally intensive parallel applications running under the OpenCL and OpenACC APIs. The suite exercises the performance of the accelerator, host CPU, memory transfer between host and accelerator, support libraries and drivers, and compilers. •SPEC MPI2007, MPI2007 is SPEC's benchmark suite for evaluating MPI-parallel, floating point, compute intensive performance across a wide range of cluster and SMP hardware. The suite consists of the initial MPIM2007 suite and MPIL2007, which contains larger working sets and longer run times than MPIM2007. •SPEC OMP2012, The successor to the OMP2001, designed for measuring performance using applications based on the OpenMP 3.1 standard for shared-memory parallel processing. OMP2012 also includes an optional metric for measuring energy consumption. SIP •SPECsip_infrastructure2011. SPEC's benchmark designed to evaluate a system's ability to act as a SIP server supporting a particular SIP application. The application modeled is a VoIP deployment for an enterprise, telco, or service provider, where the SIP server performs proxying and registration.

SPEC Tools

•Server Efficiency Rating Tool (SERT). Intended to measure server energy efficiency, initially as part of the second generation of the US Environmental Protection Agency (EPA) ENERGY STAR for Computer Servers program. Designed to be simple to configure and use via a comprehensive graphical user interface, the SERT uses a set of synthetic worklets to test discrete system components such as memory and storage, providing detailed power consumption data at different load levels.

•Chauffeur Worklet Development Kit (WDK). Chauffeur was designed to simplify the development of workloads for measuring both performance and energy efficiency. Because Chauffeur contains functions that are common to most workloads, developers of new workloads can focus on the actual business logic of the application, and take advantage of Chauffeur's capabilities for configuration, run-time, data collection, validation, and reporting. Chauffeur was initially designed to meet the requirements of the SERT. However, SPEC recognized that the framework would also be useful for research and development purposes. The Chauffeur framework is now being made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities.

•PTDaemon. The SPEC PTDaemon software is used to control power analyzers in benchmarks which contain a power measurement component.

Committees for Future Benchmarks

Handheld

•Handheld, SPEC has formed a committee chartered for the development of, and support for, a compute intensive benchmark suite for handheld devices.

SOA

•SOA. SPEC has formed a new subcommittee to develop standard methods of measuring performance for typical middleware, database and hardware deployments of applications based on the Service Oriented Architecture (SOA).

Retired Benchmarks

Culture

SPEC attempts to create an environment where arguments are settled by appeal to notions of technical credibility, representativeness, or the "level playing field". SPEC representatives are typically engineers with expertise in the areas being benchmarked. Benchmarks include "run rules", which describe the conditions of measurement and documentation requirements. Results that are published on SPEC's website undergo a peer review by members' performance engineers.

References

  1. ↑ "SPEC Frequently Asked Questions". Retrieved 15 March 2010.
  2. ↑ "The SPEC Organization". Retrieved 15 March 2010.
  3. ↑ "SPEC Membership". Retrieved 15 March 2010.
This article is issued from Wikipedia - version of the 11/10/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.