This ASQ TV episode covers the basics of benchmarking, reviews the recommended six phases of a benchmarking process, and explains one vital ingredient in benchmarking: metrics.

Itasca, IL — Improving methods of tracking musculoskeletal disorders, continuously monitoring and assessing physical risk factors, and sharing best practices can help workplace MSD prevention programs have real impact.

Stimulated by informal conversations at the XVII International Small Angle Scattering (SAS) conference (Traverse City, 2017), an international team of experts undertook a round-robin exercise to produce a large dataset from proteins under standard solution conditions. These data were used to generate consensus SAS profiles for xylose isomerase, urate oxidase, xylanase, lysozyme and ribonuclease A. Here, we apply a new protocol using maximum likelihood with a larger number of the contributed datasets to generate improved consensus profiles. We investigate the fits of these profiles to predicted profiles from atomic coordinates that incorporate different models to account for the contribution to the scattering of water molecules of hydration surrounding proteins in solution. Programs using an implicit, shell-type hydration layer generally optimize fits to experimental data with the aid of two parameters that adjust the volume of the bulk solvent excluded by the protein and the contrast of the hydration layer. For these models, we found the error-weighted residual differences between the model and the experiment generally reflected the subsidiary maxima and minima in the consensus profiles that are determined by the size of the protein plus the hydration layer. By comparison, all-atom solute and solvent molecular dynamics (MD) simulations are without the benefit of adjustable parameters and, nonetheless, they yielded at least equally good fits with residual differences that are less reflective of the structure in the consensus profile. Further, where MD simulations accounted for the precise solvent composition of the experiment, specifically the inclusion of ions, the modelled radius of gyration values were significantly closer to the experiment. The power of adjustable parameters to mask real differences between a model and the structure present in solution is demonstrated by the results for the conformationally dynamic ribonuclease A and calculations with pseudo-experimental data. This study shows that, while methods invoking an implicit hydration layer have the unequivocal advantage of speed, care is needed to understand the influence of the adjustable parameters. All-atom solute and solvent MD simulations are slower but are less susceptible to false positives, and can account for thermal fluctuations in atomic positions, and more accurately represent the water molecules of hydration that contribute to the scattering profile.

Conformational heterogeneity of biological macromolecules is a challenge in single-particle averaging (SPA). Current standard practice is to employ classification and filtering methods that may allow a discrete number of conformational states to be reconstructed. However, the conformation space accessible to these molecules is continuous and, therefore, explored incompletely by a small number of discrete classes. Recently developed heterogeneous reconstruction algorithms (HRAs) to analyse continuous heterogeneity rely on machine-learning methods that employ low-dimensional latent space representations. The non-linear nature of many of these methods poses a challenge to their validation and interpretation and to identifying functionally relevant conformational trajectories. These methods would benefit from in-depth benchmarking using high-quality synthetic data and concomitant ground truth information. We present a framework for the simulation and subsequent analysis with respect to the ground truth of cryo-EM micrographs containing particles whose conformational heterogeneity is sourced from molecular dynamics simulations. These synthetic data can be processed as if they were experimental data, allowing aspects of standard SPA workflows as well as heterogeneous reconstruction methods to be compared with known ground truth using available utilities. The simulation and analysis of several such datasets are demonstrated and an initial investigation into HRAs is presented.

At the tops of many mountains and along numerous roads across the USA are small brass disks called benchmarks.  These survey points are critical for mapping the landscape, determining boundaries, and documenting changes, and there are hundreds of them in Yellowstone National Park!

Location: Electronic Resource- 

Benchmarking is an important topic in quantum computing. There’s consensus it’s needed but opinions vary widely on how to go about it. Last week, IBM introduced a new tool — […]

The post IBM Develops New Quantum Benchmarking Tool — Benchpress appeared first on HPCwire.

Benchmarking the Accounting & Finance Function: 2014

Data isn’t meaningful unless you do something with it. Let’s say your average RevPAR for June is $150. Alone, that doesn’t mean very much. But what if you compare that number to the average RevPAR, globally and within your region? If you measure whether you’re improving year-on-year, month-on-month? If you look beyond RevPAR to accompanying key metrics?

National Bureau of Economic Research

High-growth firms (HGFs) – firms able to grow fast over a short period of time – contribute to most new jobs in advanced economies.

High-growth firms (HGFs) – firms able to grow fast over a short period of time – contribute to most new jobs in advanced economies.

High-growth firms (HGFs) – firms able to grow fast over a short period of time – contribute to most new jobs in advanced economies.

To fill a void of relevant benchmarking data, Anders and our accounting association, the Leading Edge Alliance (LEA), have launched the third annual National Manufacturing Outlook Survey, and we are requesting participation. Created specifically for privately-held manufacturers, this is the… Read More

The post Anders and LEA Launch National Benchmarking Survey for Manufacturers appeared first on Anders CPAs.

Real-world application scenarios like modern Internet services consist of diversity of AI and non-AI modules with very long and complex execution paths. Using component or micro AI benchmarks alone can lead to error-prone conclusions. This paper proposes a scenario-distilling AI benchmarking methodology. Instead of using real-world applications, we propose the permutations of essential AI and non-AI tasks as a scenario-distilling benchmark. We consider scenario-distilling benchmarks, component and micro benchmarks as three indispensable parts of a benchmark suite. Together with seventeen industry partners, we identify nine important real-world application scenarios. We design and implement a highly extensible, configurable, and flexible benchmark framework. On the basis of the framework, we propose the guideline for building scenario-distilling benchmarks, and present two Internet service AI ones. The preliminary evaluation shows the advantage of scenario-distilling AI benchmarking against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from the web site url{this http URL}.

I noticed some very interesting news last week, widely reported in the technical press, and you can find the source press release here. In a nutshell, the Embedded Microprocessor Benchmark Consortium (EEMBC) has formed a group to look at benchmarks for ultra low power microcontrollers. Initially chaired by Horst Diewald, chief architect of MSP430^TM microcontrollers at Texas Instruments, the group's line-up is an impressive "who's who" of the microcontroller space, including Analog Devices, ARM, Atmel, Cypress, Energy Micro, Freescale, Fujitsu, Microchip, Renesas, Silicon Labs, STMicro, and TI.

As the press release explains, unlike usual processor benchmark suites which focus on performance, the ULP benchmark will focus on measuring the energy consumed by microcontrollers running various computational workloads over an extended time period. The benchmarking methodology will allow the microcontrollers to enter into their idle or sleep modes during the majority of time when they are not executing code, thereby simulating a real-world environment where products must support battery life measured in months, years, and even decades.

Processor performance benchmarks seem to be as widely criticized as EPA fuel consumption figures for cars - and the criticism is somewhat related. There is a suspicion that manufacturers can tune the performance for better test results, rather than better real-world performance. On the face of it, the task to produce meaningful ultra low power benchmarks seems even more fraught with difficulties. For a start, there is a vast range of possible energy profiles - different ways that computing is spread over time - and a plethora of low power design techniques available to optimize the system for the set of profiles that particular embedded system is likely to experience. Furthermore, you could argue that, compared with performance in a computer system, energy consumption in an ultra low power embedded system has less to do with the controller itself and more to do with other parts of the system like the memories and mixed-signal real-world interfaces.

EEMBC cites that common methods to gauge energy efficiency are lacking in growth applications such as portable medical devices, security systems, building automation, smart metering, and also applications using energy harvesting devices. At Cadence, we are seeing huge growth in these areas which, along with intelligence being introduced into all kinds of previously "dumb" appliances, is becoming known as the "Internet of Things." Despite the difficulties, with which the parties involved are all deeply familiar, I applaud this initiative. While it may be difficult to get to apples-to-apples comparisons for energy consumption in these applications, most of the time today we don't even know where the grocery store is. If the EEMBC effort at least gets us to the produce department, we're going to be better off.

Pete Hardee 

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Dec 10, 2019

In this report, the authors conduct a benchmarking analysis for energy and CO2 emissions intensity of the steel industry among the largest steel-producing countries.

Benchmarking the Accounting & Finance Function: 2014

This paper uses data envelopment analysis (DEA) to assess the efficiency of welfare spending in a sample of OECD countries around 2012, focussing on health care, secondary education and general public services.

Online Resource

Barker Library - TE7.N275 no.902