Properties of Benchmarking and Yardstick Analyses

You're in the section: Price Level Regulation -> Properties of Benchmarking and Yardstick Analyses

Benchmarking1 quantifies the relative historical performance of organizations or divisions, controlling for external conditions. Once the purposes and uses of benchmarking are know, the first step is to survey the information this is available. Such information might include system operations, network capacity, financial flows, and outputs. Then with purposes and possibilities in mind, the regulator can choose the metrics to use in benchmarking and how they will be used to provide incentives for improved performance.

In general there are five types of metrics that can be used for benchmarking. It is important to understand the strengths and limitations of different methodologies so that they are used appropriately. Poorly performed, benchmarking may hinder good performance rather than promote it.2

One type of metric – Core Overall Performance Indicators – include measures that are generally available and simple to understand, such as output per employee, number of complaints, system loss, coverage, and key financial indicators. These indicators help regulators identify trends, but it is difficult to account for the relationships among the different factors. Regulators can use another type of metric, Performance Scores based on Production or Cost Estimates, to identify the best and weakest performers in a group of service providers. This approach can use sophisticated quantitative techniques to determine relationships between the results being measured (such as cost per unit) and the factors beyond the operators’ control that can affect the results (for example, population density). Data availability can make these types of analyses complex, as can the difficulty of fully accounting for differences between operators than are beyond the operators’ control.

A third type of metric – the model or virtual company approach – is sometimes used to establish a baseline for measuring operator performance. This method creates an optimized economic and engineering model of a company. The methods are complex and can be controversial because it is difficult to ground in reality the numerous assumptions that must be made in constructing the models. Process Benchmarking is the fourth approach and focuses on individual production processes, such as pumping, transport, and treatment in the provision of water. This approach provides regulators with detailed information on specific stages of production, but they can be problematic in a regulatory incentive scheme because they focus on management issues of how services are provided rather than the outcome issues of the costs and quality that customers experience. Generally management decisions should be left to the operator since it is the operator that is take on the risks of good or poor management decisions.

Customer Survey Benchmarking is the last form of benchmarking and focuses on customer perceptions. Customer perceptions can be measured through surveys, focus groups, complaint monitoring, and the like. This approach has the advantage of directly gauging the customer’s experience, but customers’ views can be influenced by attitudes and experiences that are beyond the operators’ control.

Footnotes

  1. To ease the exposition, the term benchmarking will be used in this portion of the narrative.
  2. It is beyond the scope of this narrative to explore the strengths and weaknesses of each metric. See the references in this chapter for information on the properties of each metric.