KPIs for Transition/Transformation States – What are the Key Performance Indicators (KPIs) for evaluating sector performance and how can they be used effectively by regulators? (For nations that are in the Transition and Transformation stages)
(Response by Sanford Berg, with input from Jemima T. Sy and Anna Aghababyan)
Selected issues faced by Fragile and Conflict-Affected States (FCSs) in relation to developing and improving infrastructure regulation are discussed in this set of Frequently Asked Questions (FAQs). This particular FAQ builds upon points made in the related FAQ, and applies those principles to nations (and sectors) that have greater stability. Context shapes the market, institutions, and the actions that are possible to take. The FAQs present general principles, guidance, and cases; where possible, guidance is distinguished for countries along a fragility spectrum using the following taxonomy: Crisis, Rebuild and Reform, Transition, and Transformation. Here the focus is on nations in the Transition and Transformation stages, though there are lessons for less stable nations here, since foundations should be laid for later developments.
This FAQ should be read in conjunction with the related FAQ on nations in Crisis and in the Rebuild and Reform Stage. The related FAQ establishes concepts useful for identifying the most important KPIs and for creating information systems that enable regulators to monitor, evaluate and incentivize infrastructure operators.
Nations in the Transition or Transformation stages have many institutional advantages: the state is in a position to perform more sophisticated tasks, with independent regulatory institutions generally implementing the broad public policies articulated in the law. In addition, social cohesion is greater, leading to a citizenry that has more trust in those organizations delivering services and those agencies providing oversight. Instruments of government facilitating inclusion and dialogue exist throughout the regions of the country, promoting some consensus regarding infrastructure as a priority.
When government is in a position to fund some investments, the Ministry of Finance and agencies providing direct oversight of local operators should become repositories of information. As service delivery by the government begins to expand beyond the nation’s capital, determining the effectiveness of local initiatives becomes a key task for the regulator (or agency monitoring resource allocation and utilization).
In FCSs, decision makers tend to lack basic data on sector characteristics and performance. But to attract private investments, information is very important to inform investors what the returns commensurate with the associated risks might be. If the regulatory objectives are unclear and changing, and the incentive systems and price structures are unrealistic or unreasonably low, investors will be unwilling to provide financial capital and entrepreneurs will turn their attention to other sectors.
The Body of Knowledge on Infrastructure Regulation shows the links among the three major financial records: income statements, balance sheets and statements of cash flows. Managers need to understand how these different reports capture the cumulative and annual inflows into and outflows from the firm. Similarly, regulators need the professional support that enables them to characterize the financial sustainability of current operations: http://regulationbodyofknowledge.org/financial-analysis/
- Income statements (particularly revenues, by source, operating expenditures and interest payments);
- Balance sheets (a snapshot at a particular point in time of assets, liabilities and net worth); and
- Statements of cash flows (money received and money outflows for a period of time).
Additionally, managers and regulators should have access to the following:
- Asset registries (installation date, capacity, location and maintenance records);
- Business plans (with targets and descriptions of managerial incentives for meeting those targets); and
- Operating statistics (number of workers, number of customers, reliability, service quality, etc.).
These basic organizational records provide the “raw material” for constructing KPIs. As was emphasized in the FAQ on information, such data will be captured in advanced information systems maintained by well-performing organizations. Such information also helps investors who are considering public-private partnerships to better understand conditions on and under the ground, reducing risks associated with “negative surprises.”
Steps for improving data collection for larger operators
Those monitoring larger operators have a different set of challenges. However, the steps are similar to those identified here. See the Related FAQ on dealing with data issues at the Transition and Transformation stages. Those monitoring larger operators should do the following:
- Create a benchmarking team: Professionals need a variety of technical and engineering skills so that the team has a comprehensive understanding of the water utility’s processes and operations.
- Establish trust:The group performing regulatory functions needs to avoid the creation of adversarial relationships. Both the operator and the regulator have the shared objective of improving sector performance, so a collaborative process involving key stakeholders is essential for a successful benchmarking initiative.
- Establish clear study objectives: Identifying reasonable targets and designing incentives to achieve them will frame the issues and identify data needs
- Select an appropriate basket of KPIs: Analysts must be aware of the strengths and limitations of the different indicators and of the methodologies for analyzing data.
- Build on existing data collection frameworks where possible: Unless current management information systems are thoroughly flawed, data collection should draw upon existing definitions and procedures, with the goal of improving procedures.
- Gather raw data for yardstick comparisons: This stage involves the selection of the appropriate comparison group of companies or standards.
- Apply data verification procedures: After data collection, techniques for verifying and comparing data need to be applied, so that bonuses and other incentives are not based on unverified data.
- Compare data sets: Sector statistics need to be compared with national statistics for validation purposes (for example, the related sector ministry may already have a data collection process that focuses on regional or particular technological issues).
- Perform data analyses: This crucial stage involves the application of methodologies appropriate for answering key questions. Such analyses would go beyond the “trend analysis” conducted in more fragile states. Statistical techniques can be applied to identify high performers and low performers. If there is only one utility, data should be obtained from the different geographic divisions. Such information is utilized by high level managers to evaluate regional managers. Regulators can use such data to determine relative performance of the various divisions—which is useful for setting realistic local (and overall) targets.
- Develop policy implications: Explore potential determinants of inefficiencies so foundations for future initiatives can be laid.
- Engage in information dissemination: Reporting the results of performance comparisons helps engage stakeholders in the process. For simplicity these results can be presented in simple stop-light figures with green, yellow and red representing different levels of performance and arrows indicating the extent of improvement (or decline) in a KPI.
- Establish rewards and penalties: There need to be clear procedures for planning and implementing incentives and corrective actions, where the incentives are commensurate with the benefits from achieving targets.
- Seek public comments and promote stakeholder awareness: Such a campaign, with media assistance, promotes improvements in utility operations. This is sometimes called “naming and shaming,” but is more appropriately identified as public reports on infrastructure performance.
- Define institutional responsibilities and assign individual tasks: Accountability requires that all parties understand their rights and responsibilities, including deadlines for completing specific tasks.
- Prepare follow-up studies as part of the performance review: Benchmarking via KPIs is an ongoing process for monitoring and evaluating performance outcomes. Reports need to be widely disseminated so that stakeholders and the press are kept apprised of developments.
Cases: Use of KPIs by Service Providers and Regulators
Transformation Stage: KPIs for Asia and the Pacific: The Asian Development Bank (ADB) reports KPIs for electricity, transport and telecommunications in http://www.adb.org/sites/default/files/publication/30358/ki2013.pdf (pp. 286–98). These illustrate how a limited number of KPIs can be useful for analyzing patterns across countries and regions over time.
Transition Stage: Case of Cambodia Water: KPIs for rural water are identified in an ADB report that includes key lessons from recent initiatives and areas for improved coordination and capacity building in rural water in Cambodia: http://adb.org/sites/default/files/projdocs/2009/38560-CAM-RRP.pdf. Accounting, auditing and reporting procedures are given some attention. In the case of the Phnom Penh Water Supply Authority (PPWSA), there was a major turnaround from 1993 to 2006. For example, nonrevenue water was reduced from 72 percent in 1996 to six percent in 2010, and staff per 1,000 connections was reduced from 22 to three during that period. The guiding coalition was spearheaded by the company general director, Ek Sonn Chan. Incentives include high staff salaries and bonuses for high performance. (http://www.ppwsa.com.kh/, http://www.ppwsa.com.kh/en/index.php?page=sharingthereformprocess)
Transformation Stage: Case of Electricity in Ethiopia: After a three-year study, in late 2013, the government of Ethiopia divided one state-owned giant electricity company into two. Power Grid Corporation India (PGCI) assumed the management of the Ethiopian Electric Service (EES) on a two-and-a-half-year contract that requires PGCI to carry out power feasibility studies to determine the highest-needed voltage-power capacities of transmission lines. Currently, Ethiopia is the biggest power exporter in the Horn of Africa and is aiming to be one of the continent’s biggest exporters upon completion of the construction of several ongoing hydropower projects. (http://addisstandard.com/ethiopia-splits-state-owned-giant-public-utility-into-two/)
Transformation Stage: Case of Rwanda: The Energy Sector Strategic Plan for Rwanda includes a large section on monitoring and evaluation, whereby an energy-sector working group and a number of other stakeholders provide opportunities for collaboration and cooperation. Systematic evaluation of performance is at the core of their current initiatives, with regular comparisons of actual outcomes to expected outcomes (targets). (http://www.minecofin.gov.rw/fileadmin/templates/documents/sector_strategic_plan/Energy_SSP_June_2013.pdf)
Starting a Benchmarking Initiative
The following figure, from the author’s Water Utility Benchmarking: Measurement, Methodologies and Performance Incentives, indicates the steps for starting a data-collection process for evaluating infrastructure performance—especially as applied to larger operators. This FAQ focuses only on KPIs and basic performance benchmarking, but more technical quantitative methods are applied in many regulatory jurisdictions.
Screening and Analyzing Data
The next step involves screening data (checking for missing or inconsistent data) and then analyzing the information. To be credible, the resulting reports should not claim to have greater precision than can be documented. The following cases illustrate activity around the world in this area.
Inappropriate Design of Incentives
Inappropriate design of incentives that focus on specific KPIs can harm overall performance (in terms of efficiency and service quality) to the extent that excessive resources are devoted to achieving a specific indicator. This is particularly relevant for countries in the Transformation stage. For example, focusing on reducing the number of staff per thousand customers could lead to a number of inefficient outcomes:
- Excessive substitution to more expensive inputs (such as capital or energy). For example, if plant availability is highly rewarded, managers will tend to maintain an excessive inventory of replacement parts, possibly raising costs. In PPPs or other operational structures, such behavior results in reduced value for money.
- Reduction in some activities (such as maintenance) that improve a particular indicator (for example, reducing OPEX per unit output) today but greatly weaken performance in the future. Such behavior underscores the importance of having a balanced set of indicators rather than a single KPI.
- Inattention to more important indicators. Other KPIs might be more difficult to measure but have a greater impact on production efficiency or are more valued, such as customer satisfaction with service quality.
Cases: How Data Collection Helps Decision-makers Set Targets and Design Incentives
Case of Zambia: The publication of data on individual utility performance relative to KPIs of other utilities has served as a major element in Zambia’s regulatory system. Zambia’s water regulator, National Water Supply and Sanitation Council (NWASCO), makes its annual sector reports available on the web on a timely basis. NWASCO collects a wide range of KPIs and has developed weights to be used for an overall performance score, with particular attention given to data quality. Clear diagrams capture relative performance: For example, in addition to presenting the numbers, stop lights and arrows are used to indicate levels and trends over time of individual KPIs. See http://www.nwasco.org.zm/jdownloads/Publications/Urban%20and%20Peri-Urban%20WSS%20Sector%20Reports/sector_report_2014.pdf (p. 4). CEO awards are presented by NWASCO, giving visibility to leaders who have achieved strong results. Additionally, the sector regulator will be collaborating with a newly established institution, the Water Resource Management Authority (WARMA), to develop a long-term approach to sustainability in the nation’s water resources, since future price and availability of water to residences, industry and commercial enterprises depends on understanding hydrological patterns and future demands.
Case of Uganda: This case illustrates the value of geographic (or divisional) data collection. In the case of Uganda, the institutional framework for rural water supply draws upon data from 112 districts. The framework facilitates harmonization of national objectives with donor support. Institutional responsibilities and 10 “golden indicators” are outlined in http://www.slideshare.net/ircuser/2-ssozi-uganda. The rural indicators provide information on access, functionality, value for money, access/use (sanitation), quality, quantity, equity, access/use (hygiene), management, gender, and water resources management compliance. For urban areas, the National Water and Sewerage Company (NWSC) utilized KPIs for setting targets and rewards for the managers of local water systems. NWSC headquarters served a coordinating, capacity-building and incentivizing role to meet targets set in a series of performance contracts that were established and monitored by a set of national ministries. Mugisha and Berg (2008) outline the steps taken by NWSC to improve performance in the absence of a formal sector regulator. Also see https://www.nwsc.co.ug/index.php/about-us/ourprofile.
Case of Kosovo: The Water and Wastewater Regulatory Office of Kosovo (WWRO, http://www.wwro-ks.org/English/index.html) has fewer than 20 employees. Established in 2004, it has worked with the operators’ association and other stakeholders to utilize KPIs to set tariffs and enforce compliance with service standards. Examination of its annual reports (http://www.wwro-ks.org/English/Publications/AnnualPerfor/RAPORTI_I_PERFORMANCES_2014_ENG.pdf) shows the emphasis on KPIs in making comparisons across the seven regional water companies. WWRO is able to evaluate performance and set targets for each utility based on data on water quality (percentage passing bacteriological tests and physical-chemical tests), water pressure, continuity of supply (24/7), pipe bursts, nonrevenue water, service coverage, percentage with meters, complaints, volume and value of water sales, unit costs (operations plus capital maintenance), investments, and other indicators. The DANUBIS water platform facilitates networking and sharing of data in the Danube region (http://www.danubis.org/).
Role of the Guiding Coalition
Decision makers taking a leadership role in developing and implementing new institutional arrangements and associated incentives for improving performance have been called a guiding coalition. This group becomes the catalyst (or focal point) for reform, represents key stakeholders, promotes citizen awareness of current and potential performance, addresses the entire production change (including environmental effects), limits the negative impacts of information silos and data limitations, and allocates leadership responsibilities so that the reform initiative does not get stalled.
Create a focal point for institutional reform: When both operations and oversight are part of the same organization (whether a ministry or municipality), pressure for strong performance is unlikely because reforms represent a public admission that past procedures were inadequate (at best) or corrupt (at worst). The points developed in this FAQ support the creation of a separate body for assuming leadership for the development of KPIs for operators. A leadership team can address the concerns of special interests while maintaining some distance from groups that are negatively affected by transparency, changes in organizational cultures, and improved sector performance. No matter how inefficient current sector performance is, someone (or a group) is benefiting from them; change is never painless. This requires strong political will, a strong security system, and institutions, and may, therefore, be more appropriate for countries in the Transformation stage. We use the term “guiding coalition” to refer to the team that is taking the initiative to strengthen the data collection and analysis process, identify targets for KPIs, and design incentives so managers and staff focus on the activities that contribute to improving performance. Of course, all of this is contingent upon there being the political will to engage in such an initiative. This implies relatively high-level support from some powerful political figures. The guiding coalition and the benchmarking team should be drawn from within the utilities, appropriate government agencies, community groups, women’s organizations, and other local stakeholders. Such representation will enhance the acceptability of their work.
Represent a variety of viewpoints while having a shared vision: The guiding coalition could be very small for a local project or larger for improving the ongoing operations of a utility in the capital city or a set of larger towns. It should be inclusive so that key stakeholders are represented and all voices are heard. Note that the regulatory system goes beyond the regulatory agency and the service provider. It is important to include stakeholders that are in a position to support, block or blunt reforms that would improve performance. In particular, domestic politics can limit the effectiveness of regulatory institutions. The guiding coalition needs to have a shared vision, but the members may have different preferred strategies for meeting objectives.
Promote citizen awareness: One way to educate the citizenry about actual performance levels (and to limit the impact of political rhetoric) is through greater transparency. Transparency also requires political will, which is difficult to achieve in FCSs. Transparency is enhanced by having better information in the form of annual reports from operators (including business plans that identify the financial sustainability of operators, given the policy objectives developed by politicians and implemented by regulators). Additionally, yardstick comparisons (through benchmarking) help identify strong and weak performance by operators, or by geographic divisions of a national utility. Furthermore, accountability to boards of directors and to those setting infrastructure policies and designing incentives reduces the likelihood that lax management or poor regulation will be allowed to continue. Finally, participation by affected parties (special interests) can be achieved through public hearings, public consultation processes, workshops and consumer advisory boards. <OECD, 2015a>
Cover the whole service delivery chain and related activities: Broad citizen awareness can help the guiding coalition gain leverage against those benefiting from current dysfunctional arrangements. For example, for the water sector, operators of water tankers serving areas that are not part of a water network are likely to oppose network expansion in the beginning. The current delivery system (pipes and tankers) needs to be examined, bringing on board those who are threatened by new and improved arrangements. It may be that tankers continue to play a major role in the provision of water for some period of time. Starting to collect data on water quality from the network and tankers provides fundamental information that would otherwise be unavailable. That information then becomes part of the process for developing realistic business plans for meeting the basic needs of the community. Similarly, comprehensive approaches are required for energy, transport and telecommunications.
Eliminate information silos: Information is power—organizations are riddled with silos and little potentates who view themselves as gatekeepers. Their self-importance depends on their ability to delay decisions and manipulate (or hide) data to their advantage. Such behavior is destructive to organizational success. It constrains sound analyses and elevates personalities and process over performance (actual outcomes). Of course, legacy systems and information silos are not automatically dismantled when new approaches to decision-making become possible, but leadership and teamwork can support organizational change.
Allocate leadership responsibilities: Within the guiding coalition, there needs to be agreement regarding which members will serve as external spokespersons for the group. Such leadership should not be authoritarian but display humility, agility, authenticity and compassion. Furthermore, one person cannot do everything. Each individual that is part of the guiding coalition generally represents some organization affected by changes in current institutional arrangements. Ultimately, however, individuals move the situation forward. Policies are not self-implementing. Those exercising leadership both “stir and steer”—they make sure that issues get raised and addressed, and they guide the group once the appropriate strategies have been identified. If there is a window of opportunity (the political, economic and social conditions are ready for change), leadership can help organizations develop initiatives (programs) that improve infrastructure performance.
Related FAQs on Targets and Incentives
- Challenges of incentive regulation—What are the key challenges that need to be addressed when introducing incentives?
- Promoting efficiency—To what extent do incentives actually lead to improved efficiency?
- Efficiency measures—How do you measure the efficiency of service provision?
- Efficiency targets—What are reasonable efficiency targets?
- Rewards for cost containment—What incentives should a regulator introduce in order to promote cost containment?
- Performance incentives—How can regulators encourage improvements in specific dimensions of performance?
- Penalties for noncompliance—What penalties are most effective when the operator is in noncompliance with regulatory rules (e.g., for providing data, setting prices, or meeting targets)?
- Incentives for improved performance—How can a regulator develop incentive to discourage energy/water losses?
- Promoting investments—How can regulatory incentives be introduced to promote investment?
- Coverage—How can regulators establish incentives for service expansion?
Appendix on Water KPIs
The World Bank Group has identified 10 dimensions of performance for the provision of water in “Possibilities and Limits of Urban Water Utilities in Africa Serving the Poor” (World Bank, forthcoming):
|1. Accessibility||A large share of the population in the utility’s service area obtains water that is safe, sufficient, reliable and convenient. [Indicators: share of population (total and poor) with access to various levels of service]|
|2. Safety||The water supplied is safe to drink (potable), with appropriate sampling, testing, verification and reporting systems in place.|
|3. Sufficiency||People get enough water to at least meet basic health requirements. [Benchmark: at least 50 lpcd in urban areas (WHO)—50 liters per capita per day]|
|4. Reliability||Water is continuously available, with few supply interruptions of limited duration. [Benchmark: 24 hours per day average supply]|
|5. Convenience||It does not take long to get water, and water does not have to be carried far. [Benchmark: on-site delivery. For off-site service, water round-trip is less than 15 minutes.]|
|6. Cost-effectiveness||The service is provided cost-effectively; that is, resources are used both effectively and efficiently. [Indicators: collection ratio, NRW %, staff per 1,000 connections]|
|7. Financial Sustainability||Sufficient resources are available to maintain, replace and expand the infrastructure so that accessibility, safety, adequacy, reliability, convenience and cost-effectiveness continue long into the future. [Indicator: operating ratio]|
|8. Affordability||Ability of poor households to afford water to meet at least basic needs. [Indicator: poor household expenditure on water/poor household total expenditure]|
|9. Responsiveness||The utility is responsive to customers.|
|10. Transparency||Customers have access to information about the utility’s activities, finances and performance.|
Parallel indicators for sewerage can also be utilized.
Appendix on KPIs for Buses and Taxis
The Rwanda Utilities Regulatory Authority (RURA) developed a scheme for comparing public transport operators to promote efficiency and effective competition. Several other public institutions were involved in developing key indicators, with RURA heading the team. Before the KPI initiative, many operators were providing public transportation with cars and buses in poor condition. The government company (ONATRACOM) providing public transportation did not have enough buses to cover all areas, and private companies could not get loans to buy new buses. By dedicating routes for a given period to an operator, it became much easier for companies to obtain bank financing.
The task force defined the criteria in conjunction with public transport operators and collected data. The ranking team included representatives from the ministry in charge of transport, RURA, the public sector, the national police, and the consumer association. During the awards ceremony, all media were invited to a dinner gala to ensure coverage, and the winner was awarded a medal along with one year of publicity on the RURA website (http://www.rura.rw/fileadmin/docs/CITY_CENTER_2014.jpg).
The following elements were utilized in the ranking scheme:
Safety and Comfort: Random checks on vehicles for a number of components, including sound tires, fire extinguishers, functioning of all vehicle lights, phone number displayed for emergencies, cleanliness in the vehicle, and condition of seats.
Customer Care: Checks on cleanliness in the working area; display of customer information, including tariff and departure schedules; having waiting chairs for vulnerable clients; having a suggestion box; having a security officer; level of ICT usage; customer-care cycle (for reception, boarding and exit); and washrooms.
Administration: Evaluating formal structures of organizations, key staff availability, and motivation and professionalism of staff.
Regulatory Compliance: License renewal met, timely submission of operational reports, timely regulatory fee payments, operation and respect for assigned routes.
Appendix on Electricity KPIs
To illustrate the wide range of potential KPIs, consider some of the items identified in Africa’s Power Infrastructure: Investment, Integration, Efficiency report, published in 2011 as a result of the Africa Infrastructure Country Diagnostic (AICD) (https://openknowledge.worldbank.org/bitstream/handle/10986/2290/613090PUB0Afri158344B097808213845at58.pdf?sequence=1). It identifies KPIs for generation, transmission and distribution.
Berg, S., Water Utility Benchmarking: Measurement, Methodologies and Performance Incentives, 2010, London: International Water Association.
Berg, S., “Advances in Benchmarking to Improve Water Utility Operations,” Water Policy, 2012.
Cabrera, E., Jr., P. Dane, S. Haskins, and H. Theuretzbacher-Fritz. Benchmarking Water Services: Guiding Water Utilities to Excellence, 2010, London: International Water Association.
IB-Net: International Benchmarking Network for Water and Sanitation Utilities, http://www.ib-net.org/.
Mugisha, S., and S. Berg, “State-Owned Enterprises: NWSC’s Turnaround in Uganda,” African Development Review 20 (2): 305–34, 2008.
Thompson, L. S. and H. Bente. “What Is Rail Efficiency and How Can It Be Changed?” International Transport Forum Discussion Paper, No. 2014-23, https://www.econstor.eu/dspace/bitstream/10419/109178/1/818402369.pdf.
“Possibilities and Limits of Urban Water Utilities in Africa Serving the Poor” (World Bank, forthcoming)