KPIs for Fragile States – What are the Key Performance Indicators (KPIs) for evaluating sector performance and how can they be used effectively by regulators? (For nations that are in the Crisis or Rebuild and Reform stages)

(Response by Sanford Berg, with inputs from Jemima T. Sy and Anna Aghababyan)

Selected issues faced by Fragile and Conflict-Affected States (FCSs) in relation to developing and improving infrastructure regulation are discussed in this set of Frequently Asked Questions (FAQs). Fragility is different in every context. Context shapes market, institutions, and the actions that are possible to take. The FAQs present general principles, guidance, and cases; where possible, guidance has been distinguished for countries along a fragility spectrum using the following taxonomy: Crises, Rebuild and Reform, Transition, and Transformation.

Who might benefit from this FAQ, and how it relates to other FAQs

The points developed in the FAQs on Data Availability and Setting Performance Targets apply to this question about key performance indicators (KPIs), since data collection, authentication, and quality improvement are the foundations for information systems. In the Crisis or Rebuild and Reform stages, development partners are typically major sources of funds for infrastructure investment. They will expect reports on progress—with information on changes in coverage, operations, and finances. Funds need to be accounted for, so the basic information systems required by development partners and NGOs represent the starting point for KPIs. When government is in a position to fund some investments, the Ministry of Finance and agencies providing direct oversight of local operators should become repositories of information. For nations in the Crisis or Rebuild and Reform stages, the first step in this process involves identifying and prioritizing key objectives. Decision-makers need to define the data required to evaluate performance, identify source documents, create data formats, and convince local leaders and managers that KPIs represent valuable tools for improving outcomes.

A companion FAQ deals with more mature institutions. Both these FAQs present lessons from international experience. They should be read together, since even nations in the Transition or Transformation stages are likely to have some infrastructure sectors that lag behind the others.

This FAQ includes a list of essential resources for water, energy, telecommunications, and transportation. The accompanying FAQ on KPIs and Benchmarking presents procedures for collecting KPIs, making comparisons over time and across service providers, and starting incentive design. It also recommends steps for the creation of a guiding coalition.

Performance Objectives and Key Performance Indicators

In general, infrastructure regulation is concerned with the sustainable delivery of service to a policy-defined level of quality and quantity. Regulators will be concerned that KPIs are focused on inputs, process or outputs that drive access, service performance, and compliance to standards and in more mature markets, price and competition. In some ways, KPIs are a realistic expectation (target) for the quality and quantity of infrastructure service at a given price.

For a performance indicator to be “key,” it must capture some dimension of performance that is important to customers receiving service and to those who hope to have access in the future. These key performance objectives are just as important to service providers and to investors, because both managers and investors need to know which outcomes are valued by political leaders who develop, and by regulators who administer, public policy. Those implementing policy should establish incentives for achieving important targets, but service providers can help them by giving feedback regarding measurement issues and by providing estimates of the relative costs of meeting the targets.

Those responsible for overseeing infrastructure services and for delivering those services should reach a consensus regarding what is valued and what is feasible. In the Crisis and the Rebuild and Reform stages, it is likely that a priority will be to rehabilitate infrastructure services and be able to achieve as much coverage of service as cost-effectively and as quickly as possible. Key performance objectives could include availability and access, network outage impact reduction, robustness of operations to reoccurrence of extreme events (including weather and conflicts), quality of service and reduction in downtime, etc. More mature sectors could include performance objectives on customer satisfaction, integrity of billing and collection processes, reliability, affordability, efficiency, productivity, innovation, security and safety. These are then associated with a set of KPIs.

Once the objectives are prioritized and data collection has commenced, the performance evaluation process can begin, starting small and slowly increasing the number and accuracy of KPIs. Another advantage of starting with objectives is that “easy to measure, but relatively unimportant” indicators will not become the focus of performance evaluation.

KPIs for infrastructure systems enable decision makers to track trends and identify areas needing changes, including operational procedures, maintenance/inventory practices, and capacity investments. These indicators represent valuable information regarding whether infrastructure performance is improving or worsening. In the former case, those groups responsible for particular performance outcomes should be rewarded for meeting targets. Making information available to citizens and to politicians increases the likelihood that investments will not go to projects that have low payoffs for society. For FCS, KPIs allow for fair comparisons that make it easier to avoid situations where those in power give preferential treatment to specific groups. This requires that transparency be prioritized as part of KPI development and dissemination. As noted in the FAQ on information, without data, it is not clear what managers are actually managing.

Using KPIs to Assign Responsibility to Departments within Organizations

KPIs can be classified into a set of categories that allow managers to establish primary responsibilities within the key departments of the regulated business (whether public or private) – i.e. to technical, operational, financial and commercial teams. Since performance outcomes are inter-related and inter-dependent, while specific departments might be responsible for collecting the data, performance will rely on the activities of others. Thus, the associated key performance outcomes refer to overall technical and financial operations, service quality or customer experience, for example: hours per day of service (technical), nonrevenue water or line losses (operational), cash flows and collections (financial), and customer complaints (indicating value for money, from the users’ standpoint). The following figure illustrates the information flows required for setting targets and incentives for infrastructure operations.

KPIs for PPPs. The framework above applies generally to regulation systems. In a PPP framework, in addition to performance standards following general regulations, there will be contractual KPIs that will have specific incentives linked to them. In PPPs, KPIs play more explicit roles in driving value for money in the delivery of a service. For PPPs, KPIs will define the roles and risks being assigned (or not assigned) to the private entity throughout the investment life cycle of a specific asset (or group of assets) associated with the PPP and payments to the private sector participants will be fully or partially based on meeting KPI targets. KPIs can be linked to payments, penalties and bonuses, as well as be considered as material conditions of the contractual obligation of the private sector. There could also be mutual obligations on performance between the public and the private sector. It is important in the PPP framework that KPIs and incentives be clearly identified. Parties need to agree on the definitions of the KPIs; how they are measured; and, who will conduct the measurement. In some cases, third party independent auditors can be called in to lend objectivity to measuring performance outcomes.

Five Fundamental Lessons from International Experience

The following recommendations about KPIs are based on numerous case studies and interviews.

1. Keep it simple at the start, and build the database over time. Indicators are a means to an end, not an end in themselves. This point has already been emphasized in the FAQ on information and is absolutely fundamental. Organizations in low-income, fragile and conflict-affected states are likely to face a number of constraints limiting their ability (or motivation) to collect decision-relevant data. Operators are generally severely under-resourced, and those serving as regulators (whether a ministry or a semiautonomous sector regulator) may lack the incentive to gather and publicize performance because the indicators could indicate weak performance and poor oversight. Once a serious initiative to develop a system of KPIs has been undertaken, the guiding coalition should focus on 10 to 15 data elements. These KPIs would be based on inputs and outputs such as financial information (revenues, sources of funds, breakdown of operating expenditures, collections, etc.), information on the customer (number and types of customers, average bills by customer type, percentage with meters/usage measurement devices, etc.); and information on operations and service (quantity produced, technical and commercial losses, etc). When the Association of Water Regulators of the Americas (ADERASA) began a collaborative data collection project in the late 1990s, those managing the project started with a goal of more than 130 different variables. Most water regulators in the region already had data on basic KPIs, but the jump to more than 100 items, used to create more than 30 KPIs, was probably excessive. On the plus side, this initiative allowed the regional regulatory network to promote comparisons within and across countries.Some broad areas (and associated examples) need to be addressed by decision makers:

a. Access or coverage of service – measured by number or portion of people with access to the service
b. Reliability of supply—measured by the number and duration of interruptions in the supply of public utilities.
c. Performance effectiveness—measured by the quantity of losses during the transit, transmission, or other types of provided services.
d. Quality of provided services—measured in different ways. For example, for electricity supply, it may be measured by voltage and its stability, as well as by temporary or impulse overvoltage. For water supply, it may be measured by the quality (meeting World Health Organization standards) and quantity of water (hours per day of service).
e. Commercial quality—measured by the quality of relationship between the supplier and customer (reflected in customer complaints and satisfaction surveys).
f. Revenue sufficiency—measured by efficiency and different financial indicators, such as interest coverage.

2. There is no need to reinvent the wheel; numerous freely available sources provide definitions and identify steps for improving data collection. Here the issue involves prioritizing the collection, authentication and analysis of data. Although a number of common indicators can be used to identify financial and operating trends, sector-specific variables will be needed to capture the unique features of water, energy, telecommunications, and transportation.

a. Water: Information is available from the International Benchmarking Network for Water and Sanitation Utilities. Issues around safety and health standards are particularly important for water and sanitation. The site provides guidance on KPIs (including definitions) and provides links to a number of resources. Also see the Appendix on Water Utilities.

b. Energy: Electricity information is available from a World Bank project that documented the performance of electric utilities in Sub-Saharan Africa (ESMAP 2009). An appendix illustrates the wide range of variables used in developing electricity KPIs. A detailed discussion of the Energy Kiosk approach to decentralized provision is also available.

c. Natural gas information will be similar to electricity. Particular emphasis should be given to safety records.

d. Telecommunications KPIs are available from the International Telecommunications Union. The business model for telecommunications involves a much higher ratio of revenue to fixed network costs; furthermore, current prices are not anchored at prices that yield revenues that are less than operating costs (which is often the case for energy and water). Innovations in the information and communications technology (ICT) sector have reduced scale economies and made multiple suppliers possible. Those providing oversight can focus on network interconnection issues and on the provision of price and performance information that reduces information asymmetries in the system. There is also a KPI Library that includes telecommunications and other infrastructure sectors.

e. Transportation indicators for railroads are defined in a recent study by Thompson and Bente (2014). <link> Once again, the important point is that those setting policy need baselines and trends on financial performance, passengers, distance-weight hauled, and other operating statistics, including adherence to schedules and other dimensions of quality. For roads and highways, it is necessary to have information regarding congestion and road surfaces as well as regarding accidents and other dimensions of transport performance. In the case of buses and taxis, some sample KPIs recently introduced in Rwanda are listed in an appendix.

3. It is better to have a rough estimate of an important dimension of performance than a precise calculation of some metric that has minimal implications for producers or consumers. Given this point, data are necessary to address the following questions:

a. What are the key inputs affecting output and cost? (The production process needs to be carefully delineated.)
b. How do input prices translate into costs?
c. What are the basic outputs?
d. What are the best measures of service quality in the situation facing policy makers? (In some cases, qualitative indicators reflecting citizen perceptions may be all that is feasible.)
e. How are inputs related to outputs?
f. Which specific ratios can be used to rank utilities?
g. How does information on trends affect performance evaluations?

4. Risk mitigation can be facilitated by the effective use of KPIs. In the context of FCSs, one issue addressed by KPIs involves risks and how they might be mitigated. First, what is the most important risk to service delivery given the fragility context? Second, what is the risk that the KPI implementation/enforcement is not able to provide a credible commitment framework between the regulated entities and the state?Identifying risks to service delivery: In FCSs where resources are limited and capacity of institutions may be constrained, a number of steps are needed to reduce the likelihood that a reform initiative will fail. In defining KPIs, the regulator, in consultation with stakeholders, needs to reach agreement regarding the prioritization of objectives. Many things will fail due to extreme circumstances: For example, at Crises and Rebuild and Reform stages it may not be reasonable to expect service providers to recover their operations and maintenance costs from charges to customers, but revenue sufficiency is critical for services to continue. Some way of closing the revenue gap must be found or the target needs to be adjusted—otherwise, financial viability is threatened. Or it may be necessary to demonstrate progress on equity in access for different groups, requiring both data collection and presentation to those regional or tribal or political groups. Similarly, it might be impossible to restore a high level of service immediately post-conflict and more important that access to service is restored to a large portion of the population, even though the level of service is not ideal. In such cases, the focus of the regulator could be to lead a dialogue with a broader set of stakeholders around how such risks will be mitigated. If there will be revenue shortfalls, will the government or donors be able to finance them while the regulated entity slowly achieves financial equilibrium? If so, then the regulator will need to help put in place this gradual increase in targets and set performance expectations for the future.Credibility of commitments within the regulatory framework: The second set of risk mitigation questions in the FCSs context is whether KPIs and the system around their implementation/enforcement is able to provide a credible commitment framework between the regulated entities (which could be a private-sector PPP grantee) and the state (which is a representative of consumers as well as a benefactor/granting of authority under a PPP). The regulator must have a level of self-awareness about whether KPIs are only monitored (via record-keeping) or whether they can be effectively monitored, authenticated, and enforced (via incentives) through the existing regulatory institution. Will the regulator be able to enforce the KPI in a credible way (from the perspective of the private entities, the consumers, and the public entities) given their situation, the existing legal structure, and the fragility context? If not, then the regulator and the guiding coalition will need to consider how the system can be bolstered by other mechanisms. For example, if data is an issue and judgments cannot be made clearly on the basis of uncontested data, then are dispute resolution systems or systems of expert inputs in place? (Readers can assess their situation regarding dispute resolution systems by completing the taxonomy assessment <Add link to Taxonomy here>). Do the parties need to think about taking/requiring performance guarantees from third parties? Is there is likelihood that one party is not able to deliver its part of the bargain that affects the other party’s KPI? For example, if the state is unlikely to let tariffs adjust in a high-inflationary environment, can the service provider count on the government covering the shortfall? If not, it is useful to publish business plans that are feasible under different funding scenarios—including the impacts on KPIs.

   Clearer contract/license obligations and provisions could also be helpful in dealing with contingencies that may be expected in FCS contexts. For example, regulation may require that the surplus of regulated public entities must be set aside in a contingency fund to cover future problems.

Procedures for Collecting KPIs, Making Comparisons over Time and across Service Providers, and Designing Incentives

The procedures for obtaining KPIs and designing incentives begin with data sources that document the operations of the service provider. The extent of process formalization depends to some extent on the size and age of the organization(s) under review. Although the basic steps of creating a team, working with stakeholders, selecting KPIs, authenticating information, and disseminating informative reports apply to all types of organizations, the contexts differ, so some of the tasks differ, depending on the maturity and size of the operator.

In situations where there is a lack of technical and institutional capacity, several options are commonly used for collecting and storing KPIs. One option consists of training and empowering Community-Based Organizations (CBOs) in data collection techniques. For example, a water access and quality program implemented by the Asian Development Bank in Pakistan requires empowered local water users to provide biannual reports for monitoring and evaluation purposes. Another option consists on putting donor agencies in charge of KPI collection, at least during the early stages. An example of this is the Danubis Water Platform[1], an online repository that includes KPIs for countries in the Danube region. The data is collected by the countries themselves, but the website, which includes KPIs, is managed by the World Bank and the International Association of Water Supply Companies in the Danube River Catchment Area.

[1] www.danubis.org

How to Commence KPIs Development in the Context of Fragmented Service Delivery

In countries in the Crisis or Rebuild and Reform stages, service is delivered through dispersed and low-capacity operators serving small communities, often informally.  Developing the KPI and incentives framework could be challenging.  Those monitoring smaller, dispersed operators should do the following:

• Create a benchmarking team: The emphasis should be on collaboration between those providing oversight and the local group actually responsible for delivering service. Since the nature of their operations is likely to be relatively simple, the team should focus on very basic processes.
• Establish trust/promote stakeholder awareness: Small-scale operations depend heavily on community support, and identifying reasonable targets and designing incentives to achieve them requires substantial local input. Working with stakeholders is therefore a crucial step in the process, especially in fragile states where stakeholder inclusion can contribute to building resilience (and the opposite – the feeling of exclusion may exacerbate tensions). Establishing trust, and promoting stakeholder awareness, requires very strong political will, which may be difficult to achieve in FCS.
• Select a small set of KPIs: The available records are likely to be quite limited, so capacity building must accompany the selection of KPIs.
• Gather raw data for yardstick comparisons: This stage involves the selection of the appropriate comparison group of companies or standards.
• Apply data verification procedures: After data collection processes have been instituted, techniques need to be developed so that procedures ensure that measurements improve over time.
• Perform data analyses: This crucial stage involves the application of methodologies appropriate for answering key questions—for fragile, low-income states establishing baselines and identifying trends will be an important step as this can address concerns about whether the new governance systems can support professionalism, transparency and continuity of service compared to the past.
• Develop policy implications: Explore potential determinants of inefficiencies so that the foundations for future initiatives can be laid.
• Engage in information dissemination: Reporting the results of performance comparisons helps engage stakeholders in the process. Results can be presented in simple formats, such as using a stop-light analogy, with green, yellow and red representing different degrees of improvement.
• Determine the types of incentives most likely to be both acceptable and effective: Clear procedures are necessary for gaining local support in the planning and implementation of incentives and corrective actions (the best “carrot” for meeting goals is the provision of future funds to expand and improve the current set of services).
• Prepare regular reports for local beneficiaries and for the external organizations supporting the initiative: data collection and analysis are ongoing for monitoring, communicating, and evaluating performance outcomes. Avoid actions that will be perceived as bureaucratic micromanagement. Those with oversight responsibilities need to be viewed as partners in the development process, rather than as adversaries.

Data Collection Helps to Set Targets and Design Incentives

Unless early steps are taken to gather and analyze data, local operators may not adequately maintain the networks for which they are responsible.  If those providing oversight have no evidence of inefficiencies or financial problems, they may not be in a position to inform other groups (such as development partners or NGOs) about emerging issues.  Limiting waste is a key by-product of serious data collection initiatives.  However, reports should not sit on shelves.  They should be publically available, so those affected by managerial actions (or inaction) are aware of the situation.  Furthermore, when targets are set, they can be based on attainable objectives—where some operators provide evidence of best practice operating procedures.  Those providing oversight can also promote study tours and workshops, where the most effective organizations share their experience with those operators who are still struggling.

The cases of Zambia, Uganda, and Kosovo developed in the related FAQ represent stable situations, where regulators or the ministerial team that designed the performance contract have been able to develop data frameworks that provide information on a wide range of outcomes.  For example, the water regulator in Kosovo “grades” utilities partly on the basis of the timeliness and accuracy of reports to the agency.  Such institutional arrangements promote transparency, making citizens aware of how well their service providers are complying with rules and whether performance trends are in line with comparable utilities.

Source Documents for KPIs in Conflict and Rebuild and Reform Stages.

In Conflict or Rebuild and Reform stages, information is often lacking for various reasons:

• Records might have been destroyed during a conflict;
• Data may be scattered and hard to access (stored in “bad” conditions, such as high humidity or “information silos”);
• Data collection and authentication might not be a high priority relative to just getting service to citizens;
• Staff may not have the capacity to perform basic accounting and information systems functions;
• Current management might avoid transparency in order to obscure the true state of operations; and
• Current management might avoid transparency so that potential corruption will not be revealed.

The absence of data documenting financial and operational trends does not mean that nothing can be done. Data collection and reporting procedures need to be initiated as a requirement for further external funding. Basic targets can be set based on comparable operations in other regions or nations. Such targets should reflect the priorities of policy makers and the capabilities of existing operators. Consultations with different stakeholders can help paint a picture of current levels of performance. Stakeholders also need to have input into the process so benefits and costs can both be factored into the setting of realistic targets.

Who Takes the Initiative?

The related FAQ for nations in Transition or in Transformation develops in some detail the role of a “guiding coalition” in promoting data collection, authentication, review, analysis, and dissemination.  In a nation that has been torn apart by conflict or a natural disaster, the national leaders may not prioritize data collection relative to more pressing tasks.  Nevertheless, some component of the relevant ministry needs to take responsibility for encouraging the collection of data through quarterly (or annual) reports regarding operating and financial statistics.  A culture of measurement and management can be initiated by small operators or by ministries, but to sustain data collection initiatives, the information needs to be used—by managers and by those providing oversight.

The natural focal point for institutional reform is the ministry requiring reports on the use of government funds or on funds from NGOs.  This information initiative is basically the start of an accountability program.  Professionals in the ministry are in a position to address concerns of special interests (who may be consumers from different regions or ethnic groups).  The group that becomes the data repository will later evolve into a benchmarking team.  Stakeholders that need to be kept informed include utilities, appropriate government agencies, community groups, women’s organizations, and other local groups. Such representation gives citizens greater confidence in the emerging regulatory system.

The related steps of promoting citizen awareness and covering the entire service delivery chain are discussed in greater detail in the related FAQ.  No matter how dysfunctional current production arrangements seem to be, some group is likely to be benefiting from the existing situation.  It may not be possible to turn potential opponents of reform into allies, but those taking the initiative on obtaining and verifying data need to develop some way to neutralize such stakeholders.  Such groups include alternative providers of service that are not regulated, for example, small generators whose livelihood would be affected by greater electricity coverage.  These stakeholders will need to see the value in participating in a more formal process of regulation and service delivery, such as the establishment of a level playing field.

Policies are not self-implementing.  In the case of a fragile state, one of the valuable outcomes of focusing on data, is that those closest to ultimate customers begin to focus on trends.  With only a few KPIs, local leaders become aware of the consequences of their decisions about operations, customer satisfaction, pricing, and subsidies.  Decision-makers are now able to document performance and develop business plans for the next phases of infrastructure remediation and expansion.   If there is a window of opportunity and the political, economic and social conditions are ready for change, leadership can help organizations develop initiatives (programs) that improve data collection and analysis. For more detail on specific KPIs, see the related FAQ.