Targets for Fragile States – How should performance targets be set, and how can performance be monitored when there is limited data availability and/or the quality of the data is uncertain?
(Response by Sanford Berg and Michelle Phillips, with input from Sara Ahmed, Jemima T. Sy, Ana Aghababyan, and Shyamala Shukla)
Selected issues faced by Fragile and Conflict-Affected States (FCSs) in relation to developing and improving infrastructure regulation are discussed in this set of Frequently Asked Questions (FAQs). Fragility is different in every context. Context shapes the market, institutions and the actions that are possible to take. The FAQs present general principles, policy guidance, and cases; where possible, guidance have been distinguished for countries along a fragility spectrum using the following taxonomy: Crisis, Rebuild and Reform, Transition, and Transformation.
Who Might Benefit from This FAQ, and How it Relates to Other FAQs
This FAQ focuses on initiating and improving data-collection processes, in order to set targets and monitor performance when significant data limitations exist. Those setting and implementing infrastructure policy can expect data availability and quality to be a problem. In FCS, data collection systems could be destroyed or not exist at all. Low-income states are likely to have problems with data reliability and accuracy. A key lesson to draw from this FAQ is that “people manage what they measure.” The absence of systematic data collection processes means that information in not being utilized. Improving data collection processes is an important aspect of management. The very process of recording basic data on output volumes, revenues, and operations gives the local managers a better grasp of how their project is affecting the local community. The data collection process establishes a set of baselines which can then be used to evaluate how donor or government funds are being utilized. Without such information, agencies supporting initiatives in FCS are unable to determine the cost-effectiveness of programs that are supposed to help restore confidence in the future for families facing great challenges.
Specific indicators and their use in designing incentives are addressed in other FAQs: FAQ on Key Performance Indicators and Benchmarking addresses the creation of information systems that can capture data on key performance indicators and shows how decision-makers can use this information to make comparisons over time and across comparable operating companies. Data on actual and possible levels of performance, used to set the targets, are part of this process. A follow-up FAQ to this one goes into more detail on the types of KPIs warranting attention in energy, telecom, water, and transportation—in situations where the specific sector or country has moved beyond the transition stage. The FAQ on Tariff Setting and Incentives focuses on using data to design incentives that promote quality improvements, cost containment and network expansion.
This FAQ identifies some techniques for data acquisition and authentication and outlines steps for introducing performance targets for countries in the Crisis and the Rebuild and Reform stages where basic regulatory institutions are emerging and independently functioning, data is beginning to be used for improved operations, and basic infrastructure services are starting to be provided through the state (Transition stage).
Why Good Data are Important, and Why the Data-Collection Process Matters
Data drives managerial and regulatory processes such as setting targets and creating performance incentives. In the crisis stage, monitoring inputs, outputs, revenues and costs can be challenging, but important for building accountability frameworks. Data informs different stakeholders about the nature of opportunities and risks and is particularly relevant to forging partnerships with the private sector. In mature transition or transformation stages, considerations will shift to how inputs are related to outputs; how input prices translate into costs; and what external factors affect costs, outputs and returns. Forecasts of financial and operating performance will affect the willingness of private sector to put their money at risk. Moszoro et al. (2014) wrote: “Controlling for economic characteristics, we find that overall private participation of infrastructure financing increases with freedom from corruption, rule of law, quality of regulations, and decreases with court disputes.”
While information systems are not free, they can have tremendous value in allocating resources effectively. Infrastructure is often funded by government (taxpayers), so evidence regarding the cost effectiveness of these scarce funds is one determinant of more predictable government transfers (and loans). Private investors will want to have evidence that management will use funds wisely and citizens (and politicians) consider the financial sustainability of the operator to be important. Demonstrating objectivity and expertise with the use of data is one way ministries and regulators can signal their intentions regarding new investment.
Regulators are concerned with a broad set of data—those that inform decisions regarding market structure (and opening up services to private competition); information needed to determine price/tariffs; service-quality indices required for the monitoring of service standards; and data that enable the evaluation of performance over time (for developing incentives for cost containment and investment).
The data collection process also matters. The ministry or oversight agency needs to establish an expectation that data must be submitted and that it will be reviewed. This expectation may have to be adjusted to the context: for example, a national operator faces different challenges than do community-based infrastructure service providers in rural or peri-urban areas.
The data needs to be retained in an easily accessible format. Information on trends will inform potential investors about whether existing operations have gained the confidence of the community (as reflected in bill collection rates) and indicate areas for potential partnership (such as operations, maintenance, service quality, and coverage). The data collection/reporting process enables regulators to engage stakeholders in a process of joint-sector reviews that bring different groups together and help them identify common goals. This is particularly important in FCS where the process helps animate the guiding coalition.
In the crisis stage, if the state is not providing infrastructure services, it may be the function of donors or NGOs to ensure basic data collection initiatives are begun.
Crisis stage case: Pakistan Rural Water Supply: What if there is no regulatory institution in place? In states that are in either the Crisis or Rebuild and Reform stages that are lacking in data collection systems and regulatory authorities, community-based organizations have been used to keep track of data. In Pakistan, the Punjab Rural Water Supply and Sanitation Sector Project trained the community in data collection and monitoring, making it possible to monitor the results of their clean water access project (Blume 2004).
The operator will generally collect data, but an oversight group needs to use data to plan, course correct or develop policies that will lay the foundations for better services. When the nation moves into the Rebuild and Reform stage, the data process can be transferred to the appropriate ministry or regulatory agency.
However, the data-collection process requires clear definitions of data, gaining acceptance of the need for collecting and utilizing data, developing schedules for collecting data, and (ultimately) reviewing information on trends so decision-makers reach a clear consensus regarding the patterns that are revealed by the information.
The key point in considering limited (and inadequate) information is to begin improving data collection and analysis procedures so that future decision makers can build on solid foundations.
Crisis Stage Case (Iraq electricity): An example of a FCS (Crisis Stage) that is currently using output from data collection initiatives to generate valuable information is Iraq. The Ministry of Electricity of Iraq has data on electricity losses as a share in total production by year. This data has allowed them to pinpoint problem areas, which include technical losses, due mainly to obsolete equipment, and non-technical losses, which “reflect encroachments on the electricity grid by inhabitants from slum areas.” Their data on subsidies to electricity has also allowed them to see that their subsidy level is high and unsustainable compared to other Middle Eastern countries. It has also allowed them to plan for changes in their subsidy system to address this issue (IMF 2015).
If an institution is not currently collecting data on key performance indicators, collection efforts should begin as soon as possible. Like the Chinese proverb says: “The best time to plant a tree was 20 years ago. The second best time is today.”
Why Good Data are Important for Promoting Private Participation
Data about assets, customers, performance, and operations such as maintenance and remediation programs will help define the scope of nature of risks for private initiatives. Thus, information improves the ability of project planners to mitigate risks, estimate the costs for doing so, and decide whether private participation can address them in the most cost-effective manner.
Investors entering the infrastructure sector in countries falling under the Crisis stage face several risks. The state of road networks is usually damaged during conflicts, making construction of new infrastructure projects and access to materials difficult. Access to the electricity grid is limited. Projects involving rural off-grid electrification are challenging given low incomes and a general lack of credit facilities. Additionally, investors perceive war-affected economies as high risk, which is further corroborated by international credit rating agencies. War, banditry, and guerrilla activities make it unlikely for private investors to participate in infrastructure projects (Hoeffler 1999).
Rebuild and Reform stage case: Iraq Energy and Afghanistan Telecommunications. Iraq’s Ministry of Energy estimates that the cost of expanding the electricity sector to meet rising demand is at $25 billion. However, international investment is limited (mostly financed by donors) given poor security conditions (IMF 2015). Private investment in infrastructure service is expected to arise once countries are at peace and have established adequate legal and regulatory frameworks. Given their risk-reward profile (high demand and short payback periods), telecommunications projects are often the first sectors to attract private financing in FCSs (Hoeffler 1999). Afghanistan provides an example of a FCS that was able to expand telecom access by introducing competition after the war in 2002. This was made possible by the government’s quick move to approve and implement a Telecommunications and Internet Policy in 2003 and donor support (Bhatia and Gupta 2006).
Data sensitivity in FCSs. While regulators will need to start compiling comprehensive sets of data, the agency must also be sensitive to how it handles data in FCS because information could exacerbate conflicts or put certain groups at risk. The regulator must have clear policies about what data it will and will not disclose, and when. Although not specifically focused on regulatory data or fragile countries, a draft disclosure framework (see (http://pubdocs.worldbank.org/pubdocs/publicdoc/2015/11/773541448296707678/Disclosure-in-PPPs-Framework.pdf) may provide some guidance on what to consider in disclosing information.
Dealing with Dispersed Data in FCSs
Especially, in Crisis or Rebuild and Reform stages, service is delivered through dispersed and low-capacity operators serving small communities, often informally. Collecting data will be challenging.
Regulators can work to increase the value of data collection to the local leadership and community. Information about customers, delivered output, cash flow, and other indicators are essential if citizens are to be in a position to participate and gain confidence in the state building efforts. Citizens will be able to tell if government and political efforts are moving in the right direction in terms of service quality, access, and sustainability.
Capacity building can have substantial leverage. Capacity is often lacking: informal or weak providers are likely not to have technical or financial management skills to support a well-managed information system. Recruiting and training people to collect, authenticate and report data is essential for the long-term success of local projects and provides the foundation for future expansion and improvements. A simple business plan establishes operating parameters and targets that can be reviewed at the end of each period. Without such plans (and associated procedures and a clear organizational chart indicating roles and responsibilities), the project can flounder and not meet citizens’ expectations.
Government and development-partner funding for capacity building can have substantial leverage, this is particularly true in the Transition stage. These groups often provide financial resources for creating facilities in conjunction with local operators, community partners or non-governmental organizations. Part of such initiatives should involve ongoing data collection for a few key variables. Such projects require some record keeping in a common format, such as accounting for external funding and identifying sources of internal funds and outlays for operations and investments. Recent initiatives emphasize leveraging access to mobile phones and information and communications technology (ICT) to enable even limited capacities in data collection and analysis; when this happens, trends can be identified and future support can be targeted to groups that are able to document the effective use of resources to meet community needs.
Folding private providers under an incentive framework can drive information systems improvements
Generally, the precision of collected data depends on the scale of the information system. The private-sector participants in a PPP have the incentive to collect data in order to justify a claim for payment. In fact, where there are many informal and small providers, one of the ways to work and build their capacity is to formalize them as sub-contractors/sub-concession holders under a larger, more capable private entity. The oversight organization will need to regulate fewer entities and the larger private firm plays a role in internal oversight.
Rebuild and Reform Stage Case (Senegal’s rural water systems): This case illustrates how underperformance by community user groups in managing rural water systems has triggered a reform initiative that is moving operations to PPPs, with community organizations (Association des Usagers des Forages [ASUFORS]) serving as monitors. When the ASUFORS was responsible for delivery, management was weak and did not “handle important issues such as the recovery of operating costs, appropriate tariffs and pricing, and renewal of infrastructure.” Furthermore, no separation of user representation from governance and operations was seen, leading to the unsustainability of the local systems. In recent reforms, service areas were aggregated under a larger contract with a private operator who could partner with ASUFORs eithers as bulk water supplier, or directly deliver water supply to customers formerly served by ASUFORs. The case illustrates the need both for capacity building and for appropriate design of accountability mechanisms within a sound system of governance. One result of weak performance was a renewed emphasis on a shared information system to facilitate performance updates and yard-stick comparisons.
Organize institutions around decision requirements and data management will follow. Certain types of information need to reach decision makers promptly and in an organized fashion. In the renewable energy and natural gas sectors, producers are often independent from distributors. Because energy cannot be stored cheaply, it needs to be generated at the same time it is consumed. In the United States, reliability regions have been set up to manage the flow of information (and energy).
How Can Data Quality be improved?
The primary task for those providing oversight involves initiating a data-collection program, so baselines (and targets) can be established. Stakeholders need adequate data of sufficient quality, since inadequate data will result in inaccurate representation of the status of service delivery. How can data quality be improved?
Conduct audits and data consistency checks. Regulators primarily rely on data being reported by the regulated entities. In a PPP setting, the private counterparty has a built-in incentive to collect data, because payments are linked to performance indicators. In these cases, measures should be taken to ensure the data are of sufficient quality. Ideally, an audit would be conducted by an independent party. In Mali, the Ministry of Mines, Energy and Water uses monitoring firms to augment its data collection capacity and to support regulated entities in assessing their own performance status. If hiring independent auditors is not feasible, other steps can be taken. For example, decision makers can look for inconsistent definitions, missing data, or extreme data values. If, for instance, a value usually falls in the 30,000 to 40,000 range, and the value for this year is 3,500, decision makers should ask themselves if this could be a coding issue.
Seek additional information sources. Another alternative would be to see if other data sources are consistent with the current data source. Does the utility provide data to other government departments? Is it possible to obtain those data? In the Crisis stage, there are probably few other entities collecting data on small operations. So this strategy is not available. However, in the Rebuild and Reform and the Transition stages, other groups such as media may be reporting about performance. Such information sources should be utilized by those monitoring performance. (More detail in the follow-up FAQ)
Assess data accuracy and reliability. Once the data have been obtained, it is important to assess their accuracy and reliability. Highly accurate and reliable data will allow people to make meaningful assessments and comparisons among utilities (or for the same utility over several years). For more on data accuracy and reliability see the follow-up FAQ.
Rebuild and Reform Stage Case (Uganda water metering program): In Entebbe, Uganda, data metering measures were implemented to reduce non-revenue water involving data accuracy and reliability. These included: regular spot checks on large customers, rotation of meter readers every 6 months, a meter/age database which allowed them to replace old meters first, and universal metering of all customer accounts. This allowed them to improve their non-revenue water situation drastically (USAID 2010).
Crisis Stage Case (Nigeria’s water sector): In Nigeria, state water utilities demonstrated weak performance over the years, and a major reason had been their inability (or unwillingness) to collect, store, and report data for critical decision making. As a result, the Federal Ministry of Water, in conjunction with development partners, recently introduced all state water agencies to the International Benchmarking Network (IBNET) system and has encouraged data collection using a basic set of KPIs. Initially, data quality was a major issue, but the culture of data collection is gradually becoming entrenched. This has resulted in higher operational efficiencies.
Review patterns and compare with other benchmarks. Data, organized in patterns, if carefully analyzed and interpreted, yields insights that can be used to develop realistic targets. If no historical record exists regarding financial and operating statistics, then no starting point can be identified to use as a baseline for gauging the impacts of external or internal targets and incentives. Generally some numbers are available, but the data are likely to be uneven in quality and not very timely. In such situations, targets will have to be based on comparable utilities in other regions or countries.
Include the baseline process as an initial deliverable in the PPP contract. In PPPs, establishing the baseline can often be included as part of the contract’s first key deliverable. Targets and performance incentives can then be confirmed once this baseline is available. If what is found in the baseline represents a material difference, contract terms may be amended according to pre-agreed procedures. Similar flexibility may need to be built into a licensing regime.
Crisis Stage Case (Pakistan Monitoring Water Access): In Pakistan, monitoring of the Punjab water access project by Community Based Organizations (CBO) included baseline data collection which enabled them to calculate a 90% reduction in reported water-related diseases, an averaged increased household income of 24%, and as much as an 80% increase in school enrollment of children, after the project was implemented. All of these statistics are available thanks to a benefit and evaluation report biannual requirement of the project (Blume 2004).
Crowd-source Analysis and Review. Creating data-collection systems is a key responsibility of those providing sector oversight, whether it be a regulatory agency (for countries in the Transformation and Resilience stages), Community Based Organizations or donor agencies (for countries in the Crisis and Rebuild and Reform stages). However, increasing access to data can lead to performance patterns being analyzed by other parties beyond these agencies, such as by academic researchers. Graduate students in university programs can provide “free consulting” via their research papers—gaining the input of professors and making academics aware of the important issues facing infrastructure decision makers. Those responsible for oversight are now in a position to conduct studies before developing targets and incentives.
Use secondary sources of data such as surveys. Many FCS have limited technology, capacity, and training for storing large amounts of data. Survey methodologies funded by donor agencies can be used to obtain data useful in gauging the situation in countries where capacity is limited. For example, the Asian Foundation conducted a survey of the Afghan people in 2012. This survey contained information on several infrastructure-related topics such as: availability of means of transportation, main sources of energy for cooking and heating, and access to communications technology (Tariq et al 2012). This information can be used to establish baselines.
Crisis Stage Case (Open Data for Resilience Initiative): The Global Facility for Disaster Reduction and Recovery provides a platform for data collection in FCS. The program uses crowdsourcing and open data platforms and is used by the World Bank Regional Disaster Risk Management Teams. For example, policymakers, government and consumers in Haiti can access and submit data for the country at Haiti Data. Haiti’s data includes a section on resources and planning, that includes Haiti’s road network (2006), which was created by the Haitian National Centre for Geospatial Information.
Rebuild and Reform case: Rwanda Governance: Another example, from Rwanda, a country in the Rebuild and Reform stage, is the eRwanda program. eRwanda, an ICT project funded by the World Bank at the request of the government of Rwanda, had the goal of improving the efficiency and effectiveness of targeted internal government processes. This project included several activities, including surveys to measure user perceptions; this was part of a local university’s capacity building program for monitoring and evaluating performance (Kelly and Souter 2014).
Crisis Stage Case (data collection in Afghanistan’s ICT sector): Insights from Afghanistan’s successful ICT expansion offers some lessons that could also be applied to data collection. These include: drawing on local strengths, establishing a policy at early stages, focusing on developing talent drawn from universities and training programs, ensuring that public agencies build their human and institutional capacities, and seeking support from donors early (Kelly and Souter 2014).
Rebuild and Reform Case (survey of Bangladesh’s small scale private sector providers of water and electricity): The World Bank funded a survey of small scale private sector providers in several countries, including Bangladesh, to better understand them and their ability to provide infrastructure services. The survey was exploratory and based on qualitative data from face-to-face interviews with providers, which included water kiosks and battery charging stations. Results showed that water kiosks have the largest impact on the poor, and that lack of legal standing is problematic. Survey results were then used to provide recommendations for the future (Izaguirre 2009).
Rebuild and Reform and Transition Case (networks of water operators in the Danube region): In the Danube region, associations of water operators in numerous countries represent another source of data. Those setting and implementing policy should focus on maintaining positive relationships with all stakeholders. It is important that those collecting data not be viewed as adversaries but as partners in improving sector performance.
Why do data systems fail in regulated entities and how can this be mitigated?
Managers responsible for making operating and investment decisions are also responsible for collecting data. They may have weak information systems (because of excessively low revenues relative to operating expenses), or they may fear that the provision of data will negatively affect their status or personal position. If reporting is not done regularly, the existing governance system is ineffective.
Several explanations can be given for data gaps: (1) the group with oversight responsibilities (a ministry, municipal council, or regulatory commission) lacks either the authority, political will, or motivation to “extract” data from the operating company; (2) the managers need training in the fields of accounting, finance, data acquisition, and data analysis; (3) the operator has limited funds and information systems are a low (short-term) priority relative to other tasks that must be performed.
Crisis Stage Case (water for entrepreneurial women in Pakistan): The Punjab Rural Water Supply and Sanitation Sector Project, funded by the Asian Development Bank, provided access to clean drinking water to 800,000 people using low-cost community-based projects. The local people participated in the planning and construction stages, and later became fully responsible for operation and maintenance (O&M) costs. CBOs were formed to take care of O&M, which included financial management, technical operations, and data collection on, for example, water quality monitoring. The CBO was in charge of project evaluation, which included collection of baseline data, information on performance indicators, data analysis, and evaluation. The CBO also prepared biannual reports monitoring benefits and evaluating results (Blume 2004).
Rebuild and Reform Stage Case (ICT in Rwanda): The government of Rwanda’s National Information and Communication Infrastructure Plan, was intended to increase ICT access. The project was delegated to the Rwandan Utilities Regulatory Authority and the Rwandan Information Technology Authority. A national training center at the Kigali Institute of Science and Technology and ICT training at the National University were established. The government was able to use ICT indicators to track its progress. This resulting information showed that ICT indicators did not improve by much. The government used this information to assess deficiencies in plan implementation and realized that there were deficiencies in infrastructure and human resources, coupled with low ICT-awareness and readiness by the general public and government officials. The government then implemented a second ICT plan, which included plans for a national data center. This initiative, which involved country and local level governments and donors such as the World Bank, led to improvements in several ICT indicators. However, the government’s evaluation of the program showed that several shortcomings remained, including a lack of systematic monitoring and evaluation (Kelly and Souter 2014).
Identify reasons for lack of reporting. Those developing and implementing public policy should not begin in an adversarial relationship with current operators. Rather, the guiding coalition that is responsible for new initiatives needs to determine the reason for non-reporting (one of the three explanations identified above) and take steps to remedy the situation. In preparing a reform initiative, the coalition will need to be able to quantify the existing performance of infrastructure suppliers, so some preliminary data collection is required. A collaborative workshop that focuses on current information systems could serve as a catalyst for getting high-level managers to review their information systems. Lower-level managers often operate in data silos, where the individual with access to data uses this exclusivity to gain favors or just to experience the sense of power that accompanies “control.” To establish credible information systems, broad political support must be mustered for improving efficiency in the provision of infrastructure services.
Develop a guiding coalition for data collection. Motivated staff (with skills in accounting, engineering and other fields) can identify causes of data limitations and propose budgetary and organizational initiatives that can strengthen data collection, authentication and analysis. A guiding coalition seeking performance improvements should promote professionalism within the operating company through training and continued monitoring of information flows. Data teams should be established and their work should be given priority within the operating company. A sound data system involves at least five elements: accountability, prioritization, integration, attention to cost effectiveness, and casting a wide net. Of course, such systems tend to be appropriate for more formal operations, as in the electricity distributor in national capitals, rather than in rural areas. The basic point for nations in crisis is that a “culture of information” needs to be encouraged from the start. Assigning a specific individual the task of collecting, authenticating, assembling, and reporting the data gives that person some status. The key is to avoid creating an “information silo” where access to data is severely limited. For small operations, the duplication of data systems is likely to be minimal. Finally, the ministry or agency providing oversight needs to be sensitive to the cost of data collection—so obtaining a limited quantity of important information is much more valuable than a vast amount of unaudited (and poor quality) data. It is then up to those developing and implementing public policy to analyze and utilize that information to improve sector performance.
Establish a knowledge center and promote training. The Water Regulator for Rivers State (Nigeria, a country in the Crisis Stage) has created a knowledge center for the water sector because of the complete absence of relevant historical data. The center’s purpose is to document all information for the water supply and waste-water sector. Cooperation with local universities should result in improved analysis of hydrological, engineering operations, and financial sustainability. To improve local capacity building, the regulator is initiating data-collection training programs for local young people. The focus is on data authentication and presentation to improve short-, medium-, and long-term decision making. Because the work is given priority, the data collection staff sees its role as strategic for the utility’s future. The acquisition and deployment of modern information and communication technologies are being encouraged for use in the data collection and management program.
How Can Evidence-Based Decisions be Fostered by those Implementing Reforms?
It is said that “the fewer the facts, the stronger the opinion.” This suggests that data and supporting analyses can defuse some potential conflicts. Decisions based on managerial discretion and political pressures characterize settings where data are not systematically collected. Evidence-based decisions cannot be made without historical statistics on finances (cash flows, income statements, and balance sheets) and operations (inputs, delivered outputs, service quality, customers, etc.). Therefore it is important to publicize information about trends over time and performance patterns across suppliers. Ultimately the question of data availability and data quality relates to how well inputs (networks, maintenance, labor, etc.) are translated into outputs (infrastructure services that are delivered to residences, industrial customers, and commercial demanders). Without financial and operating statistics, it is difficult (if not impossible) to evaluate sector performance and to identify the strengths and weaknesses of current regulatory and managerial arrangements.
Promote stakeholder participation to build credibility of oversight. The role of regulation in promoting transparency and the rule of law is critical. It is important that oversight agencies are not perceived by the public as being “in the hands of politicians.” In the first stage of regulatory reform, regulation can be viewed as being in the start-up stage (involving staff recruitment, training, informational workshops, and stakeholder meetings). Regulatory leadership tends to play a catalytic role for building the new institutional system. By improving transparency and promoting stakeholder participation, the new agency can gain the trust of civil society and powerful groups within the political system. Without external support, the regulator is likely to have difficulty obtaining information and winning public confidence that a new process has been created that is designed to improve sector performance.
Help stakeholders become more informed about utility operations. When citizens and elected officials can more easily compare the contexts and outcomes experienced by other nations in the region, it is harder for rhetoric to drive decisions. Policy dialogues are improved when evidence can be compared showing how the architecture of governance systems is (or is not) associated with particular types of outcomes. Additionally, development partners are likely to assess how transparency and communication promotes citizen education and managerial effort. Published data on indicators provide evidence of commitment to making changes in operations and investment planning. Ministries and Regulators can turn to donor organizations such as the World Bank, USAID, and the Asian Development Bank for assistance in such projects.
Conduct stakeholder workshops and meetings. In the total absence of data, stakeholder meetings can be used to document customer perceptions as well as the expectations of those without service. Even though “opinions” are being shared, both the operator and customers are able to listen to one another; both have obligations—the operator to provide service and the customers to pay for it. Managers can explain why service quality is low or network coverage has been slow to expand. Customers can identify their priorities regarding the service. Note that in some cases the direct customers are few (as with a shipping port or a pipeline to a power plant). In other cases, there are many retail customers, such as is the case for water, electricity or toll roads. In either case, the press can highlight infrastructure performance, giving the public greater awareness regarding the economic and social implications of poor service. An ongoing series of meetings probably will trigger better data collection and reporting, so managers are in a position to respond to public criticisms by documenting improvements in KPIs.
Require submission of business plans. An infrastructure provider without a business plan is like a jet plane without engines—it cannot get off the ground. Thus the absence of a business plan is evidence of weak management. In particular, attention needs to be given to the cost-accounting system, which is central to whether targets can indeed be met within current financial constraints. Data can be analyzed using an Excel spreadsheet, and then the internal rate of return can be compared with the cost of capital; inconsistencies would suggest that the targets are unrealistic in terms of their values or timing.
Develop policies for confidential information. Disclosure is very important from the private-sector point of view, but especially in the FCS context where poor data handling can exacerbate conflicts. This situation is still likely to leave some discretion to the negotiating official. The law should establish a general presumption of full disclosure, while providing for exemptions, such as to ensure commercial confidence for PPPs. For example, bidding details may be viewed as proprietary information by those submitting bids, yet there needs to be some way to inform affected parties about the selection process. The standard contract provisions would begin with the application of the relevant Freedom of Information Act to PPPs. The standard provisions need to provide specific language for confidential information based on the general exemptions provided in the act, but pointing to specific elements that in the context of the project would be considered confidential. In addition, it is important to specify the time period during which information is to remain confidential.
For countries in the Rebuild and Reform stage: These countries should aim for partial disclosure comprising the following: basic project information; procurement information plus disclosure of the RFQ and RFP documents; and information on government support. Another option is to place the full PPP contract and its schedules in the public domain. Disclosing all contract documents does not require a high level of skill/capacity on the part of the government agency doing the disclosing, but it does require political will. Care must be taken, however, to ensure that confidential clauses or contract schedules are redacted prior to disclosure.
For countries in the Transformation and Transition stage: It is recommended that in addition to disclosure as recommended for countries in the Rebuild and Reform stage, moderate-capacity countries should also attempt to disclose information on tariffs, service and fulfillment of the KPI targets. This would ensure that members of the public get headline information on the basic characteristics of the project; the level of taxpayer money going into the project; why they are expected to pay the stated tariff; and the services they would expect to receive against their payments.
Avoid regulatory capture. Setting appropriate targets and monitoring performance requires more than data; it also requires that these functions are carried out by an unbiased, professional group. Because the oversight agency and the operator will be in regular contact, and especially in case of regulating PPPs where the private party will have better access to information, there is a danger that targets or performance evaluation will be influenced by close relationships between whoever is serving the regulatory function and the operator’s management. Regulatory capture is always a danger, because those developing and implementing policy begin to identify too closely with those providing infrastructure services. Especially in an FCS context, the public perception of regulatory fairness is critical, because fragility often stems from groups being disenfranchised. A perception of capture, particularly by privately owned regulated entities, can be a flashpoint. Small consumers are rarely, if ever, competently represented, and academics in institutions that might enrich and provide intellectual discipline for the regulatory process are seldom involved. The result is that the regulatory process, in too many cases, if left to its own devices is “captured,” and the likelihood of political interference is enhanced. Additionally, especially when corruption is a problem. This issue is examined in greater detail in the FAQ on regulatory autonomy and political interference.
How Can Target Setting become More Realistic?
Unrealistically challenging targets frustrate managers; on the other hand, excessively low hurdles do not affect behavior. Identifying baselines and realistic targets requires data, but actually obtaining data is the first challenge facing those attempting to monitor and improve infrastructure performance. Setting targets is just one element in establishing incentives for performance improvements, although it goes to the heart of risk allocation, value for money and price premiums in PPPs.
Targets need to be based on the reality that is captured in past trends, current patterns across comparable service providers, and best practice. The group setting the policy must do the following:
- Collect historical financial and operating data (with clear definitions and authentication procedures);
- Analyze data across comparable operators (where topology, customer density and mix, and system ages are similar, or fully accounted for);
- Identify the preeminent supplier (so the feasibility of meeting the target is well established); and
- Determine the time frame for meeting the target, with explicit dates (because adjustments to complex delivery systems cannot be made quickly).
Prioritize data needs. Fewer targets can help simplify the process of measurement and management. This will require consensus on priorities.
Transition Stage case: Uganda Water. The Ugandan NWSC experience underscores the importance of identifying KPIs and starting the process. Uganda’s NWSC formulated only a few KPIs related to financial improvement (collections), operational efficiency (leakage), and staff productivity to ensure detailed data collection on this limited number of KPIs. The selection of indices reflected the performance situation when the reform program was initiated.
Transformation Stage Case (Jamaica water commission targets): In 2003, the Office of Utility Regulation (OUR) established a number of targets for the National Water Commission (NWC), including the following:
- Net receivables not to exceed 25 percent of revenues, and bad debt provision to be eight percent;
- Employee costsnot to exceed 35 percent of revenues (within two years of determination);
- Unaccounted-for water to be reduced to 55 percent by the end of fiscal year 2004–05 and thereafter by at least two percentage points per year;
- Collection rate to be 92 percent of billed revenues; and
- Water quality compliance to be 99 percent of the IJAM standards.
Clarify stakeholder responsibilities. A number of different stakeholders could have responsibility for setting performance targets: the relevant government ministry; the operator’s board of directors (representing owners, which might be a municipality or a ministry); managers of the company (to establish targets that trigger staff bonuses); or the sector regulator. These stakeholders have different interests and capabilities. Conflicts among various groups can lead to problems for the operator. Accountability needs to be clearly outlined to avoid conflicts and duplication of effort. (For more on this issue, see the follow-up FAQ.)
Rebuild and Reform Stage Case (Rwanda Utilities Regulatory Authority): Stakeholders (including other government agencies) play a role in collecting and authenticating information in Rwanda. A team in the Rwanda Utilities Regulatory Authority (RURA) identifies data to be monitored on a regular basis. Through a benchmarking process, the team prepares definitions and determines how the elements can be monitored. The team calls for a consultation meeting with all telecom operators during which they share the documents, including data to be monitored, and also seek comments about the substance and the process. Once a consensus is achieved, a template for technical data is sent to operators for monthly or quarterly data collection. In the case of network coverage, RURA obtains information provided by operators but also cross-checks that information by requesting local administrators through the Ministry of Local Administration (supervising ministry) to let RURA know about areas not covered by service providers. Thus, the regulator knows exactly which cells (the lowest entity in Rwanda’s administration system) are not covered by a cellular system—helping it evaluate the full impacts of competition.
Obtain data for comparable situations. A starting point for setting targets could be based on international benchmarking for countries in comparable situations. Such investigations can bound the reasonable targets. It should be made clear to the operator that future tariff adjustments will be conditioned on the provision of better data. Such an approach should gradually reduce information asymmetries unless the current management has no stake in improving performance and wants to avoid the political push-back from some future (justified) price increase. For water, IBNET represents one source of data on KPIs for water around the globe. For Africa, see a 2009 report that lists KPIs. The Africa Infrastructure Country Diagnostic represents a major initiative to develop comparable data for the region. As another example, the Nigerian Electricity Regulatory Commission has a report on KPIs in generation for that nation.
Institute rewards for meeting targets commensurate with the efforts required to meet the targets. The issue of incentives is addressed more fully in another FAQ that focuses on KPIs and incentives for achieving targets. Nevertheless, it is important to note that declaring a target is just the first step in the process. Incentives must then be devised and implemented. In PPPs, these incentives are built-in through payments and remuneration. It is not so straightforward for publicly managed entities wherein service performance is not as directly linked to the remuneration of utility managers and staff. These incentives could be established by an external authority (such as a ministry, municipality, or sector regulator) or within the infrastructure firm (by the CEO or the board of directors). Performance contracts can serve the goal of incentivizing managers to meet targets to achieve bonuses for teams within their organizations.
For a general primer on gaining data, the reader should review Measuring and managing for results in fragile and conflict-affected states and situations. That report provides guidance on monitoring and evaluation in FCS and low-income nations. It recommends a number of principles for those concerned with data collection and authentication:
- Use official data sources where possible
- Work with local partners
- Work with the private sector
- Work with the military
- Invest in data
- Involve beneficiaries in monitoring
- Use opinion polls
- Use perception surveys, but with care
- Triangulate (use multiple sources to cross-check data)
- Innovate (e.g. use mobile phone technology, aerial photography, and other technologies)
References
Bhatia, Bhavna, and Neeraj Gupta, Transforming Telecoms in Afghanistan: Expanding Affordable Access by Introducing Competition, April 2006, PPIA Gridlines, World Bank, Washington, DC. http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2006/09/28/000310607_20060928112947/Rendered/PDF/375370AF0Gridlines0101PUBLIC1.pdf
Biswas, Asit K., and Cecilia Tortajada, Water Supply of Phnom Penh: A Most Remarkable Transformation, 2009. http://www.thirdworldcentre.org/wp-content/uploads/2015/04/phnompenhrep.pdf
Blume, C., and M. Dayrit, Bringing Water to the Poor: Selected ADB Case Studies. Ladies First: Accessible Water for Entrepreneurial Women in Pakistan, Asian Development Bank. http://www.adb.org/publications/bringing-water-poor-selected-adb-case-studies
Diallo, Oumar, Levers of Change in Senegal’s Rural Water Sector, World Bank Water and Sanitation Program Report, June 2015, World Bank, Washington, DC. http://www.wsp.org/sites/wsp.org/files/publications/WSP-Levers-of-Change-Senegal-Rural-Water-Systems.pdf.
Hoeffler, Anke, Challenges of Infrastructure Rehabilitation and Reconstruction in War-Affected Economies, Background paper for the African Development Bank Report, 1999. http://www.afdb.org/fileadmin/uploads/afdb/Documents/Publications/00157630-EN-ERP-48.PDF
Iraq: Selected Issues, IMF Country Report No. 15/236, 2015. https://www.imf.org/external/pubs/ft/scr/2015/cr15236.pdf
Izaguirre, Ada Karina, A Survey of Small Scale Private Service Providers of Water and Electricity in Bangladesh, in Opportunities and Challenges for Small Scale Private Service Providers in Electricity and Water Supply: Evidence from Bangladesh, Cambodia, Kenya, and the Philippines (ed. J. Baker), World Bank, 2009.
Kelly, Tim, and David Souter, The Role of Information and Communication Technologies in Postconflict Reconstruction, the World Bank, 2014. https://openknowledge.worldbank.org/bitstream/handle/10986/16679/9781464800740.pdf?sequence=1&isAllowed=y
Manager’s Non-Revenue Water Handbook for Africa: A Guide to Understanding Water Losses, USAID and World Bank Institute, March 2010. http://www.pseau.org/outils/ouvrages/usaid_manager_non_revenue_water_handbook_for_africa_en.pdf
Managing results in conflict-affected and fragile states: a stock-take of lessons, experience, and practice. http://www.agulhas.co.uk/cms_files/14/InterimGuidanceNote-Managing_results_in_conflict-affected_and_fragile_states_a_stock-takeoflessonsexperienceandpractice.pdf
Marian Moszoro, Gonzalo Araya, Fernanda Ruiz-Nuñez and Jordan Schwartz, Institutional and Political Determinants of Private Participation in Infrastructure, International Transport Forum Discussion Paper 2014-15, October 2014, OECD.
Mugisha, Silver, and Sanford Berg, Turning around Struggling State-Owned Enterprises in Developing Countries: The Case of NWSC-Uganda,” in Going Public: Southern Solutions to the Global Water Crisis, pp. 15–25, 2007, London: World Development Movement.
Ssozi, Disan, National Monitoring of Water and Sanitation in Uganda, Ministry of Water and Environment, 2013.
Tariq, Mohammad Osman, Fazel Rabi Haqbeen, and Palwasha Lena Kakar, Afghanistan in 2012: A Survey of the Afghan People, The Asia Foundation, 2012. http://asiafoundation.org/resources/pdfs/Surveybook2012web1.pdf
Water and Wastewater Services in the Danube Region: A State of the Sector,” May 2015, www.danube-water-program.org.