Selecting and Using Performance Measures
This section discusses the importance of using performance data to make decisions. It highlights the role of performance measures and identifies how they are used to establish achievable performance targets. A more detailed discussion of Transportation Performance Management can be found in Chapter 2.
Performance Management Framework
As discussed in Chapter 2, transportation agencies have embraced the use of performance data to drive investment decisions. A performance-based management approach enables agencies to select and deliver the most effective set of projects for achieving strategic objectives, while also improving internal and external transparency and accountability.
A typical performance management framework includes:
- A clear idea of the agency’s strategic objectives.
- The use of performance measures to assess performance.
- Methods to evaluate and monitor performance results.
- The evaluation of factors with capacity to improve long-term performance.
- The allocation of funding to achieve agency objectives.
- Ongoing processes to monitor and report progress.
A fundamental component of the framework is the use of performance measures to evaluate system performance and the importance of establishing business processes to evaluate, monitor, and use the data to influence agency decisions. These are achieved by aligning decisions at all levels of the organization with the agency’s strategic objectives and ensuring that the right performance measures are being used to drive decisions. This alignment helps to ensure that resource allocation decisions and the day-to-day activities of agency personnel support the agency’s priorities and the interests of external stakeholders.
The existence of a regular, ongoing processes to monitor and report results is critical to identifying and implementing improvements to system performance or to further the effectiveness of the performance management process. The continual monitoring and update of a performance management framework is reflected in Figure 6.1, which illustrates inputs to performance targets and how ongoing monitoring and adjustments are fed back into the framework to adjust future targets. The surveys conducted regularly to support a pavement, bridge or maintenance management system are examples of the types of performance monitoring activities fundamental to an effective performance management organization.
Agencies with a performance management framework in place have benefited from:
- Maintaining a clear and unified focus for making agency decisions based on agency priorities, public input and available resources.
- Using available funding more effectively to preserve or improve system performance while lowering life cycle costs.
- Allocating available resources based on analysis of past performance and expected conditions to address areas most in need of attention.
- Having the data to confidently defend funding requests or explain the impact of reduced budgets.
- Building a transparent and accountable organization by communicating the basis for making resource decisions.
- Meeting legislative requirements.
It is important to select performance measures that are meaningful to the agency and that can directly inform decisions. This may vary depending on the agency context, culture, and TAM maturity.
In 2001, during the development of a long-range transportation plan (LRTP), the Arizona DOT took a strategic approach to how investments should be made. Under the new approach, Arizona DOT established the following three investment categories:
- Preservation, including activities that preserve existing transportation infrastructure.
- Modernization, including improvements that upgrade the efficiency, functionality, and safety without adding capacity.
- Expansion, including improvements that add transportation capacity by adding new facilities or services.
To implement the new initiative, the Arizona DOT developed a report titled “Linking the Long-Range Transportation Plan and Construction Program” or” P2P Link” that applied financial constraints to the long-term vision. Through a collaborative process that involved a consultant, local and regional governments, and transit agencies, the Arizona DOT published an implementation plan for putting the P2P Link into practice. The resulting process includes scoring projects based on both a technical and policy score that are added together to determine a project’s ranking. The technical score is generated by the asset owner based on an analysis of the data while the policy score is determined based on each project’s contribution to LRTP goals and performance measures. The process helps to ensure that projects are ranked in accordance with the agency’s strategic objectives using only the most meaningful criteria in a transparent and defensible way.
Arizona DOT’s Link Between Strategic Objectives and Investment Decisions
Source: ADOT. 2014. Linking the Long-Range Plan and Construction Program P2P Link Methodologies & Implementation Plan.
Performance measures are used within a performance management framework to allocate resources and provide feedback on the effectiveness of the activities in achieving overall objectives. Performance measures are indicators used for evaluating strategies and tracking progress. A performance measure can be an indication of asset condition, such as a pavement condition rating, or an indication of an operational characteristic, such as the annual number of fatalities on a facility.
The most effective performance measures drive decisions that are important to the success of the program. For example, maintenance departments may use performance measures that track actual expenditures to planned expenditures to ensure that available funding is directed towards the highest-priority items, as shown in the Colorado DOT practice example.
It is also important that the measures drive the desired performance within an organization. For instance, a performance requirement that measures whether pavement or bridge designs are submitted on time might cause incomplete or incorrect submittals to meet a deadline, leading to an increase in construction modifications. A more effective measure might focus on a minimal number of design modifications during the construction phase of a project.
Effective performance measures should also primarily be outcome-based rather than output-based, meaning that they focus on the result or impact of an activity rather than the inputs that went into the activity. Several examples of outcome- and output-based measures are shown in the sidebar on Page 6-8. Outcome-based measures are generally preferred because they indicate the effect on the traveling public resulting from the actions taken, so they usually relate to user priorities such as the length of time for a road to be cleared after a snow event or the absence of litter and graffiti. They are developed based on a description of what an agency wants to achieve as a result of the actions undertaken. Outcome-based measures are commonly used for managing ancillary assets such as drainage assets and signs. For instance, the performance of drainage assets might be reported in terms of the percent of pipes/culverts greater than 50 percent filled or otherwise deficient and the performance of signs might be reported in terms of the percent of signs viewable at night.
Output-based measures, on the other hand, track the resources used to achieve the outcome, such as the number of hours of labor used or the number of light-bulbs changed in a month. While the data is important information for managing resources, it does not necessarily drive outcomes that would matter to the public. For instance, travelers on a highway are much more interested in knowing when the road will be cleared of snow than how much overtime went into the operation.
When possible, agencies should use performance measures that are leading measures rather than lagging measures to influence future decisions. A leading measure uses changes in performance to provide insights into potential changes that might influence a future decision one way or another. For example, knowledge that a ramp meter has exceeded the manufacturer’s suggested service life might drive a decision to replace that meter. Similarly, increases in equipment downtime might indicate risks due to an aging fleet are growing or that planned operational activities will not be performed as planned. A lagging measure, on the other hand, looks back on the results of past investment strategies after the decisions have been made. Because a lagging measure is recorded after the fact, there is a delay (lag) in the agency’s ability to adjust its practices and improve performance. Bridge and pavement condition measures are examples of lagging measures because the reported conditions reflect the impact of decisions made several years in the past. Lagging measures are commonly used to evaluate a program’s effectiveness or to verify that actual investments achieved projected results.
In transportation, an agency might have a lagging measure for tracking complaints responded to within a 48-hour window. The measure provides an indication of the public’s satisfaction with the road network and is easy to monitor and report. However, if an agency really wants to effect change, it might develop leading measures to track the percent of complaints not worked on within a two-hour window or the percent of complaints that can’t be resolved by the initial point of contact and must be passed to someone else. Focusing on these types of measures could drive agency decisions to ensure complaints are being worked on quickly and are being assigned to the right people. General characteristics of effective performance measures are presented in Table 6.1.
North Carolina DOT
The North Carolina DOT authorizes its divisions to determine how funding will be used for maintenance activities and uses performance data to assist with this activity. Each year, Division Engineers submit annual plans detailing what work will be accomplished; these plans are reviewed quarterly with the Chief Engineer to discuss actual versus planned work. Their accomplishments are also displayed in a dashboard for internal use, as shown in the following image. Public-facing dashboards are also available showing overall conditions and performance trends. The Division Engineers are also held accountable for their performance, since their planned and actual performance data are incorporated into their annual evaluations.
Source: Leading Management Practices in Determining Funding Levels for Maintenance and Preservation. Scan Team Report, NCHRP Project 20-68A, Scan 14-01, National Cooperative Highway Research Program, May 2016.
Use of Performance Measures
Performance measures are used to:
- Connect agency policies and objectives to investment decisions.
- Establish desired and targeted levels of service that consider past performance, current and future demand, stakeholder priorities, and anticipated funding.
- Align agency policies, investments, and day-to-day practices in a meaningful and easily understood manner.
- Prioritize investment needs.
- Monitor and report progress towards desired objectives to both internal and external stakeholders in a consistent, cost-effective, and transparent manner as illustrated in practice examples from the Washington State, North Carolina, and Virginia DOTs.
Outcome-based measures better relate to performance characteristics noticed by the public and other stakeholders than output-based measures.
Table 6.1 - Desired Performance Measure Characteristics
|Measurable with available tools/data||May require no additional cost for data collection|
|Forecastable||Enables data-driven target setting based on future conditions|
|Clear to the public and lawmakers||Allows performance story-telling to customers and policymakers|
|Agency has influence over result||Measures agency activities rather than impact of external factors|
The Washington DOT uses its Maintenance Accountability Process (MAP) to comprehensively manage maintenance budgets and to communicate the impacts of policy and budget to both internal and external stakeholders. Field condition surveys are conducted annually to assess the condition of 14 assets on the highway system such as signs and signals, ITS assets, tunnels, and highway lighting. For each asset, a level of service target is established, based on expected funding levels and importance of the asset to the agency’s strategic objectives. The targeted and actual performance is summarized on a statewide basis and presented to the legislature, media, internal stakeholders, and other DOTs in a format similar to what is shown in the figure (https://www.wsdot.wa.gov/NR/rdonlyres/8EC689DF-9894-43A8-AA0F-92F49AC374F5/0/MAPservicelevelreport.pdf). In 2018, Washington State DOT achieved 77 percent of its highway maintenance targets. Targets that were not achieved are shown as red bullseyes and areas where the targets were exceeded include a checkmark with the bullseye. The results illustrate where additional investment is needed on a statewide basis and provides a basis for setting maintenance priorities during the year.
Targeted and Actual Performance Results Used to Set Maintenance Priorities
Source: WSDOT. 2017. Multimodal Asset Performance Report. Washington State DOT. https://wsdot.wa.gov/publications/fulltext/graynotebook/Multimodal/AssetPerformanceReport_2017.pdf
To support accountability, credibility, and transparency, the Washington State DOT publishes its quarterly performance report, referred to as The Gray Notebook. Each edition of the Gray Notebook presents updates on multimodal systems' and programs' key functions and analysis of performance in strategic goal areas based on information reported to the Performance Management and Strategic Management offices of the Transportation Safety and Systems Analysis Division. Washington State DOT also publishes its Gray Notebook Lite, which highlights key metrics referenced in the Gray Notebook in a format for quick reading. Examples from each of these documents are presented in the figures.
The Gray Notebook and the Gray Notebook Lite
Source: WSDOT. 2019. https://www.wsdot.wa.gov/Accountability/GrayNotebook/
Performance dashboards are also a popular way to present progress, using color-coded indicators similar to those on the dash of an automobile. An example of the interactive dashboard available from the Virginia DOT is shown in the figure. The screen reports performance in seven areas (performance, safety, condition, finance, management, projects, and citizen survey results) and the needles indicate whether the performance is within targeted ranges. Hyperlinks are available in each area if a user wants to explore historical trends or explore performance objectives in more detail.
Virginia DOT's Performance Dashboard
Source: Virginia DOT. 2019. http://dashboard.virginiadot.org/
Future Directions in Performance Measures
As agencies advance the maturity of their practices and move towards investment decisions across assets and modes (as discussed in Chapter 5), there is increasing interest in the use of leading measures and asset performance measures other than asset condition.
Asset management plans document the processes and investment strategies developed by an agency to manage its infrastructure assets. These asset management plans support an agency’s performance-based planning and programming processes for making long-term investment decisions and feed shorter-term project and treatment selection activities. Together, these activities ensure the investment decisions of an agency are aligned with performance objectives and goals.
Examples of these types of measures include:
- Financial Measures – Internationally, financial performance measures have been used successfully to express whether the level of investment has been adequate to offset the rate of asset deterioration or depreciation. For example, the Queensland Department of Infrastructure and Planning uses an Asset Sustainability Ratio defined as the capital expenditure being made on asset renewals (e.g., improvements) divided by the depreciation expense (discussed further in Chapter 4). If the ratio is less than 100 percent, the level of investment is not adequately replacing the depreciation occurring each year. Queensland also uses an Asset Consumption Ratio comparing the current value of the depreciable assets to their replacement value in order to show the aged condition of the assets.
- Life Cycle Measures – A life cycle performance measure is a relatively new leading measure, promoting the selection of sound, long-term strategies best able to maximize performance at the lowest possible cost. There are several life cycle performance measures under consideration by the FHWA, including the Remaining Service Interval (RSI), which is being validated under a research project. The RSI is based on identifying a structured sequence of the type and timing of various repair and replacement actions needed to achieve a desired LOS over a long timeframe at the minimum practicable cost. The results of the RSI evaluation may be used to generate a Life Cycle Impact Factor, summarizing the difference in life cycle costs associated with the various strategies being considered.
- Sustainability Measures – With an increased focus on identifying long-term sustainable solutions to transportation system needs, agencies may seek to develop new sustainability performance measures in order to properly indicate the impact a proposed solution may have on environmental conditions. The use of a recycling measure for gauging the amount of recycled material used in road construction is an example of this type of measure, as are measures for monitoring carbon dioxide emissions.
Making performance measures publicly available through reports, scorecards, or dashboards increases transparency into agency operations, which can serve as motivation to improve staff’s desire to meet the standards established, thereby increasing the chance of success.
North Carolina DOT
The North Carolina DOT has an interactive Organizational Performance Scorecard that provides an online indicator of the Department’s success at meeting targets in the following six core goal areas:
- Make Transportation Safer.
- Provide Great Customer Service.
- Deliver and Maintain Infrastructure Effectively and Efficiently.
- Improve Reliability and Connectivity of Transportation Systems.
- Promote Economic Growth Through Better Use of Infrastructure.
- Make NCDOT a Great Place to Work.
An example of how the information is shown; it presents the target for an overall infrastructure health index and the most recent results. As shown by the red “x” in the box on the far right, NCDOT is not currently meeting its target of a health index of 80 percent or more.
North Carolina DOT’s Organizational Performance Scorecard Website – Excerpt
Source: NCDOT. 2019.https://www.ncdot.gov/about-us/our-mission/Performance/Pages/default.aspx