6.1.2 Evaluating the Effectiveness of Performance Measures

6.1.2

Evaluating the Effectiveness of Performance Measures


Because of the important role performance measures have in supporting performance-based decisions, agencies should use care in selecting measures that drive the right types of results. This section introduces several approaches to evaluate the effectiveness of an agency’s performance measures.


Assessment

In its handbook for agency executives, AASHTO suggests an assessment of performance measures should consider the following:

  • Is the number of performance numbers reasonable? – An agency should retain performance measures addressing critical areas of importance that are maintainable with time. The Maryland and New Mexico DOTs have approximately 80 measures reviewed on a regular basis, but the Florida and Pennsylvania DOTs use approximately 15 to 20 measures to review strategic performance. Some agencies identify a small number (< 10) of KPIs selected from the pool of operational and tactical measures that best reflect an agency’s progress toward achieving its overall goals.
  • Are the measures meaningful? – Some agencies choose only to use easily measured performance activities because the information is easy to obtain. However, other measures may do a better job of driving good decision making.
  • Does the level of detail in data collection match the level of detail required to drive decisions? – Agencies should balance data availability with the analytic rigor used to make decisions. For instance, if pavement markings are replaced every year, it is not necessary to collect retro-reflectivity information annually. Similarly, collecting data on one lane of a two-lane highway may be enough for approximating the condition across the full width of the roadway.
  • Do they support the right decisions? – The performance measures should drive decisions in support of strategic objectives. For example, a performance measure based on the amount of overtime incurred after a snow event is less effective than one able to monitor the number of hours until the roads are cleared.
  • Are existing data sources reliable? – In most situations, existing data can provide the information needed for performance management, but it must be reliable and maintained regularly to be useful.

An assessment of performance measures can be important, since many organizations find that over time, the number of performance measures they are managing can become unwieldy.

Pennsylvania DOT

After using performance measures for years, the Pennsylvania DOT recognized that the number of measures being used had increased to a level that was difficult to manage. In 2011, the Pennsylvania DOT conducted an assessment of their performance measures using the following series of questions to guide their decisions as to which measures to keep, which to change, or which to delete:

  • Who is using the measure?
  • What exactly is being measured?
  • Why is this particular measure needed?
  • Whose performance is being measured?
  • Is the performance goal defined?
  • Does a similar measure already exist?
  • Is the existing measure meeting the needs and intent or should it be modified?

If a measure was needed where no measure exists, the following additional questions were used:

  • Does the measure affect continuous improvement?
  • Is the data for the measure updated as frequently as needed? Should it be updated monthly, quarterly, or yearly?
  • Is the measure easy to quantify?
  • Is the measure easy to understand?
  • Is it clear who owns the measure?
  • Does the measure provide a means of comparison?
  • Have unintended consequences been investigated?
  • Can the unintended consequences be successfully mitigated?

The process has helped to ensure that the agency is focused on the right measures to drive desired results and behaviors. The analysis found several issues that could be addressed, including eliminating duplicate or overly complicated measures, modifying measures that were driving unintended consequences, and resolving data quality issues.

SMART Evaluation

As discussed earlier, performance measures are used to set desired or targeted levels of service. Targets may be short-term, such as the 2- and 4-year targets state DOTs are required to submit to FHWA, or they may be long-term targets, such as the desired State of Good Repair (SOGR) serving as the basis for an agency’s TAMP.

Performance targets are evaluated using the “SMART” method, which evaluates whether targets are:

  • Specific. The performance is explicitly described.
  • Measurable. Progress towards the target can be monitored in a consistent manner.
  • Achievable. The target considers past performance, expected changes in demand, available resources and other considerations that make it realistic.
  • Relevant (also referenced as results-oriented). The target should be meaningful to the agency and drive the right outcomes.
  • Time-related (also referenced as timely or time-bound). There is a stated timeframe for achieving the target.

Nevada DOT

The Nevada DOT recognized that although performance measures were being reported regularly, they were not driving agency policies or decisions. The assessment evaluated the performance measures being used in each of the five key performance areas shown in the figure as well as the organizational culture to support performance management.

The study recommended improvements to emphasize the importance of messaging in order to advance the agency’s performance management culture, extend the performance culture beyond the headquarters office to field staff, and develop job performance plans emphasizing accountability at the division, office and unit levels. The study also recommended the periodic review of performance measures to ensure their continued relevance to agency business processes.

Nevada DOT’s five key performance areas and measures



Source: Nevada DOT. 2017. Adapting a Culture for Performance Management at the Nevada Department of Transportation.

Benchmarking

In simple terms, benchmarking is a process of comparing performance and practice among similar organizations as part of an agency’s continuous improvement activities. Benchmarking provides an opportunity to learn about approaches used by high-performing organizations to uncover noteworthy practices, inform target-setting activities, or to foster innovation and improvement within an agency. Benchmarking should focus on improvement and lessons learned rather than as a way to penalize underperformers.

As mentioned in Chapter 1, AASHTO has developed a comparative benchmarking tool for enabling state DOTs to compare performance outcomes and practices with peer agencies as part of their continuous improvement activities (http://benchmarking.tpm-portal.com/). This includes a peer selection tool, so agencies can compare practices to peers with similar characteristics. It also features a performance comparison tool with a number of chart options enabling agencies to compare results. For instance, an agency may elect to compare pavement smoothness characteristics with a neighboring state. There is also a portal to facilitate the exchange of practices among registered DOT users through a Notable Practice Narrative.

An example from the AASHTO TPM Portal showing a comparison of bridge deck percentage determined to be structurally deficient is shown in Figure 6.2. Similar comparisons are available for safety, environmental, and non-motorized (bicycle and pedestrian) performance measures. For transit agencies, Transit Cooperative Research Program (TCRP) Report 141, A Methodology for Performance Measurement and Peer Comparison in the Public Transportation Agency, provides specific guidance for comparing performance with other agencies.

Figure 6.2 Example Performance Comparison from the AASHTO TPM Portal



Source: TPM Portal. 2019. http://benchmarking.tpm-portal.com/compare/bridge-condition/deficient-bridges

Audits

Internationally, ISO standards include the conduct of periodic internal audits to help an agency evaluate whether its asset management program and components meet the agency’s needs, adhere to best practices and are being used to support decisions. In addition, agencies use auditing for service providers to confirm contract compliance in situations where road network maintenance and management activities have been outsourced.

TIP
The AASHTO TPM Benchmarking Tool (http://benchmarking.tpm-portal.com/) was designed to assist state DOTs with benchmarking TPM data, providing a data source and comparison tools.