RDT Evaluation

RDT Evaluation

The Department continually assesses its RD&T programs to ensure their effectiveness and performance. This section of the RD&T Annual Funding Report discusses DOT's RD&T evaluation strategy, which is depicted in Table 3.

As shown, this broad strategy comprises three principal mechanisms: (1) systematic application of the Administration's R&D Investment Criteria and Program Assessment Rating Tool (PART); (2) annual internal review of operating administration RD&T programs; and (3) external RD&T coordination and review.

R&D Investment Criteria and PART

To guide the planning and management of research across the government, the Administration has established three investment criteria for RD&T: relevance, quality, and performance. The criteria incorporate established best practices for research evaluation as identified by the National Academy of Sciences, Government Accountability Office, and others. Each criterion has both prospective and retrospective elements:

  • Relevance. Programs must have complete plans, with clear goals and priorities; must articulate their potential public benefits; and must be relevant to national and customer needs.

  • Quality. Programs must use clearly stated, defensible methods for awarding funding; those allocating funds through means other than a competitive,
    merit-based process must document how quality is maintained.

  • Performance. Programs must maintain long-term objectives, with annual measures and targets, and define appropriate outputs, outcomes, schedules, and decision points.

The PART contains questions to assess how well agencies are implementing the investment criteria. On the basis of the PART, programs are rated as Effective, Moderately Effective, Adequate, Ineffective, or Results Not Demonstrated. To date, DOT RD&T programs have been assessed through the PART in FAA, FHWA, FRA, FTA, NHTSA, and PHMSA. Each program has been rated Moderately Effective or better and is implementing all PART recommendations.

Annual Internal Program Review

Within the Department, the primary mechanism for ensuring implementation of the R&D Investment Criteria and PART is the annual review of modal RD&T programs. These internal reviews enable program managers to (1) continuously improve program management and performance; (2) identify and share best practices; (3) identify opportunities for leveraging RD&T resources and for crossmodal research; and (4) prevent duplication. The reviews are conducted by RITA through the Program Review Working Group of the DOT RD&T Planning Council and RD&T Planning Team. This working group represents each of the Department's research-performing administrations and OST.

The Program Review Working Group assessed all modal RD&T programs in FY 2005 and FY 2006 and is undergoing a new round of reviews in FY 2007. Thus far, the working group has found that all operating administrations are implementing the R&D Investment Criteria and that there is no unnecessary duplication. The working group has also identified a number of areas for greater collaboration, including geospatial technologies (applications for safety monitoring, traffic control, and incident management); freight capacity (intermodal RD&T and enhanced freight movement); security (system approach to security issues); alternative energy technologies (climate change and alternative energy sources); and sensors and materials (advanced sensor technologies, non-destructive evaluation, and enhancement of material properties).

RD&T Evaluation Best Practices

Among the purposes of the annual review process is to ensure that RD&T programs are evaluated according to established best practices. Beginning in FY 2007, annual program reviews will focus on how well operating administrations are implementing the practices summarized in Table 4.

The Program Review Working Group will assess operating administrations' adherence to these practices using a set of formal RD&T evaluation guidelines and agreed-upon implementation criteria. The schedule for reviews in FY 2007 is as follows:

  • FAA, FHWA— February 28, 2007

  • FRA, FTA—March 20, 2007

  • FMCSA, NHTSA—April 17, 2007

  • OST, PHMSA, RITA—May 23, 2007

As in previous years, RITA will report the results of the reviews to the RD&T Planning Council. Results will be published in next year's RD&T Annual Funding Report.

External Coordination and Review

A critical element of each of the best practices cited above is systematic consultation and engagement with external stakeholders and experts. Such activities avoid duplication, uphold the technical quality of research, and ensure that RD&T programs are wise public investments that address critical needs.

Within the operating administrations, external coordination and review are essential for establishing RD&T priorities, programmatic activities, and performance metrics. Table 5 summarizes operating administrations' RD&T evaluation processes, the results of their most recent evaluations, and a schedule for reviews in FY 2007.