Office of the Auditor General of CanadaOAG reports published in the past are available through Publications.gc.ca.
2004 November Report of the Auditor General of Canada Chapter 3—National Defence—Upgrading the CF-18 Fighter Aircraft
2004 November Report of the Auditor General of Canada
Chapter 3—National Defence—Upgrading the CF-18 Fighter Aircraft
3.1 Fourteen years will have elapsed from the time National Defence identified the need to modernize the CF-18 until Phase 1 upgrades are completed on 80 of 119 fighter aircraft in 2006. Phase 2 concludes the modernization and is scheduled for completion in 2009, after which National Defence expects to fly the CF-18 until 2017 or longer. Delays in the approval processes, the budget cutbacks of the late 1990s, and the increasing cost of maintaining existing equipment have contributed to the length of time taken before the aircraft's deficiencies could be fixed.
3.2 In Phase 1, we found some problems with project and risk management, staff shortages, and approval delays. These concerns need to be addressed so that they do not become impediments to the successful completion of Phase 2. If they aren't addressed, the final delivery of fully upgraded CF-18s could be delayed beyond 2009. The current CF-18 airframe has a limited amount of flying hours left, so the Department needs to take full advantage of its investment in the modernization by ensuring upgrades are installed and available to pilots as soon as possible.
3.3 We looked at the largest-dollar contract for each of the five upgrades and found them to be within cost. We found that the work being done on the aircraft was addressing critical deficiencies and National Defence officials were satisfied that the aircraft being delivered at the time of our audit were meeting the Department's performance expectations.
3.4 When delays and staff shortages threatened certain testing milestones, operational and technical test staff at the Department worked together to overcome those problems and meet their deadlines.
3.5 Three of the five Phase 1 upgrades are proceeding on time; two are behind schedule. One, a flight simulator training system, was to be ready for pilot training by the time Phase 1 upgraded aircraft were delivered; instead, the system is delayed by up to two years. As a result, the Department will forgo savings expected by using the old training system until the new one arrives and may see increased fatigue on the aircraft due to added flying training hours.
3.6 In order for National Defence to get full advantage of the improved operational capabilities until 2017 or longer, it must ensure that it can address existing pilot shortages, shortages of air technicians who maintain the aircraft, shortages of spare parts to keep the aircraft flying, and budgetary pressures on operational funding. Until these concerns are resolved, National Defence cannot get assurance that the $2.6 billion investment in the CF-18 will enable it to meet operational demands until 2017 or longer.
Background and other observations
3.7 National Defence is modernizing 80 of its CF-18s to fix capability deficiencies that have existed since the early 1990s. The $2.6 billion multi-year, multi-project upgrade will enable the Air Force to fly these aircraft until 2017, or longer, with improved avionics, weapons, and communications systems.
3.8 When purchased in 1980, the CF-18 life expectancy was up to 2003. However, by 1992, after deploying the aircraft to the Gulf War in 1991, the Department had concerns about several deficiencies.
3.9 With ongoing maintenance, some upgrade work, and structural fatigue life management, the Department planned to prolong the life of the pre-modernized fleet to 2010, recognizing it could continue to fly, but its capabilities would be limited. In 1998, National Defence granted internal approval to begin modernizing the CF-18 aircraft fleet through a series of incremental upgrades and modifications. These would occur between 2001 and 2009 and address critical deficiencies such as identifying friend or foe aircraft, effectively interoperating with other aircraft in joint operations, communicating on continually secure channels, and defending against jamming of its radio and radar. The number of aircraft to be modernized was based primarily on affordability. Plans for the 39 remaining aircraft were not finalized at the time of our audit. Some of these aircraft have been used as a source of spare parts.
3.10 In our 2001 Report, Chapter 10, National Defence In-Service Equipment, we reported on the availability of military equipment and looked at the performance of the CF-18. We examined abort rates, which are the number of failures per 1,000 flying hours that result in cancelled missions, and found that the CF-18 was experiencing a growing number of aborts. Aging and reduced funding combined to restrict the performance and availability of these aircraft.
The Department has responded: National Defence agrees with all the recommendations and has committed to taking action to address concerns we raise in this chapter.
3.11 The primary roles of the CF-18 aircraft are to maintain air sovereignty over Canada, help defend North America, provide tactical support for joint operations, and contribute to North Atlantic Treaty Organization (NATO) missions. Because of these roles, the aircraft needs equipment to communicate and work with allied aircraft and ground forces, defend against attacks, and provide surveillance.
3.12 The CF-18 is an aging aircraft but, according to National Defence, it can operate until 2017 or longer if upgraded and maintained. However, it must compete for funding with other equally aging military platforms. National Defence has had to balance its demands for spending with available funding.
Capital spending has continued to decline
3.13 We last reported on the need for National Defence to increase the capital portion of the defence budget in our April 1998 Report, Chapter 3, National Defence—Equipping and Modernizing the Canadian Forces. Since then, spending on capital equipment has decreased both as a percentage of overall funding and in real terms. Funds for updating or buying new equipment continue to be pressured by increasing operational costs. The Department pointed out in an internal analysis of capital spending that "the lack of stable capital funding can lead to the costly and inefficient cancellation, deferral, or smoothing of cash flows of capital equipment acquisitions."
3.14 The budget for National Defence has grown in the last few years, mainly to cover the increasing costs of operations and personnel (Exhibit 3.1). During this time, capital spending remained relatively stable at $2 billion annually but decreased from around 19 percent of the overall defence budget to 15 percent.
3.15 Addressing equipment needs. As more and more equipment ages and becomes operationally obsolete, the demand for upgrades or replacements increases. Projects are deferred when funds are not available. We reported in 1998 that the number of deferred projects was growing and outpacing the ability of National Defence to address its equipment deficiencies or modernize its capabilities. This problem is not unique to Canada. Equipment rust-out is a challenge faced by many allies.
The CF-18 Incremental Modernization Project
3.16 National Defence is upgrading 80 of its fleet of 119 CF-18s in two phases over nine years from 2001 until the planned completion date in 2009. The CF-18 Incremental Modernization Project consists of 15 projects. We examined five of the projects, which were scheduled for completion between 2001 and 2006. Of the five projects, three involve modifications to the aircraft, the fourth involves the acquisition of a network of training simulators for a fully modernized CF-18, and the fifth is a developmental project for multi-purpose colour displays in the cockpit.
3.17 Phase 1 is planned for completion in 2006 and will cost approximately $1.5 billion. We found that three of the five projects were on track. The simulator project experienced significant delays but is now underway, and the Department moved the installation phase of the multi-purpose colour displays to Phase 2 after it fell behind schedule. Without the Phase 1 projects that modernize the aircraft systems and software, Phase 2 upgrades cannot be completed. In order for Phase 2 to start on time, Phase 1 must be completed on time.
3.18 Given the number of projects, their individual complexity, and their degree of integration, both phases present difficult challenges (exhibits 3.2 and 3.3). Some Phase 1 upgrades provide the foundation for Phase 2 projects.
Focus of the audit
3.19 We focussed on the projects included in Phase 1 (2001-2006) of the CF-18 upgrade. We examined the acquisition process for three on-aircraft projects, the Advanced Distributed Combat Training System (ADCTS) project for flight simulation training, and the development of the new colour displays. The development work for a sixth project, Data Link, was added after we started the audit, so we did not examine it. We did not assess the military decision to modernize the CF-18 aircraft but rather focussed on whether the upgrades will address identified deficiencies.
3.20 We examined the work by National Defence, Public Works and Government Services Canada, and the Treasury Board of Canada Secretariat to identify needs, examine options, approve expenditures, award contracts, manage contracts, manage project risk, and oversee programs. Further information about the audit objectives, scope, approach, and criteria can be found at the end of the chapter in About the Audit.
Observations and Recommendations
Upgrading aircraft capability
Dealing with CF-18 deficiencies
3.21 National Defence identified the CF-18 deficiencies that, in its view, needed to be addressed. In our audit, we looked at whether the contracts for the CF-18 Incremental Modernization Project for Phase 1 addressed those deficiencies, which consisted of the following:
- Supportability. The aircraft industry no longer produces most of the original CF-18 avionics components. Equipping the CF-18 with modern avionics will allow the Department to maintain the aircraft into the future.
- Interoperability. Many of Canada's allies are updating their aircraft. To continue to communicate and operate effectively with allies, the CF-18 requires similar updates.
- Operational capability. The CF-18 needs upgrading to continue to perform as an effective fighter aircraft. Some of the potential threats to Canadian security are new since the CF-18 was first built.
- Survivability. Certain components that increase the survivability rate of the aircraft and its pilots are now obsolete. The Phase 1 upgrades will contribute to improved survivability.
3.22 We noted that other defence projects address some deficiencies; for example, the CF-18 Crash Survivable Flight Data Recorder is included as part of a separate Air Force-wide project.
Need to modernize first recognized in 1990–91
3.23 National Defence first recognized the need to modernize its CF-18s during the 1990-1991 Gulf War but could not get an upgrade program started until 1998 (Exhibit 3.4). In 1992, it proposed the CF-18 Mid-Life Update project to replace the following components: new mission computers, a new operational flight program, a new stores management system that enables the aircraft to use newer weapons, new communication systems, and new radar. The Department brought forward these equipment requirements again in 1994 under the new name Systems Life Extension Project to emphasize the focus on mission systems. Finally, in 1998, the Department received approval for its new strategic approach of a series of incremental projects, namely, the CF-18 Incremental Modernization Project. Although they were part of that project, the new mission computers and operational flight program projects were approved separately that year. The stores management system, communication systems, and radar were all approved as parts of the ECP-583 project in 1999.
3.24 Decision to upgrade 80 aircraft. We expected to find an analysis to support why the Department chose 80 aircraft as the number to modify. We expected that this analysis would take into account the many variables affecting the estimated useful life of the aircraft, including its expected attrition rate, age, and roles as defined in National Defence policy. We were unable to find such an analysis. However, Department officials told us that upgrading 80 CF-18s was reasonable, financially. Yet, new threats to North America in light of global events over the past two years may increase the demands on the upgraded 80-aircraft fleet and may put even greater pressure on the Air Force to manage its fatigue life, maintenance, and flying hours.
3.25 Air Force analysis indicates that modernizing 80 aircraft does not mean that 80 aircraft would be available on a daily basis. The planned allocation of the 80 CF-18s is four operational squadrons of 12 aircraft each, with the remaining 32 available for training, testing and evaluation, and depot level maintenance. Of the 48 aircraft in operational squadrons, only 70 percent, or 34, are normally mission-ready on a daily basis. With an expected attrition rate of one aircraft every two years, National Defence has recommended a review of how well the modernized 80-aircraft fleet will meet Canada's ongoing commitments, particularly in a post-September 11, 2001 environment.
3.26 Recommendation: We recommend that the Department of National Defence review the modernized fleet's ability to meet Canada's commitments, particularly subsequent to September 11, 2001.
National Defence's response. National Defence appreciates the importance of ensuring that the modernized fleet will have the capacity to meet Canada's existing commitments. The Canadian Forces and National Defence have processes and analysis tools to monitor and match resources with commitments. One tool, developed by the Department's Operational Research division, is the Air Force Structural Analysis (ASTRA) model. ASTRA is an analytical model that calculates the resources required to meet specified commitments. Employing these processes and analysis tools, and taking into account September 11, 2001 and other relevant influences, National Defence will review the capacity of the modernized fleet to meet Canada's existing commitments.
Management of individual projects
Three of five projects on schedule
3.27 Projects on time. We found that the mission computers, operational flight program, and Engineering Change Proposal 583 (ECP-583) projects were on schedule. Equipment for these projects is being installed simultaneously.
3.28 The ECP-583 is the main upgrade that replaces obsolete systems and installs better radios, radar, a better weapons management system, and a friend-or-foe identifier. It will cost about $1 billion to complete. National Defence was able to take advantage of work already done by the contractor to develop and install an ECP-583 for the United States Navy and the Royal Australian Air Force. Thus, this was mainly an off-the-shelf purchase of a known product by an experienced contractor.
3.29 The new operational flight program software runs the upgraded mission computer, which consists of the navigation computer and weapons delivery computers. Both projects are on track and on time.
3.30 National Defence is receiving CF-18s with the ECP-583, the operational flight program, and the mission computer work completed according to schedule from the contractor. As a result, the Department has been able to meet its own target to deliver two squadrons of Phase 1 upgraded aircraft that are capable of performing North American Aerospace Defense Command (NORAD) roles. At the time of our audit, the contractor had delivered 33 aircraft, upgraded to the Department's satisfaction and on time.
3.31 Projects behind schedule. However, the remaining two Phase 1 projects are behind schedule. The Department has pushed back the multi-purpose display group project to Phase 2. The project involves installing state-of-the-art colour displays to provide more and faster operational information to pilots in an easy-to-understand format. The Department is developing the project in collaboration with the Royal Australian Air Force (RAAF), which is also upgrading its F/A-18s. Under the terms of a memorandum of understanding, Canada is responsible for the development of pre-production displays. Responsibility will then transfer to the RAAF, which will lead the development and testing phases. The Department chose collaboration so that it could share costs with a partner. Department officials report that the delay will have no operational impact, and they remain confident that this joint project will fit in the Phase 2 time frame.
3.32 The Advanced Distributed Combat Training System (ADCTS) will provide simulators for pilot training. However, it is two years behind schedule and the Department now plans delivery for 2005 (see Costly delays: The Advanced Distributed Combat Training System project).
Delays pose problems
3.33 Overcoming testing delays. National Defence worked with the United States Navy and the contractor to test its ECP-583 upgraded aircraft. During late 2002, the Department faced delays getting a prototype aircraft ready for testing. A Combined Test Force, composed of operational and technical test staff, worked around staff shortages and an absence of baseline data with which to compare results. It kept expertise together and enabled test personnel to share resources to ensure that the work could get done. It allowed test personnel to complete all necessary test steps and evaluate the results. Because of this group, the impact of the delay in receiving the prototype aircraft was minimized.
3.34 Simulator delays. National Defence began working on the Advanced Distributed Combat Training System project in late 1999 and planned to begin using the simulators in April 2003 to assist with pilot training for upgraded CF-18s as the aircraft came into service. However, the project fell behind schedule when it went before a series of departmental project committees, each with its own mandate and review processes, for approvals. The project received final departmental approval in November 2001.
3.35 In May 2002, the government approved the $200 million purchase of ADCTS. The purchase included ten portable "part-task" simulators and six full simulators for pilot training on the upgraded CF-18s. The Request for Proposal process began in September 2002. Despite the need for the simulators, the contracting process took longer than expected, and it was not until March 2004—almost two years after project approval—that the contract was awarded.
3.36 Because of these delays, National Defence was expecting to receive the first of the part-task simulators in September 2004 and the full simulators in September 2005—more than two years after pilots began flying modernized CF-18s.
3.37 Plans to improve the acquisition process. The project approval process is one of many components of the acquisition process. CF-18 project staff indicated that the main challenge to staying on schedule was that process, over which they have no control.
3.38 National Defence has targeted the lengthy acquisition process for reform. One of the Department's main plans to shorten the acquisition process is to streamline the approval process and make it more responsive and effective.
3.39 A 1998 department study revealed that most capital equipment projects took, on average, 16 years from concept to project completion. The Department acknowledged that this is an unacceptable length of time and committed to shortening the process by at least 30 percent—to 11 years. In December 2003, the Department developed a new project approval process to reduce internal approval times and in 2004 was working on an implementation plan.
Managing the acquisition process
Testing and evaluating results are acceptable
3.40 According to National Defence, the primary purpose of testing and evaluation is to identify, understand, and manage the technical and performance risks associated with equipment design, manufacture, and in-service support. The director of technical airworthiness is the delegated technical airworthiness authority under the Aeronautics Act and ensures that the airworthiness program achieves an acceptable level of aviation safety from a technical perspective. We reviewed the Department's testing and evaluation processes to identify whether it carried them out according to its policies. We did not audit whether the testing and evaluation processes were comprehensive and accurate; rather, we reviewed the test plans and results with the appropriate officials and found that the Department was satisfied that the aircraft was operationally acceptable.
3.41 National Defence conducted two types of testing and evaluation: engineering and operational. Engineering verified the technical airworthiness of the new design and that the modified aircraft met performance expectations. Operational verified the operational effectiveness and suitability of the changes made to the test aircraft. Although the start of testing and evaluation was delayed by three months due to the late completion of the test aircraft, the testing and evaluation team met their milestones, as described earlier.
3.42 Test engineers found some problems with the new software and the radios. The Department was investigating these problems at the time of our audit. National Defence officials told us the Director of Technical Airworthiness was examining the engineering test and evaluation trouble reports and had provided a provisional technical airworthiness clearance. This clearance signifies that there are no significant airworthiness issues and is provided pending resolution of the remaining issues. The project staff and technical airworthiness staff were working toward full technical airworthiness clearance in the fall of 2004.
3.43 National Defence told us that it is satisfied with the performance of the modernized aircraft now in service.
Department followed contracting policy and procedures
3.44 Contracts. We examined the largest-dollar contract in each of the five projects. All are within costs and the payments are on schedule. Contract payments are based on the delivery of aircraft, spare parts, documentation, hardware, and training that have been delivered by the contractor according to either milestones or on a time-and-material basis, as required by the contract.
3.45 For the ECP-583, the government authorized $152 million for training, spare parts, contract amendments, and other costs. Approximately $51 million in contract amendments had been approved at the time of the audit, mainly for maintenance training. Expenditures are well within the $152 million.
3.46 We reviewed whether the five contracts complied with trade agreements by examining whether the sole-sourced procurements in four of the five contracts were exempt from the North American Free Trade Agreement and the World Trade Organization rules. We did this by comparing the contract requirements against the lists of goods from the trade agreements that may not be sole sourced. In all four contracts, the decision to sole source was based on the ownership of the technical rights associated with the procured goods. Finally, for the lone competitive contract we ensured that the steps taken in the contracting process met those established in the Public Works and Government Services Canada (PWGSC) Supply Manual.
3.47 We verified that PWGSC had posted an Advanced Contract Award Notice to ensure that potential bidders had the opportunity to challenge the decision to sole source the contract. We checked a sample of contracts and amendments and verified that the Department had followed the signing authority requirements. We reviewed exceptions made to the standard limitation of liability contract clause and found that PWGSC had complied with the associated requirements of Treasury Board risk management policy.
3.48 In November 2003, the original ECP-583 sub-contractor completed the sale of its division that had been responsible for the ECP-583 installations. We found that PWGSC demonstrated due diligence by ensuring that the assignment of the sub-contract to the new sub-contractor did not result in additional technical, financial, and legal risks to the government of Canada.
3.49 Fairness monitor documentation. PWGSC engaged a fairness monitor to review the solicitation and associated documents and to monitor the procurement process for the Advanced Distributed Combat Training System. We asked the Department to provide us with the fairness monitor's working papers and were informed that they did not have any. PWGSC explained that it does not require working papers because assurance is derived from the ongoing participation of the fairness monitor in the bid evaluation process and is supported by their professional qualifications and experience. However, in the absence of such documentation, we are unable to determine the level of reliance that we, or the Department, can place on the fairness monitor's observations.
3.50 Status reports. We also found that the requirement for reporting to senior management on contract status was not standardized. Of the five contracts, there were monthly progress reports for only two. Of the remaining three, one contract had not been signed yet, but we were told that reports would commence when the contract was awarded. We expected to find regular status reporting on more than two of the four signed contracts because of the interdependencies of the upgrades. However, PWGSC officials told us that the reporting requirement for each contract is "based on contract complexity and political sensitivity."
3.51 Despite a few concerns, we are satisfied that the Department has carried out its work in the contracting areas we examined in all material respects and in accordance with the Supply Manual.
Better project management required
3.52 Better use of project management tools is necessary. Because the CF-18 modernization is technically complex, very costly, and has many interdependent activities, we expected to find a mature project management process in place. However, we found that National Defence needs to better use project management tools to track progress, monitor performance against milestones, and determine appropriate staff resources.
3.53 The Project Management Plan, produced by project staff, is a key document in National Defence's project management framework. The Department uses it to guide both project execution and project control. The Master Implementation Plan, produced by operational staff, directs modernization activities and guides the implementation of the modernized CF-18 into service with minimal impact on operations. These important documents provide an overall plan of schedules and deliverables, how the work will be done, performance expectations, responsibilities, training, communications, and certain financial aspects of the CF-18 modernization. We found that, although some of the individual elements were produced in different forms, a Project Management Plan was not produced due to a personnel shortage and the Master Implementation Plan remained in draft form until February 2004.
3.54 The project staff use project management software to maintain a CF-18 master schedule to monitor individual project schedules. However, the master schedule contains only the major activities for each of the individual projects; thus, its usefulness at the working level is limited. In addition, the master schedule does not identify baseline dates for project milestones and so deviations from the schedule cannot be tracked, measured, or reported to senior management.
3.55 The Treasury Board Secretariat has issued project management guidelines to departments to encourage good management practices. Departments need to use a suitable database system to track key objectives and numerical information, such as deviations from the schedule, cost, and scope objectives. National Defence has a Capabilities Initiatives Database to track how well projects are performing, but we found the information it contains is not always reliable and is easily changed to reflect actual rather than expected performance. As a result, it is not possible to measure whether a project is meeting expectations or if it needs help. Senior management cannot rely on the database to determine if projects are meeting milestones or where delays are occurring.
3.56 Recommendation. National Defence project management staff should prepare a project management plan that clearly indicates the critical path among projects and project activities to ensure the reporting of reliable project information to senior management and the appropriate application of resources to meet Phase 2 target completion dates.
National Defence's response. National Defence agrees. As part of Phase 1 of the CF-18 modernization, elements of an overall project plan were prepared and used to guide the project. The project team is now working on a consolidated and expanded master document. This document should serve to further enhance the management of the project as well as improve oversight by senior departmental officials.
3.57 Staffing limitations. The Department limits the number of military staff who can work on equipment acquisition projects to about 460, rather than matching the number of staff needed for active projects. The Assistant Deputy Minister (Materiel) organization is concerned that this number of staff is too low and may be putting equipment projects at risk. The Assistant Deputy Minister (Materiel) has started a study to fully understand skill and capacity gaps. The results of the study will be used to determine whether the ceiling should be raised.
3.58 Because of limits on staffing, the current military staffing system cannot ensure that project offices receive the right people, with the right skills, at the right time, to achieve optimal project management delivery. Project managers have no assurance that vacancies, even critical ones, will be filled by qualified candidates. The military staffing system depends upon the availability of personnel with the technical skills required by a particular project. For example, aerospace engineers were much in demand at the time of our audit; as a result, there were not enough experienced engineers to meet operational and project demands—including the demands of the CF-18 modernization.
3.59 Internal Department reviews in 2001, 2002, and 2003 note the lack of experienced staff for acquisition projects and enough staff overall. We found that staff shortages have existed since the CF-18 modernization began. We found only about half of the positions staffed, and the available staff did not always have the experience or skills needed. In order to cope, the Department has hired contract employees to fill voids.
3.60 On occasion, project staff assigned to work on Phase 2 were reallocated to help with Phase 1 in order to keep Phase 1 on track. Early on in the project, project staff expressed concerns to senior management that the CF-18 modernization was short staffed, and that the shortage would jeopardize Phase 1 and future work. Senior project staff report that the staffing situation may become worse for Phase 2.
3.61 Training required. Project management experience is not common and we found that about 80 percent of the CF-18 project staff arrived with little or no project management experience. Internal reports to the Assistant Deputy Minister (Materiel) group identified the lack of experienced staff as a serious problem facing many projects. Even though the Department's acquisition project offices are staffed mainly by military members, there is no long-term training path for developing project manager or director skills. National Defence needs a project management progression path so that staff can learn skills and be ready to apply them to large, complex projects such as the CF-18 modernization, rather than spending much of their project time learning about this. Staff could start by working on smaller projects to gain this experience and demonstrate their capacity to progress to larger, more complex projects.
3.62 Recommendation. National Defence should examine its support for large and complex projects to ensure that it is not limiting success by failing to provide skilled and experienced people with knowledge of good project management practices.
National Defence's response. National Defence agrees. The Assistant Deputy Minister (Materiel) continuously reviews and updates current project management training courses and, in addition, provides support to individuals who seek to acquire higher education and qualifications on their own initiative.
The Department also maintains a Project Management Office staffed by a limited number of project management specialists. The purpose of this office is to support capital projects, particularly new projects, and to provide assistance during prolonged absences of assigned project management personnel due to training, postings, or job rotations. Providing specialist support in the early phases of large and/or complex projects helps bridge the learning curve faced by newly assigned project management personnel.
The Department will further undertake to resource positions critical to the successful management of the CF-18 modernization. The Department continues to work to ensure flexibility of assignment of expert resources between projects as well as ensuring that common expertise is appropriately consolidated. In this respect, the Department's Chief of Review Services has been asked to perform an independent study and to make recommendations pertaining to the organization and allocation of project management resources and expertise.
National Defence and Public Works and Government Services Canada need to improve their risk management
3.63 Risk management plan only recently produced. We are concerned that a risk management plan was not developed at an earlier stage in the project to identify and manage risks to the successful completion of the CF-18 modernization. National Defence is at an early stage of risk management activity and only recently drafted its plan to track and manage risks.
3.64 Relying on the contractor. Public Works and Government Services Canada and the Treasury Board Secretariat relied on National Defence for the assessment of risk at the funding approval stage. National Defence, in turn, relied on the contractor who had identified their risk exposure as low. Therefore, all three organizations used a low risk rating for the CF-18 upgrades. As noted in the Treasury Board policy on project management, we expected National Defence and Public Works and Government Services Canada to be able to provide us with evidence to support the consultative process used to establish the government's low risk rating. For instance, we expected National Defence to have identified critical risks that could affect the project schedule, such as lack of staff, technical difficulties, or delays in validating and verifying the aircraft for acceptance and airworthiness. Similarly, we expected PWGSC to be able to provide us with documents to support the risks it had identified and how they had been mitigated through the various clauses within the related contracts.
3.65 We also expected to find a PWGSC risk management plan to address contractual risk, but the Department told us that it identified and dealt with potential risks on a continuing basis. The Department explained that a risk management plan must be produced by the contractor; therefore, the plan provided by the contractor is the formal government document.
3.66 Criteria. We evaluated the risk management practices at National Defence and PWGSC through two sets of criteria. The first set was made up of six risk management themes drawn from the Project Management Institute. The Treasury Board had adopted these themes as department policy for ensuring the use of sound project management principles, including risk management practices. The second set of criteria is made up of federal and departmental risk management policy requirements that are specific to the entity audited. We gathered the results of these evaluations through interviews and document reviews.
3.67 We found that while National Defence and PWGSC met some of the criteria for good risk management, none were met fully. Both departments need to improve risk management in each of the six areas shown in Exhibit 3.5.
3.68 Treasury Board Secretariat oversight. Treasury Board Secretariat officials informed us that they oversee the CF-18 modernization to ensure the integrity of the expenditure process. They stated that they also ensure that National Defence identifies and assesses project risks and puts in place measures and strategies to deal with risks, but they don't manage the risks associated with individual projects.
3.69 The Treasury Board Secretariat did not provide us with evidence to support its review of National Defence's identification and assessment of project risks for the ECP-583 project. Secretariat staff told us that they attended the National Defence ECP-583 briefings and that this demonstrated due diligence in its oversight role. While attending National Defence briefings does represent some oversight, we also expected the Secretariat to provide evidence supporting the other work they do in carrying out their oversight role.
3.70 We found that the availability of such supporting evidence started to increase in 2003. We particularly note that the government recently approved a CF-18 project on the condition that National Defence submit annual reports. Annual reports for all CF-18 projects are now to highlight progress on milestones and any changes to scope, and they should include an updated risk assessment. We view this as a positive step.
Impact on Phase 2
Risks to Phase 1 threaten timely completion of Phase 2
3.71 In Phase 1, the contractor for the major project—the ECP-583—had already successfully installed and tested it on the F/A-18 fleets of two other nations. National Defence was able to take advantage of a ready-to-go package for the integration of mature avionic systems that the contractor had developed for the U.S. Navy. Phase 1 required National Defence to address unique technical modifications, co-ordinate testing, and develop integrated logistic support, among other things.
3.72 Phase 2 involves the installation and integration of systems that have not been in production as long as the Phase 1 systems. The ECP-583R2 consists of the integration and installation of Data Link, multi-purpose displays, helmet mounted displays, and upgrading of the Counter Measures Dispensing System (Exhibit 3.2). The CF-18 software will also be upgraded. At the time of the audit, work was being done on Data Link and the multi-purpose displays to integrate them with the CF-18s. In order for the contractor to start the Phase 2 production line as the Phase 1 line finishes, Phase 2 was following an aggressive schedule.
3.73 Department officials told us that the risks for the ECP-583R2 could be higher than the ECP-583, but National Defence still assessed it as low. Project staff were developing plans to manage risk. The departmental approval process for the ECP-583R2, originally planned for fall 2004, was delayed. The Department now anticipates awarding the contract in early fall 2005. If approval for the ECP-583R2 is further delayed, there is little room to adjust to keep the modernization on track.
3.74 Funding issues must be resolved. Originally, a new Defensive Electronic Warfare Suite, which consisted of a counter measures dispensing system, a radar warning receiver, and an electronic pulse jammer, was to be installed on the CF-18. The full Suite would equip the CF-18 for operations in a high-threat scenario. However, at the time of the audit, only the counter measures dispensing system portion of the Suite was included as part of the ECP-583R2 in Phase 2. The radar warning receiver and the jammer did not proceed through the approval process due to funding issues. Defence officials told us that the CF-18s would not be deployed to a high-threat scenario without a full Suite, subject to government direction. National Defence plans to have a full Suite in its CF-18s by 2009.
National Defence needs to resolve resource issues
3.75 We found that there are concerns outside the scope of the CF-18 upgrades that could threaten the overall success of the modernization. They need to be addressed to ensure full use of the aircraft until the end of its useful life.
3.76 Technicians. National Defence is experiencing a shortage of skilled aircraft technicians. Because of the lack of technicians, maintenance work is taking longer, which means that aircraft turnaround times have increased. The longer the turnaround time, the fewer the possible sorties and available flying hours for pilots in training. Demographic indicators show that this problem may get worse as older technicians retire, leaving more inexperienced technicians to maintain the aircraft.
3.77 To ensure that aircraft do not degrade because of a lack of proper maintenance, National Defence must have the technicians and the spare parts available when needed. It was not clear at the time of this audit what the projected use of spare parts would be for the modernized CF-18 and whether there were sufficient spare parts to meet demand. The Department has begun developing a Logistics Management Plan to forecast requirements.
3.78 Pilots. Pilot shortages are a challenge that we reported on in the past. We were informed that because of pilot shortages last year, the Air Force was not able to use all of the available flying hours for the CF-18s. To ensure that the 80 upgraded aircraft are used fully, National Defence must have enough trained and available pilots to fly the missions. Pilot shortages are felt particularly at the training units, which often suffer first when there are not enough experienced pilots for both squadron operations and training unit demands.
3.79 The Department caps yearly flying hours at 182.7 per pilot, which is sufficient to maintain proficiency for a medium-threat scenario. Pilots no longer train for high-threat scenarios, which require about 240 flying hours per year. With the introduction of simulators in 2005, training personnel envision that additional operational training objectives may be met.
3.80 Funding. The ability of National Defence to sustain and support established CF-18 operational capability and capacity in the future may be affected by the resources available. National Procurement funding is used for spare parts and repairs and overhaul contracts; however, funding uncertainties have an impact on the ability of the Department to plan. The spring 2004 Aerospace Management Committee indicated that there is a projected gap of $100 million between the Air Force funding demands and available National Procurement funds. As well, the Committee projects a decrease in National Procurement funding such that by 2007–08, available funds will be about 65 percent of identified demand. This may introduce uncertainty and volatility in the Department's ability to maintain and continue flying the CF-18s to meet operational commitments.
3.81 Recommendation. National Defence should ensure that sufficient funding is allocated to support the CF-18 and that a sufficient number of trained technicians and pilots are available to maximize the value of the investment in the CF-18 modernization.
National Defence's response. National Defence accepts the recommendation and will continue to analyze resource requirements on an annual basis to optimize resource allocation. The CF-18 fleet will be allocated a portion of National Procurement funding based on forecast maintenance and support in order to optimize utilization and achieve value for money relative to the costs of the modernization project. Furthermore, we recognize the importance of ensuring that the value of the investment in the CF-18 modernized fleet is maximized. To that end, additional funding has been allocated to increase the number of technicians produced by our training process and, as a longer-term solution, an initiative entitled Air Technician Training Renewal is commencing.
The production of a pilot is a complex and multi-faceted process that includes recruiting; training at the basic, advanced, and operational levels; and combat ready training at an operational squadron. At the end of 2003, a multi-million dollar recruiting campaign was directed at, among other things, potential pilot candidates. In conjunction with this initiative, the Air Force is currently in the process of developing better selection tools to improve the likelihood that potential candidates will be successful during their training. In addition, new measures are also under consideration with a view to retaining more of our qualified pilots.
3.82 Phase 1 of the CF-18 Incremental Modernization Project is the result of a process that reasonably links mission requirements to the upgrades undertaken. The five projects we reviewed were all progressing within cost; however, two of the five projects were behind schedule. The upgraded aircraft were being delivered on schedule and, according to National Defence, were meeting its performance expectations.
3.83 Public Works and Government Services Canada has managed the critical parts of its contracting responsibilities. However, it could better document the work it carries out as the government's contracting agent.
3.84 Although we found it difficult to obtain the documentary evidence to support the Treasury Board Secretariat's oversight role in the early stages of Phase 1, since 2003, when evidence concerning the performance of its oversight role became more readily available, we concluded that the Secretariat was carrying out an oversight role.
3.85 Improvements to project and risk management are needed before National Defence continues with the Phase 2 upgrade projects. There are impediments to the successful outcome of Phase 2 that can be addressed by a review of staffing, better project control through planning and reporting, a better understanding of risk management, and implementing the risk management plan it now has. If these areas are not addressed, the Department is putting at risk its ability to get full value from its $2.6 billion investment in the CF-18.
3.86 There are also factors outside the direct scope of the CF-18 modernization project that need to be addressed by the Department to ensure that it attains the full investment value of upgrading the CF-18s. The Department needs to address the personnel shortages that threaten its ability to fly the CF-18s and keep them properly maintained. As well, it needs to resolve support funding issues.
About the Audit
The overall objective of our audit was to assess whether the CF-18 Incremental Modernization Project is being effectively managed.
To do this, we addressed whether
- National Defence effectively identified a valid capability need;
- the Department managed the project within performance, cost, and schedule expectations;
- Treasury Board of Canada staff provided adequate initial scrutiny of submissions and whether their monitoring and oversight, given the resulting projects, match the risk, materiality, and technical complexity of the projects;
- Public Works and Government Services Canada managed the contracting portion of the defence acquisition effectively such that the contracting process complies with government policy and contracting regulations and that for processes under its control, the project has remained within performance, cost, and schedule expectations; and
- the project met or will meet the standards associated with effective planning, including training, maintenance, materiel support, and risk management.
Scope and approach
We carried out our audit primarily at National Defence headquarters and included visits to the 4 Wing fighter base in Cold Lake Alberta, 1 Canadian Air Division, and the modernization facility in Mirabel, Quebec.
The audit team interviewed personnel from National Defence, PWGSC, and the Treasury Board Secretariat. We examined department files, relevant documents, and reviewed the experiences of other countries who modernized their fighter aircraft.
When we planned this audit in the summer of 2003, the sub-projects and projects contributing to the modernization of the CF-18 consisted of two phases, Phase 1 (2001–06) and Phase 2 (2004–09).
The five Phase 1 projects:
- mission computers
- operational flight program
- Engineering Change Proposal 583
- multi-purpose displays group units
- Advanced Distributed Combat Training System
The criteria for the audit included the extent to which
- National Defence conducted adequate requirement and option analyses to ensure that it is acquiring or upgrading a valid capability need;
- the Department undertook risk analysis to identify and manage risks;
- test and evaluation processes are sufficient to ensure that the capability delivered meets the original specifications;
- National Defence and PWGSC followed government contracting policy and regulations;
- equipment implementation plans are comprehensive enough to ensure the most efficient and effective introduction of the equipment into operations, including optimal equipment availability, support, maintenance, and training throughout the equipment's life cycle; and
- National Defence gave due consideration to, and introduced, acquisition reforms where appropriate at all internal levels as they related to the selected project.
Assistant Auditor General: Hugh McRoberts
Principal: Wendy Loschiuk
Director: David Saunders
For information, please contact Communications at (613) 995-3708 or
Avionics—The onboard electronics used for piloting aircraft. (Back)
Sole source contracts—There are four instances when a government department may sole source a contract instead of asking for bids from more than one supplier. The Treasury Board allows for sole sourcing in an emergency situation, when the estimated cost is below a certain amount, when soliciting bids would not be in the public interest (for example, for security reasons), and when only one company or person is capable of performing what is required by the contract. The Treasury Board's Contracting Policy states, "This last exception . . . should be invoked only where patent or copyright requirements, or technical compatibility factors and technological expertise suggest that only one contractor exists." (Back)