This Web page has been archived on the Web.

2003 November Report of the Auditor General of Canada Chapter 7—Human Resources Development Canada and the Canada Employment Insurance Commission—Measuring and Reporting the Performance of the Employment Insurance Income Benefits Program

2003 November Report of the Auditor General of Canada

Chapter 7—Human Resources Development Canada and the Canada Employment Insurance Commission—Measuring and Reporting the Performance of the Employment Insurance Income Benefits Program

Main Points

Introduction

Observations and Recommendations

Service and program effectiveness

Service to Canadians

Program effectiveness

Conclusion

About the Audit

Appendix A—Principal changes to Employment Insurance, 1971-2003

Appendix B—Access to Employment Insurance income benefits: Two approaches

Exhibits:

7.1—Income benefits statistics, 2002-03

7.2—The growth of the surplus credited to the Employment Insurance Account

7.3—Administrative costs of Employment Insurance

7.4—Percentage of calls when the caller got a busy signal when trying to speak to a service representative

7.5—Speed of payment—initial and renewal claims

7.6—Percentage of files that meet all quality objectives of Human Resources Development Canada, April-August 2002

7.7—Timeliness in scheduling appeal hearings with the Board of Referees

7.8—Example of lack of balance in effectiveness reporting

Main Points

7.1 The Employment Insurance (EI) Income Benefits program provided $12.7 billion in benefits to Canadians for temporary income replacement in 2002-03. We looked at how program performance was measured and reported from two perspectives: service to Canadians and effectiveness.

7.2 Human Resources Development Canada (HRDC) puts considerable effort into measuring the performance of the EI Income Benefits program and has a good deal of information with which to manage it. However, aspects of performance are not measured or are not measured well. For example, the key measure for call centres does not give the whole picture of access to this service. It did not take into account that 65 percent of calls in 2002-03 resulted in the caller getting a busy signal when trying to speak to a service representative.

7.3 The available performance information tells a mixed story. In 2002-03, over 95 percent of total payments to claimants were correct, and a 2001 opinion survey showed that a majority of respondents were generally satisfied with the service they received. However, performance of key aspects of service varied considerably among regions—in particular, access to call centres and the quality and timeliness of the processing of claims. Service in some regions was significantly and chronically below performance targets. The numerous efforts to improve performance in meeting service targets have made only a small difference.

7.4 The Canada Employment Insurance Commission and HRDC have not yet provided Parliament with a comprehensive and clear picture of the impact of the 1996 changes to the Employment Insurance Act. While many effectiveness issues have been evaluated, certain key ones have not. Although Parliament is provided with more information on the EI Income Benefits program than is required of most programs, we identified issues that still need to be addressed when reporting to Parliament. Reporting of evaluation results is often selective. The Commission and HRDC have not clearly reported what savings have resulted from the changes to the Act in 1996. Nor have reports to Parliament described other important issues, such as the uneven service across the country for processing claims and what HRDC plans to do about it.

Background and other observations

7.5 Over the past 60 years, the EI Income Benefits program has been the main income security program for working Canadians. The principal objective of this program is to provide temporary income support to insured Canadians who involuntarily lose their jobs. Over the years, the program has evolved, recognizing that workers face other employment risks related to childbirth, parenting, and illness.

7.6 Employment Insurance is paid for by workers and their employers. Over the past few years, the amount of premiums they paid has exceeded the amount of benefits paid out, and a surplus of $43.8 billion is now credited to the EI Account. Our Office has brought the size of the surplus to Parliament's attention every year since 1999. The administrative costs of the program were $1.5 billion in 2002-03.

7.7 The Minister of Human Resources Development is responsible for the program. The Canada Employment Insurance Commission has specific responsibilities under the Employment Insurance Act. HRDC helps the Minister and the Commission, and manages and delivers the EI Income Benefits program across the country.

Human Resources Development Canada and the Canada Employment Insurance Commission have responded. The Department and the Commission agree with our recommendations and have indicated in their responses the actions they have planned or that are underway to address the recommendations. An overall response is also provided at the end of the chapter.

Top

Introduction

7.8 The Great Depression of the 1930s meant hardship for millions of Canadians. It was a time of mass unemployment and seriously depressed standards of living. In 1933, at its worst, nearly one quarter of the country's labour force was out of work, and an estimated 15 percent of the population was getting some form of relief. At first, the needs of the unemployed were met through improvised community relief efforts, supplemented by private charities. These measures were followed by government-funded make-work projects, then relief through the use of vouchers, and finally, direct cash payments.

7.9 Among the many changes brought by the Depression was a shift in public policy regarding the jobless. Many Canadians knew what it meant to face the risk of unemployment due to forces entirely outside their control. Unemployment was a common threat and one against which collective action could and should be taken. By the end of that decade, unemployment was no longer considered solely an individual problem and a local responsibility. However, converting these public attitudes into a national program required a constitutional amendment. A 1935 effort by the government of R.B. Bennett to establish a federally run employment insurance scheme was declared by the courts to be outside the federal government's jurisdiction.

7.10 The May 1940 Speech from the Throne set out the federal government's priorities of industrial stability, justice, and social security. The government announced its intention to seek an amendment to the British North America Act to give it the authority to establish a national program of unemployment insurance. The Act was subsequently amended, clearing the way for a federal unemployment insurance scheme. In the House of Commons' debate, the Minister of Labour stated that the fundamental purpose of Unemployment Insurance was to promote the economic and social security of Canadians by supporting workers between jobs. In August 1940, the Unemployment Insurance Act was given Royal Assent.

7.11 Unemployment Insurance came into being during the high-employment era of World War II and helped the federal government manage the transition to a postwar economy. It began with a focus on the income support needs of regular, full-time, year-round workers who might find themselves without work temporarily. Over the next 30 years, it expanded to cover almost all employed workers. It became an important vehicle for the federal government to address a wide range of income support issues. While retaining its primary focus, it has been adapted to provide more active forms of support, such as training. Appendix A shows the principal changes to the EI Income Benefits program from 1971 to 2003.

An overview of Employment Insurance in 2003

7.12 Employment Insurance provides two general types of benefits: income benefits and employment benefits.

7.13 Income benefits. Under Part I of the Employment Insurance Act, income benefits are a temporary replacement of income to claimants while they look for work. This provision includes work-sharing agreements for temporary work shortages to allow employees to receive pro-rated EI benefits while working for part of a week, thereby avoiding layoffs. Employment Insurance also provides three types of special benefits: maternity benefits, payable to biological mothers for work missed as a result of pregnancy and childbirth; parental benefits, payable to both biological and adoptive parents for the purpose of caring for a newborn or adopted child; and sickness benefits, payable to claimants who are too ill to work. Exhibit 7.1 shows key statistics for income benefits for 2002-03.

7.14 Various approaches are used to measure the extent to which Canadians have access to EI income benefits. Appendix B shows how two of these approaches are calculated. While workers in an employer-employee relationship are required to contribute to Employment Insurance, the receipt of benefits in the event of becoming unemployed depends on meeting the requirements of the Act.

7.15 Employment benefits. Part II of the Employment Insurance Act provides a set of employment benefits and support measures tailored to meet the needs of individuals and local circumstances. The Government of Canada has labour market development agreements with the governments of most provinces and territories. These enable provincial and territorial governments to assume direct responsibility for the design and delivery of benefits or to take part in co-management arrangements with the federal government. The Act allows the government to spend up to 0.8 percent of the total estimated insurable earnings on employment benefits. About $2.2 billion (0.6 percent of total insurable earnings) was spent on employment benefits in 2002-03.

Program delivery

7.16 The Minister of Human Resources Development is responsible for the EI Income Benefits program. The Minister is assisted by the Canada Employment Insurance Commission, which has specific responsibilities under the Employment Insurance Act. The Minister and the Commission, in turn, are assisted by Human Resources Development Canada (HRDC). The Department manages and delivers the EI Income Benefits program across the country.

7.17 The administration of the EI Act is split between HRDC and the Canada Customs and Revenue Agency (CCRA). HRDC and the Commission are responsible for applying the benefit provisions. The CCRA is fully responsible for collecting EI premiums from employers and employees; collecting benefit repayments from claimants whose income for the year is higher than the threshold, through the assessment or reassessment of their income tax returns; and providing decisions on insurability under the Act. A Memorandum of Understanding between HRDC and the CCRA lists detailed services to be provided to and paid for annually by HRDC.

7.18 Under the EI Act, the Canada Employment Insurance Commission has numerous responsibilities for delivery of the EI Income Benefits program. The Commission has four members. The Chairperson and Vice-Chairperson (the Deputy Minister and Associate Deputy Minister of Human Resources Development Canada) represent the interests of the federal government. The commissioners for workers and employers represent the interests of workers and employers, respectively. The Commission has responsibility for the following areas under the Employment Insurance Act:

Prior to 2002, the Commission set the annual EI premium rate on the recommendation of the Ministers of HRDC and Finance.

7.19 The delivery of the EI Income Benefits program is largely dispersed to the community and regional levels. More than 8,000 full-time equivalents deliver and support the program across Canada. There are 320 points of service (100 main HRDC centres and 220 satellite offices where limited services are offered), 11 call centres, 6 insurance payment operation centres, and 4 information technology centres across the country.

7.20 The 1996 reform to the Unemployment Insurance Act. In 1996, Parliament approved wide-ranging changes to the Unemployment Insurance Act (which became the Employment Insurance Act). The EI Income Benefits program maintained its role of providing temporary income support. However, it was redesigned to promote stronger attachment to the labour force and to introduce stronger insurance principles into the system, with a view to softening the impact on low income families with children. The program was also adjusted to better reflect the changing nature of work in the last decade.

7.21 Changes were made to the eligibility system, the benefit structure, and the financing framework. Under the new system, eligibility was based on hours of work, rather than weeks, and there were stricter eligibility requirements for new entrants and re-entrants to the labour market. Changes to the benefit structure included new rules for frequent claimants. Changes also included a reduction and a freeze of the maximum insurable earnings and the introduction of premium refunds. Finally, changes were made to how the premiums were set. Taken together, these changes contributed to making it possible to accumulate large surpluses.

Top

The surplus in the Employment Insurance Account

7.22 The surplus in the EI Account grew by $3.3 billion in 2002-03, reaching $43.8 billion (Exhibit 7.2). Every year since 1999, we have drawn attention to this issue in the auditor's reports on the EI Account's financial statements and in the Public Accounts of Canada. In our view, Parliament did not intend for the Account to accumulate a surplus beyond what could reasonably be spent on Employment Insurance, given the existing benefit structure and providing for an economic downturn. The current surplus is about three times the maximum reserve that the Chief Actuary of Human Resources Development Canada considered sufficient in 2001.

7.23 In the 2003 federal Budget, the government announced that it would consult on a new rate-setting process to be implemented for 2005. Our concerns about the surplus are discussed fully in Chapter 10, Other Audit Observations, of this Report.

The cost of administering the Employment Insurance Act

7.24 Section 77 of the Employment Insurance Act specifies that the costs of administering the Act are to be charged to the EI Account. The Minister of Human Resources Development is responsible for reporting on the EI programs to Parliament. The Canada Customs and Revenue Agency, which collects premiums and benefit repayments and provides decisions on insurability under the Act, shares the administration of the EI programs. HRDC, the CCRA, the Treasury Board Secretariat, and the Department of Justice all supply services that support management and delivery of programs under the EI Act.

7.25 The costs to provincial and territorial governments of administering employment benefits and support measures under the labour market development agreements are also charged to the EI Account (Exhibit 7.3).

The Modernizing Service for Canadians initiative

7.26 HRDC is currently working on a multi-year effort to improve its performance by better responding to the changing needs and expectations of Canadians. The goal of the Modernizing Service for Canadians initiative is to renew HRDC's programs, policies, services, and service delivery by focussing on what citizens need in a way that supports their full participation in the workplace and community. While the initiative will touch virtually all aspects of HRDC's operations in some way, parts of it specifically involve Employment Insurance. For example, the Department is planning to develop a fully automated, on-line process for applying for, adjudicating, and paying EI benefits. In addition, it intends to automate fully the process for providing Records of Employment and allow employers to submit their records using the Internet.

Focus of the audit

7.27 We carried out this audit to determine whether

7.28 We did not examine the employment benefits programs of HRDC; nor did we look at the CCRA's role in EI administration. More information on our audit can be found at the end of the chapter in About the Audit.

Top

Observations and Recommendations

Service and program effectiveness

7.29 The government has stated that a modern management agenda requires managers to look beyond activities and outputs to focus on actual results—the outcomes of their programs. Managing for results means clearly defining the results to be achieved, delivering the program or service, measuring and evaluating performance, and making adjustments to improve both efficiency and effectiveness. It also means reporting on performance in ways that make sense to Canadians.

7.30 The foundation of results-based management is accurate and timely performance information. The government expects departments and agencies to implement an information regime that measures, evaluates, and reports on key aspects of programs and their performance in core areas. It also holds managers accountable for achieving results.

7.31 We looked at how performance of the EI Income Benefits program was measured and reported from two perspectives: service to Canadians and program effectiveness.

Top

Service to Canadians

7.32 We examined the four distinct operational components of program delivery: answering phone inquiries at call centres, processing EI claims, processing appeals, and ensuring compliance with the EI Act and regulations. We looked at the set of measures HRDC uses to track its performance for these activities. From these measures, HRDC chose a set of key ones for reporting to Parliament in 2002-03. We examined whether reliable data had been collected for the key measures and whether the data provided a good indication of service performance.

7.33 We expected that HRDC would have a set of measures covering the main dimensions of service performance for each activity—the amount, timeliness, and quality of the services provided—and that these measures and associated targets would apply nationally. We also expected that management would be analyzing and using this performance information for accountability purposes and program improvements.

Performance of EI call centres needs to be improved

7.34 HRDC encourages people to inform themselves about the EI Income Benefits program by contacting one of its 11 EI call centres, and it has been expanding the range of services that call centres provide. Callers initially reach an automated message system that may answer their questions, or they can request to be transferred to a service representative. Good service from call centres means that callers can access the information they need reasonably quickly.

7.35 Key measure provides partial information. HRDC's key performance measure for call centres is how long callers wait to speak to a service representative. The target is to have 95 percent of callers trying to reach a representative do so within three minutes. According to HRDC officials, this target was set to align the level of service provided by HRDC's Income Security (Canada Pension Plan, Old Age Security) and EI programs. We examined the controls for data processing for this measure at the national level, and while we found weaknesses, there was limited impact on the overall reliability of the data.

7.36 However, we found that none of the call centres met this service target in the last three years and performance varied considerably among regions. One region responded to 88 percent of calls in three minutes in 2002-03, while another met the target only 49 percent of the time. Moreover, this measure does not give the whole picture of access to call centres. It measures only the waiting time for those able to enter the queue to speak to a service representative—not the number of callers who get a busy signal and have to call back. HRDC received 19.8 million calls in 2002-03 when the caller tried to transfer to a representative, but this figure includes redials. It is not known how many callers these calls represented. As a result, HRDC does not have a good measure of the actual demand for call centre services.

7.37 Exhibit 7.4 shows the percentage of calls by region for the last three years when the caller got a busy signal when trying to speak to a service representative. In most regions, a substantial percentage of calls were not transferred to a representative because the queue of waiting calls was full. Moreover, the percentage is growing. In 2000-01, 34 percent of calls nationally that attempted to enter the queue got a busy signal; in 2002-03, this figure had risen to 65 percent.

7.38 Improvement needed in set of measures for call centres. We looked at the set of measures that management uses for call centres and found that certain aspects of service are not systematically measured. HRDC monitors aspects of call centre performance, including numbers of calls received and answered, and timeliness. However, there is no consistent approach among call centres to assess whether callers who speak with a service representative are receiving complete and accurate information. Managers have developed their own quality measures and targets for service quality. One of the four call centres we looked at had discontinued monitoring of quality. As a result, program management has difficulty comparing the quality of service among call centres.

7.39 More needs to be done to improve performance. Regional and national program managers have identified the following reasons for problems meeting performance targets in call centres:

7.40 To improve performance, management has upgraded the capability of the automated system and trained staff to help clients make more use of it. It has also worked on clarifying standard mail correspondence to reduce unnecessary inquiries at call centres. Further, it is examining ways of linking call centres to better manage call volumes and improve service.

7.41 These efforts have not had a significant effect on performance in meeting service targets. Notwithstanding the problems we have identified with this measure, call centres are far from meeting the three-minute target. In addition, as we have seen, the number of calls getting a busy signal when the caller was trying to speak to a service representative has increased over the past three years.

Performance in processing EI claims varies among regions

7.42 An application for EI income benefits can be made in person at a Human Resources Canada Centre. The applicant provides a Record of Employment as well as other supporting documentation as required (for example, a medical certificate). Applications are also sent by mail or by Internet through Appli-Web, an electronic application form available on HRDC's Web page. For claims processing, good service means paying the correct amount of benefits reasonably quickly to eligible claimants.

7.43 HRDC's key measure for timeliness in processing benefits (speed of payment) is the percentage of first payments for initial and renewal claims made within 28 days of when the applicant becomes eligible for EI. HRDC uses the Comprehensive Tracking System (CTS) as a way to monitor payment accuracy.

7.44 Target for speed of first payment not met. The national target is to make 75 percent of payments within 28 days. This measure was introduced in 2001-02 to replace an older one that measured elapsed time from the date the claimant applied for benefits. Management identified reliability problems with this older indicator.

7.45 HRDC starts counting elapsed time from when the claimant becomes eligible for benefits. For example, if the applicant waits 14 days to apply, then the Department has only 14 days left to meet its target. HRDC officials informed us that the figure 75 percent was chosen because it was expected to both challenge the Department and take into account that client behaviour would have an impact on the timeliness of processing Employment Insurance benefits. For example, some claimants delay filing their claims and returning their reports on earnings and availability. Regions and individual offices changed their work practices to meet this new measure, but most have not been able to meet it. Performance among regions varied in 2002-03 from 44 percent in Alberta to 77 percent in New Brunswick (Exhibit 7.5).

7.46 We examined the controls for data processing for the 75 percent target and found that they are generally adequate. However, the measure does not cover all claims. It excludes cases, for example, where a client report on earnings and availability has not been submitted and processed more than 14 days after the date it was issued. Based on preliminary findings by HRDC, a substantial portion of cases may be excluded from the measure. Management has been examining ways to include these claims in this measure.

7.47 Ninety-five percent of payments are correct. Based on a national sample of claims, HRDC uses the Comprehensive Tracking System to estimate overpayments and underpayments caused by claimants, employers, and the Department and presents the data in the financial statements for the EI Account. The CTS is examined as part of our financial audit of the EI Account each year. In 2002-03, the system found that 95.5 percent of payments were accurate.

7.48 Reasonable set of measures. We looked at the set of performance measures that HRDC uses for processing claims. We found that they reasonably cover service demand and the timeliness and quality of the service provided. However, the quality measure discussed below is not yet producing fully reliable information.

7.49 Many claims do not meet all quality objectives . HRDC added to its performance measures in 2001 when it began the Strategic Directions initiative. This is a standard, program-wide approach to reviewing the quality of claims processed by its staff. HRDC recently audited the initiative to determine whether it produced reliable performance information. The audit work indicated that improvements were needed in how the initiative was implemented because the rate of quality deficiencies it had found was 10 percent higher nation-wide than what Strategic Directions had found. In our view, these findings show that the initiative is not yet producing fully reliable information.

7.50 Nationally, 61 percent of the files met all quality objectives for processing claims; in three regions, about half the files had one or more quality deficiencies (Exhibit 7.6). While many of these problems do not result in payment errors, they can have other consequences such as increased processing costs to the Department or slower service to claimants.

7.51 The Comprehensive Tracking System results also provide information on the quality of claims processing. For example, in 2002-03, despite a payment accuracy rate of 95.5 percent, the CTS showed that 28.5 percent of the files had an error leading to an overpayment or underpayment of benefits. The incorrect payments were only 4.5 percent of the total amount of benefits paid out because most claim errors involved small sums of money. Management has used these results to identify areas for improvement in the past on a national basis and to initiate national training and procedures. However, these national results can also be broken down at least to the level of the geographical areas and show some variation in performance. In our view, the CTS results could have also been used to focus improvements in quality control on specific areas needing attention.

7.52 Limited success in improving performance. Regional and national program managers offered the following reasons for the program's difficulty in meeting its targets for timeliness and quality of claims processing:

7.53 There are a number of ongoing efforts to improve performance in claims processing. Appli-Web allows claimants to apply on-line. Records of employment can also be submitted on-line by some employers. Both of these initiatives are meant to improve the speed and accuracy of processing. Chapter 1 of this report discusses our audit of Government On-Line, which includes a case study on Appli-Web.

7.54 However, notwithstanding these efforts, the available measures provide a mixed view of service performance for claims processing. Speed of payment has improved somewhat over the last two years in some regions but is still below target in most regions. While the overall accuracy of payments has improved from 94.4 percent in 2001-02 to 95.5 percent last fiscal year, the recent audit work done by HRDC found that a significant percentage of files were not meeting all quality objectives.

Timeliness of processing appeals is a concern

7.55 Claimants have the right to appeal within 30 days of being advised of HRDC's decision on their benefits. Employers can also appeal a decision (for example, if an employer believes that a former employee should not receive benefits). An independent body, the Board of Referees, decides on appeals. A decision by the Board can be appealed to the Umpire, an administrative tribunal that hears appeal cases. Persons not satisfied with a decision rendered by an Umpire may apply for judicial review in the Federal Court of Appeal and finally appeal to the Supreme Court. For appeals, good service by HRDC means providing timely service and accurate information.

7.56 Timeliness targets not met. HRDC's key measure for appeals services is the percentage of hearings scheduled with the Board of Referees within 30 days of the appeal being filed. Departmental officials explained that the 90 percent target for this measure provides for the 10 percent of cases that require additional fact-finding. We examined the controls for data processing for this measure at the national level. While we found weaknesses, there was limited impact on the overall reliability of data. Performance among regions varied considerably. While certain regions exceeded the 90 percent target, others fell short (Exhibit 7.7).

7.57 Regional and national program managers explained that delays in processing appeals often result from problems in quality with the original claim decision—for example, the information supporting the decision is incomplete and additional fact-finding needs to be done. In addition, some local managers place a higher priority on processing claims than on appeals and allocate staff accordingly. Management is implementing initiatives to reduce the number of appeals received by improving the quality of the initial claim decision. It has not, however, developed a specific plan to meet its timeliness targets.

7.58 HRDC also measures the percentage of client appeal files (referred to as a docket in the EI regulations) received at the office of the Umpire within 60 days from the date the appeal was filed. The target for this measure is set by the regulations to the EI Act at 100 percent of cases. In 20 percent of cases in 2002-03, HRDC did not comply with the regulations.

7.59 Recommendation. Human Resources Development Canada should comply with the regulations to the Employment Insurance Act in filing appeal dockets with the Umpire.

Department's response. The Department agrees with the recommendation and is implementing the necessary steps. An enhanced policy on appeals review has been put in place in 2003-04 and is in the process of being implemented.

7.60 Quality of service not measured consistently. We looked at the full set of measures used for processing appeals. Timeliness of the appeals process and some aspects of quality are measured. However, no consistent quality review of Board of Referee appeals was being done across the regions. During our audit, management began implementing a standard approach to quality review of appeals.

Top

Objectives for savings related to investigation and control activities need to be risk-based

7.61 Investigation and control includes prevention, detection, and deterrence activities to ensure that claimants and employers comply with the EI Act and regulations. Good service for investigation and control means that the EI Account is protected from fraud and abuse and that claimants and employers are dealt with fairly and in a timely and professional way.

7.62 Problems with key measure. The key performance measure for investigation and control is total savings, which captures direct and indirect savings to the EI Account resulting from this work. Direct savings refer to the dollar value of overpayments identified and penalties imposed as a result of investigation and control activities. Indirect savings occur when an investigation prevents or reduces inappropriate benefit payments. Accordingly, these savings are derived from actuarial estimates of how long the claimant would have actually collected benefits.

7.63 Each year, the actual savings objective is negotiated between the Treasury Board Secretariat and HRDC based on the level of resources that HRDC plans to invest in investigation and control activities. This objective is then divided among the regions, based on past savings results. The objective for 2002-03 was $539 million.

7.64 However, we could not be assured that this objective was reasonable. While the objective is results-based, there is no comprehensive national or regional assessment of risk supporting it. A risk-based approach would start with an assessment of the likelihood and seriousness of non-compliance with the EI Act and regulations and HRDC's expected results for detecting and deterring it. The regions would then plan for the appropriate mix of prevention and detection activities, and, based on this, they would forecast expected savings. While HRDC has introduced other performance measures for investigation and control activities, the focus has remained on savings. In our view, generating savings has become an end in itself as savings objectives are based on previous savings achieved and not on program compliance objectives.

7.65 HRDC estimated $523.5 million in savings last fiscal year. We examined the quality controls in place to ensure the reliability of this information. Controls were adequate, but there is a problem with how savings are calculated. For a variety of reasons, claimants can have their benefits stopped. The unpaid benefits or overpayments that result from these cases are counted as savings for performance purposes. Many decisions are subsequently revised or reversed. In these cases, however, there were no savings, but the amount recorded as savings is not adjusted accordingly. The savings are therefore overestimated. Furthermore, the problem with how savings are calculated could also lead to inappropriate behaviour by investigation and control officers. Regional staff told us that there was a risk that payment stoppages could be recommended and then reversed, resulting in overestimated savings.

7.66 Unclear link to program performance. Without a clear link to program objectives, it is difficult to determine how information on savings is used to improve program performance—in particular, improvements in program compliance. There is no national guidance on the balance between investigation and control activities that focus on detection of fraud (direct savings) and those that focus on prevention (indirect savings). The regional objectives for direct and indirect savings are based on the previous year's targets. We noted a wide variation among regions. For example, direct savings account for about 60 percent of total savings in Quebec, while in Ontario they represent about 40 percent. Since there is no risk assessment of the nature and extent of non-compliance with the EI Act and regulations, HRDC cannot usefully compare savings results among regions or determine whether there is an optimal split among regional activities.

7.67 Inconsistent approach to quality measurement. HRDC has a number of measures for its investigations and control activities—for example, dollar savings, number of prosecutions, and claimants attending information sessions. However, monitoring of the quality of investigation and control activities is inconsistent. While some regions review how investigations were handled, others do not do so consistently. This makes it difficult for HRDC to compare quality among regions.

7.68 Recommendation. Human Resources Development Canada should base its objectives for savings from investigation and control on an assessment of compliance risks and the Department's expected results for detecting and deterring non-compliance with the Employment Insurance Act and regulations.

Department's response. The Department agrees with the recommendation, which will build upon an existing strong track record regarding prevention, detection, and deterrence activities. HRDC is currently in the process of introducing a risk management approach with respect to the investigation and control function and the accompanying savings involved.

An opinion survey showed majority of clients were satisfied with the service

7.69 The Service Improvement Initiative is a government-wide effort to improve satisfaction with service quality by 10 percent by 2005. As part of this initiative, HRDC conducted an opinion survey on client satisfaction in June 2001 to establish a baseline for the EI Income Benefits program. While the survey was general in nature and there was a low response rate, the majority of respondents were satisfied with the service that they had received at an HRDC office or from a call centre. No further surveys have been done to determine whether there has been a change in client satisfaction with HRDC services.

Cost information for EI services is not based on a comprehensive approach

7.70 We looked at whether management was measuring the cost of EI services. The Department tracks unit cost for claims processing on a monthly basis. However, the information presently compiled is not based on a comprehensive approach taking into account all the relevant costing elements that would allow valid comparisons to be made. Management told us that work is underway to establish a comprehensive approach to calculating EI unit costs.

7.71 HRDC is also required to keep track of the costs of administering the EI programs. We examined whether the charges to the EI Account for this purpose were reasonably accurate. We concluded that the formula used by the Department was providing a fair picture of the actual costs of administering the EI programs.

Increased efforts needed to address outstanding key performance issues

7.72 HRDC has made a considerable effort to measure the performance of the EI Income Benefits program and has a good deal of useful information with which to manage it. However, there are aspects of performance that are not yet measured or are not measured properly. We found that HRDC has a reasonable set of performance measures for processing EI claims. However, for call centres, investigation and control, and appeals, information on the quality of service provided across regions is inconsistent. There are also problems with certain measures. Management has made some efforts to assess the causes of problems in meeting targets and to implement remedial action.

7.73 Despite these efforts, the available performance information tells a mixed story. On the one hand, 95.5 percent of payments to claimants were correctly made. Moreover, a 2001 opinion survey on satisfaction found that respondents were generally satisfied with the EI services they received. On the other hand, the program at a national level did not meet the service targets for key aspects of its performance over the last three years. While the results vary by region and activity, service performance was often significantly and chronically below targets. This indicates that despite local, regional, and national initiatives, the Department had not made an effective effort to meet performance targets in all regions for key aspects of service delivery. It is those outstanding performance issues that management must find a way to address.

7.74 Recommendation. Human Resources Development Canada should develop a reasonably complete picture of service performance by adding new measures where needed and correcting problems in existing measures.

7.75 Recommendation. Human Resources Development Canada should ensure that performance targets are met across the country by further assessing the cause of performance problems and by implementing effective solutions.

Department's response. The Department agrees with the above recommendations. It has already set in place a number of targeted measures to bring about improvements in service performance, as part of the process of continuous review and improvement in this area. Within the current review process, an assessment of the adequacy and comprehensiveness of the existing array of performance indicators will be undertaken.

In addition to the above, further important and far-reaching steps are currently in the planning stage under the Department's Modernizing Service for Canadians initiative. This multi-year effort is aimed at renewing the full range of HRDC's programs and services (including Employment Insurance) to meet the changing needs of Canadians.

Parliament is not given the full picture of the service's performance

7.76 HRDC uses three documents to report to Parliament. The Report on Plans and Priorities presents HRDC's planned results, while the Departmental Performance Report presents and explains actual results. The Monitoring and Assessment Report (MAR) is required by the EI Act and presents various information on the EI programs.

7.77 In our view, these reports have not given Parliament the full picture of the service performance of the EI Income Benefits program. They have not described important performance issues, such as the uneven speed and quality of processing claims across the country. Currently, HRDC reports only national averages for key measures, giving parliamentarians only a very broad view of performance. For call centres, it reports the percentage of calls answered by a service representative within three minutes. But it does not report the larger percentage of calls that cannot get into the queue. It also does not report how it plans to meet its service targets in all areas of the country.

7.78 Recommendation. Human Resources Development Canada should report measures that better capture service performance in sufficient detail to meet the information needs of parliamentarians. The Department should describe plans to meet performance targets where required.

Department's response. Agreed. See response to recommendations 7.74 and 7.75.

Top

Program effectiveness

The Commission has a mandate to assess the effectiveness of the EI Income Benefits program

7.79 The EI Act requires the Canada Employment Insurance Commission to measure the effectiveness of EI benefits as well as the impacts of the 1996 legislative reform. HRDC carries out this work on behalf of and under the direction of the Commission.

7.80 Specifically, the Commission must monitor and assess

7.81 HRDC also undertakes policy analysis and research studies as part of its oversight role for the EI Income Benefits program and carries out evaluations as required by the Treasury Board policy on program evaluations.

7.82 In the following paragraphs, we examine how the Department and the Commission plan and co-ordinate these efforts to assess the effectiveness of the program, how it ensures the quality of the key effectiveness measurements, and how the Commission fulfills its responsibilities to report on effectiveness.

An evaluation plan needs to be developed for the program

7.83 The EI Income Benefits program is a large and complex program that affects Canadians and the economy in a number of ways. This fact as well as the Commission's clear mandate to measure the program's effectiveness warrants a systematic approach to evaluating the program's effectiveness over a reasonable period.

7.84 Branches within HRDC plan their own work relative to their responsibilities. Communication between branches takes place through the committee co-ordinating the Monitoring and Assessment Report and through the formal planning of each branch. These efforts, however, have not resulted in a common departmental evaluation plan for the EI Income Benefits program.

7.85 The Commission and the Department have not taken advantage of the benefits that could be derived from an evaluation plan for the EI Income Benefits program. A plan would ensure that all the key components of this EI program would be evaluated while avoiding undue overlap. It would also enable the Department to establish clear measurement priorities, co-ordinate its activities, and allocate resources. Further, it would facilitate reporting on effectiveness annually.

7.86 In the absence of a departmental evaluation plan for the program, we conducted an extensive review of the evaluation and policy literature on EI to establish what the Commission and the Department could have measured. We compared the efforts of HRDC and the Commission since the 1996 reform to measure the program's impacts and overall effectiveness with our review of the program's potential impacts. We found that HRDC had undertaken 68 studies addressing many of the effectiveness issues that we identified. However, we noted that it had still not addressed, or had only partly addressed, some key issues seven years after the reform to the Act. For example, it did not do an analysis of the cost-effectiveness of the EI Income Benefits program, including the impact of the EI surplus. It did only a limited analysis of the impact of the benefit repayment provision since 1996 and of the impact of EI on employers' management of their workforce.

7.87 We also noted that some of the studies overlapped. For example, several studies examined the impact of EI on new entrants and re-entrants (persons returning to the workforce). Given that some issues have not been evaluated, there may have been opportunities to allocate resources differently to ensure more complete coverage.

7.88 During our audit, HRDC began planning a comprehensive evaluation of EI and the EI reform to be completed over the next three years. The Department indicates that this evaluation will identify gaps in knowledge of the effectiveness of the EI Income Benefits program and that studies will be done to address them. The studies will be used to produce synthesis reports on EI and the EI reform.

7.89 Recommendation. Human Resources Development Canada should design, implement, and communicate an evaluation plan for the EI Income Benefits program.

7.90 Recommendation. The Canada Employment Insurance Commission should ensure that the impact of all the key changes to the Employment Insurance Act are assessed and reported in the Monitoring and Assessment Report.

Human Resources Development Canada's and the Canada Employment Insurance Commission's response. The Commission and the Department agree with the recommendations. Plans are currently being implemented to produce an extensive evaluation of the program's longer-term impacts. The work is scheduled to proceed over the next two years. As soon as a comprehensive evaluation framework for this work has been developed, it will be reported to Parliament through the Monitoring and Assessment Report.

As results from the evaluation work become available, the Commission will ensure that the impacts of key changes in the Employment Insurance Act are reported in the Monitoring and Assessment Report. This will proceed by building on the significant body of evaluation results information already developed for the Employment Insurance Income Benefits program over the last several years.

Top

External peer reviews of evaluations and studies are done

7.91 Program evaluation is one of the most important efforts undertaken by a department to measure a program's effectiveness. Accordingly, we expected HRDC to have in place reasonable and appropriate quality controls for the information it produced on effectiveness issues.

7.92 We examined the extent to which post-reform evaluations and studies focussing on EI were subject to a peer review process. HRDC management told us that 49 out of 57 evaluations and MAR-related studies, and all the research studies published by the Applied Research Branch, were peer reviewed. We examined how HRDC conducted five of these peer reviews and found that useful technical comments were provided and used to improve the evaluations and the studies.

7.93 We analyzed the quality of the evaluation studies. We concluded that the five evaluations we reviewed were generally well done, with clear statements of assumptions and results, and reasonable conclusions drawn from the results. We have some concerns about the methodology used in one study, for which no peer review was done. These concerns are mitigated, however, by the fact that a previous study, conducted on a similar program design but using a more reliable methodology, produced a range of results that are consistent with the results of the second study.

The Commission is required to report to Parliament on the effectiveness of EI

7.94 In addition to its legal responsibilities to monitor and assess the effectiveness of the EI reform, the Commission is required to report to the Minister annually. The Minister, in turn, tables the report in the House of Commons. We expected that the Commission's reports to Parliament over several years would provide a comprehensive and clear view of the program's performance. We focussed our assessment on the information provided in the six monitoring and assessment reports that have been tabled in Parliament so far.

7.95 The MAR is an important document. Normally departments and agencies are required to report on the performance of their programs in the Departmental Performance Report. The MAR provides Parliament with more information than is required of most programs. The report annually summarizes information coming from a number of sources. This information is intended to provide an overview of the labour market and of various developments associated with the EI programs while presenting an array of information on the effectiveness of the benefits and assistance provided under the Act.

7.96 In our view, the Monitoring and Assessment Report has improved over the years; this was particularly noticeable in the last MAR. However, we believe that some important issues still need to be addressed in future MARs.

The savings expected from the 1996 EI reform were not clearly reported

7.97 In its 1995 Budget, the government announced that strong economic performance and unemployment insurance reform would reduce the overall size of the program "by a minimum of ten percent." In 1996 the government estimated that the EI reform alone would generate $1.925 billion in savings by 2001-02, when all the new measures would have matured. Savings would come from the following:

7.98 The EI Act requires the Canada Employment Insurance Commission to monitor and assess whether the savings expected as a result of the changes made in the Act are being realized and to report on its assessment. Therefore, we expected that the Commission, with HRDC's help, would have tracked those savings in order to report on the extent to which they are being realized.

7.99 Estimates of savings related to a reduction in income benefits were done. Those savings represent 80 percent of the total savings of $1.925 billion promised. For the most part, tracking these savings was difficult. They were based on a series of complex assumptions, because the effects of the 1996 changes to the Act had to be separated from subsequent changes to the Act and from the growth of the economy, which also generated savings through lower unemployment rates.

7.100 Estimates for some components of the savings resulting from increased sanctions for fraud and from enhanced services to claimants were not done.

7.101 The estimated savings mentioned in paragraph 7.97 have been reported but in an inconsistent and unclear manner. Except for the first MAR, subsequent MARs did not clearly outline the savings targets and did not provide clear year-to-year progress toward targeted savings. As a result, Parliament was not clearly informed of the extent to which the expected savings were realized.

7.102 Contributing to this lack of clarity is the fact that over the years, the MARs have reported savings other than those related to the 1996 reform. For example, savings related to investigation and control activities are reported in most of the MARs without specifying that most of these activities were occurring before the reform; therefore, only a fraction of these savings could be related to the $1.925 billion in savings promised.

7.103 Similarly, the 2002 MAR reported $650.8 million in unpaid benefits as savings without clearly stating that the unpaid benefits reported are "gross figures" (that is, figures representing the full amount of benefits that beneficiaries could theoretically collect if they claimed all the weeks for which they are entitled—something that only occurs in a minority of cases). Findings from an ongoing evaluation by HRDC in this area suggest that the net savings (that is, an estimation based on the actual number of weeks that were not paid) related to unpaid Part I income benefits are in fact much smaller than the reported "gross figures."

7.104 Recommendation. With the help of Human Resources Development Canada, the Canada Employment Insurance Commission should clearly report, in the 2003 Monitoring and Assessment Report, the extent to which the savings expected from the 1996 changes to the Employment Insurance Act were achieved.

The Canada Employment Insurance Commission's response. The Commission and the Department agree that the 2003 MAR will report on the extent to which the total savings from the 1996 changes to the Employment Insurance Act have been achieved. This will build upon the extensive work that has already been undertaken in this area and reported in earlier MARs.

Lack of balance in effectiveness information reported

7.105 When reporting to Parliament, the Commission has not made full use of the numerous studies done by the Department on the effectiveness of the EI Income Benefits program. We compared the findings of the studies with what was reported about them in the MARs. While the information reported is factually correct, in many cases the Commission did not report all the key findings and emphasized positive findings. Management told us that these studies are available on the Internet. However, we found that over the years many evaluations were not available until several months after the MARs were published (see Exhibit 7.8 for an example).

7.106 We found that the other information developed for the MARs was reported in a more balanced manner than the effectiveness information. For example, the MARs have consistently raised issues about the access to the fishing benefits. With the reform, the eligibility system switched from one based on weeks of work to one based on earnings. Stricter requirements apply for new entrants and re-entrants. Before the reform, the majority of fishers qualified with just the minimum number of insured weeks. The MARs indicated that virtually all claimants for fishing benefits have more earnings than the minimum required. The reports stated that these results suggest that earnings-based entry requirements may be too low given the level of revenue in the fishery.

7.107 Finally, the MARs include little discussion of the quality of performance information and the reliability of information sources, such as the studies and the surveys quoted, which would provide a basis for judging the credibility of data.

7.108 Recommendation. The Canada Employment Insurance Commission should ensure that the monitoring and assessment reports fully reflect the key findings of the studies done on program effectiveness.

7.109 Recommendation. The Canada Employment Insurance Commission should ensure that the monitoring and assessment reports provide sufficient information on the reliability of the effectiveness information reported.

The Canada Employment Insurance Commission's response. The Commission is pleased that the Auditor General has recognized that the MAR is a valuable report and that it is improving over time. The Commission will ensure that the report continues to improve. As part of this, the Commission will ensure that key findings from effectiveness studies are more fully reported and that sufficient information on the reliability of results measurements undertaken is provided.

Top

Conclusion

7.110 Over the past 60 years, Employment Insurance has been the main income security program for working Canadians. The principal objective of the Employment Insurance Income Benefits program is to provide temporary income support to insured Canadians who involuntarily lose their jobs.

7.111 Employment Insurance is paid for by workers and their employers. Over the past few years, the amount of premiums has exceeded the amount of benefits, to the point that there is now a surplus of $43.8 billion credited to the EI Account. Our Office has reported to Parliament on the size of this surplus every year since 1999. Since 2000-01, the administrative costs of the EI programs have increased from $1.4 billion to $1.5 billion.

7.112 We looked at the measurement and reporting of the performance of the EI Income Benefits program. We found that HRDC put considerable effort into measuring the program's performance and had a reasonable set of performance measures for processing EI claims. However, one measure related to the quality of claims is not yet producing fully reliable information. Important aspects of performance were not well measured for call centres, investigation and control, and appeals. For example, in the last two years, the majority of calls to a service representative at a call centre got a busy signal.

7.113 We examined the controls in place to ensure reliability of performance data for four key measures. Although we found weaknesses in the controls for two of the key measures, they had limited impact on the overall reliability of data. In addition, the way investigation and control savings are calculated results in their overestimation. We also examined how HRDC was tracking the costs of EI services. The Department has recognized that this cost information is not based on a comprehensive approach and has undertaken a project to address this issue.

7.114 Service to Canadians was often well below performance targets and, in many regions, chronically so. There was considerable variation among regions in timeliness and quality of processing initial and renewal claims. However, other aspects of performance were positive. In 2002-03, over 95 percent of total EI benefit payments were correct; a 2001 opinion survey showed that a majority of respondents were generally satisfied with the service they received.

7.115 We expected that HRDC would not only be measuring its performance but would also have taken action to improve it. While local, regional, and national efforts have been made to meet service targets, they have not been effective. In our view, HRDC has not made adequate use of performance information for program management and accountability purposes.

7.116 We are also concerned that Parliament does not have a complete picture of how well the EI Income Benefits program is serving Canadians. Reports to Parliament have not described important performance issues, including the uneven speed and quality of service across the country for processing claims.

7.117 The EI Act requires the Canada Employment Insurance Commission to measure the effectiveness of EI benefits as well as the impacts of EI reform, and report its assessment through the Minister of Human Resources Development to Parliament. However, seven years after the reform to the Act, the Commission and the Department have not yet developed and reported a comprehensive and clear picture of the program's impact. In addition, certain key effectiveness issues have not yet been studied. We found that the monitoring and assessments reports have improved over the years and this was particularly noticeable in the 2002 MAR. However, we found that reporting of evaluation results is often selective and that HRDC has not reported clearly to Parliament on whether the savings expected from the 1996 reform to the EI Act were realized.

Human Resources Development Canada's and the Canada Employment and Insurance Commission's overall response. The Commission and the Department are pleased to note that the auditors acknowledge that considerable effort is being put into measuring the performance of the Employment Insurance Income Benefits program and that, as a result, there is a good deal of information with which to manage it. As noted by the audit, despite its complexity, the program has a strong track record in getting the appropriate payments to its clientele for income replacement—over 95 percent of total program payments to claimants are confirmed as correct. Further, a 2001 opinion survey showed that Employment Insurance claimants were generally satisfied with the service they received.

With respect to areas in need of improvement, the issue of differences in service performance across regions has been further confirmed by the audit. For some time, the Department has been undertaking, and is continuing to undertake, a series of targeted and systems-wide initiatives to bring about improvements in service. As the auditors note, the most important and far-reaching steps are currently in the planning stage under the Department's Modernizing Service for Canadians (MSC) initiative. This multi-year effort is aimed at renewing HRDC's programs, policies, services, and service delivery by focussing on what citizens need in a way that supports their full participation in the workplace and the community. While the MSC initiative will touch virtually all aspects of HRDC's operations in some way, important parts of it will specifically involve Employment Insurance.

Elsewhere, the Commission and the Department are pleased that the audit has confirmed that important information is being made available to Parliament on the Employment Insurance Income Benefits program through the annual Monitoring and Assessment Report (MAR). Further, it is acknowledged that recent versions of the MAR have demonstrated significant improvements. The audit also has confirmed that the Department's program evaluations of Part 1 of the Employment Insurance Act are well done and are making available for parliamentarians clear statements of results and reasoned conclusions regarding the program's effects.

The Commission and the Department approach the reporting of program results to Parliament as a process of continuous improvement. Consequently, the areas identified by the audit where Parliament needs to be better informed will be addressed and responded to as a matter of priority.

Top

About the Audit

Objectives

Our audit objectives were to determine whether

Scope and approach

The scope of the audit is the Employment Insurance Income Benefits program. We looked at how performance of the program was measured and reported from two perspectives: service to Canadians and program effectiveness. We examined how service to Canadians is measured for four distinct operational components of the program delivery: answering phone inquiries at call centres, processing Employment Insurance (EI) claims, processing appeals, and ensuring compliance with the Employment Insurance Act and regulations. We did not examine the employment benefits programs of HRDC; nor did we look at the Canada Customs and Revenue Agency's role in EI administration.

We interviewed officials and reviewed documentation at HRDC headquarters and at the Canada Employment Insurance Commission. We examined reports to Parliament, including the evaluations and the studies on which they are based. We visited some regional and local offices and call centres where we conducted interviews with officials and analyzed various types of documentation. To assess the reliability of four key measures, we conducted tests at headquarters and in all four information technology centres on systems used by HRDC for managing performance information. In addition, we relied on internal audit work done by the Department.

Criteria

Our audit was based on the following criteria:

Audit team

Assistant Auditor General: Maria Barrados
Principal: Peter Simeoni
Directors: Yves Genest, Michelle Salvail, Suzanne Therrien

Steve Ashfield
Marise Bédard
Doreen Deveen
Anne-Marie Dionne
Susan Gomez
John McGrath
Paul Pilon
Lise Tremblay

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll-free).

Top