Evaluation of Economic Analysis and Statistics

Project number: 96261
Final report
January 16, 2020

Evaluation of Economic Analysis and Statistics
(PDF - 1.0 MB)

Table of contents

• 1.0 Profile of the Economic Analysis and Statistics function
• 2.0 Evaluation objective and scope
• 3.0 Evaluation findings
• Evaluation findings: Program results
• Evaluation findings: Contribution to decision making
• Evaluation findings: Information attributes are critical
• Evaluation findings: Fisheries data affects EAS
• Evaluation findings: Gaps
• Evaluation findings: Gender based analysis+
• Evaluation findings: Efficiency
• Evaluation findings: Governance
• 5.0 Conclusions and recommendations
  • Conclusions
  • Recommendations
• 6.0 Appendices: Evaluation methodology and management action plan
• 7.0 References

1.0 Profile of the Economic Analysis and Statistics function

For the purpose of this evaluation, the Economic Analysis and Statistics (EAS) Function includes all EAS programs and services as well as their governance structure(s). This includes all economic analysis, statistical information and advice provided at National Headquarters (NHQ) and in regions, on a wide range of issues of concern to Canada’s fisheries, oceans’ sectors and the Canadian Coast Guard (CCG), to support evidence-based decision making.

Starting in 2018-19, three new programs in the Departmental Results Framework (DRF) were initiated by the EAS Directorate at NHQ to facilitate delivery of economic analysis and statistical services at the national and regional levels, to inform departmental decision-making. The three programs are as follows:

EAS Function encompasses a range of services, including mandatory services, provided to varying degrees by the EAS Directorate and each region:

• Economic and socio-economic impact analyses
• Cost-benefit analysis for regulatory proposals under any act, and for, Species at Risk Act (SARA) Action Plans^{Footnote 1} 
• Data entry, data integration and maintenance of statistical databases
• Economic profiles, e.g., key exports, trading partners
• Economic modeling and forecasting to provide forward-looking assessments
• Framework and policy development
• Input to Memorandum to Cabinets (MCs), Treasury Board Submissions (TB Subs), scenario/briefing notes
• Internal statistical reports, guidelines, frameworks, other materials
• Development and application of economic tools
• National Headquarters EAS advice, guidance to departmental sectors, regions on economic analyses, tools, processes, etc.
• Research projects to strengthen economic analyses, e.g., test methodologies, assess approaches to economic valuation
• Trade agreement analysis

Economic analysis, statistical information and advice are provided to DFO and CCG by the EAS Directorate at National Headquarters (NHQ) and independently by economic analysis units within the Policy and Economics or Strategic Services groups in DFO regions (i.e. Newfoundland and Labrador, Maritimes, Gulf, Quebec, Central and Arctic and Pacific).

The EAS Directorate provides services mainly to clients at NHQ, in response to client requests. It also provides framework and guidance documents to assist the regions and sectors. Examples of frameworks and guidance include a Framework for Socio-Economic Analysis to Inform Integrated Fish Management Planning and Fish Harvest Decisions (2008) a Framework for Integrating Socio-Economic Analysis in Species at Risk Act Listing Decisions (2016) and Guidance on incorporating economic use information into marine protected area network design (2017). Regional EAS services are provided on a response basis.

Four of the regions i.e. Newfoundland and Labrador, Maritimes, Gulf, and Central and Arctic include fisheries statistics as part of their economic analysis, statistical information and advice. In the other two regions i.e. Quebec and Pacific, statistics are part of the Fisheries Management program.

Much like the rest of DFO and CCG, there is no line reporting between NHQ EAS Directorate and the regional Policy and Economics or Strategic Services units.

This means that the EAS Function staff report to the Director General EAS at NHQ or to the Regional Directors General in DFO’s six regions. EAS Function full-time employees (FTEs) increased by 20% from 2014-15 to 2018-19, from about 69 to 83.

In the figure below, the numbers on the left indicate the total number of staff involved in the EAS Function. The numbers on the right represent the number of staff involved in fisheries data entry, a function that is unique to DFO among the peer departments. Expenditures related to economic analysis, statistical information and advice are presented below. The graph on the left shows EAS Function expenditures including the NHQ and regions. The graph on the right shows EAS program expenditures, i.e., for the EAS in NHQ only (EAS programs are only resourced at the NHQ as reflected in the DRF).

EAS Function Expenditures 2014-15 to 2018-19

2.0 Evaluation objective and scope

Evaluation objective

The evaluation assessed the extent to which the EAS Function addresses DFO and CCG needs for economic analysis, statistical information and advice to support evidenced-based decision-making.

Evaluation scope

This is the first evaluation of the EAS Function. It covers the five-year period April 2014 through March 2019 and focuses on the service needs of key internal DFO/CCG clients only – clients external to the department were not included. The evaluation includes the Ice Assistance Emergency Program.

The evaluation does not address DFO data governance issues other than those specific to the EAS Function.

Fisheries statistics are part of the Fisheries Management program in two regions, Pacific and Quebec. That program is not part of this evaluation.

Evaluation questions

Evaluation questions are aligned with Treasury Board’s 2016 Policy on Results and address areas of interest to senior management. The evaluation assessed factors having a bearing on the , effectiveness and efficiency of the services provided.

This included an assessment of the capacity of the EAS Function to address client needs, and the interface between the EAS Directorate and DFO programs, sectors, regions and the CCG.

Specifically, the evaluation answered the following questions:

1. To what extent do the three programs address ongoing needs within DFO/CCG?
2. To what extent are economic and statistical data and analysis considered to be timely, current, accurate, comprehensive and accessible^{Footnote 2}?
3. To what extent do clients understand the economic and statistical information provided and consider it reliable and relevant to their needs^{Footnote 2}?
4. To what extent and in what ways have the economic and statistical information and advice contributed to support and inform decision making within DFO/CCG?
5. To what extent are Gender Based Analysis (GBA+)*  considerations included in economic and statistical data, analyses, and advice?
6. What are the factors that have facilitated or hindered the programs’ ability to inform DFO/CCG decisions?
7. In what way are the three programs an effective and efficient means of providing services?

Evaluation methodologies, limitations and mitigation strategies are discussed in Appendix A.

* Gender-based analysis plus (GBA+) involves considering diverse, under-represented groups in the development, delivery and reporting on results of departmental policies and programs. The “plus” in GBA+ indicates it encompasses more than gender. For example, Indigenous Peoples, remote communities and low-income coastal communities are DFO-relevant GBA+ groups, in addition to women and possibly other groups.

3.0 Evaluation findings

Evaluation findings: Program results

Overall, the evidence demonstrates that two of the three programs have met their targets and are effective in providing EAS services. The third program is intended to address Canadian Coast Guard needs regarding economic analysis, statistical information and advice. These needs have yet to be defined.Results

Tracking of results for reporting purposes in the Performance Information Profiles (PIPs) for the three EAS programs has been done only for the EAS Directorate thus far. The results indicate that the Fisheries Economics and Statistics program and Aquatic Ecosystems Economics program are meeting their targets of addressing 100% of requests received.

While the results indicate they have met their targets and are effective, the indicator (percentage of requests completed) does not provide a comprehensive assessment relevant to the quality and complexity of analyses and advice, the workload and resource allocation, or whether the programs support decision-making. In addition to the need for meaningful indicators, EAS Function programs and PIPs need to address all four departmental Core Responsibilities, as the EAS Function is relevant across the department.

The numbers below include a large number of requests for statistical data and analyses.

Marine Operations Economics Program

Based on the evidence available, it was not possible to assess the effectiveness of the third program. Requests for services from CCG has decreased to zero, and the planned FTE has been reallocated to other priorities.

CCG’s National Strategies’ Economic Industry Intelligence notes that it does not conduct its own economic services as defined by the EAS Function. An example of their work would be examining trends in Canada’s marine trade (e.g. at Canadian Ports, top commodities shipped/handled at ports, value of shipped goods) and related marine traffic trends to enhance awareness of Coast Guard client needs, and to inform program & service delivery. The evidence suggests that the EAS Function has not provided any services to the CCG in the last fiscal year.

Evaluation findings: Contribution to decision making

The EAS Function contributes to senior management decision making by addressing routine requests, such as cost-benefit analyses for regulatory proposals and SARA Action Plans, and ad hoc requests for fisheries information and analyses.

Economics: Significant role

Economics is widely recognized as an important element of fisheries and ocean conservation, one that informs legislative, regulatory, policy and management decisions in many countries.

• ‘Economics of fisheries’ is a field that has informed policymakers about harvest levels since the mid-1950s. (Oxford University 2016 article)^{Footnote 3}
• Economics is critical to the success of marine biodiversity. (IIFET 2015 Annual Newsletter)^{Footnote 4}

Contribution to DFO decision making

Economic analysis, statistical information and advice have contributed to informing senior level decisions and have helped senior management understand the consequences of decisions. The EAS Function fulfills two mandatory requirements, specifically cost-benefit analyses as required by Treasury Board for regulatory proposals, and a similar but less prescribed analysis required by DFO for SARA Action Plans.

Economic analysis, statistical information and advice have served an essential role in informing, among other things, Memoranda to Cabinet, Treasury Board Submissions, regulatory proposals, Integrated Fisheries Management Plans, Total Catch Allocations, treaty negotiations and fishery closures.

Examples

Cost-benefit analysis

• The EAS Function’s cost-benefit statement in the 2018-19 regulatory proposal for the Banc-des-Américans Marine Protected Area notes
  • the regulation will limit or mitigate impact of human activities, conserve and protect marine species, habitat, ecosystems and water quality;
  • low incremental costs, the estimated cost over a 30-year period, and the average annual cost; and
  • the sectors impacted, e.g., commercial communal fisheries, tourism industry.

Ad Hoc analysis

• The EAS Function’s socio-economic analyses have informed senior management about the impact of invasive species, including Asian carp, Chinese mitten crab and zebra mussels.
• The EAS Directorate’s model for forecasting global fishery markets has contributed to informing trade negotiations and the potential impacts of domestic and international policies that affect production or prices. It has also identified emerging opportunities for the Canadian fish and seafood sector as well as policy issues.
• In some instances, EAS analyses have helped decision makers understand the consequences of their decisions, rather than specifically inform decisions, e.g., understanding trends in trade, trade capacity and the economic impact of science-based decisions on communities.

Evaluation findings: Information attributes are critical

The EAS Function contributes to senior management decision making by addressing routine requests, such as cost-benefit analyses for regulatory proposals and SARA Action Plans, and ad hoc requests for fisheries information and analyses.

The eight attributes below, which Treasury Board has used to define quality information, provided a basis for assessing the performance of the EAS Function.

Framework to assess performance: Information attributes

These attributes are prerequisites…

• 1. Timely
• 2. Current
• 3. Accurate
• 4. Comprehensive
• 5. Accessible

…for the following 3:

• 6. Understandable
• 7. Reliable
• 8. Relevant

A DFO Mission-critical risk

According to the 2019-20 Corporate Risk Profile, information for decision making is a mission-critical risk. A lack of access to complete, accurate and timely data and information could impede evidence-based decision and policy making and impair DFO’s and CCG’s ability to deliver on their mandates and fulfill their core responsibilities.

Clients’ perception

Clients’ perception of the extent to which the eight information attributes are addressed is positive, although there is room for improvement. EAS Function directors closely matched client ratings, except for timely and relevant which they rated on the scale closer to ‘considerable’ extent (3.8).

Several EAS Function directors noted a six-month to two-year lag for some fisheries statistics. They see this as affecting the comprehensiveness, accuracy, reliability and of EAS analyses.

Evaluation findings: Fisheries data affects EAS

The EAS Function’s ability to provide reliable and relevant analyses is dependent in part on the quality of fisheries data available.

EAS Function is affected by the quality of statistical data available

Well-documented statistical data issues affect the EAS Function. Four reports since 2007 identify significant fisheries data issues that compromise the availability of accurate and current data on a timely basis, particularly at the national level. The most recent report, the September 2018 Data Governance Workshop Report, lists key data-related challenges for the three EAS programs (p. 94), also noted in a 2015 report.

Examples

• Data capture errors, outdated conversion factors, and inconsistencies in methodologies and terminology across regions are significant impediments to the accuracy and completeness of data and analyses.
• A 2007 DFO Economic Analysis and Statistics Policy Sector report notes, “it has been over twenty years since a full review and revision of [conversion factors] have been conducted.”
• Standard Conversion Factors of Landed Species is one of three DFO guidance documents currently posted on the Economic Analysis and Statistics Division’s intranet site. The standards date from June 1984.
• Work on introducing e-logs to improve catch and effort data has been ongoing since 2006-07. According to a 2018 DFO audit report, full national implementation of e-logs had yet to be realized.
• The methodologies used to estimate the value of landings and price data by species is not consistent across the regions. The value of landings is an important variable in identifying the fishing industry economic contribution both nationally and internationally.
• DFO’s Statistical Services website does not present up-to-date national statistical data, e.g., current to within a week or a few days, as for some regional statistics.
• National reports range from several months to three years old, i.e., dated from 2016 to 2018 at the time of writing this report.
• The EAS Directorate’s Statistical Services Unit is responsible for compiling and disseminating accurate and coherent national fisheries statistics on a timely basis using information drawn from regional statistical systems.
• Given various fisheries data issues and the time required at the national level to clean and prepare a cohesive set of data from regional datasets that are not consistent, the Unit is challenged in fulfilling its responsibility.

Evaluation findings: Gaps

Gaps exist in terms of planning, capacity and services, and there is a lack of awareness of the EAS Function on the part of clients and potential clients. This limits the EAS Function’s capacity to inform DFO/CCG decision-making.

The evaluation evidence demonstrates that there are gaps/factors that impact the EAS Function and they related to planning, capacity and services.

Planning

Proactiveness

• There is a need for the EAS Function to proactively meet with clients in all regions, sectors and CCG, to assess client needs and be better positioned to address departmental priorities and to plan economic analyses. The EAS Function works informally with internal clients (meetings, working groups, work plans, emails, etc.) to assess client needs and issues arising but this is not consistently done across the EAS Function nor is it done on a formal basis. In addition, to providing a more complete understanding of client needs, such interaction can contribute to improving clients’ awareness and understanding of the EAS services available to them and overall planning of EAS service delivery.
• The survey yielded an interesting result with respect to awareness. The survey was sent only to those who were identified by the EAS Function as being clients, yet 31% of respondents that participated in the survey indicated they were not EAS clients.

Understanding needs

• Understanding client requests, and clients’ inability at times to clearly explain their needs, have impacted the EAS Function’s capacity to readily respond and inform decision-making. Interestingly, survey respondents indicated that they determine their needs 6 to 12 months in advance. Understanding client needs was noted in another DFO evaluation^{Footnote 5} as being a challenge, and lessons learned from that evaluation may assist the EAS Function with future best practices.
• Some interviewees identified missed opportunities in serving potential clients, since there was none to limited economic analysis, statistical information and advice provided. Canadian Coast Guard, Conservation and Protection, and Corporate Services were discussed as areas where economic advice and information could potentially help to inform decision-making.

Tools

• Science and Evaluation were raised as examples of using multi-year planning as a planning tool which includes regular consultations with clients.
• Memorandum of Understanding (MOUs) and Service Level Agreements (SLAs) generally were not perceived as tools that could help the EAS Function plan, as the type of work (ad hoc) typically does not lend itself to agreements. Other government departments (OGDs) have similar views with regards to the use of MOUs and SLAs. However, one region has found SLAs to be a useful tool.

Capacity

Resources and data

• Resource limitations affect the ability to meet increasing client demand. For example, for Species at Risk (SAR), there has not been enough capacity in EAS to meet client needs which resulted in delays in preparing regulatory packages. This was not an issue of functional capability, but a lack of capacity to handle a lot of regulatory work due to an insufficient number of staff.
• As noted previously, the EAS Function is affected by the quality of statistical data available.

Research and analysis

• Interviewees mentioned that the quality of cost-benefit analysis and other analyses at DFO are good, but would benefit from more research and analysis to support statements and improve consistency. This could involve undertaking studies to provide information that will be used repeatedly e.g., the value of protecting a given species. As an example, ECCC invested in valuing the social impact of carbon, which has reduced the time required for related regulatory proposals. The information, which is used repeatedly, is validated periodically.
• Interviewees, including senior management, noted a need for the EAS Function i) to be flexible in providing some analyses more quickly and ii) to address increasingly complex, emerging and broad issues, e.g., long-term and predictive decision-making, the valuation of ecosystems, economic policy work, more in-depth qualitative socio-economic analysis as well as quantitative (i.e., address social, cultural aspects), scenarios, projections, etc. Emerging issues that may affect EAS services include, for example, expanding analyses for Marine Protected Areas (MPAs) to address the impact of closures more fully, aquaculture, aquatic invasive species (AIS), Bill C-68 changes, quantifying the non-market value of non-economic assets that help protect biodiversity, such as the benefits of sponge filtering environmental toxins, and behavioural economics analysis.

Services

Awareness of EAS services

• Survey respondents identified the extent to which they received each of 15 different EAS services over the five-year period, e.g.,  economic and socio-economic analysis, fee analysis, economic modelling and forecasting, and so forth. Most services are not used. Of the 29 respondents,
• On average across services 21 respondents received a service either never or rarely
• 12 received economic and socio-economic analyses and economic profiles ‘sometimes’
• 11 received other analyses, data or advice ‘sometimes’
• 3 or 4 received six of the services ‘frequently’ or ‘very frequently’.

A common list and descriptions of the services offered by the EAS Function (the regions and EAS Directorate) would contribute, in part, to improving clients’ understanding of EAS services available.

CCG needs

• CCG’s needs regarding economic research, analysis and advice have not been defined. A few interviewees mentioned there is a need for economicresearch, analysis and advice that is not being met presently, e.g., when CCG develops MCs and TB subs.

GBA+

• The gaps identified with regards to GBA+* include, in particular, an absence of discussion in cost-benefit analyses about impacts on women. The literature review revealed that women’s involvement in fisheries, aquaculture and marine conservation in many countries extends beyond those directly employed in these sectors and is typically underestimated and overlooked.

Rule of five

• Privacy concerns mean that the EAS Function must follow the rule of 5**. This is problematic as much of the data is for sparsely populated areas, which impacts the EAS Function’s ability to accurately inform decision-making. In addition, the rule of 5 affects the EAS Function’s ability to provide information to external parties, e.g., environmental community, non-governmental organisations (NGOs), and academics.

EAS advice in briefing material

• Previous DFO briefing note templates for the Deputy Minister and the Minister did not include a section for EAS advice as they did for Science. Some interviewees indicated that adding a section of EAS advice in briefing note templates would be valuable to inform decision-making. Recent changes to memoranda departmental templates suggest socio-economic analysis as one piece of evidence to be part of the considerations and rationale to support recommendations.
• Survey respondents were asked if they verify the validity of any economic information they provide for senior management decision making with the EAS Function and just over 50% of respondents indicated that they ‘sometimes’, ‘rarely’ or ‘never’ follow-up with the EAS Function.

* GBA+ analysis is currently represented to a limited extent in economic analysis, and program leads are not yet fully taking GBA+ into account during the early phases of program, policy, or regulatory design.  As guidance on GBA+ implementation becomes available from the GBA+ Centre of Expertise, shared responsibilities will become more clear.

**To protect privacy, the rule of 5 is applied, meaning there must be a minimum number of 5 units in each category. For example, if an analysis involves fishers, vendors and buyers, there must be a minimum of 5 in each group, not just one or two of the groups.

Evaluation findings: Gender based analysis+ 

GBA+ is addressed by the EAS Function to a limited extent, due in part to a lack of available data. Economic analysis, statistical information and advice focus mostly on Indigenous peoples and remote communities. DFO and CCG need to clarify what is required for GBA+ for the department. Women’s full contribution to fisheries, aquaculture and marine conservation, beyond employment, is overlooked and underestimated by many countries including Canada.

The EAS Function is required to provide cost-benefit analyses in support of regulatory proposals and SARA Action Plans. A key component is the socio-economic analysis which addresses the impact of proposals and plans on specific populations, such as Indigenous peoples and remote or low-income communities. The federal government requires consideration of impacts on various groups of men and women, otherwise referred to as GBA+.

The EAS Function has taken GBA+ into consideration to some extent, focusing mostly on Indigenous peoples and remote communities in some of its analyses. Generally, the term is not well known or understood, and it is not clear whether there are under-represented populations relevant for DFO and CCG beyond Indigenous men, Indigenous women, low-income or remote communities, and non-Indigenous women.

GBA+ data is usually obtained from Statistics Canada, Canada Revenue Agency or Employment and Skills Development Canada as they typically collect demographic data. Whether these data sources can provide whatever additional GBA+ data is deemed relevant to DFO or if DFO needs to collect the data itself needs to be determined and is a decision involving Strategic Policy and CCG.

The EAS Function has some GBA+ databases that it could be using more extensively and which others might wish to access as well.

“Indigenous Women…Catalyze Change in Fisheries Governance on Canada’s Pacific Coast”
– 2018 IIFET Newsletter, p. 17

Of seven EAS Directorate guidance documents (e.g. the Framework for Cost-Benefit Analysis of Aquatic Species at Risk Regulations (2016)), three guidance documents mention gender as a factor to consider, among others, e.g., Indigenous peoples, official-language minorities, recent immigrants and lower income Canadians.

In many countries, including Canada, women’s full contribution is often overlooked even though they are a substantial and essential part of the industry. Worldwide, they comprise half the total working population in fisheries and aquaculture, including processing and other related services. Aside from harvesting and processing, women in European fisheries have been involved in at least five main categories including overall management, communication, bookkeeping, marketing, and logistical support, such as picking up equipment and crew.

In Canada, women are involved in attending industry association meetings, preparing license applications, bookkeeping and possibly other essential activities, which are not captured in DFO or other available data.

Evaluation findings: Efficiency

The inclusion of the three programs in the Departmental Results Framework has given the EAS Function access to funds for its economics and statistical work, which had not been previously available through enabler funding. In addition, DFO’s EAS Function resources, when compared to other Economic Analysis and Statistics units in other departments, are proportionately lower.

Efficiency of the EAS function

The inclusion of three EAS programs in the DRF has given the EAS Function much needed access to funds for its economics and statistical work, which had not been previously available through enabler funding. Regional Directors also commented on the importance of the EAS programs as a means of securing funds. The EAS Function at NHQ and in the regions does not receive enabler funding unless directed by the ADM, whereas the EAS programs allows the EAS Function to secure funding through Memoranda to Cabinet.

The economic analysis and statistics units in three other comparable government departments do not have EAS programs as part of their DRFs but instead operate under enabler funding. When the four year average (2015-16 to 2018-19) of expenditures,which includes large data purchases, and FTEs for the economic analysis and statistics units are roughly compared to the total expenditures and total FTEs of the other three departments, the results suggest DFO’s EAS Function resources are proportionately lower compared to the three other government departments.

* 21 FTEs are from NHQ

Evaluation findings: Governance

The EAS Directorate and the regional Economic Directors together provide oversight of DFO’s EAS Function facilitated, in part, by a National Economic Directors Committee. However, greater clarity is required with respect to the EAS Directorate’s role and the NEDC’s in terms of providing guidance, direction and governance of the EAS Function including the programs.

Governance

The EAS Directorate and the regional Economic Directors together provide oversight of DFO’s EAS Function. In the absence of line reporting, the NHQ EAS Directorate and the regional EAS Function directors have established good working relationships. This is facilitated, in part, through a National Economic Directors Committee (NEDC) which holds regular monthly meetings and an annual face-to-face meeting. The committee is unique in that there is no comparable Strategic Policy committee bringing together regional directors and NHQ Strategic Policy counterparts.

According to the NEDC’s Terms of Reference, its mandate, in part, is to be responsible for establishing national processes and guidance documents. The Terms also identify key activities as including, annual economic workplans for the upcoming fiscal year and the development, review and approval of national processes and frameworks. This suggests the NEDC has a key role in the governance of the EAS Function of the Department. In practice, this is not the case. Instead, the NEDC has evolved into more of an information-sharing body, although it has been involved in developing guidance material posted on the EAS Directorate’s intranet webpages.

The EAS Directorate indicates on DFO’s intranet that it is the Departmental Centre of Expertise for economic and statistical analysis and research. This suggests the EAS Directorate has the lead role and not the NEDC. As such, greater clarity is required with respect to their respective roles.

Half of the EAS Function directors are of the opinion that it is not clear if the NEDC or the EAS Directorate should establish national processes and guidance documents. The lack of line reporting between the regions and NHQ is seen as contributing to the lack of clarity between the NEDC and the EAS Directorate. Two-thirds noted there is a lack of clarity with respect to the NEDC’s role regarding the three EAS programs and the performance indicators as these are not part of NEDC discussions.

The three OGDs selected for comparative purposes have a centralized EAS function, which they believe contributes to the efficiency of their operations. While the NEDC helps offset the decentralized nature of DFO’s EAS Function to a limited extent, there is need for greater clarity regarding its role and that of the EAS Directorate for the EAS Function and the EAS programs.

A Different approach

In the Pacific Region, the Policy Analysis and Treaty Support (PATS) unit within the Reconciliation and Partnerships Branch undertakes economic analyses separate from the rest of the EAS Function. PATS provides economic information related to Food, Social and Ceremonial allocations and economic/commercial access provided to Indigenous groups as part of treaty and reconciliation agreements and packages. PATS also provides data with respect to catch reporting which is used to develop financial and allocation mandates for treaty and reconciliation agreements.

Governance of the EAS Function relies on regional staff working closely with national staff to ensure consistency in analytical approaches, which is not the case in this instance (Figure 1). Instead, there is a “wall” between PATS and the EAS Directorate. Lack of a relationship with PATS and access to its data hinder the EAS Directorate’s ability to provide analyses at a national level and ensure consistency in informing the Deputy Minister’s decision-making on treaties.

Figure 1: Disconnect between PATS and EAS Directorate hinders EAS analyses at national level

5.0 Conclusions and recommendations

Conclusions

The EAS Function has established a base in serving client needs and contributing to senior management decision-making. The creation of EAS programs has facilitated access to funding, but there is a lack of awareness and understanding of the services that are available.

According to the evaluation evidence, the EAS Function’s contribution department-wide could be strengthened by two means:

Taking a proactive approach to developing an awareness and understanding of economic analysis, statistical information and advice services and identifying needs across the department; and,

Anchoring the EAS Function around DFO’s mandate, long-term priorities and senior management challenges.

Interviewees commented that a proactive approach would include annual EAS workplans and regular meetings with clients and potential clients department-wide. This would contribute to identifying the data that is required on an ongoing basis to support economic analyses. This in turn identifies the data holders and partners the EAS Function needs to engage with in addition to Statistics Canada, CRA and ESDC, such as Transport Canada.

The EAS Function needs to cultivate a better understanding of how it can be of service to inform MCs, TB submissions and address significant and increasingly complex issues having a bearing on DFO’s and CCG’s mandate and priorities. The EAS Function is best positioned to determine how economics can contribute to DFO and CCG programs and services.

Recommendations

Recommendation #1: EAS function planning

It is recommended that the Senior Assistant Deputy Minister, Strategic Policy, establish a department-wide annual planning process for multi-year workplans for the EAS Function to proactively serve all departmental programs, and Internal Services as applicable, based on the Departmental Results Framework. Workplans and processes for managing emerging priorities should support the EAS Function in ensuring high-quality, consistent and timely economic and socioeconomic information and advice is available across the department to inform senior management decisions, departmental mandates, commitments, challenges and other long-term priorities.

Rationale: A department-wide, proactive EAS Function will facilitate i) developing an awareness and understanding of services available, ii) establishing annual priorities for services, iii) confirming performance indicators or indicating a need for modifications, iv) identifying ongoing data requirements, v) meeting federal GBA+ requirements, vi) addressing complex issues and opportunities arising, and vii) leveraging existing economic analysis, statistical information and advice work. Anchoring EAS workplans and services around the Departmental Results Framework and long-term priorities serves to underpin the of the EAS Function.

Recommendation #2: Performance indicators

It is recommended that the Senior Assistant Deputy Minister, Strategic Policy, ensure the Performance Information Profiles for EAS Function programs incorporate meaningful performance indicators that provide information significant to the achievement of results and management of the programs.

Performance indicators should address Core Responsibility #3 Marine Navigation, as the current

Performance Information Profile does not consider this Core Responsibility. It would either be a separate profile for a fourth EAS program or the programs be amalgamated as one program in the DRF, specifically an all-encompassing program for the EAS Function covering all four departmental Core Responsibilities.

Rationale: Current indicators and targets speak to outputs (number of requests received and dealt with) not whether programs are achieving their intended results or the quality of results. They also do not provide insight on capacity to help inform workload and resource decisions. The EAS Function has for all four departmental Core Responsibilities.

6.0 Appendices: Evaluation methodology and management action plan

Although the evaluation encountered some methodological challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence.

Administrative data

A review of EAS expenditures was undertaken to understand the funding of the EAS programs and services, including in the regions. As the programs were newly created, the financial management systems did not have all the expenditures information for the scope of the evaluation. In addition, EAS in the regions are often part of a directorate which includes other divisions, so expenditures had to be manually segregated. To mitigate this situation, the evaluation team requested the information from the EAS Directorate and the six regions and estimates were used for the analyses. In addition, responsibility for fisheries statistical data in two regions is with a separate sector, Fisheries Management, which is not an EAS program or service. Its resources, therefore, were excluded from the administrative data that was gathered.

Given the recent introduction in 2018-2019 of the EAS programs, the Performance Information Profiles (PIPs) afforded very limited information on results and excluded the information from the regions. To mitigate this challenge, the evaluation assessed the usefulness of the performance indicators and actual results in relation to the intended results and triangulated findings from other lines of evidence.

Document and website review

The evaluation team reviewed information relevant to the EAS Function, such as diagnostic and audit reports on DFO fisheries statistics and data governance, and a review of posted oceans, fisheries and aquaculture information, research, reports, and guidelines.

Mapping exercise

A mapping exercise, during the planning phase of the evaluation, provided the evaluation team with initial insight on the perspectives of the EAS Directorate and regions regarding economic services and activities at the regional and national levels. When possible, the results were used to inform the development of interview guides, the comparative analysis as well as some of the evaluation findings.

Interviews

EAS directors (including from regions) involved in EAS were interviewed. DFO and CCG senior management were also interviewed to gather their views on the services received. Interviews were undertaken with clients that received EAS services from April 2014 to March 2019 (identified by EAS in NHQ and in the regions). Sampling was carried out to ensure that perspectives from different clients (e.g. science, fisheries management, etc.) were gathered to support the results from the survey. Clients who were not interviewed were surveyed.

Comparative analysis

A comparative analysis with three other government departments provided insight on the type of economic analyses and statistical work they undertake, their capacity to respond to requests, whether their economic and statistical services are delivered through a program (i.e., identified as a program on their Departmental Results Framework), policies and processes, GBA+ considerations, whether these departments face similar challenges as DFO, and whether there are other approaches the EAS Directorate might consider.

Documented examples

A brief, focused analysis was undertaken on four examples which contributed to identifying factors that have facilitated or hindered EAS to inform DFO/CCG decisions.

The documented examples were focused on the following themes:

1. Ice Assistance Emergency Program (IAEP)
2. Oceans Protection Plan (OPP)
3. Oceans Program
4. Treaty Support on the West Coast

Survey

A survey with EAS’ internal DFO National Headquarter and regional clients, i.e., to directors and managers, provided input on their needs for economic analysis and statistics services, whether there are issues needing to be addressed and if EAS services have helped to inform DFO/CCG decisions.

The survey was made available online to 140 EAS clients who were identified by EAS directors, and 29 completed survey responses were received which represents a 20% response rate. The survey was administered between July 15 and August 11, 2019. CCG staff were not included in the survey as they were interviewed instead. The survey responses were used to triangulate findings from other lines of evidence. Although there was a low response rate, especially for some questions, findings were used to triangulate evidence from other lines of evidence. In addition, interviews with DFO’s senior management were undertaken as they are the main decision makers and ultimately users of EAS information for evidenced-based decision making.

Understanding of EAS activities

Understanding the EAS activities in the Marine Operations Economics Program was challenging as there were limited activities taking place and CCG clients were not clearly identified. This situation was mitigated by consulting with CCG senior management to understand their capacity and needs regarding EAS as well as triangulating with other lines of evidence when possible.

Management Action Plan (MAP)

Recommendation 1:

Recommendation:

It is recommended that the Senior Assistant Deputy Minister, Strategic Policy, establish a department-wide annual planning process for multi-year workplans for the EAS Function to proactively serve all departmental programs, and Internal Services as applicable, based on the Departmental Results Framework. Workplans and processes for managing emerging priorities should support the EAS Function in ensuring high quality, consistent and timely economic and socioeconomic information and advice is available across the Department to inform senior management decisions, departmental mandates, commitments, challenges and other long-term priorities.

Rationale: 

A department-wide, proactive EAS Function will facilitate i) developing an awareness and understanding of services available, ii) establishing annual priorities for services, iii) confirming performance indicators or indicating a need for modifications, iv) identifying ongoing data requirements, v) meeting federal GBA+ requirements, vi) addressing complex issues and opportunities arising, and vii) leveraging existing economic analysis, statistical information and advice work. Anchoring EAS workplans and services around the Departmental Results Framework and long-term priorities serves to underpin the of the EAS Function.

Management response

Across NHQ and the Regions, the EAS Function conducts a mix of longer-term anticipatory work, as well as a large amount of responsive work related to emerging and urgent policy and management issues. An opportunity exists to improve awareness of EAS services across the Department, and to better align longer-term projects with Program and Departmental priorities.

Link to larger program or departmental results (if applicable)

Addressing this recommendation will allow the EAS Function to support the Department’s progress towards addressing the mission-critical risk of “Information for Decision Making”, which spans all four core responsibilities.

Recommendation 2:

Recommendation:

It is recommended that the Senior Assistant Deputy Minister, Strategic Policy, ensure the Performance Information Profiles for EAS Function programs incorporate meaningful performance indicators that provide information significant to the achievement of results and management of the programs. 

Performance Indicators should address Core Responsibility #3 Marine Navigation as the current Performance Information Profile (PIP) does not cover this Core Responsibility. To do this, a fourth EAS program can be created or the existing programs can be amalgamated as one program in the Program Inventory. Specifically an all-encompassing program for the EAS Function covering all four departmental Core Responsibilities.

Rationale: 

Current indicators and targets speak to outputs (number of requests received and dealt with) not whether programs are achieving their intended results or the quality of results. They also do not provide insight on capacity to help inform workload and resource decisions. The EAS Function has for all four departmental Core Responsibilities.

Management response
Management accepts the recommendation to consolidate the economics programs, and to investigate opportunities for the creation of performance indicators that would provide senior management with actionable information on program performance. Additional efforts will be directed towards aligning EAS functions in both NHQ and regions with the consolidated economics program.

Link to the larger program or departmental results (if applicable)

N/A

7.0 References