Evaluation of the Aquatic Climate Change Adaptation Services Program (ACCASP)

Final Report
Project Number 96265
March 10, 2020

Evaluation of the Aquatic Climate Change Adaptation Services Program (ACCASP)
(PDF - 2.0 MB)

Table of Contents

• 1.0 Evaluation context
  • 1.1 Overview
  • 1.2 Links to previous evaluations and audits
  • 1.3 Evaluation scope
  • 1.4 Evaluation methodology
• 2.0 Program context
• 3.0 Broader climate change context
• 4.0 Summary of key findings
• 5.0 Evaluation findings
  • 5.1 The ACCASP responds to Federal Government and DFO needs to some extent
  • 5.2 Resources are allocated to the three priority areas as designed
  • 5.3 Knowledge gaps remain within and beyond ACCASP priority areas
  • 5.4 Projects were completed as planned
  • 5.5 Gender-based analysis plus (GBA+)
  • 5.6 ACCASP information and tools are used by DFO and external organizations
  • 5.7 ACCASP information and tools are accessible in various ways
  • 5.8 ACCASP information is being used, however, it remains underutilized
  • 5.9 Opportunities to increase the value added of ACCASP research
  • 5.10 Communication challenges
  • 5.11 Internal capacity of the ACCASP is limited
  • 5.12 ACCASP contributions exceed program responsibilities
• 6.0 Lessons learned
• Appendix A: Methodology
• Appendix B: Evaluation matrix
• Appendix C: Performance measurement under the horizontal management framework for clean growth and climate change
• Appendix D: Broader climate change context

1.0 Evaluation context

1.1 Overview

This report presents the results of the Evaluation of Fisheries and Oceans Canada’s (DFO) Aquatic Climate Change Adaptation Services Program (ACCASP).

The evaluation was conducted as identified in DFO’s 2019-20 to 2023-24 Departmental Evaluation Plan, and as required by the Treasury Board Secretariat for the Horizontal Roll-up Evaluation of the Adapting to the Impacts of Climate Change Initiative, which is led by Environment and Climate Change Canada.

The evaluation was conducted by DFO’s Evaluation Division between August 2019 and January 2020 in accordance with the Policy on Results (2016) and included all DFO regions: National Capital Region (NCR); Newfoundland and Labrador (N&L); Maritimes; Gulf; Quebec; Central and Arctic (C&A); and Pacific. The evaluation covered a period of four years from 2016-2017 to 2019-2020.

1.2 Links to previous evaluations and audits

The 2017 Horizontal Evaluation of the Adaptation Theme^{Footnote 1} assessed the overall and performance of the federal government’s Clean Air Agenda Adaptation Theme to which the ACCASP contributed as one of 10 program elements. In response to the recommendations of the evaluation, the ACCASP developed a management action plan^{Footnote 2}; then implemented it by 2018-19.

The 2017 Commissioner of the Environment and Sustainable Development (CESD) Report “Adapting to the Impacts of Climate Change”^{Footnote 3} examined the extent to which 19 federal departments and agencies implemented the Federal Adaptation Policy Framework. The framework states that all federal departments and agencies must take action and consider climate change impacts in their programs, policies, and operations.

• DFO was one of the five departments that identified and assessed its climate change risks and initiated measures to adapt to climate change in their areas of responsibility. The ACCASP played a key role in the departmental efforts and results.

DFO’s Evaluation of Science Funding (2018-19)^{Footnote 4} assessed the efficiency of 16 funding programs, including the ACCASP, of the Ecosystems and Oceans Science (EOS) sector and whether the application, selection and/or administration processes support equity and diversity.

• The evaluation made recommendations related to the overall research funding allocation process of the EOS sector. In response, EOS is developing a new funding allocation system by March 2021. The new system will increase efficiencies, standardize communication and establish a formal priority setting mechanism within key client sectors.
• The current evaluation incorporates findings from the Evaluation of Science Funding. For the year that the ACCASP used a competitive funding process (2018-19), ACCASP data was disaggregated and analyzed.

1.3 Evaluation scope

The evaluation was designed as an outcome evaluation and covered the period from 2016-2017 to 2019-2020. The evaluation assessed the ACCASP’s achievements toward the expected results, and the efficiency of the program in delivering on its objectives.

Each of the four years within the scope of the evaluation reflects a different funding context for the ACCASP.

• 2016-17: The ACCASP was extended for 2 years under the Clean Air Agenda. Budget 2016 allocated $2.8 M to the program.
• 2017-18: The ACCASP was renewed under the Pan-Canadian Framework. Budget 2017 allocated $700K to the program in 2017-18, in addition to the $2.8M allocated in 2016-17. Total program funding was $3.5M.
• 2018-19: The ACCASP began receiving A-base funding of $3.5M per year under ECCC’s Adapting to the Impacts of Climate Change Initiative.
• 2019-20: The ACCASP continued to receive $3.5M A-base funding. Only a portion of this year falls within the scope of the evaluation^{Footnote 5}.

1.4 Evaluation methodology

The evaluation examined the questions presented below. It used multiple lines of evidence, both qualitative and quantitative, which were triangulated to mitigate potential limitations. The methodology included: document review, interviews, administrative data analysis, financial analysis, webometric analysis, and analysis of data obtained from DFO’s Evaluation of Science Funding (2018-2019).

1. What needs does the ACCASP address?
2. To what extent is the research funded by the ACCASP available to users?
3. To what extent is the research funded by the ACCASP being used?
4. How efficient is the ACCASP research funding cycle?
5. To what extent are ACCASP resources appropriate to support the achievement of its objectives?
6. To what extent does the ACCASP address challenges and explore opportunities for improvements?
7. Are the application and approval processes for ACCASP research funding equitable to all potential applicants?

The evaluation methodology and limitations are presented in Appendix A. The evaluation matrix is presented in Appendix B.

2.0 Program context

The ACCASP is an internal science-funding program that allocates research funds to DFO scientists. The ACCASP conducts scientific research and monitoring activities necessary to identify climate change impacts and vulnerabilities, improve ocean forecasting and develop adaptation information and tools for vulnerable coastal regions. The objective of the ACCASP is to inform DFO’s adaptation decisions related to fisheries, ocean management and coastal infrastructure.

The ACCASP receives $3.5M per year (A-Base) under Environment and Climate Change Canada‘s (ECCC) Clean Growth and Climate Change – Adaptation (CGCC) initiative. Eight (8) full time equivalents (FTEs) are funded under the program. Six (6) FTEs, one in each region, conduct regional research activities and participate in national ACCASP working groups. Two (2) FTEs, based in the NCR, are responsible for program management, oversight and coordination at the regional and national level. The NCR team also acts as the point of contact for internal and external requests related to climate change adaptation.

The ACCASP contributes to the objectives and activities of DFO and the Government of Canada’s Pan-Canadian Framework for Clean Growth and Climate Change (PCF) by supporting the aquatic climate change research, communication and collaboration that lead to data acquisition and adaptation knowledge creation. Relevant and reliable science information is needed to inform decision-making and respond to climate change commitments from a departmental and federal government perspective.

At the federal level:

• The ACCASP reports program outcomes to ECCC under the Horizontal Management Framework (HMF) for Clean Growth and Climate Change. All relevant horizontal outcomes and indicators are presented in Appendix C.
• The ACCASP has contributed to the Adaptation and Resilience Pillar of the PCF since 2017-18.

At the departmental level:

• The ACCASP falls under the Ocean and Climate Change Science Program within DFO’s Departmental Results Framework. The program contributes to the Aquatic Ecosystems Core Responsibility to conserve and protect Canada’s oceans and other aquatic ecosystems and species from human impact and invasive species.
• The ACCASP directly supports the departmental result that ‘Scientific information on Canada’s oceans and other aquatic ecosystems is available to inform management decisions’.

The program is designed to advance science activities in three closely interlinked priority areas:

1. Ocean Chemistry: ACCASP collects ocean chemistry data to assess the current and expected extent of ocean acidification and hypoxia; study the interaction of ocean acidification and hypoxia with other climate stressors and conditions (i.e., temperature, ocean circulation, fresh water input); and help understand how these interactions affect fish and other aquatic species.
2. Vulnerability of fisheries and coastal infrastructure to climate change: ACCASP conducts:
   • Fisheries vulnerability assessments that consider levels of exposure to climate stressors and the resiliency and adaptive capacity of fish species and stocks to determine how these factors could change their distribution, productivity and abundance; and
   • Vulnerability assessments for coastal infrastructure and coastal areas that consider the climate change impacts of sea-level rise, frequent storm surges and changing ice conditions.

   ACCASP develops adaptation tools and methods to inform planning and management decisions when developing adaptation strategies in order to address and minimize risks to fisheries and coastal communities.

3. Refining applied ocean models. ACCASP produces ocean models and simulations that incorporate past and present assumptions as well as direct observations to predict future climate and ocean conditions (i.e., temperature, currents and ocean chemistry). The ACCASP also refines ocean models to improve the accuracy of predictions, develop smaller-scale predictions that are specific for individual communities and extend model simulations over longer timescales.

The three priority areas were determined based on previously identified risks. In 2013, the ACCASP conducted risk-based assessments of climate change impacts to Canada’s four large aquatic basins over a 10, and a 50 year horizon.

The Arctic, Pacific, Freshwater, and Atlantic aquatic basin risk assessments considered how climate change could impact the four regions in terms of six pre-identified climate change risks to DFO sectors, infrastructure and operations^{Footnote 6}:

3.0 Broader climate change context

The ACCASP operates within a context of high demand for aquatic climate change research to inform adaptation efforts at international, national, departmental, and regional levels. The ACCASP does not have direct responsibilities beyond supporting aquatic climate change research for DFO. Nevertheless, the ACCASP contributes a unique expertise at the federal level. The department is required to ensure scientific information is available to support broader responsibilities and commitments that may go beyond ACCASP science, climate change science in general, and adaptation.

Demand for aquatic climate change adaptation services is driven at the international, national and departmental levels.

International climate change drivers

• Commonwealth Blue Charter (2018)
• High Level Panel for a Sustainable Ocean Economy (2018)
• North-American Leaders Summit (2016)
• UN 2030 Agenda for Sustainable Development (2015)
• UNFCC Paris Agreement (2015)
• International Panel on Climate Change (IPCC)(1998)
• Arctic Council (1996)
• UN Framework Convention on Climate Change (1992)

Canadian climate change initiatives

• Federal Sustainable Development Strategy (2019-22)
• Greening Government Strategy (2017)
• Pan-Canadian Framework on Clean Growth and Climate Change (2016)
• Federal Adaptation Policy Framework (2011)

DFO responsibilities

• Legal obligations under the Oceans Act (1996); Federal Sustainable Development Act (2008); Fisheries Act (1985), including Bill C68 amendments (2019)
• Federal commitments under the Federal Sustainable Development Strategy (2019-22), Greening Government Strategy (2017); Pan-Canadian Framework (2016); and Federal Adaptation Policy Framework (2011)
• Minister of Fisheries, Oceans, and the Canadian Coast Guard Mandate Letters
• Commitments under the Horizontal Management Framework for Clean Growth and Climate Change (2016)
• DFO’s Corporate Risk Profile (2019-20)

More details on the evolution of these drivers and resulting pressures on the ACCASP are presented in Appendix D.

4.0 Summary of key findings

Under its current mandate and design, the ACCASP performed well. However, in the evolving climate change context, the needs for aquatic climate change science information and tools have also evolved within and beyond the ACCASP’s Ocean Chemistry, Vulnerability Assessment and Ocean Modelling priority areas.

The ACCASP responds to needs to some extent

• The ACCASP responds to needs for aquatic climate change science information and tools. ACCASP-funded projects and ongoing monitoring activities advance aquatic climate change science to support adaptation efforts, as per the program’s objectives. Most activities have been completed as planned and on schedule.
• However, the demand for ACCASP-generated information is greater than what the program can provide under its existing design and resources. Outstanding and evolving science needs for evidence-based decision-making in support of adaptation efforts for DFO and the Government of Canada are broader than the ACCASP’s current mandate.
• The practice of distributing research funds by DFO’s six regions is in line with the actual regional distribution of scientific expertise; however, it results in differential knowledge acquisition across Canada's three coasts because four of the six DFO administrative regions are located in the Atlantic coast.

ACCASP information and tools are underutilized

The ACCASP produces research of high scientific value. ACCASP research has been used by DFO, and across governments and private organizations. The broader science community in Canada and internationally also uses ACCASP research and data. Nevertheless, ACCASP-generated tools and information are not well known and therefore remain underutilized.

• ACCASP information and tools are primarily used by early adopters while potential users have been identified and are likely to increase as more DFO programs and external organizations increase their climate change adaptation efforts.
• A number of challenges, common to science-oriented programs, impede the communication of ACCASP information and tools to a broader audience to inform decision-making in support of adaptation efforts. Moreover, the ACCASP does not carry out formal client engagement or outreach strategies since the program is not designed or funded for this purpose.

The ACCASP fulfills a role beyond its mandate

• ACCASP management staff fulfill a growing role within the department and are involved in activities that are driven by an evolving broader climate change context. These activities, which have increased continuously since 2017, exceed the program’s direct responsibilities.
• The program does not have the sufficient internal capacity to fulfill this expanding role and this further detracts resources from being focused on ACCASP-specific activities.

5.0 Evaluation findings

5.1 The ACCASP responds to Federal Government and DFO needs to some extent

Adapting to the expected impacts of climate change and minimizing associated risks requires aquatic climate change science information to support evidence-based decision-making.

At the federal government:

• The federal government is a unique source of authoritative climate change science information supporting adaptation efforts in Canada.
• The ACCASP is the only federal program that advances research in the area of aquatic climate change science in support of federal adaptation efforts, including ECCC’s Horizontal Initiative on Clean Growth and Climate Change and the Pan-Canadian Framework.
• Under the PCF and the Horizontal Initiative on CGCC, the ACCASP, on behalf of DFO, is responsible for activities that support adaptation in vulnerable coastal and northern regions.

At Fisheries and Oceans Canada:

• By supporting DFO’s departmental result 2.2, Scientific information on Canada’s oceans and other aquatic ecosystems is available to inform management decisions, the ACCASP contributes information that supports several of the department’s key priorities.
• ACCASP research also contributes to DFO commitments to use good scientific evidence and traditional Indigenous knowledge when making decisions affecting fish stocks and ecosystem management (2019 DFO Mandate Letter), and to ensure effective use of research resulting from restored federal funding for freshwater research, federal ocean science and monitoring programs among others (2018 DFO Mandate Letter).

A number of factors suggest that, overall, science needs to support evidence-based decision-making within the ACCASP’s three priority areas will increase

• Climate-induced changes in the ocean are occurring. There will be a continuous need for monitoring activities to provide a comprehensive assessment of the current state and extent of changing ocean conditions.
• The body of knowledge on aquatic climate change science is advancing, and with it, new gaps and areas of importance. For example, the prominence of research on ocean acidification had not been anticipated ten years ago.
• The technology that is available to conduct aquatic climate change research is evolving. This will further drive science needs as, for example, areas that were previously inaccessible become more accessible.
• Awareness of climate change risks and adaptation needs is increasing among potential users within and outside the department. The need to assess vulnerabilities and develop adaptation tools for new clients is likewise expected to increase.
• Demand for research that is integrated across ACCASP’s complementary priority areas and research themes is increasing.
• Demand for research that integrates additional factors in conjunction with ocean chemistry, vulnerability assessments and modelling activities is increasing. Factors such as socio-economic impacts on affected communities are critical and will be needed to better understand complex phenomena and adopt effective measures.

Evolving needs lie beyond the current mandate of the ACCASP to address them

• ACCASP priorities and resources were determined based on assessments of DFO climate-change risks as of 2016, as well as the state of technology and scientific knowledge at that time.
• Meanwhile, the broader climate change adaptation context and related science needs continue to evolve. DFO and the federal government are expected to respond to these evolving demands and responsibilities based on sound science advice.
• As the only federal program advancing the aquatic climate change science body of knowledge, the ACCASP is unable to address evolving needs within the existing program design and resources.

More details on the increased pressures on the ACCASP as a result of the evolving broader climate change context are presented in Appendix D.

5.2 Resources are allocated to the three priority areas as designed

The ACCASP has advanced research in ocean monitoring, vulnerability assessment and modelling activities.

• DFO undertook a science advisory peer-review process to develop a framework to enable the systematic integration of climate change, oceanographic and ecological stressor data and fish stock vulnerability information into stock assessments to enable climate-ready decision-making in fisheries resource management. 22% of fisheries stock assessments now incorporate climate change considerations. The program is on track to reach a target of 100% by 2026.
• Long-term baseline ocean chemistry data has been collected for offshore areas; more coastal ocean acidification monitoring is required. Foundational knowledge on the acidification of Canada’s oceans has been augmented.

Since 2017-18, the total budget identified in the ACCASP foundational documents is $3.5M per year and includes salaries for eight FTEs, costs for internal services, and operations and maintenance (O&M). The O&M research budget lies between 58% and 66% of total program funds. The number of FTEs has remained unchanged. The figure represents the planned percent distribution of the total ACCASP budget, by year. The actual percent distribution of the O&M research funds is also provided for comparison. The variance between the planned and the actual O&M distribution ranges from 4% to 13%.

Long description

Four pie-charts represent the planned percent distribution of the total ACCASP budget, by year, for the four years 2016-17, 2017-18, 2018-19 and 2019-20. In 2016-17, the planned percent distribution was: 26% salary; 16% internal services, and 58% O&M. Furthermore, the O&M funds were distributed by the three priorities as follows: 33% Ocean Chemistry (OC); 46% Vulnerability and 21% Monitoring. In 2017-18, the planned percent distribution was: 21% salary; 13% internal services, and 66% O&M; and the O&M funds were distributed by the three priorities as follows: 30% Ocean Chemistry (OC); 50% Vulnerability (V) and 20% Monitoring (M). In 2018-19, the planned percent distribution was: 22% salary; 15% internal services, and 63% O&M; and the O&M funds were distributed by the three priorities as follows: 35% Ocean Chemistry (OC); 44% Vulnerability and 21% Monitoring. In 2019-20, the planned percent distribution was: 22% salary; 15% internal services, and 63% O&M; and the O&M funds were distributed by the three priorities as follows: 35% Ocean Chemistry (OC); 44% Vulnerability and 21% Monitoring.
Four smaller pie-charts represent the actual % distribution of research funds by year and priority area, as follows: 2016-17: OC 37%; V 38%; M 25%; 2017-18: OC 25%; V 58%; M 17%; 2018-19: OC 30%; V 38%; M 32%; 2019-20: OC 28%; V 38%; M 34%; The distribution of FTEs by priority area is: 2 for OC; 4 for V and 2 for M.

Long Description

The bar graphs represent the actual O&M expenditures by year, priority, and total, in $K. They are: $1602K in 2016-17; $2252K in 2017-18, $1846K in 2018-19 and $1836K in 2019-20. A dashed line shows the planned total O&M amounts. They are: $1577K in 2016-17; $2231K in 2017-18, and $2234K in 2018-19 and 2019-20. In 2018-19 and 2019-20 the actual O&M expenditures are 83% and 82% of the planned O&M, respectively.

O&M actual expenditures are broken into four categories: Modelling (M), Ocean Chemistry (OC), Vulnerability (V), and NCR support^{Footnote 7}.

In 2016-17 and 2017-18, the O&M budget included a dedicated amount of $90K per year (in addition to the research funding) to conduct national synthesis and communication of research findings. Since 2018-19, communication expenditures and other NCR support have been sourced from O&M research funds.

Since 2018-19, actual expenditures supporting ACCASP research activities were slightly above 80% of the planned O&M budget. Several factors likely contributed to that:

• As an A-Base funded program, the ACCASP is subject to variable reductions that take place after the initial annual budget has been allocated (e.g., financial support for corporate initiatives to enhance infrastructure and financial pressures at the sector/regional level)^{Footnote 8}.
• The cost of employee benefits increased. This was not anticipated in the initial program budgets.

5.3 Knowledge gaps remain within and beyond ACCASP priority areas

Long description

There are three ring graphs showing the percentage of responses by ACCASP priority areas. For Ocean chemistry, 79% of responses are “very important”; 16% - “important”, and 5% - “somewhat important”. For Vulnerability: 89% of responses are “very important”; 6% - “important”, and 5% - “do not know”. For Modelling: 58% of responses are “very important” and 42% - “important”.

Vulnerability assessments were viewed as most critical based on the magnitude, scope, and urgency of the departmental needs and gaps that are and will continue to be addressed (e.g., annual demands for information to enable adaptive responses to the climate change induced events that are already taking place).

Aquatic climate change science in support of adaptation efforts could be advanced within each priority area to address current knowledge gaps as well as emerging science needs that lie beyond the ACCASP’s current design.

5.4 Projects were completed as planned

Monitoring activities advance research objectives under the Ocean Chemistry priority; directed funds^{Footnote 9} are provided on an annual basis per region.

Research projects advance objectives under Vulnerability Assessment and Ocean Modelling priorities; projects must demonstrate a clear link to a priority area and are assessed based on different criteria such as scientific rigor, feasibility, and timing.

• Prior to 2015-16, a competitive funding process^{Footnote 10} was used.
• In 2016-17 and 2017-18, the funding allocation process was informal due to the timing of the program renewal. The regions proposed new projects, recommendations based on DFO risks were made by the NHQ National Manager and Regional Science Managers, then the Science Executive Committee made decisions.
• In 2018-19, a competitive funding process was used and 16 successful research projects received funding for multiple years. Planned funds for 2019-20 were distributed for Year 2 activities of these multi-year projects.

Long description

The number of funded projects by year and priority area is presented, as follows:

• In 2016-17: OC – 6 projects; V – 11 projects; M – 12 projects; total – 29 projects.
• In 2017-18: OC – 6 projects; V – 20 projects; M – 11 projects; total – 37 projects.
• In 2018-19: OC – 6 projects; V – 10 projects; M – 7 projects; total – 23 projects.
• In 2019-20: OC – 6 projects; V – 9 projects; M – 7 projects; total – 22 projects.

Long description

The status of 89 funded projects from 2016-17 to 2018-19 is as follows:

• In 2016-17: 25 - completed as planned; 1 - completed with a delay, and 3 - unknown status;
• In 2017-18: 32 - completed as planned; 1 - completed with a delay, and 4 - unknown status;
• In 2018-19: 16 - completed as planned; 7 – On track as planned.

Note: Year-end reports for projects funded in 2019-20 were not submitted as of December 2019.

• 90% of research projects were completed as planned and on schedule.
• In 2016-17 and 2017-18, seven projects were funded for which researchers did not provide reports; hence, their status is unknown. Under the current funding requirements, there are no consequences if funded researchers do not submit reports, or reports are delayed.
• In 2018-19, the 16 projects that are on track were funded for multiple years.

5.5 Gender-based analysis plus (GBA+)

The Evaluation of Science Funding conducted a GBA+ analysis^{Footnote 12} that considered the differential success rate of diverse groups of applicants in the 2018-19 EOS funding cycle (n=69). The evaluation explored gender and other identity factors to assess barriers to participation.

• ACCASP applicants identified as being a man or a woman in similar proportions, while 12% preferred not to answer.
• ACCASP applicants are more often late career researchers (52%) than mid (31%) and early (10%) career researchers^{Footnote 13}.

The evaluation found that 57% of applicants were successful at receiving funds from the ACCASP. Successful ACCASP applicants were approved for funding in the same proportion by gender, age, and career status indicating there are no barriers with respect to these identity factors.

By proportion of successful applicants, women’s success rate (50%) was higher for the ACCASP compared to the overall EOS sector (Men: 57% and Women: 43%).

Long description

This pictogram shows that 45%(n=31) of the applicants were female and 43% (n=30) were male.

This pie chart shows that 50% of successful applicants were men; 42% were women. 8% of successful applicants did not specify their gender.

Long description

This bar graph shows the career status of ACCASP applicants: 10% - early career; 31% - mid-career; 52% - late-career, and 7% -other.

This pie chart shows the career status of successful applicants who were awarded funding: 11% - early career; 26% - mid-career; 55% - late career; and 8% - other.

Long description

The pie-chart on the left shows that the % distribution of applicants by DFO region is: 16% - Central and Arctic; 7% - Gulf; 29% - Maritimes; 17% - N&L; 24% - Pacific; and 7% - Quebec.

The pie chart on the right shows that the % distribution of successful applicants by DFO region is: 17% - Central and Arctic; 4% - Gulf; 31% - Maritimes; 17% - N&L; 21% - Pacific; and 10% - Quebec.

This indicates there are no barriers related to geographic location.

Long description

• 21% of all research funds (or $1429K) from 2016-17 to 2019-20 were allocated in the Pacific coast;
• 13% of all research funds (or $889K) from 2016-17 to 2019-20 were allocated in the Arctic coast; and
• 66% of all research funds (or $4407K) from 2016-17 to 2019-20 were allocated in the Atlantic coast.

• When research activities are grouped by Canada’s coasts, more projects are funded on the East coast since four out of six DFO administrative regions are located in the Atlantic coast (Newfoundland & Labrador, Maritimes, Gulf and Quebec) versus one each on the Pacific and Arctic coasts.
• ACCASP researchers on the East coast benefit from potential opportunities to collaborate based on research synergies (for example, focal species of interest). Based on geography, there are less potential opportunities for knowledge-sharing and collaboration in the Pacific and Central & Arctic regions.
• ACCASP research in the Pacific and C&A regions is also affected by the differential cost of conducting research activities due to the size and geography of the regions. For example, Arctic research is critical for informing subsistence and commercial harvesting practices in areas where climate change impacts food security, as per DFO’s mandate^{Footnote 15}. However, the Arctic lacks necessary baseline monitoring data and costs associated with working in the Arctic are significantly higher than any other region. Research activities on the West and East coasts include arctic components, yet the rate of knowledge acquisition is slowest in the C&A region.
• Resulting gaps in the aquatic climate change science available to support adaptation efforts may impact DFO’s ability to meet commitments regarding vulnerable coastal and northern regions outlined in the 2019-22 Federal Sustainable Development Strategy and the Pan-Canadian Framework.

5.6 ACCASP information and tools are used by DFO and external organizations

ACCASP research has been used to develop adaptation tools for internal and external users to inform decisions related to managing fisheries, oceans and coastal infrastructure and help prepare for expected future changes. Tools are available via the Turning Science Into Action page, accessible on ACCASP’s Aquatic Climate Change Science website. ACCASP research is also available on www.sealevelrise.ca, which was developed by the Ecology Action Centre in partnership with DFO and other partners to inform Atlantic Canadians and help local authorities and developers enhance future community planning.

The ACCASP has developed three main tools:

1. Coastal Infrastructure Vulnerability Index (CIVI)^{Footnote 16}: The index reflects a site’s vulnerability to sea-level rise, storm surge, and other effects of climate change. It incorporates harbor engineering and socio-economic indicators. The CIVI is not publicly accessible.
2. Canadian Extreme Weather Water Level Adaptation Tool (CAN-EWLAT)^{Footnote 17}: The tool provides sea-level rise projections for Canada’s coastline and advice on how much higher to build coastal infrastructure to accommodate sea-level rise projections. CAN-EWLAT is available online to the public.
3. Fish Stock Climate Vulnerability Assessment Tool (FSCVAT)^{Footnote 18}: The tool helps fisheries managers determine which commercially valuable species are most vulnerable to climate change. FSCVAT is not publicly accessible.

Examples of users of the ACCASP tools include:

• DFO’s Small Craft Harbors Program uses the CIVI and CAN-EWLAT to incorporate climate change adaptation measures in all sites, except one in the Arctic.
• DFO’s Real Property is exploring the potential for using ACCASP tools to assess risks to DFO’s coastal infrastructure.
• Municipal partners access ACCASP information through Sealevelrise.ca to plan for flood risk, disasters, and predicted changes in fisheries distributions.
• The Geological Survey of Canada applies the CIVI to inform their Coastal Climate Geoscience Program to help coastal communities plan for regional sea level rise.
• Port authorities and private insurance companies have sought out ACCASP information to develop emergency plans, conduct environmental assessments, and implement insurance policies based on flood risk vulnerability.

5.7 ACCASP information and tools are accessible in various ways

Following the recommendations of the 2017 Evaluation of the Horizontal Adaptation Theme, the ACCASP updated the program’s Aquatic Climate Change Science website^{Footnote 19}. The program facilitated access to information and tools for different levels of users: waders, swimmers, and deep divers.

• The program has made efforts to increase the interpretability of its information. The website now includes a series of videos and infographics related to Ocean Acidification and Hypoxia.

• The program has made efforts to increase the accessibility of its information. The ACCASP website continues to be developed to include all completed research projects, including project summaries and links to relevant academic articles, technical reports^{Footnote 20} and manuscript reports^{Footnote 21}.
• 75% of ACCASP science products were available on its website within three months of publication^{Footnote 22}.

The following several sections describe how ACCASP information is accessible through the program website, various reporting initiatives and academic journals.

Long description

The graph shows the number of visits to four key ACCASP webpages from Nov.2017 to Aug.2019: ACCASP Index Page; Changing Ocean Chemistry; Impacts on ecosystems and fisheries; and Understanding aquatic climate change risks.

The visits to the ACCASP Index page increased from 50 in Nov.2017 to about 400 in Nov.2018, and varied within the range 50-100 per month since then. A direct source generated more than 50% of the peak traffic to the ACCASP index page between Sept.2018 and Jan 2019.

The traffic to the Changing Ocean Chemistry page and the Impacts on Ecosystems and Fisheries page varied in similar levels between 50 and 150 per month. Google searches generated 73% of the increasing traffic to the Changing Ocean Chemistry page and 71% of the traffic to the Impacts on Ecosystems and Fisheries page in October 2019.

Google searches generated 80% of the traffic to the Changing Ocean Chemistry page in November 2018. The traffic to the Understanding Aquatic Climate Change Risks page was around 50 visits per month most of the period.

• The entire ACCASP site reached peak traffic (> 700 visits) between October and November 2019. Most likely this is due to the increased number of visits to the Changing Ocean Chemistry and Impacts on Ecosystems and Fisheries page.
• ACCASP tools are available in the Turning Science into Action. This page has generated low traffic (171 visits) since its launch in July 2018.
• Traffic to the Canadian Extreme Weather Water Level Adaptation Tool (CAN-EWLAT) summary page has been moderate (60-140 visits) but consistent for the past two years. This likely demonstrates the tool has a modest, but dedicated audience.

Long description

The sources of traffic to the CAN-EWLAT tool are: Google (42.5%); Sealevelrise.ca (19%); Direct source (18.9%); DFO website (2%); Other (18%).

• As of November 2019, 84 of the 129 scientific research publications attributable to ACCASP are available on the Web of Science.
• On average, each article has been cited 5.15 times. The number of citations for the 84 indexed journal articles has been increasing^{Footnote 24}.
• ACCASP research is used around the world. ACCASP research is largely cited by Canadian universities and federal departments, research institutes, international partners (e.g., National Oceanic and Atmospheric Administration (NOAA)), and foreign governments.

Long description

The geographical regions of citing articles are: Europe (41%); Canada and USA (38%); Asia (12%); Australia (5%); New Zealand (2%); South Africa (1%) and Brazil (1%).

The top 7 institution groups for articles citing ACCASP research are: Canadian Universities (28%); DFO (25%); Research institutes (22%); Universities (12%); NOAA (7%); US Government (5%) and ECCC (2%).

The number of citations has been steadily increasing from 1 in 2012, to 141 in 2019.

Since 2015, the ACCASP provided input to 49 external and DFO reports; many in response to requests from the Canadian Science Advisory Secretariat. The program is meeting a target of one departmental and/or national report that has incorporated aquatic climate science research findings per year starting in 2017-18.

The list contains some of the reports where ACCASP contributed:

Departmental reports

• State of the Atlantic Ocean Report (2018)
• Framework for Incorporating Climate Change Considerations in Stock Assessments (2018)
• Atlantic Zone Monitoring Program Science Advisory Report (2017)

National and federal reports

• Natural Resources Canada. Canada’s Changing Climate Report (2019)
• Northwest Territories. Status of the Environment Report (2019)
• Canadian Meteorological and Oceanographic Society. Special Reports (2015 and 2019)
• Arctic Monitoring and Assessment Programme. Arctic Ocean Acidification Assessment; Case 5: Climate change impacts on subsistence fisheries in the Western Canadian Arctic (2018)
• Natural Resources Canada. Canada’s Marine Coasts in a Changing Climate (2016)
• Expected in 2020-21: Environment and Climate Change Canada. National Climate Change Science and Knowledge Plan

International reports

• UN International Panel on Climate Change (IPCC). Special Report on Oceans and Cryosphere (2019)
• Arctic Monitoring Assessment Programme. Arctic Ocean Acidification – Second report (2018)
• International Ocean Institute. Ocean Acidification in Canadian Waters, Ocean Yearbook “Future of Ocean Governance and Training” (2018)

5.8 ACCASP information is being used, however, it remains underutilized

DFO – Ecosystems and oceans science sector

• The EOS sector is the primary intended user of ACCASP information. ACCASP research activities complement ongoing science programs and contribute to EOS objectives and results.
• However, ACCASP contributions to the EOS may not always be clearly identified or explicitly articulated.

ACCASP applicants are encouraged to identify clients in research proposals. Nonetheless, interviewees believe that the program has low visibility and is not well-known by clients.

• The ACCASP also provides science advice to the EOS sector through the Canadian Science Advisory Secretariat (CSAS) process^{Footnote 25}. While a CSAS response usually involves the EOS sector at large, the ACCASP provides input specific to climate change related management needs.

41% of the ACCASP applicants consulted for the Evaluation of Science Funding expected that almost half of ACCASP projects result in a CSAS Science Advisory report.

DFO – Other non-science sectors

• The Small Craft Harbors Program uses ACCASP information and tools for asset management, harbor planning and construction, and implementing climate change adaptation measures in the coastal areas. CIVI was specifically developed for this purpose.
• Resource Management uses ACCASP’s vulnerability assessments, tools and frameworks to inform decision-making related to sustaining and protecting fisheries and fish stocks of commercial value.

ACCASP has contributed to the Ecosystem Approach to Fisheries Management (EAFM), lead by Fish Population Science, since its inception in 2019. EAFM is a great example of cross-sector collaboration for policy development based on sound aquatic climate change science advise.

• Real Property has identified collaboration with ACCASP as a potential area for future work and is currently in the stage of informal discussions. ACCASP information and tools could be used to address DFO’s and the Canadian Coast Guard’s infrastructure risks along the coastline that may result from the impacts of extreme weather, sea-level rise or ice conditions.

While Real Property is a key player for addressing DFO commitments under the Greening Government Strategy (GGS)^{Footnote 26}, the scope of the requirements are broader. Thus, other DFO groups (Ocean Management, Canadian Coast Guard’s Search and Rescue, Aquaculture Management) have been identified as potential users of ACCASP information.

Federal government departments

• ECCC is a key partner of the ACCASP as the Department lead for the Horizontal Initiative on CGCC, to which the ACCASP reports on results.

ACCASP is collaborating with ECCC to develop the National Climate Change Science and Knowledge Plan to support the delivery of the PCF. The ACCASP leads on the Aquatic Systems section.
ECCC’s Canadian Centre for Climate Services (CCCS) will house products and services to provide Canadians with authoritative information and tools to respond to climate change.

• The ACCASP also collaborates with Natural Resources Canada on their Adaptation Platform.

The Adaptation Platform promotes ACCASP research and tools such as CAN-EWLAT to help Canadians, non-governmental organizations, as well as provincial, territorial and municipal governments adapt to climate-change impacts.

Other canadian users

Early users of ACCASP tools and information were identified among public and private organizations in Canada. They include northern and coastal communities; provincial/territorial governments; insurance, fishing and engineering companies; and port authorities. Information is used in planning and design, infrastructure and asset management, emergency preparedness, and business decisions.

Academia (canadian and international)

Universities and research institutions from Canada and abroad access ACCASP publications in scientific journals and databases to support their research activities^{Footnote 27}.

International users

ACCASP contributes research findings to a broader international domain of data and information on oceanic climate change impacts. International organizations also share ACCASP input to a worldwide audience:

• The National Oceanic and Atmospheric Administration (NOAA) collaborates with the ACCASP to coordinate monitoring efforts and knowledge in the Pacific and Atlantic coasts. It also advances the understanding and responses to climate impacts on fisheries. The National Centre for Environmental Information (NCEI) data portal hosts ACCASP data produced in collaboration with NOAA.
• The Global Ocean Acidification – Observation Network (GOA-ON) maintains an international portal for ocean monitoring data to which ACCASP contributes.
• The Arctic Council’s Arctic Monitoring Assessment Programme (AMAP).

ACCASP provides Canadian information on UN Sustainable Development indicator 14.3 Ocean Acidification. This information will be available through the Federal Sustainable Development Data Hub hosted by Statistics Canada.

5.9 Opportunities to increase the value added of ACCASP research

ACCASP information is available to support decision-making at departmental, federal and international levels. However, under the current design, the ACCASP does not engage with current and potential users which results in low visibility for the program. Nevertheless, user engagement still occurs in three key ways:

1. Researchers receive ad-hoc requests from potential users based on existing personal working relationships. This type of engagement is opportunity-based and varies by region.
2. Researchers co-develop research projects with northern communities and land-claim co-managers in the C&A region. This leads to face-to-face meetings and training sessions with Territorial governments and independent northern organizations.
3. Researchers provide input related to climate-change to CSAS requests for the EOS sector.

Several approaches have been suggested to increase the value-added of ACCASP research. However, a number of factors impede the implementation of these approaches. However, some of the potential solutions are beyond the scope of the current program.

5.10 Communication challenges

As subject-matter experts, the program could coordinate efforts to improve the communication, interpretability and applicability of ACCASP research. However, the following challenges impede broader communication of ACCASP research.

Low degree of awareness of the ACCASP at DFO

• The ACCASP is not widely known. ACCASP research has low visibility since program information is largely integrated into broader reporting initiatives and therefore program contributions are not obvious or advertised.
• The program’s ability to increase its profile within the department is limited. The ACCASP lacks the internal capacity necessary to enable collaboration and coordination efforts across the EOS sector and DFO.
• DFO programs are not systematically incorporating climate change considerations in their program design and delivery since there is no departmental guidance to do so. Program-specific science needs for aquatic climate change research in support of adaptation efforts are therefore not known.

Information synthesis and mobilization efforts are required

Expertise is required to develop targeted tools and information

• Resources and capacity are required to either develop or engage expertise in: communication outreach, client services and engagement, and tool development.
• This is not necessarily the role of scientists who have different priorities and lack the specific skillset needed to carry out broader activities related to communication and information dissemination. Therefore, the role of scientists in disseminating information beyond contributions to the primary literature is unclear.

5.11 Internal capacity of the ACCASP is limited

Strategic thinking is required to advance a number of program aspects:

Enhancing collaboration between scientists

There is a desire for more collaborative and networking opportunities at the working level. Despite examples of good working relationships, most projects are conducted in silos. Increased networking activities could facilitate opportunities to integrate research efforts between regions and research themes.

Enhancing the quality of the ocean monitoring information

The ACCASP is in the process of developing a formalized National Ocean Chemistry Monitoring Program to standardize sampling and data collection methods and protocols across the regions.

Data management practices are also required at the national level. Reviewing and establishing national management practices for ACCASP ocean chemistry data will allow the program to provide input into various international data platforms.

Strengthening internal tracking and reporting practices

Internal reporting practices could be improved to ensure that information on the progress of funded research projects is readily available to the NCR team. Reports are not always provided by scientists and this creates challenges when consolidating information about funded projects. Timely and complete reports allow the management team to gain insight about issues and challenges that could be addressed to improve efficiency.

Developing the ACCASP’s path forward

Within its current mandate, the program recognizes a need to discuss opportunities for strategic planning. Allowing for long-term planning to address urgent and time sensitive research topics could increase the value of ACCASP research efforts.

5.12 ACCASP contributions exceed program responsibilities

The ACCASP has been involved in activities beyond the program’s direct responsibilities. Moreover, this involvement has significantly increased since 2017 due to the evolution of the broader climate change adaptation context (See Appendix D). This involvement further detracts resources from ACCASP activities and improvements.

In addition to overseeing activities related to the program’s core mandate, the NCR team also:

1. Provides advice and input to strategic planning and briefing documents for DFO senior management;
2. Responds to varied ad-hoc requests for climate change-related information beyond those related to aquatic climate change science;
3. Provides support to PCF interdepartmental governance and coordination activities and committees, including participation in activities not specific to the Adaptation Pillar; and
4. Collaborates on national and international initiatives that exceed expectations for ACCASP contributions. (e.g., IPCC Special Reports, National Climate Change Science and Knowledge Plan, GGS Risk Assessments)

Regular involvement and contributions since 2017 that are beyond the ACCASP’s mandate include:

• Coordination of the National State of the Ocean Working Group; National Ecosystem Approach to Fisheries Management Working Group; National Arctic Science Committee and the Science Website Renewal at DFO.
• Support to the PCF’s Deputy Minister and Assistant Deputy Minister Oversight Committees (all four pillars).
• Input to the Convention on Biodiversity - an international legally-binding treaty for the conservation and sustainable use of biodiversity.
• Reviewing and coordinating DFO’s input to the IPCC Special Report on Ocean and Cryosphere in a Changing Climate (2019).

The ACCASP also participates in:

• The PCF’s Director Level Adaptation and Resilience Task Group, and the DG Adaptation and Resilience Committee, which coordinates federal adaptation activities at the Director and senior manager level, including aspects of Provincial/Territorial engagement, results measurement and capacity building.
• Working groups and discussions leading to the development of the Aquatic Systems component of the National Climate Change Science Plan.
• ECCC/TBS Working Group on Risk Assessments to provide guidance on the departmental risk assessment requirements of the Greening Government Strategy.
• Interdepartmental Natural Infrastructure Working Group and Climate Engineering Working Group.
• IPCC meetings and working groups, as required.

6.0 Lessons learned

Overall, the evaluation finds that the ACCASP faces many challenges that are beyond its purview and control since the program is not designed to respond to growing needs for aquatic climate change science to inform decision-making in support of adaptation efforts. Although the ACCASP is performing well under its current mandate, it faces and will continue to face challenges while it fulfills a broader role to support Departmental climate change initiatives.

Departmental considerations:

Appendix A - Methodology

The evaluation used multiple lines of evidence, both qualitative and quantitative, which were triangulated to mitigate potential limitations. The methodology and limitations are described below.

Document review

• Internal ACCASP documents such as tools and templates, application guides, calls for proposals, steering committee minutes and operating frameworks were reviewed.
• Broader national and international initiatives were reviewed to understand how the program and its objectives align with climate change adaptation priorities and activities of broader national and international initiatives.

Interviews

• 23 scoping and key informant interviews were conducted with ACCASP scientists, ACCASP program managers, DFO senior management, as well as internal and external end-users of ACCASP data and information.

Administrative data analysis

• An analysis of 111 research projects funded between 2016-17 and 2019-20 was conducted to assess whether research was completed as originally planned and according to schedule.
  • Limitation: Administrative data represents only a one year funding cycle (2018-19) which poses a challenge for assessing trends.

Financial analysis

• Financial data from DFO’s Chief Financial Officer Sector was used to assess the extent to which the program received funding to deliver on its objectives.
  • Limitation: Only partial information was available as a base of this analysis as there is only one complete year of program activities following the transition period. In addition, the analysis of ACCASP financial data indicates that a discrepancy exists between the financial data obtained from DFO’s corporate systems and from the program which precluded a detailed analysis of financial information.

Data obtained from the evaluation of science funding (2018-19)

• Secondary data was obtained from the Evaluation of Science Funding^{Footnote 29} which collected information from applicants, reviewers and end-users of all 16 Ecosystems and Oceans Science (EOS) sector funding programs for the 2018-19 funding cycle.
• The data was filtered by program to facilitate a meta-analysis of ACCASP-relevant information from responses to the applicant survey (n=83) and end-users survey (n=18)^{Footnote 30}. Using survey data, identity factors including gender, career status and geographical location were explored to assess potential barriers to funding access for applicants.
  • Limitation: The Evaluation of Science Funding addressed the overall efficiency of all 16 EOS funding programs. Therefore, it is challenging to ascertain the extent to which responses are specific to the ACCASP as opposed to the whole suite of science funding programs.

Webometric analysis

• The program’s publications and website traffic were analyzed to assess the availability, reach, and potential use of ACCASP-funded research.
• To conduct the webometric analysis, a list of 129 academic articles, 55 DFO reports, and 17 external reports to which the ACCASP has contributed was compiled and vetted by the program. With the support of DFO’s Library Services, a citation analysis of 85 publications indexed on the Web of Science was conducted. For publications that were not indexed on the Web of Science, a cited reference search was conducted separately.
• To analyze the impact of several websites supported by the ACCASP, web metrics were generated with the support of DFO’s Communication Services. This included links to the main ACCASP website and links to several online tools.
  • Limitation: For effective citation analysis, it is recommended a minimum window of 3 to 5 years after the date of publishing has passed. The list of provided publications contains many recent publications, therefore the webometric analysis results should be considered a preliminary snapshot of potential use. Citation analyses are also restricted to publications available on the Web of Science.

Appendix B – Evaluation matrix

Appendix C – Performance measurement under the Horizontal Management Framework for Clean Growth and Climate Change

The 2017-18 outcomes and indicators below support the long-term outcome for the Adapting to impacts of Climate Change Horizontal Initiative.

Appendix D – Broader climate change context

1. International climate initiatives

• Canada is a signatory of the UN 2030 Agenda for Sustainable Development, which defines 17 global sustainable development goals with 169 targets. Under this initiative, countries implement national sustainable development strategies, monitor progress toward goals and targets, and make performance data available for international sharing and analysis. The ACCASP reports Canadian data on target 14.3: Ocean Acidification.
• Canada signed the UN Framework Convention on Climate Change (UNFCC) and the UNFCC Paris Agreement (2015), and committed to address climate change through mitigation and adaptation measures.
• The UN International Panel on Climate Change (IPCC) provides scientific expertise internationally in support of the Convention. ACCASP researchers contribute to the activities of IPCC Working Groups.
• Canada has been a member of the Arctic Council since 2008. ACCASP research contributes to two of the Council’s priorities: Monitoring and Assessment of Arctic Ocean Acidification, and Ecosystem Approach to Management in the Arctic.
• In line with the decisions of the North-American Leaders Summit (2016), Canada has committed to strengthening cooperation on ocean management as well as the integration of ocean observation systems, oceans and climate change research.

How are international drivers evolving?

• In line with the UN 2030 Agenda for Sustainable Development, a Decade on Ocean Science and Sustainability (2021-30) has been declared.
• The UN International Panel on Climate Change (IPCC) has increased the intensity and prominence of research and reporting related to adaptation and vulnerability. Canada is contributing to these activities with regards to the oceans, cryosphere, and also more broadly.
• Collaboration with the Arctic Council is expected to increase based on the recognition that the Arctic faces significant knowledge gaps and experiences climate change at a faster rate. The ACCASP will contribute to the 2020 Synoptic Arctic Survey in conjunction with the Arctic Monitoring Assessment Programme Working Group.
• Collaboration with the National Oceanic and Atmospheric Administration (NOAA) on Ocean Acidification and fisheries vulnerability is ongoing.
• International data sharing collaboration through the Global Ocean Acidification Observing Network (GOA-ON) is ongoing.

2. Canadian climate change initiatives

3. DFO responsibilities

A number of commitments at the departmental level are driving a need for ACCASP-generated research.

• DFO has legal responsibilities under the Oceans Act (1996), Species at Risk Act (2002), Federal Sustainable Development Act (2008), and Fisheries Act (1985), including the 2019 amendment, Bill C68.
• The DFO Minister’s Mandate Letter outlines DFO commitments to use good scientific evidence and traditional Indigenous knowledge when making decisions affecting fish stocks and ecosystem management (2019) and ensure effective use of research resulting from restored federal funding for freshwater research, federal ocean science and monitoring programs among others (2018).
• In the past several years, the ACCASP has been addressing many requests for climate change adaptation action within the department. The scope of these requests is broader than the science and adaptation aspect. They include implementation requirements and representation on behalf of the department at the federal level related to the Federal Adaptation Policy Framework and the PCF.
• Recent ACCASP research builds on the risk assessments of the four water basins conducted by ACCASP in 2013. The 2012 DFO Climate Change Risk Profile identified 6 key risks for the department, and the scientific needs to address these risks. In the 2019-20 DFO Corporate Risk Profile, the risks associated with changing aquatic environments have been elevated as mission critical.

How are DFO needs and responsibilities expected to evolve?