Evaluation of the Canadian Science Advisory Secretariat (CSAS)

Project Number: 96175
Final Report
March 4^th, 2019

NOTE: The Evaluation Canadian Science Advisory Secretariat has an alternative report format that uses graphics and visuals, which is available in pdf format.

1.0 Program Profile

1.1 Canadian Science Advisory Secretariat (CSAS)

The Canadian Science Advisory Secretariat (CSAS) is the main, formal process within Fisheries and Oceans Canada (DFO) for obtaining science advice to inform policy and management decisions for all departmental priorities (FRS Evaluation, 2011). The objectives of the Secretariat are:

1. To ensure that the client’s needs for science advice are met in a timely, cost-effective and reliable manner;
2. To provide all clients with stable and consistent service, with roles and responsibilities clearly understood by all participants; and,
3. To have full accountability to its clients and DFO, while maintaining independence from policy influence.

The CSAS achieves its objectives by coordinating scientific peer-review assessments on a multitude of issues related to fish stock dynamics, species at risk, invasive species, marine and freshwater ecosystems, marine protected areas, and aquaculture, and by providing advice through one of the following science advisory products:

• Science advisory report: Summarizes the advice from the working paper(s) and the peer-review assessment. The summary bullets are usually completed during the assessment.
• Science response: Summarizes the advice and discussions from the Special Science Response Process (SSRP) including the conclusion for the client. The SSRP often does not involve the same level of analysis or peer-review as they are usually conducted for: new and urgent requests; requests that do not require a thorough and inclusive advisory process; requests that require a review on exiting information; or, when DFO is not the final advisory body.

CSAS clients comprise staff from policy and management programs within DFO as well as scientists working in DFO’s Ecosystems and Oceans Science (EOS) sector on occasion.

Several other organizations employ a similar process for developing and providing science advice. For example, the International Council for the Exploration of the Sea (ICES) utilizes a network of scientists from 20 member countries to provide science to decision-makers that helps them to make informed choices on the sustainable use of the marine environment and ecosystem.

1.2 The SAGE principles

The Science Advice for Government Effectiveness (SAGE) Principles were developed by the Council of Science and Technology Advisors (CSTA)^{Footnote 1} to support the effective use of science advice in making policy and regulatory decisions. These six principles serve as the foundation of the CSAS science advisory process.

SAGE principles:

1. Early identification: The government needs to anticipate, as early as possible, those issues for which science advice will be required, in order to facilitate timely and informed decision making.
2. Inclusiveness: Advice should be drawn from a variety of scientific sources, from experts in relevant disciplines and from external and international sources. Due weight also needs to be given to ‘traditional knowledge’ of local peoples.
3. Sound science and science advice: The government should employ measures to ensure the quality, integrity and objectivity of the science and science advice it uses, and ensure that science advice is considered in decision making.
4. Openness and transparency: The government is expected to employ decision-making processes that are open as well as transparent to stakeholders and the public, and that the public has access to the findings and advice of scientists as early as possible.
5. Review: Subsequent review of science-based decisions is required to determine whether recent advances in scientific knowledge have an impact on the science advice used to reach the decision.
6. Uncertainty and risk: Science in public policy always contains uncertainty that must be assessed, communicated, and managed. Government should develop a risk management framework that includes guidance on how and when precautionary approaches should be applied.

1.3 The CSAS science advisory process

The CSAS science advisory process as outlined in program documents and described by key informants and case study participants is cyclical, operates on an annual schedule, and is comprised of eight stages. Figure 1 shows the CSAS science advisory process.

Figure 1: The CSAS science advisory process

1.4 The three phases of the science advisory process

The eight stages of the CSAS science advisory process can further be grouped into three phases by which the CSAS develops and provides science advice to its clients. The three phases of the CSAS science advisory process form the structure of this report. The three phases are the following:

• Phase one: Addressing requests for science advice.
• Phase two: Developing the science advice.
• Phase three: Knowledge dissemination.

Figure 2: The three phases of the CSAS process

1.5 The structure of the CSAS

The main CSAS office (hereafter referred to as the Secretariat) is located at DFO’s National Headquarters in Ottawa and works in partnership with the Centres for Science Advice (hereafter referred to as the centres) in each region.

• The Secretariat coordinates any science advisory processes that occur in the national capital region, while the centres coordinate science advisory processes in their region.
• The Secretariat publishes all the science advisory products and supporting documents received from the regional centres.

The number of full-time equivalent (FTE) positions varies across the Secretariat and the centres based on their available budget and the average number of requests for science advice received each year. Figure 3 shows the regional distribution of CSAS resources.

Figure 3: Structure and resources of the CSAS

^{Footnote 3}

Assigned to CSAS

Funded by CSAS budget

Vacant positions^{Footnote 2}

“Employees at CSAS are doing multiple jobs because of staffing issues.” Internal Stakeholder

2.0 Evaluation Context

2.1 Evaluation scope

The evaluation will cover fiscal years 2013-14 through 2017-18 and will be carried out by the DFO Evaluation Division between March, 2018 and January, 2019

2.2 Evaluation approach

Principles-focused evaluation, specifically the extent to which principles informing the CSAS science advisory process are articulated, adhered to and are leading to desired results.

2.3 Evaluation objectives

• Examine the of the SAGE principles 20 years after their development and to assess whether the policies, guidelines and structures deployed by the CSAS adhere to these principles. Additionally, the evaluation will assess whether the CSAS science advisory process, particularly peer review meetings, reflects current best practices in Canada and internationally.
• Examine CSAS’ compliance with its policies and guidelines.
• Document trends in the end-user demand for CSAS advisory services, over the last five years.
• Explore and document regional and/or national activities focused on broadening the range of perspectives consulted with regards to current and future science advice and/or information generated through the CSAS process.

2.4 Evaluation questions

1. What resources are invested by DFO in support of CSAS-related activities?
2. Do the SAGE principles and guidelines developed by the Council of Science and Technology Advisors (CTSA) continue to be relevant in the current context for government decision-making?
3. To what extent do the policies, guidelines and structures of the CSAS science advisory process adhere to the SAGE principles?
4. To what extent does the CSAS support the development and provision of science advice and information to decision-makers?
5. How do end-users address their needs when their requests for science advice are not selected?

3.0 Evaluation methodology

3.1 Lines of evidence

3.1.1 Case studies

• Completed site visits in the Maritimes and Pacific regions to observe peer-review assessments and to conduct interviews with CSAS staff and management, as well as CSAS clients, scientists, and external participants of the peer-review assessments.
• A total of 28 interviews across both sites.

3.1.2 Interviews

• Conducted interviews with CSAS staff and management, as well as CSAS clients, scientists, and/or external participants of peer-review assessments in the regions not involved as one of the case studies.
• A total of 29 interviews.

3.1.3 Surveys

• Administered a survey to CSAS clients to assess the extent to which their requests for science advice are addressed; the quality of the science advice provided; as well as any factors that facilitate or challenge the CSAS science advisory process.
• Response rate of 20% with 149 respondents out of a possible 736 individuals.

Applicants to DFO Science Funding Programs

• Survey of applicants (i.e. SERES, BI, PC, CH, VM, EG groups) to assess the extent to which they are involved with the CSAS science advisory process and whether they use the products generated by this process.
• Response rate of 24% with 254 respondents out of a possible 1073 individuals.

3.1.4 Data analysis

• Requested and examined data from the CSAS related to the number of requests received and addressed, as well as the number of CSAS documents submitted for publication and published between 2013-2014 and 2017-2018.

3.1.5 Document review

• Reviewed relevant documents related to the CSAS including policies, guidelines, templates and training materials.

3.16 Comparative analysis

• Literature review of science advisory processes in other jurisdictions (international) and federal government departments as well as interviews with representatives from the International Council for the Exploration of the Sea, the National Oceanic and Atmospheric Administration, and Environment and Climate Change Canada’s Wildlife Management and Regulatory Affairs Division.

3.2 Limitations and Mitigation strategies

Limitations and Mitigation Strategies

In order to mitigate limitations as much as possible, data was triangulated from multiple lines of evidence. This approach was taken in order to demonstrate the reliability and validity of the findings and to ensure that conclusions and recommendations are based on objective and documented evidence. Other mitigation strategies are described below and, when relevant, at various other points in this report.

3.2.1 Limitation and mitigation 1

Limitation

The evaluation team experienced challenges with the analysis of the request and publication data due to a lack of systematic tracking and unreliable data. Additionally, data for the 2013-14 fiscal year was unavailable and was therefore, not included in the analysis. There were also some uncertainties regarding the reliability of submission and publication data in the year 2015-16. It was also not possible to link these two data bases in order to draw conclusions about the entire science advisory process from request for science advice to the submission and publication of the advice.

Mitigation

To mitigate the challenges regarding the request and publication data bases, CSAS coordinators were consulted at various points of the analysis. Additionally, with respect to the request database, data pertaining to submitted requests were manually matched to requests included on the science advisory schedule to verify how many requests were addressed by the CSAS.

3.2.2 Limitation and mitigation 2

Limitation

The evaluation team also faced challenges with regards to the survey of CSAS clients, as there was no population available.

Mitigation

To mitigate this challenge a sample was created by extracting any client names noted in the request database provided by the Secretariat, as well as the names of Fisheries and Oceans employees working within policy and management programs available in the Government Electronic Directory Services (GEDS) including: Aquaculture Management; Resource Management; Aboriginal Fisheries; Ecosystem Management; Small Craft Harbours; Fisheries Protection Program, Species at Risk, Oceans Management, Habitat Protection; Oceans Division; Policy & Economics; and, Aquatic Resources Division. A sample of 736 potential respondents was created using this method.

3.2.3 Limitation and mitigation 3

Limitation

The evaluation team experienced challenges conducting a comparative analysis of the CSAS science advisory process as there were no comparable programs found to exist within Canada. In particular, the inclusion of peer review assessments to develop the science advice and the participants invited to these assessments appeared to be unique to the CSAS.

Mitigation

The evaluation team conducted a limited comparative review exploring science advisory processes in other jurisdictions (e.g. the United States and Europe) and best practices around peer review assessments. Additionally, greater emphasis was placed on examining the CSAS science advisory process in more detail, including its adherence to the SAGE Principles.

4.0 Evaluation Findings: CSAS & SAGE

4.1 Financial resources and other considerations

Financial Profile – CSAS Direct Costs:

• The total direct cost of delivering the CSAS is unknown as it is not possible to track the expenditures of the DFO research institutes, laboratories, experimental centres and client sectors that participate in and in some cases fund science advisory processes^{Footnote 4}.
• The costs outlined in the figure represent an estimate of the annual expenditures of the Secretariat and centres, which averaged approximately $2.2M over the last five years with salary costs accounting for a little over 80% of total expenditures.

Other Considerations & Examples of Unknown Costs:

• Impacts of unstaffed positions: This factor contributes to reducing the costs of CSAS.
• EOS staff time: DFO scientists are playing a pivotal role in support of CSAS since the process draws mainly from internal expertise. Applicant survey results show that on average, staff spend 45 working days of their time supporting CSAS science advisory processes (median = 25 working days). The costs associated with the time committed by DFO scientists is unknown.
• Impacts of increased translation costs: As a result of an investigation from the commissioner for Official Languages, as of April, 2018 all CSAS documents published on its website must be translated in both official languages. Coupled with DFO’s recent change to the Translation Bureau for translation services the average annual translation expenditures for the CSAS is expected to increase from $263,310.00 to approximately $1,068,486.00 about 49% of the current CSAS budget. Figure 4 shows the CSAS estimated expenditures between 2013-14 and 2017-18.

Figure 4: Estimated CSAS expenditures (2013-14 to 2017-18)

4.2 The value of the CSAS science advisory process

The benefits of the CSAS process includes the following elements:

• Critical mechanism for the provision of advice: Overwhelmingly key informants and case study participants perceive the CSAS as an important mechanism for developing and providing science advice in support of decision-making.
• Unique nature of the CSAS: Findings from the comparative analysis, key informant interviews and case studies suggest that while other programs for providing science advice exist within the federal government (e.g. Wildlife Management and Regulatory Affairs at Environment and Climate Change Canada) and internationally (e.g. Northwest Atlantic Fisheries Organization) they do not employ the same structure or engage in peer-review in the same manner as the CSAS.
• Supports industry: Provides credible stock assessment data which sometimes feeds into the eco-certification process assessing whether a fishery is well-managed and sustainable.
• CSAS aligns with DFO’s mandate: The work of the CSAS supports the Government of Canada’s expectation that “Decisions will be informed by scientific evidence” (Speech from the Throne, 2015) and DFO’s mandate to “use scientific evidence…when making decisions about fish stocks and ecosystem management” (Minister of DFO Mandate Letter, 2016 & 2018).
• The CSAS is used to demonstrate the achievement of DFO’s mandate: There are four performance indicators in the Departmental Results Framework that relate to the CSAS process; thereby, illustrating the perceived value of the CSAS in terms of supporting the achievement of the department’s expected outcomes.

4.3 Alignment with best practices and government priorities

The SAGE principles are often used by Secretariat and centre staff, as well as participants of the peer-review assessment as a guide for the science advisory process and/or to navigate difficult or uncertain situations during the process, such as when one or more participants of the peer-review assessment adopt an advocacy role and fail to remain objective.

A comparative review of the science advisory process of other organizations with a comparable mandate to DFO suggest that their policies and practices are quite reflective of the SAGE principles:

• The National Oceanic and Atmospheric Administration (NOAA) has legislation, policies and practices that support the use of the best scientific information available, the use of peer-review, the articulation of uncertainties and risks, review, as well as the transparency and openness of their science advice.
• The International Council for the Exploration of the Sea (ICES) employs policies and engages in practices related to inclusiveness, sound science advice, uncertainty and risk, as well as openness and transparency.
• New Zealand Fisheries is focused on building relationships between science and policy and does so through practices that support considerations around uncertainty and risk, the provision of sound science, as well as openness and transparency.

The SAGE Principles also align with several priorities of the DFO and the federal government including, but not limited to:

• Open Government - making government including data and report more accessible (i.e. openness and transparency).
• Reconciliation – exploring new ways of working together including meaningful engagement with Indigenous peoples…when Canada proposes to take actions which impact them and their rights, including their lands, territories and resources (i.e. inclusiveness).
• Strengthening science- and evidence-based decision making (i.e. sound science)

4.4 SAGE – alignment with best practices and government priorities

Adherence to the SAGE principle

Evidence from the document review, key informant interviews and case studies suggest that the extent to which the CSAS science advisory process adheres to the SAGE principles tends to vary across regions and types of request. This variance is generally attributed to an insufficient number of policies, guidelines or structures that operationalize the SAGE principles to ensure the systematic implementation of a science advisory process in compliance with these principles across Canada.

Opportunities for change

In particular, there are opportunities for greater clarification regarding the operationalization of specific principles at each phase of the process, namely early issue identification, inclusiveness, sound science advice, as well as openness and transparency. These opportunities will be explored throughout the report during the examination of the different phases of the CSAS process. Figure 5 shows opportunities for greater clarification of the SAGE principles at specific stages of the CSAS process.

Figure 5. Opportunities for greater clarification of the SAGE principles at specific phases of the CSAS process

5.0 Phase 1 of the CSAS process - Findings

5.1 Overview of phase 1

The first phase of the process includes the following three main stages:

• Stage 1: Every year the Secretariat and centres send out a call for requests for science advice across DFO.
• Stage 2 : CSAS clients develop formal requests for science advice around specific issues or questions that will support decision-making. Requests are often submitted by clients in priority order.
• Stage 3: Requests received are evaluated against the Prioritization Framework for Peer Review Requests to identify their level of risk and achievability. The science advisory schedule is developed and it is expected that priority is given to very high, high and moderate risk requests that are deemed achievable. The science advisory schedule is reviewed and approved by DFO’s Senior Management Committee (SMC).

Figure 6. Phase 1 of the CSAS Science advisory process: Addressing requests for science advice

5.2 Extent to which the CSAS is able to address requests for science advice

Results from the analysis of the CSAS administrative data illustrates that between 2013-14 and 2017-18, 49% (610) of the 1247 requests for science advice submitted to the CSAS were included as part of the science advisory schedule either the same year they were submitted or in subsequent years as resources to provide the advice became available^{Footnote 5}.

Between 2013-14 and 2017-18 the number of requests for science advice submitted to the CSAS increased by 31%. Figure 7 shows the number of requests not addressed, addressed in subsequent years, and addressed same year from 2013-14 to 2017-18.

Figure 7. Count and proportion of requests addressed from 2013-14 to 2017-18

5.3 Extent to which requests for science advice are addressed, disaggregated by client sector

An analysis of the extent to which requests are addressed by DFO sectors was also performed. Figure 8 shows the ability of CSAS to address requests by client sector.

Figure 8. CSAS ability to address requests by client sector and fiscal year

5.4 Extent to which high risk and unachievable requests are addressed

Classifying requests according to their risk and achievability level

Of the 1247 requests received by the CSAS between 2013-14 and 2017-18 931 (75%) were classified according to their risk level and achievability^{Footnote 8}.

Of these 931 requests 220 (24%) were classified as high risk and while the majority were deemed to be achievable, 44 (20%) of the high risk requests were deemed unachievable.

The remaining 711 (76%) requests were classified as medium or low risk and 161 (23%) were deemed unachievable.

In some cases unachievable requests were added to the science advisory schedule as:

• Scientists were able to find time in their schedules to address certain requests; and/or
• Legal requirements were completed allowing potential meetings to move forward.

Between 2013-14 and 2017-18 45 (28%) unachievable requests that were not high risk and 14 (32%) of the unachievable, high risk requests were eventually added to the CSAS science advisory schedule. Figure 9 shows the proportion of risks classified according to their risk level and achievability.

Figure 9. Count and proportion of requests classified according to their risk and achievability level

5.5 What clients do when their request for science advice is not addressed

When clients and end-users were asked the extent to which the CSAS was able to address their requests for science advice 68% of survey respondents (n=65) stated that CSAS addressed their requests to a little or moderate extent. Almost half of these respondents, 48% (n=31) are from Oceans Management, Species at Risk or the Fisheries Protection Program, while 40% (n=26) are from Fisheries Resource Management or Aquaculture.

When Requests for Science Advice are Not Addressed

The majority, (77%) of clients and end-users that participated in the survey indicated that there are times when they will make decisions in the absence of science advice from the CSAS.

Mitigation Strategies

Key informants and case study participants noted that CSAS clients will sometimes conduct their own literature reviews to help inform upcoming decisions.

Unintended Outcomes

5.6 Factors that challenge the extent to which requests are addressed

Capacity Within EOS to Address the Research Question

Results from the key informant interviews, case studies, as well as the client and end-user survey indicate that a lack of capacity within EOS is the main issue as to why certain requests for science advice are not addressed.

Issues around capacity often result from a lack of:

• availability by EOS scientists (75% of survey respondents),
• available data (47% of survey respondents); and,
• expertise within EOS on the subject matter in question (34% of survey respondents).

According to key informants and case study participants, EOS capacity issues are slightly more prevalent for requests from the FPP, and Oceans and Species at Risk as their requests are often more complex and there are fewer EOS resources available to conduct the research within required timelines. The lack of EOS resources is primarily attributed to the fact that in the past research efforts at DFO were primarily focused on stock assessments. Findings from the survey support these assertions as 42% of respondents from these three client groups indicated that there is a lack of expertise within the EOS sector to address their requests as compared to 24% of respondents from DFO’s Fisheries Resource Management and Aquaculture client groups.

Prioritization of Requests

Interviews with CSAS program staff indicate that the extent to which the risk prioritization framework is applied to assess requests for science advice varies by request and by region. This is often because the main factor that determines whether a request is addressed, by CSAS, is the degree to which a request is achievable based on available EOS resources. Level of risk is often the secondary prioritization consideration. There are however, no clear guidelines regarding how to address very high or high risk requests that are determined to be unachievable. Consequently, high risk requests may not be addressed despite the urgency for science advice.

Requests that do Not Require Science Advice

CSAS is an obvious entry point within EOS where clients can seek science advice. The CSAS may therefore, receive requests that do not align with its mandate (e.g. does not require peer-review; is not a request for advice, but for information) or that cannot be addressed through a peer-review assessment because of capacity issues within EOS9^{Footnote 9}. As there is no formal process to address these requests other than accept or reject them for inclusion as part of the science advisory schedule such requests may not be addressed and clients may find themselves without the science required for decision-making.

Formulation of the Research Question

At times the research question outlined in the requests for science advice cannot be answered by EOS as the question is too broad, too complex, unclear or beyond the scope of the CSAS. Approximately 14% of respondents of the client and end-user survey noted that their request could not be answered due to the manner in which the question was written. All of these respondents are staff from the FPP, Oceans and Species at Risk programs.

Some of the clients interviewed indicated that they are unfamiliar with the science information that EOS can provide and/or struggle to “make a management question into a science question.” Internal Stakeholder

In-Year Requests

The ability for clients to submit in-year requests provides a certain level of flexibility to the CSAS process to be able to respond to emerging issues. Such requests however, “create significant additional workload and work planning issues and also have a bearing on the type of analyses that can be conducted, which can, in turn, affect the quality of advice” (CSAS, 2016). Because in-year requests are not tracked in the same manner as other requests it is not possible to determine full extent to which these types of request present as a challenge to the CSAS science advisory process.

There are concerns that in-year requests are not subject to the same oversight as requests submitted through the annual call as they are not reviewed by the Secretariat and approved by the SMC. Consequently, the CSAS may address requests that do not require a peer-review assessment. They may also supplant or displace requests that are already part of the science advisory schedule and/or require the re-allocation of already strained EOS resources.

5.7 Implementing structures and processes to support early identification – potential solutions

POTENTIAL SOLUTION

Formalized communication between the CSAS, its clients and EOS scientists during the development and assessment of the science advisory schedule. In particular:

• A requirement for consultation and collaboration between CSAS clients and the EOS sector to develop research questions that address the needs of the client and that can be answered by EOS. The Secretariat and centres may support these discussions provided they have the available resources. This expectation should be clearly articulated and communicated to clients and EOS.
• Implement bi-annual meetings in each region between the centres or Secretariat, the DG/RDS and the Regional Directors (RD) from each client division to review all of the submitted requests for science advice and the science advisory schedule.
• Provide clients with systematic feedback as to why requests are not being addressed including a point of contact for further inquiry such as other methods for accessing the science information required.

Expected Benefits

It is anticipated that the implementation of formalized communication between the CSAS, its clients and EOS scientists will:

• Improve the triaging of requests in terms of identifying requests that require science advice through a peer-review assessment versus requests that require advice or information that may be provided through other mechanisms.
• Establish opportunities for clients to seek out other forms of science advice or information when their requests cannot be addressed through a CSAS science advisory process (e.g. when other requests are prioritized, when there is no data).
• Help clients formulate their research questions so that they can turn a management question into a science question.

POTENTIAL SOLUTION

The implementation of a multi-year CSAS science advisory schedule in all DFO regions.

• Would allow the EOS sector to identify the immediate and long-term needs for science advice within DFO. The EOS sector could work with the CSAS and clients to determine the best process for addressing requests including those for which the data may not be available and requests that do not require science advice.
• It can be pre-populated with requests that are: (1) carried over from previous years; (2) recurring; and/or, (3) legislative or regulatory requirements. Consequently, CSAS clients may not be required to re-submit requests for science advice they continue to require as long as the request (i.e. questions to be answered) have not changed. The CSAS could however, encourage clients to review and reprioritize ongoing requests until they are added to the SAS.
• Several regions are using multi-year schedules with some success. Review lessons learned before standardizing any practices across all regions.

Expected Benefits

It is anticipated that the implementation of a multi-year science advisory schedule will:

• Support better planning in terms of allocation of resources to minimize the impact of requests on EOS sector resources by identifying gaps in available data and/or expertise within the sector.
• Reduce the frequency of in-year requests submitted by clients.
• Provide the Secretariat and centres with other mechanisms for addressing very high or high risk requests that are deemed unachievable and thus cannot be addressed through a CSAS science advisory process.

6.0 Phase 2 of the CSAS process – Evaluation findings

Overview of phase 2

The second phase of the process includes the following three main stages:

• Stage 4: Each request included as part of the science advisory schedule is assigned a science lead who may work with a team of scientists to conduct the research required to respond to the request. The science lead works in collaboration with the client and the Chair of the peer-review assessment to: develop a terms of reference (ToR) for the process; a list of potential participants for the peer-review assessment; identify the working paper(s) required; and, confirm meeting logistics. The terms of reference is submitted to the Secretariat for review and publication.
• Stage 5: The science lead or team conduct the research required and draft the required working paper(s) outlining the available data and analysis in response to the request.
• Stage 6: This scientific information is reviewed and discussed during the peer-review assessment with the objective of reaching consensus around the content of the science advice that will inform fisheries management decisions and fisheries policy.

Figure 10. Phase 2 of the CSAS science advisory process: Developing the science advice

6.2 Peer-review assessments – best practice and strength of the CSAS

When asked about the strengths of the CSAS science advisory process the inclusion of peer-review assessments for developing the science advice was noted by almost all of the respondents (91%) of the client and end-user survey. This sentiment was echoed by the majority of key informants and case study participants who perceive the peer-review assessments as providing the greatest value to the CSAS science advisory process.

Peer-review is a highly respected and accepted process to assess and ensure the quality, objectivity and reliability of scientific methods and information. It is often used as a quality control measure within the scientific and decision-making communities and is considered a best practice when using scientific information to inform fisheries management decisions (Penney, 2010). Peer-review is also considered to be the cornerstone of the SAGE principle of Sound Science and Science Advice (Council of Science and Technology Advisors (CTSA), 2009).

Evidence from the key informant interviews, case studies, document review, and the survey with clients and end-users suggest that there are several policies, guidelines and structures supporting the CSAS science advisory process that align with the general conditions required for successful peer-review including:

• Determining the terms of reference with specific technical questions to guide the process;
• Assessors should strive for consensus and meet transparency requirements throughout the process; and,
• All information intended for use to inform decision making should be subject to peer-review including information provided by stakeholders, such as industry and non-governmental organizations (Penney, 2010).

Additional evidence however, suggests opportunities for improving peer-review assessments including, but not limited to:

• Clarification regarding who should participate in peer-review assessments.
• Additional guidelines and mechanisms to mitigate potential conflicts of interest around what is discussed.

6.3 Expertise and inclusiveness – who should participate

Expertise and Diversity of Participants – Strength & Challenge of the CSAS Science Advisory Process

The majority (73%) of respondents from the client and end-user survey indicated that the peer-review assessments involve an appropriate range of perspectives, as well as participants with the expertise to support the development of science advice to a good or great extent.

The expertise and diversity of participants were also considered significant strengths of the CSAS science advisory process by 85% and 71% of respondents respectively, as well as two of the main factors challenging the process.

Restricted participation

Findings from key informant and case study interviews help explain the contradictory nature of the survey findings as they highlight a tension among internal and external stakeholders of the CSAS program including who should participate in peer-review assessments. In particular, some stakeholders believe that only scientists and/or individuals who are able to understand and contribute to the scientific knowledge presented during these assessments should participate.

6.4 Expertise and inclusiveness – further considerations

Engaging External Stakeholders at Other Stages or Phases of the Science Advisory Process

While some stakeholders believe that only scientists and/or individuals who are able to understand and contribute to the scientific knowledge should participate in peer-review assessments, many of them also feel that external stakeholders representing local and traditional knowledge have a role to play within the CSAS science advisory process.

For instance, several key informants and case study participants suggested that the participation of external stakeholders may be better during the earlier phase of the CSAS science advisory process, such as the development of the request for science advice or even in the collection and analysis of the data that will be used to develop the working paper informing the science advice.

Reflecting the Diversity of Participants in the Science Advice

According to the majority (62%) of client and end-user survey respondents, the science advice received only reflects diverse knowledge systems pertaining to the subject matter under review to a little or moderate extent. Several key informants noted that if the peer review assessments are expected to be inclusive of a diverse group of participants, the science advice should reflect the diversity of the knowledge provided.

Several key informants, case study participants and respondents of the CSAS client and end-user survey noted that they are struggling with how to reflect the diversity of participants and their contributions to the peer review assessments in the science advice. While there are some ongoing efforts to consider and incorporate diverse knowledge, additional guidance on how to be inclusive when drafting the science advice is desired.

Current Activities:

• Centres and scientists in some regions are looking to increase opportunities to work with AAROM organizations in the collection and/or analysis of the data that will inform future CSAS science advisory processes.
• EOS has hired a Mitacs Science Policy Fellow to examine how the CSAS can meaningfully integrate Indigenous Traditional Knowledge within the science advisory process.

6.5 Expertise and inclusiveness – looking to best practices

The lack of guidance regarding the notions of expertise and inclusiveness resulted in variations across centres, client groups and science assessment teams in terms of how they determine who should be invited to participate in peer-review assessments. For instance, the presence of industry and/or Indigenous participants are more prevalent in certain regions. Additionally, some regions will always attempt to involve at least one external reviewer with a strong science background in the subject matter of the assessment, while others may only invite external reviewers for certain assessments.

Best Practices for Inclusiveness and Balance of Expertise for Peer-Review

Peer-review processes for developing science advice should involve a wide range of scientific expertise from relevant disciplines within and outside government. Peer-reviewers should have the expertise required to review, challenge and contribute to the science information under discussion (Penney, 2010).

Representatives from external stakeholder or interest groups may be invited to participate as relevant local and traditional knowledge should also be acknowledged as part of the peer-review (Penney, 2010). “The knowledge and expertise of representatives from the different [external] stakeholder or interest groups that is used to inform the scientific debate should be identified as such, and may then be duly reflected in the science advice provided” (Ministry of Fisheries, 2011, p. 13).

Clients/Managers may also be invited to participate in peer-review assessments. The participation of clients/managers may improve understanding, encourage buy-in and streamline science-policy communication; however, it is important to note that like external stakeholders clients/managers cannot be expected to act completely impartially.

6.6 Expertise and inclusiveness – opportunity to address the tension and lack of guidance around participation

POTENTIAL SOLUTION

The Secretariat should develop clear guidelines to support the operationalization of the SAGE principle of inclusiveness. In particular, the guidelines should include criteria regarding which participant groups should be invited to the different types of science advisory processes (i.e. SSRP compared to the more comprehensive peer-review assessment resulting in a science advisory report) and during which stages of the process they should be invited to participate.

Further Considerations:

• Consultations with internal and external stakeholders of the CSAS science advisory process may provide greater insight regarding who should participate in peer review assessments according to the nature of the request and the breadth of knowledge required.
• When developing these guidelines it is important to consider best practices, as well as the priorities and requirements outlined in the Minister’s Mandate Letter, including the “use of scientific evidence, traditional Indigenous knowledge, and the precautionary principle…when making decisions affecting fish stocks and ecosystem management” (2018).
• Reflect on how science advisory processes related to fish stocks and ecosystem management in other jurisdictions (e.g. International Council for the Exploration of the Sea, the National Oceanic and Atmospheric Administration New Zealand Fisheries) engage external stakeholders in their process and how they reflect diverse perspectives in their science advice.

6.7 Managing external participants and conflicts of interest

Managing External Participants

Some key informants and case study participants are concerned that external stakeholders (e.g. industry, non-governmental organizations) participating in the peer review assessments may attempt to capitalize on their participation to advance their own agenda. Consequently, these individuals would be in a conflict of interest with the science advisory process^{Footnote 10}.

Peer-reviewers are expected to act in an independent manner to provide impartial and objective review, and not as advocates or representatives for any interest group. As noted previously however, evidence in the literature indicates that external stakeholders cannot be expected to act completely impartially as they are representatives from other organizations and/or are likely to have a vested interest in the outcome of the science advice. All real or perceived conflicts of interest therefore, must be defined, identified and managed to ensure that the science advisory process remains objective and credible (Penney, 2010) .

Managing Conflicts of Interest in Science Advisory Processes – Best Practices & SAGE

Best practices highlight that it is important for government departments to document and examine all potential conflicts of interest of participants involved in science advisory processes (Penney, 2010).

Several international science advisory processes related to fish and oceans management (e.g. New Zealand Fisheries, ICES) rely on the Chair of their peer-review assessment to manage any potential conflicts of interest that may arise. They also have robust conflict of interest guidelines and/or policies to further mitigate situations where participants may act more as advocates for their organization instead of objective peer-reviewers.

The provision and enforcement of conflict of interest guidelines by government departments is also a requirement of the SAGE Principle of sound science and science advice, and “advisors are required to declare any conflicts of interest prior to serving in an advisory capacity and to update such declarations throughout the term of their service” (CTSA, 1999, p. 6).

6.8 Challenges managing conflicts of interest in the CSAS science advisory process and potential solutions

Challenges Managing Conflicts of Interest in the CSAS Science Advisory Process

• The CSAS does not have a conflict of interest policy or guidelines and the management of potential conflicts is the responsibility of the Chair of the peer-review assessment.
• According to key informants and case study participants while most Chairs are quite skilled at managing potential conflicts of interest, some appear to be less comfortable with such situations.
• Not all of the individuals who chaired a peer-review assessment completed the CSAS Chair training and therefore, may not have the knowledge and/or skills to mitigate potential situations of conflict of interest.
• Findings from the applicant survey indicate that of the 49 scientists who chaired at least one CSAS peer-review assessment, 13 scientists (27%) had not completed the CSAS Chair training.
• The centres or Secretariat must sometimes rely on individuals who have not completed the training to Chair a peer-review assessment as it is a volunteer position that is dependent on the availability of DFO staff.

POTENTIAL SOLUTION:

The Secretariat develops a conflict of interest policy with clear guidelines on the roles and responsibilities of peer-review assessment participants including the Chair. In particular, the guidelines should include expectations for documenting potential conflicts of interest and how the Chair is expected to manage participants who may be acting in a conflict of interest. Consequently, if the Chair of a peer-review assessment has not completed the CSAS Chair Training they would have the guidelines outlined in the policy to help them mitigate potential situations of conflict of interest.

POTENTIAL SOLUTION:

That anyone who assumes the role of the Chair for a science advisory process has completed the CSAS Chair training. This will help ensure that the Chair has the skills and knowledge required to mitigate potential situations of conflict of interest.

6.9 Evaluation findings: Quality of the science advice

Client Perspectives Regarding the Quality of the Science Advice

CSAS clients were asked to assess various aspects of the science advisory process. The survey results are presented in Figure 11, below.

Figure 11: Client perspectives regarding the quality of the science advice

Overall the majority of clients perceive that the science advisory reports and/or science responses are clear and easy to understand (67%), provide the information required (67%) and align with the scope of the terms of reference (66%). The uncertainties and limitations of the advice are also clearly outlined (74%) and provided with sufficient detail (66%).

Additional analysis reveals that the greater the alignment between the terms of reference and the resulting science advisory report or science response the more likely it is that the science advice is useful to and used by the client.

6.10 Opportunities to improve the quality of the science advice

Unable to Use the Science Advice

In some cases clients were unable to use the science advice received as it did not align with their needs or was unclear. Some contributing factors in these situations include:

The terms of reference is developed once research is underway and the client compromises what they need in terms of what they can get without realizing the potential implications on future decision-making.

The extent to which EOS is able to address the clients’ question(s) evolves once the research is underway.

The client’s needs change between the time the terms of reference is drafted and the peer-review assessment.

POTENTIAL SOLUTION

Formalized check-in points between the drafting of the terms of reference and the peer-review assessment. Will support ongoing communication between the science assessment lead/ team and the client to ensure that the question(s) and objectives continue to be relevant for the client and that if there are any changes to the extent to which the questions can be answered by EOS scientists the client, CSAS and the Chair are informed of these changes. The Secretariat and centres may support these discussions provided they have the available resources.

For Consideration:

• The type of check in point can be flexible based on the risk level of the request and/or the preferences of the assessment team and client.
• Ideas provided throughout the evaluation include: email updates; multiple Steering committee meetings; and, informal meetings.

7.0 Phase 3 of the CSAS process – Findings

7.1 Overview of phase 3

The third phase of the process includes the following two main stages:

• Stage 7: Following the peer-review assessment the Chair of the peer-review assessment is responsible for ensuring the science advisory product and the proceedings document are finalized, reviewed and/or approved by participants of the assessment before submitting the documents to the appropriate regional centre or the Secretariat. The science lead is responsible for ensuring that the research document is finalized with any updates discussed during the peer-review assessment and submitted to the appropriate regional centre or the Secretariat for publication.
• Stage 8: Once documents are received the regional centre or Secretariat ensure they are complete, formatted as required and available in both official language for publication. Regional centres then submit all documents to the Secretariat for publication. The Secretariat publishes all submitted documents on the CSAS website ensuring that the science advice is publicly available.

Figure 12: Phase 3 of the CSAS science advisory process: Knowledge dissemination

7.2 The importance of open and timely science advice

The CSAS science advisory process aligns with best practices and adheres to the SAGE principle of openness and transparency by striving to make all science advice and accompanying documents publicly available through the CSAS website.

• Best practices articulate that “[a]ll scientific findings and the analysis underlying management decisions should be readily accessible to the public” (Penney, 2010, p. 74), and that this information should be provided in a timely manner.
• Such practices are aligned with the SAGE principle of openness, particularly the expectation that the “public has access to the findings and advice of scientists as early as possible” (CTSA, 2009).

Evaluation findings however, illustrate that there are opportunities for improvement when it comes to ensuring the CSAS documents are available to the public in a timely manner.

• For instance, the analysis of the CSAS publications database reveals that the majority of CSAS documents are not submitted for publication or published within required timelines outlined in the CSAS Policy on Timelines for Submission and Publication of Documents.

7.3 Evaluation findings: Submission and compliance rates

Submission Compliance Rates - Targets

CSAS documents must be submitted to the Secretariat for publication as soon as possible following the peer-review assessment and no longer than:

• 8 weeks for the Science Advisory Reports (SAR) or Science Reponses (SRR).
• 4 months for the Research Documents (RES) and Proceedings (PRO).

Figure 13, 14 and 15 show the submission compliance rates for all of the CSAS regions and for all the types of document.

Figure 13: The proportion of CSAS documents submitted late to CSAS for all regions and types of document.

Of the 77% of documents that were submitted and published, 62% of them were submitted late. When broken down by document type, the science advisory reports and research documents have the lowest compliance rates.

Figure 14: The proportion of CSAS documents submitted on time, separated by document type

Figure 15: The median and mean amount of time documents are submitted to CSAS late

7.4 Challenges impacting submission compliance rates

According to key informants, case study participants and applicant survey respondents, CSAS documents are often submitted past expected timelines as a result of the following challenges:

Compliance with Federal Government and CSAS Policies for Communications and Web Publications

Compliance with the Government of Canada’s policies related to communications and web publications requires significant time. Examples of compliance requirements include formatting documents and ensuring they conform to web accessibility standards.

On April 1st, 2018 the CSAS Policy on the Translation of Documents was implemented. This policy was noted as being one of the main factors delaying the submission of documents as compliance requires significantly more time than what was originally conceived when the Policy on Timelines for Submission and Publication of Documents was drafted.

Increased Pressure on Workload

EOS Scientists

Completing CSAS documents was noted as a cumbersome process, which further constrains the workload of EOS scientists and impacts their ability to engage in other research-related activities. Scientist(s) responsible for completing CSAS documents may also forgo completing CSAS documents as they must turn their attention to tasks put on hold during the CSAS process.

CSAS Staff

Sometimes documents are received unformatted or partially formatted and CSAS staff must work with scientists to finalize the documents or in some cases finalize the documents themselves. This may significantly impact the workload of centre or Secretariat staff, particularly when there are vacant positions.

Participation in the CSAS science advisory process is not perceived by EOS scientists as contributing to their career development.

Key informants, case study participants and applicant survey respondents noted that the completion of CSAS documents is not a high priority for some EOS scientists who do not consider these documents as contributing towards their career progression.

7.5 Potential solutions to mitigate the challenges impacting submission and compliance rates

POTENTIAL SOLUTIONS

Extend timelines for submitting documents to the Secretariat for publication in consultation with CSAS clients and EOS scientists.

In particular, the timelines for the science advisory reports and the science responses, which are half the timeline of the proceedings and research documents.

Extending submission timelines may help:

• Provide additional time for translation and formatting, especially for the research documents, which can be very large.
• Provide the individuals responsible for completing the documents flexibility to complete the documents and commence new tasks.

The notion of extending timelines is intended to increase compliance rates by considering activities that were not originally required. At the same time it is important to consider the best practice of ensuring documents are publicly available in a timely manner and that clients receive the science advice within their required timelines.^{Footnote 12}

Collaboration between the Secretariat, regional centres and management in the EOS sector to dispel myths that participation in the CSAS science advisory process does not contribute towards the career progression of EOS scientists.

This may include:

• Providing EOS sector staff with education and training related to their involvement within the CSAS science advisory process including the potential impact on their career.
• Updating the Careers Progression Management Framework for DFO researchers to emphasize the role of the CSAS science advisory process in terms of career progression. Currently, the framework refers to science advice, but not the CSAS science advisory process specifically.

Capacity review

• The EOS sector EOS sector consider the capacity needed to address the workload around the submission of CSAS documents.

7.6 Publication compliance rates

Publication Compliance Rates - Targets

CSAS documents must be submitted to the Secretariat for publication as soon as possible following the peer-review assessment and no longer than:

• 10 working days for the Science Advisory Reports (SAR) or Science Reponses (SRR).
• 3 weeks for the Research Documents (RES) and Proceedings (PRO).

Figure 16, 17 and 18 show the publication compliance rates all of the CSAS regions and documents

Figure 16: The proportion of CSAS documents published online by CSAS late

Of the 77% of the published CSAS documents, 66% of them were published late. When broken down by document type, the science advisory reports and science response have the lowest compliance rates.

Figure 17: The proportion of CSAS documents published on time, separated by document type

Figure 18: The median amount of time documents are published by CSAS late

7.7 Challenges impacting publication compliance rates and opportunities for improvement

CHALLENGE - Secretariat Resources

Several of the FTE positions within the Secretariat remain vacant resulting in a lack of human resources to support activities and tasks related to the publication of CSAS documents. In addition to publishing all CSAS documents, Secretariat staff have other responsibilities such as, managing the submission of all of the CSAS documents related to national peer-review assessments, as well as running queries and producing reports related to the requests and publications databases. While they do their best to complete all of their tasks within required timelines it is sometimes not feasible to do so, particularly when multiple documents are submitted for publication within a short time period.

POTENTIAL SOLUTION

Capacity review

The EOS sector consider the capacity needed to address the workload around CSAS documents and specifically publications.

CHALLENGE – External Notification and Approval

Findings from the evaluation highlight that publication delays may occur as a result of a number of reasons that are external to the Secretariat, examples of which include: the approval process for media lines connected with a document; providing senior management with two weeks’ notice before CSAS documents are posted; and, the PDF proof approval by the centres. Such delays are especially relevant for science advisory reports and science responses, which should be published within ten working days once they are submitted to the Secretariat.

POTENTIAL SOLUTION

Timelines

Extend publication timelines to provide the required time for notifications to and approval from senior management. Timeline extensions may also help mitigate some of the current issues within the Secretariat related to human resources until vacant positions are filled.

Using CSAS documents to support DFO activities

In addition to ensuring that CSAS documents are publicly available it is very important that documents are shared with CSAS clients and other-end users in a timely manner as these individuals are the primary users of the science advice for decision-making or to inform other research activities.

86% of respondents of the client and end-user survey and 75% of respondents of the applicant survey noted that they use one or more CSAS documents to support their work.

It was noted that several clients also use the research documents and appreciate having access to the data and analysis that informed the science advice. It is also valuable to have the Research Document publicly available during the consultation process for transparency purposes.

The proceedings documents are generally not used by the clients.

Supporting the use of science advice: Moving beyond publications

Receiving documents

Key informant interviewees and case study participants noted that about a third of the time clients are not receiving CSAS documents within their required timelines; thereby, limiting the extent to which the advice is used for decision making or delaying management decisions, which can, in turn, undermine the credibility of the science advisory process (CSAS, 2016).

• When documents are unavailable clients may use draft versions; however, they are uncertain as to what extent these documents can be used or shared given that they are not published (i.e. publicly available).

Interactive Forms of Communication

7.10 Other forms of knowledge dissemination - potential solutions

POTENTIAL SOLUTIONS

Technical briefings between the EOS scientist(s) who conducted the research that fed into the science advice and the client(s) following the peer-review assessment. This is an opportunity to review and discuss the draft science advice before it is finalized and submitted for publication. It is expected that such briefings would support the use of the science advice by providing:

• CSAS clients with the opportunity to ask any outstanding questions and to confirm that have a clear understand of the science advice before using it in the decision-making process.
• EOS scientists with the opportunity to learn how their work will be used for decision making.

According to the results of the applicant survey a little more than half of EOS scientists (56%) are supporting CSAS clients as they use science advice for decision making. Consequently, there are opportunities to improve the science-policy interface at this point in the CSAS science advisory process.

A standardized approach for disseminating the final versions of CSAS documents to clients, particularly the science advisory report or science response.

While some clients receive documents directly from the Chair, the EOS science lead, the regional center or the Secretariat, others noted that they only become aware that final versions of the document are available when they find them on the CSAS website.

Best practices around the timeliness for providing science advice to decision makers indicates that at times this requires the information to be shared before it is published, or in some cases even before it is finalized depending on the timelines for management decisions (Penney, 2010).

8.0 Conclusions and recommendations

CONCLUSION

The CSAS science advisory process is generally perceived as a unique and important mechanism for developing and providing science advice in support of decision-making. It also supports industry by providing credible stock assessment data that is required by industry when seeking eco-certification for their product. The CSAS aligns with DFO’s mandate and four performance indicators in the Departmental Results Framework related to the CSAS process are used to demonstrate the achievement of the department’s expected outcome.

The SAGE Principles serve as the foundation for the CSAS science advisory process and reflect best practices for the provision of science advice. They also align very well with the priorities of DFO and the federal government including: open government; reconciliation; and, strengthening science.

One of the main strengths of the CSAS science advisory process is the inclusion of peer-review assessments for developing the science advice. Peer-review is a highly respected and accepted process to assess and ensure the quality, objectivity and reliability of scientific methods and information and is considered a best practice when using scientific information to inform fisheries management decisions.

The number of requests addressed by the CSAS remains relatively stable; however, the number of requests received increased by 31% between 2013-14 and 2017-18 suggesting that there is a greater demand for science advice within DFO.

Over 60% of clients and end-users feel that CSAS is doing a good or great job providing them with the information they require and that this information: aligns with the scope of the questions in the terms of reference; outlines and provides details regarding the uncertainties and limitations of the advice; and, is clear and easy to understand.

86% of respondents of the client and end-user survey and 75% of respondents of the applicant survey noted that they use one or more CSAS documents to support their work.

By striving to make all science advice and accompanying documents publicly available through its website the CSAS science advisory process aligns with best practices and adheres to the SAGE principle of openness and transparency.

While the CSAS is valued and well-respected within DFO there are several opportunities for improving the science advisory process.

Conclusions

Findings from the evaluation indicate that the CSAS science advisory process may be improved through the implementation of mechanisms that standardize and formalized communication between EOS scientists and CSAS clients during each phase of the science advisory process.

Phase 1

Communication between the CSAS, its clients and EOS scientists during the development and assessment of requests for science advice may improve the alignment between management questions and science advice and the triaging of requests submitted to the CSAS.

Phase 2

Check-in points between the drafting of the terms of reference and the peer-review assessment is expected to help ensure that the question(s) and objectives outlined in the terms of reference continue to be relevant for the client and that if there are any changes to the extent to which the questions can be answered by EOS scientists the client, CSAS and the Chair are informed of these changes.

Phase 3 The implementation of technical briefings between the EOS scientist(s) who conducted the research that fed into the science advice and the client(s) following the peer-review assessment is expected to support the use of the science advice by client in decision-making.

Recommendations

It is recommended that the ADM, Ecosystem and Oceans Science:

1. Develop mechanisms to support the standardization of formal communication between EOS scientists and CSAS clients during each phase of the science advisory process. These mechanisms should allow for flexibility in terms of the type and extent of communication during each phase of the science advisory process based on the risk level and complexity of the request, as well as the relationship between and the preferences of the science assessment lead and client.

Conclusions

Several of the factors that challenge the extent to which the CSAS is able to address the requests for science advice received each year are linked to the SAGE Principle of early issues identification and may be mitigated with the implementation of multi-year science advisory schedules. In particular, a multi-year schedule would allow the EOS sector to identify the immediate and long-term needs for science advice within DFO, which may support: better planning and allocation of EOS resources; triaging of requests; a reduction in the frequency of in-year requests; and, addressing high risk requests.

A multi-year schedule may also beneficial as it can be pre-populated with requests that are: (1) carried over from previous years; (2) recurring; and/or, (3) legislative or regulatory requirements. Consequently, CSAS clients may not be required to re-submit requests for science advice they continue to require as long as the request (i.e. questions to be answered) have not changed. The CSAS could however, encourage clients to review and reprioritize ongoing requests until they are added to the schedule.

Currently, some regional centres are employing multi-year schedules with various levels of success. It is advisable to consult with these regions and assess lessons learned before standardizing any practices across all regions.

Recommendations

It is recommended that the ADM, Ecosystem and Oceans Science:

2. Implement a multi-year science advisory schedule as part of the CSAS science advisory process to support adherence to the SAGE Principle of early issues identification and to mitigate some of the challenges impacting the extent to which the CSAS is able to address requests for science advice.

Conclusions

Findings from the evaluation revealed that there is a tension among CSAS stakeholders regarding who should participate in peer review assessments. Some stakeholders believe that only scientists and/or individuals who are able to understand and contribute to the scientific knowledge presented during these assessments should participate. Other stakeholders believe that the peer review assessments should be more inclusive of participants from industry, academia, Indigenous communities and non-governmental organizations.

The main contributing factor for the tension around participation in peer-review assessments is the lack of guidance regarding the operationalization of the SAGE principle of inclusiveness, particularly around the definition of expert.

Best practices suggest that peer-review processes for developing science advice should involve:

• a wide range of scientific expertise from relevant disciplines within and outside government.
• Representatives from external stakeholder or interest groups as relevant local and traditional knowledge should also be acknowledged as part of the peer-review.
• Clients/managers as their participation may improve understanding, encourage buy-in and streamline science-policy communication.
  • It is important to note that like external stakeholders clients/managers cannot be expected to act impartially.

Recommendations

It is recommended that the ADM, Ecosystem and Oceans Science:

3. Develop clear guidelines to support the operationalization of the SAGE principle of inclusiveness. In particular, the guidelines should include criteria regarding which participant groups should be invited to the different types of science advisory processes (i.e. SSRP compared to the more comprehensive peer-review assessment resulting in a science advisory report) and during which stages of the process they should be invited to participate.

Conclusions

Some key informants and case study participants are concerned that external stakeholder groups (e.g. industry, non-governmental organizations) participating in the CSAS process may attempt to use the process to advance their own agenda.

A comparative analysis of best practices revealed that several international science advisory processes related to fish and oceans management also rely on the Chair of their peer-review assessment to manage any potential conflicts of interest that may arise. At the same time however, they also have robust conflict of interest guidelines and/or policies to further mitigate situations where participants may act more as advocates for their organization instead of objective peer-reviewers.

The CSAS does not have a conflict of interest policy or general guidelines on how to mitigate potential situations of conflict of interest. Currently, the Chair of the science advisory process is responsible for addressing conflicts of interest and is taught how to do so through the CSAS Chair training. Not all of the active Chairs however, have participated in this training and as a result some Chairs may lack the skills and knowledge to properly mitigate conflicts of interest. If however, the CSAS had a conflict of interest policy with clear guidelines around how the Chair is expected to manage participants who may be acting in a conflict of interest any Chair of a peer-review assessment who had completed the CSAS Chair Training could use the policy to help them mitigate potential situations of conflict of interest.

Recommendations

It is recommended that the ADM, Ecosystem and Oceans Science:

4. Develop a conflict of interest policy to support adherence to the SAGE Principle of sound science advice with clear guidelines regarding the roles and responsibilities of peer-review assessment participants including the Chair.

Conclusions

Evidence from the evaluation suggests that the timeline targets for submitting and publishing CSAS documents as outlined in the CSAS Policy on Timelines for Submission and Publication of Documents may no longer be feasible to achieve as a result of several factors including:

• Government of Canada policies and guidelines for communications and web publications;
• the implementation of new policies (e.g. Policy on the Translation of Document);
• workload pressures;
• perceptions that writing CSAS documents does not contribute towards career progression;
• available CSAS resources; and,
• external notification and approval requirements.

Consequently, the compliance rates for submitting and publishing CSAS documents on time (i.e. as per the targets) are quite low.

Recommendations

It is recommended that the ADM, Ecosystem and Oceans Science:

5. Review timeline targets and develop mechanisms to increase compliance rates for the submission and publication of CSAS documents.

9.0 Management Action Plan

9.1 Recommendation no 1

Recommendation 1: It is recommended that the ADM, Ecosystem and Oceans Science develop mechanisms to support the standardization of formal communication between EOS scientists and CSAS clients during each phase of the science advisory process. These mechanisms should allow for flexibility in terms of the type and extent of communication during each phase of the science advisory process based on the risk level and complexity of the request, as well as the relationship between and the preferences of the science assessment lead and client.

Rationale: Findings from the evaluation indicate that the CSAS science advisory process may be improved through the implementation of mechanisms that standardize and formalized communication between EOS scientists and CSAS clients during each phase of the science advisory process. In particular, improved communication during the development and assessment of requests for science advice may improve the alignment between management questions and science advice and the triaging of requests submitted to the CSAS. Check-in points between the drafting of the terms of reference and the peer-review assessment is expected to help ensure that the question(s) and objectives outlined in the terms of reference continue to be relevant for the client and that if there are any changes to the extent to which the questions can be answered by EOS scientists the client, CSAS and the Chair are informed of these changes. Finally, the implementation of technical briefings between the EOS scientist(s) who conducted the research that fed into the science advice and the client(s) following the peer-review assessment is expected to support the use of the science advice by client in decision-making.

Strategy

In consultation with the CSAS clients, opportunities to enhance communication at each phase of the science advisory process will be explored. Mechanisms to improve communication and how CSAS interfaces with its clients will be incorporated into CSAS national and regional procedures which will be monitored over time to assess if there has been improvements or if further enhancements are required.

Management actions

Consultations with clients on information needs and support structures for a renewed CSAS process

Due Date (by end of month)

September 2019

Management actions

CSAS Office to develop, integrate, and communicate new expectations regarding communications with client sectors

Due Date (by end of month)

January 2020

Recommendation no 2

Recommendation 2: It is recommended that the ADM, Ecosystem and Oceans Science implement a multi-year science advisory schedule as part of the CSAS science advisory process to support adherence to the SAGE Principle of early issues identification and to mitigate some of the challenges impacting the extent to which the CSAS is able to address requests for science advice.

Rationale: Several of the factors that challenge the extent to which the CSAS is able to address the requests for science advice received each year are linked to the SAGE Principle of early issues identification and may be mitigated with the implementation of a multi-year science advisory schedule. In particular, a multi-year schedule would allow the EOS sector to identify the immediate and long-term needs for science advice within DFO, which may support: better planning and allocation of EOS resources; triaging of requests; a reduction in the frequency of in-year requests; and, addressing high risk requests.

Strategy

A multi-year science advisory schedule will enhance the CSAS to manage and plan processes while aligning to Departmental priorities. Some regions are currently following this approach but it is not used nationally. To develop a national approach, a review of regional practices is required and then a process will be developed to facilitate a national multi-year scheduling for fiscal year 21/22.

Management actions

• Call for a multi-year schedule

Due Date (by end of month)

• January 2020

Management actions

• CSAS Office/Science Staff conclude engagement with client sectors regarding multi-year advisory needs

Due Date (by end of month)

• December 2020

Management actions

• Approval of Multi-year Advisory Schedule by ADM Ecosystems and Oceans Science and relevant Departmental governance committee

Due Date (by end of month)

• March 2021

Recommendation no 3

Recommendation 3: It is recommended that the ADM, Ecosystem and Oceans Science develop clear guidelines to support the operationalization of the SAGE principle of inclusiveness. In particular, the guidelines should include criteria regarding which participant groups should be invited to the different types of science advisory processes (i.e. SSRP compared to the more comprehensive peer-review assessment resulting in a science advisory report) and during which stages of the process they should be invited to participate.

Rationale: There is a tension among CSAS stakeholders regarding who should participate in peer review assessments. Some stakeholders believe that only scientists and/or individuals who are able to understand and contribute to the scientific knowledge presented during these assessments should participate. Other stakeholders believe that the peer review assessments should be more inclusive of participants from industry, academia, Indigenous communities and non-governmental organizations. The main contributing factor for the tension around participation in peer-review assessments is the lack of guidance regarding the operationalization of the SAGE principle of inclusiveness, particularly around the definition of expert.

Strategy

A review of the SAGE principles will be conducted with the intent to incorporate the principles into existing or new polices as applicable. These policies will explicitly demonstrate where the SAGE principles apply and how CSAS will measure and report on their application.

Management Actions

• Approve and communicate policy on participation in CSAS processes.

Due Date (by end of month)

• December 2019

Recommendation no 4

Recommendation 4: It is recommended that the ADM, Ecosystem and Oceans Science develop a conflict of interest policy to support adherence to the SAGE Principle of sound science advice with clear guidelines regarding the roles and responsibilities of peer-review assessment participants including the Chair.

Rationale: The CSAS does not have a conflict of interest policy or general guidelines and there is concern that external stakeholder groups (e.g. industry, non-governmental organizations) participating in the CSAS process may attempt to use the process to advance their own agenda. A comparative analysis of best practices revealed that several international science advisory processes related to fish and oceans management have robust conflict of interest guidelines and/or policies to mitigate situations where participants may act more as advocates for their organization instead of objective peer-reviewers.

Strategy

The need for a conflict of interest policy or guideline for the CSAS processes is evident. A draft policy will be developed using best practices of known science advisory processes.

Management Actions

• Approve and communication a conflict of interest policy for participation in CSAS processes

Due Date (by end of month)

• Septembre 2019

Recommendation no 5

Recommendation 5: It is recommended that the ADM, Ecosystem and Oceans Science review timeline targets and develop mechanisms to increase compliance rates for the submission and publication of CSAS documents.

Rationale: Timeline targets for submitting and publishing CSAS documents as outlined in the CSAS Policy on Timelines for Submission and Publication of Documents may no longer be feasible to achieve as a result of several factors including: requirements to comply with government policies; workload pressures; perceptions that writing CSAS documents does not contribute towards career progression; available resources; and, external notification and approval requirements. Consequently, the compliance rates for submitting and publishing CSAS documents on time (i.e. as per the targets) are quite low.

Strategy

Stated timelines for the submission and publication of CSAS advisory products must be balanced with client needs while meeting government of Canada requirements for accessibility and official languages. Options will be considered which will include reassessing current timelines in the context of new publishing requirements to reviewing CSAS production processes with aim to identify efficiencies which may improve turn-around-times at both the regional and national levels.

Management Actions

• Science Executive Committee approval of new national standardized procedures and products.

Due Date (by end of month)

• December 2020

Management Actions

• CSAS Office incorporates new guidelines in planning process for the inaugural multi-year call for advice.

Due Date (by end of month)

• January 2020

References

Council of Science and Technology Advisors (CSTA) (1999). Science Advice for Government Effectiveness (SAGE). Retrieved from:
http://publications.gc.ca/collections/Collection/C2-445-1999E.pdf

Government Office for Science (2010 ). The Government Chief Scientific Adviser’s Guidelines on the Use of Scientific and Engineering Advice in Policy Making. Retrieved from:
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/293037/10-669-gcsa-guidelines-scientific-engineering-advice-policy-making.pdf

Ministry of Fisheries (2011). Research and Science Information Standard for New Zealand Fisheries. Wellington, New Zealand. Retrieved from:
http://www.openseas.org.nz/wp-content/uploads/2017/06/MPI_ResearchandScienceInformationStandard _2011.pdf

National Research Council (2004). Improving the use of the “best scientific information available” standard in fisheries management. Washington, DC: National Academies Press.

Penney, A.J. (2010). Review of International Guidelines Relating to Science Quality Assurance and Peer Review. New Zealand Ministry of Fisheries. Retrieved from:
http://fish.govt.nz/NR/rdonlyres/84ED7B5D-7CD3-49D4-B89F-7F7437789446/0/International_Guidelines_Science_ Quality_Assurance.pdf.

The National Academies of Sciences, Engineering and Medicine (2016). Background information and confidential conflict of interest disclosure for studies related to government regulation. Retrieved from:
http://www.nationalacademies.org/coi/bi-coi_form-1.pdf?_ga=2.259525578.1384469451.1547491427-1976804095.1547491427