Evaluation of the Oceans Protection Plan (Phase 1)

Final evaluation report
June 24, 2019

Evaluation Division, Planning, Results and Evaluation Directorate

Fisheries and Oceans Canada

This first evaluation of the Oceans Protection Plan (OPP) focuses on information management / information technology (IM/IT) challenges within Fisheries and Oceans Canada and Canadian Coast Guard OPP initiatives. This evaluation will be followed by a second evaluation in 2019-20, and both will inform the horizontal evaluation of the OPP starting in 2021-22. 

Table of contents

• 1.0 Evaluation Context
  • 1.1 Oceans Protection Plan
  • 1.2 OPP – Four main pillars representing priority areas
  • 1.3 Evaluation scope
  • 1.4 Four initiatives were selected to examine these issues, one from each OPP pillar
  • 1.5 Methodology and evaluation questions
• 2.0 Overview of OPP Pillars and Selected Initiatives
• 3.0 Initiative profiles:
  • 3.1 Modern hydrography and charting in key areas
  • 3.2 Improving drift prediction and near-shore modelling
  • 3.3 Reducing the threat of vessel traffic on whales and other marine mammals through detection and avoidance
  • 3.4 Collaborative Situational Awareness Portal (CSAP)
• 4.0 Summary of performance for the first two years
• 5.0 Scope of initiatives’ IM/IT needs
• 6.0 IM/IT context prior to OPP
• 7.0 Evaluation findings:
  • 7.1 OPP planning for IM/IT requirements was limited
  • 7.2 There were challenges in providing IM/IT services during the first two years of OPP
  • 7.3 Solutions were developed to achieve milestones and key outputs
  • 7.4 IM&TS is developing approaches to move OPP forward
  • 7.5 Department-wide efforts regarding data management and funding are in progress
  • 7.6 Beyond IM/IT, there were other findings related to the first two years of OPP
• 8.0 Recommendations
• 9.0 Appendices
• Appendix A: Evaluation methodology
• Appendix B: Limitations and mitigation strategies
• Appendix C: Gender-based analysis plus
• Appendix D: Management action plan

NOTE: The Evaluation to Oceans Protection Plan (Phase 1) has an alternative report format that uses graphics and visuals, which is available in PDF [2.35 MB] format.

1.0 Evaluation context

1.1 Oceans Protection Plan

On November 7, 2016, the Federal Government launched a $1.5 billion national Oceans Protection Plan (OPP) to improve marine safety and responsible shipping, to protect Canada's marine environment, and to offer new possibilities for Indigenous and coastal communities.

As a whole-of-government strategy, the OPP is led by Transport Canada (TC) and is implemented by multiple departments to achieve shared outcomes, including Environment and Climate Change Canada (ECCC), Natural Resources Canada (NRCan) and Fisheries and Oceans Canada (DFO) and the Canadian Coast Guard (CCG).

The OPP consists of 57 initiatives that fall under four main pillars. DFO and CCG are involved in 41 initiatives.

1.2 OPP – Four main pillars representing priority areas

1. State-of-the-art Marine Safety System: Create a world-leading marine safety system that protects Canada’s waters, including new and preventive response measures.
2. Stronger Evidence Base, Increased Community Participation and Public Awareness: Invest in oil spill cleanup research and methods to ensure that decisions taken in emergencies are evidence based.
3. Preservation and Restoration of Marine Ecosystems: Restore and protect marine ecosystems and habitats.
4. Indigenous Partnerships: Create stronger local emergency response capacity by establishing Indigenous partnerships and by engaging coastal communities.

1.3 Evaluation scope

The scope of the evaluation covers the period 2017-18 to 2018-19 and is based on information collected up to February 2019. The evaluation is inclusive of National Headquarters and all DFO and CCG regions.

The evaluation was designed as an implementation evaluation^{Footnote 1} to respond to key information needs that were identified by senior management:

• Document early successes and implementation challenges of OPP initiatives; and,
• Gather information on the IM/IT challenges of four data-intensive initiatives.

1.4 Four initiatives were selected to examine these issues, one from each OPP pillar

1. Modern Hydrography and Charting in Key Areas
2. Improving Drift Prediction and Near-shore Modelling
3. Reducing the Threat of Vessel Traffic on Whales and Other Marine Mammals through Detection and Avoidance
4. Collaborative Situational Awareness Portal (CSAP)

1.5 Methodology and evaluation questions

The evaluation relied on multiple lines of evidence, including interviews, case studies, data analysis, site visits, and document review to answer the following questions:

1. To what extent are the OPP initiatives examined as part of the evaluation on track to achieve their milestones and key outputs in a timely manner?
2. To what extent have DFO and CCG provided adequate IM/IT support required by the OPP initiatives examined as part of the evaluation?
3. To what extent have the IM/IT needs associated with the OPP initiatives examined as part of the evaluation been costed?
4. What has been the impact of OPP on DFO and CCG IM/IT support teams’ workload and ability to meet client requests?
5. To what extent have access and participation of women and under-represented populations been considered in the OPP initiatives examined as part of the evaluation?

Details about the methodology are available in Appendices A, B, and C.

2.0 Overview of OPP pillars and selected initiatives

The table represents the program logic chain of the four selected initiatives. Each initiative contributes to the shared outcomes and targets, and to the strategic result of the respective OPP pillar in the same column (with the exception of the Near-shore Hydrographic Modelling for E-navigation project, which contributes to the outcome and result of Pillar 1). Funding information is provided for each initiative at the bottom of the table.

3.0 Initiative profiles:

3.1 Modern hydrography and charting in key areas

Modern hydrography and charting in key areas, under DFO Science, Canadian Hydrographic Service, aims to provide better navigational information in the hands of mariners.

Description

This initiative conducts activities related to the following four areas:

• Ports: Conduct intensive modern hydrographic and charting activities to provide electronic navigation charts for highly critical areas across the country, including 23 highest priority commercial ports and waterways (13 in British Columbia, seven in Quebec, and three on the Atlantic coast).
• Near-Shore Hydrography: Fill important gaps in high-resolution coastline and bathymetry in inter-tidal zones and near-shore areas to ensure the delivery of improved navigational charts and enhanced electronic navigational charts in near-shore areas (e.g., Haida Gwaii), high-risk coastal and inland water zones.
• Arctic Hydrography and Charting: Undertake extensive efforts, in collaboration with CCG and involvement of academia and private sector, to fill hydrographic data gaps in the Arctic, through the provision of new and updated navigational products and services.
• Dynamic Hydrographic Products and Services: Strengthen navigational safety and the prevention of marine incidents by delivering, in collaboration with DFO’s Ocean Science Program, dynamic hydrographic products and services (tide and water level, under-keel and overhead information) in key areas.

Key risks if delayed: Mariners would not have the most up-to-date information needed to prevent marine accidents and to protect marine protected areas.

3.2 Improving drift prediction and near-shore modelling

This initiative reflects the reliance of the federal government and external organizations on accurate ocean models to respond to environmental and maritime disasters (e.g., spills and accidents) and to support electronic navigation.^{Footnote 2}

Description

The broad objectives of this initiative are to enhance the Government of Canada’s ocean modelling activities in support of:

• Enhanced environmental protection and marine safety applications (e.g., drift prediction for oil spills):
  • This initiative will work to enhance the drift prediction capacity currently in place in support of improved oil spill (or other contaminant) mitigation. The outputs of DFO’s work to improve drift prediction will feed into improving ECCC capacity for marine emergency response and will support ECCC’s National Environmental Emergency Centre in its response related to environmental incidents.
• Enhanced safety for navigation and related activities (hydrographic e-navigation):
  • This initiative will develop high-resolution, near-shore hydrodynamic models for use by Canadian Hydrographic Services in the development of dynamic navigation products and services.

The initiative will prioritize six high-risk ports/areas: Kitimat, Port of Vancouver (Vancouver Harbour, Fraser River), Strait of Canso Port, Port of Saint John and the St. Lawrence River (Québec City-Montréal). Once work on those ports has been completed, following a risk-based approach, activities will be expanded to cover additional areas of Canadian waters (beyond the OPP timeframe). 

Key risks if delayed: Delays in this initiative would lead to limited improvement to the accuracy of drift prediction models which could lead to environmental impacts. In addition, delays would lead to limited improvement in hydrographic e-navigation charts which could have safety and economic impacts.

3.3 Reducing the threat of vessel traffic on whales and other marine mammals through detection and avoidance

This initiative will develop and test technologies able to detect the presence of whales in near-real time on both the west and east coasts.

Description

The objective of this initiative is to develop and test various acoustic and imaging technologies to detect the presence of whales in near-real time, particularly the Southern Resident Killer Whale (SRKW) on the Pacific coast and the North Atlantic Right Whale (NARW) on the Atlantic Coast.

The ability to capture this up-to-date information could help alert mariners of whales in a particular area and help to reduce the risk of vessel collisions.

This initiative is closely linked to the Marine Environmental Quality (MEQ) initiative. Both initiatives collect data for acoustic detection of whales and tracking vessels. MEQ uses the data to assess the impacts of noise on whales.

Key risks if delayed: Delays in developing, testing, and evaluating near-real time whale detection technologies could delay the development of a potential whale collision and avoidance system.

3.4 Collaborative Situational Awareness Portal (CSAP)

This initiative will establish a maritime awareness information system that will provide coastal partners and stakeholders with an accessible, comprehensive, and user-friendly picture of maritime activity, including vessel traffic.

Description

CSAP is an online platform aimed at increasing access to local maritime information for Indigenous partners, coastal communities, and stakeholders. For example, internal Coast Guard systems, such as the Marine Communication and Traffic Services centres, will share information to Indigenous communities to increase awareness and collaboration.

The CCG is conducting engagement sessions to collect feedback to inform the development of the online platform, provide training opportunities for Indigenous and coastal communities on the use of the tools and the breadth of available information, and to develop closer bilateral relationships with Indigenous communities.

The roll out plan for CSAP includes all of Canada’s coastal areas. Initial delivery of access licenses started nationally in communities with a high-level of bilateral engagement with other

OPP initiatives. The goal is to have 30 licenses for each of the three CCG regions by the end of 2019-20. This will increase until all interested communities have access.

Key risks if delayed

The greatest risk if this initiative is delayed would be to the relationship with Indigenous communities who have been engaged in the development of this portal. Delays could lead to a loss of credibility and trust with the federal government, as well as potential operational gains from increased collaboration.

4.0 Summary of performance for the first two years

Overall, the initiatives have achieved their milestones and key outputs for the first two years.

The evaluation collected evidence to:

• Examine the achievements of the initiatives during the first two years of OPP and to document early successes.
• Identify significant challenges, mainly related to:
  • The access of IM/IT resources and services;
  • Data management;
  • The holistic coordination across all areas responsible for the implementation of OPP, with the view of the overall achievement of results; and
  • Performance tracking and monitoring.
• Identify and document examples of good practices that allowed the initiatives to mitigate delays and other impacts as a result of the IM/IT challenges.
• Identify some alternative solutions that were used to mitigate the IM/IT challenges, but are not sustainable and pose significant risks.

The evaluation could not assess the extent to which key outputs, results, and targets are on track for the next three years of OPP

• Currently, the OPP budget is insufficient to address IM/IT needs and there is no feasible solution with regards to funding re-allocations (as of April 2019).
• It is not clear whether the efforts to improve data governance currently in progress are sufficient and timely enough to address OPP data management issues and associated risks.
• Individual initiatives have been dealing with challenges, developing good practices, and setting their priorities in silos with little coordination with other areas (sectors, regions, IM/IT support).
• There may be missed opportunities to coordinate efforts to realize efficiencies and achieve the overall OPP results.

A detailed description of the initiatives’ performance for the first two years is provided on the following pages.

5.0 Scope of initiatives’ IM/IT needs

Highest IM/IT needs

Modern Hydrography and Detection and Avoidance need static data storage for the large volume of incoming data (e.g., bathymetry and acoustic data). Data management is needed to ensure access, protection, stewardship, and use of data.

Drift Prediction and Detection and Avoidance/MEQ rely on High Performance Computers (HPC) for running complex models for near-shore hydrography, oil drift prediction, and whale detection.

Modern Hydrography needs specialized hydrographic software applications for the maps and uses GIS products and services for dynamic products.

Modern Hydrography and CSAP require network infrastructure and connectivity for dynamic hydrographic products and for compiling and displaying web-based information from multiple sources.

Other IM/IT needs

Dynamic data storage is essential for HPC operational data flow, but is of smaller magnitude than static data storage. CSAP will need static data storage and data management in the future. Modern Hydrography needs additional HPC capacity for 24/7 operationalization of the near-shore models. All initiatives need improved data management, connectivity, and network infrastructure to work more efficiently.

6.0 IM/IT context prior to OPP

Previous changes to the IM/IT environment had an affect on OPP.

Government-wide changes to IM/IT

• In 2012, Shared Services Canada (SSC) was created. As per the federal Shared Services Canada Act, this department became the mandatory provider of IM/IT infrastructure services to federal government departments.
• Part of all departments’ IM/IT resources has been transferred to SSC. A consolidated service delivery model was introduced. Efforts to standardize the services were initiated.
• Some DFO/CCG services relying on IM/IT infrastructure (such as ships’ operational services, national security and public safety operations) are not part of SSC’s mandate.

DFO/CCG changes to IM/IT

• Prior to 2012, DFO/CCG programs with IM/IT requirements had internal IM/IT experts with subject matter expertise to respond to their specific IM/IT needs.
• In 2012, DFO/CCG consolidated IM/IT specialists into two groups: Information Management and Technology Services (IM&TS) for DFO and Integrated Technical Services (ITS) for CCG.
• Since 2012, IM&TS has prepared and submitted program service requests to SSC for both DFO and CCG programs.

DFO/CCG investments into IM/IT

• The profile of IM/IT services at DFO is complex and includes support for over 300 applications across all sectors and regions.
• Prior to OPP, IM&TS had approximately 40 IM/IT projects per year as part of the regular business intake. At that time, the demand for IM/IT services exceeded the available capacity.
• In 2016, the department made a $2 million investment to stabilize the delivery of IM/IT applications based on 2016-17 forecasts, which did not include OPP.

7.0 Evaluation findings:

7.1 OPP planning for IM/IT requirements was limited

Finding: The magnitude and costs of the IM/IT requirements for OPP were underestimated and not well understood in the planning phase of OPP. IM/IT providers and regional expertise were not included during this phase of OPP. IM/IT costs, beyond onboarding for new employees, were not considered in most of the OPP planning documents.

IM/IT planning considerations

• Initially, OPP IM/IT requirements were not well understood. Regional representatives, who would be delivering much of the programing, and IM/IT specialists were not engaged in the early planning phase of OPP. A recent DFO Audit review (March 2019) also concluded that OPP IM/IT requirements and related costs were underestimated.
• OPP identified $1.2 million for IM/IT to support the onboarding process for new employees (e.g., new computers and laptops). Additional IM/IT costs were not included in early planning documents for OPP, except for an estimate of $2.9 million for high-performance computing (HPC) based on old funding models for internal HPC capacity.
• IM&TS received no additional funding for the delivery of IM/IT projects supporting over $1.2 billion in investments. However, during the first year of OPP, it became evident that IM/IT services were a critical resource for most OPP initiatives to deliver on their expected results.

7.2 There were challenges in providing IM/IT services during the first two years of OPP

Finding: The workload associated with OPP almost doubled the demand for IM&TS services, especially in areas requiring scientific expertise, during the first two years of OPP. Processes for acquiring SSC costs and timeline estimates were not clear. The program management of initiatives did not fully understand their IM/IT needs, including the time and costs required to deliver them. Furthermore, coordination between IM&TS, SSC, and the management of initiatives was limited.

IM&TS capacity to address the increased OPP workload

• IM&TS regular business intake from 2015 to 2017 was approximately 40 projects per year. In 2017-18, an additional 37 OPP-related IT projects were identified, which almost doubled IM&TS’s workload.
• OPP funding, which IM&TS received for the onboarding of new employees, supported the services from the categories shown in the left-hand column above. No funding was allocated for the services and resources across the remaining categories, which are expected to provide critical support to OPP activities.
• With the launch of OPP, the demand for IM/IT expertise and internal capacity increased in the areas of HPC, GIS, and cloud services.

IM/IT services through SSC during the first two years of OPP

• There was a lack of clarity and transparency in processes related to the submission of service requests and their prioritization at SSC.
• The costing process for SSC services was lengthy and unclear. It was not aligned with budgeting processes within the federal government. SSC expected confirmed funding at the time of the initial service request and did not provide itemized estimates, which restricted flexibility to adjust requirements to fit estimated costs.
• SSC’s high costs and delays in providing requested services had an impact on the ability of some areas to deliver on their activities during the first two years of OPP.

SSC could not provide the required HPC capacity during the first year of OPP to the Drift Prediction and Near-Shore Modelling initiative. As a result, the initiative’s modelling activities were delayed and $700,000 allocated to HPC in 2017-18 could not be spent. The implementation plan had to be revised to address the delays.

Limited coordination between IM&TS, SSC, and the initiatives

• Science program areas needed time to develop their research plans, identify the necessary IM/IT resources, and communicate their requests to service providers.
• Interviewees from the science initiatives indicated that, initially, both SSC and IM&TS did not understand their needs for IM/IT resources related to scientific activities. IM&TS representatives felt that program representatives did not recognize the efforts and underestimated the costs related to the IM/IT services and support they expected from IM&TS and SSC.
• Some interviewees appreciated the efforts of IM/IT providers to understand their needs, but felt that IM/IT challenges and needs go beyond OPP and are a department-wide issue.
• Within IM&TS, client portfolio managers (CPM) are assigned to specific program areas to assist them in obtaining IM/IT support based on their business requirements. In the first year of OPP, there was no dedicated CPM for OPP. As a result, OPP requests were handled individually by sector which did not promote coordination to establish efficiencies for OPP IM/IT requirements.

Since 2012, there have been few experts in DFO with a strong knowledge of both complex scientific matter and the specialized IM/IT tools and products required by the department. A knowledge exchange was necessary to bridge the communication gap, which required time, effort, and willingness on all sides.

7.3 Solutions were developed to achieve milestones and key outputs

Finding: Milestones and key outputs for the first two years of OPP were achieved by adjusting planned completion dates for some of the initiatives’ activities and by developing alternative IM/IT solutions. Where possible, IM&TS, SSC and, the program management implemented good practices to minimize delays. However, some alternative solutions have led to challenges and risks with data management.

The status of milestones for year 1 and 2 is based on self-reported performance information by the initiatives (as of February 2019). Interviewees indicated that the online Project Progress Reports (PPR) may not accurately capture initiatives’ progress, issues or challenges.

Modern hydrography

• A focus in the first two years of OPP was to collect data for 23 high-priority commercial ports. The surveys are on schedule for completion in 2020-21. Data is in the bathymetry database, cleaned, and ready for chart production.
• Dynamic hydrographic products and services are being integrated into existing ECCC infrastructure for 24/7 support, as planned.
• A large volume of LiDAR and multibeam data is currently being collected to update very old near-shore information. Insufficient data storage and HPC capacity had caused some delays until these resources were secured in December 2018.
• Some delays occurred obtaining ship time for surveys in the Arctic; completion dates have been adjusted.
• Resources were re-located from areas that were temporary delayed to other areas, so that overall work was not put on hold.
• Modern Hydrography has 71 reported milestones with 55% completed, 31% in progress, 14% not yet started and 0% requiring attention.

Drift prediction

• This initiative provides key support to many areas of OPP, which rely on complex modelling.
• Initially, two computational models were used: NEMO for drift prediction and FVCOM for e-navigation. In September 2018, a decision was made to use NEMO for both purposes, to allow for efficient operationalization of dynamic hydrographic products and 24/7 service delivery in partnership with ECCC. 
• Modelling activities were delayed in 2017-18 as HPC access was less than expected. Coordination between the regions and collaboration with SSC and ECCC allowed for some progress to be made. Program management explored opportunities to resolve the HPC issue. Funding was re-allocated in December 2018.
• As a result of these changes, research plans and timelines were revised and the initiative is now progressing well.
• Drift Prediction has 21 reported milestones with 15% completed, 52% in progress, 5% not yet started and 14% requiring attention.

Detection and avoidance

• In 2018, DFO hosted a meeting with Canadian and international experts to assess the effectiveness of existing and emerging real-time whale detection technologies within the Canadian context. The results of the meeting helped inform the development of the initiative’s research plan.
• Testing of several technologies is in progress:
• A whale tracking network based on hydrophones is used to locate SRKW in near real-time by listening to their unique vocal sounds.
• Hydrophones have been placed aboard DFO’s buoys to detect NARW.
• In partnership with academia, underwater acoustic gliders are used to monitor the presence of NARW and showcase the sightings data on an interactive map available to the general public.
• The implementation of planned research activities requires an acoustic data management system to house large amounts of data. In 2017-18, storage hardware was used in lieu of secured data storage environment. Future data storage and HPC needs remain a challenge that should be addressed.
• Detection and Avoidance has 13 reported milestones with 15% completed, 77% in progress, 5% not yet started and 8% requiring attention.

Collaborative situational awareness portal

• An engagement plan to seek feedback on CSAP from Indigenous and coastal communities has been developed and is currently being implemented. Where appropriate, engagement sessions have been integrated into the broader engagement approach of CCG and the federal government.
• For the engagement sessions in 2017-18, a CSAP prototype was developed on the existing CCG operations network (OPNET) because SSC could not respond to CSAP’s server requirements within expected timelines. This initiative’s milestones for the first year of OPP were met.
• As of February 2019, a version of CSAP has been demonstrated in engagement sessions with Indigenous communities across Canada. More sessions are in progress and will inform future releases of the CSAP web platform.
• In the first two years of OPP, CSAP did not have access to its full funding. A standard business analysis process was required to determine the parameters of the portal (expected to be finalized in the spring of 2019).
• Future CSAP needs for HR resources and specialized expertise are currently under review.
• CSAP has 8 reported milestones with 75% completed, 25% in progress, 0% not yet started and 0% requiring attention.

Good practices helped to minimize delays.

To support the achievement of milestones and key outputs, IM&TS, SSC, and the initiatives looked for solutions to address the IM/IT challenges they were experiencing.

Investing in a dedicated IM&TS resource

• In 2017-18, the Canadian Hydrographic Services reallocated funding for a dedicated client portfolio manager within IM&TS, which helped to coordinate and streamline the large volume of requests for their IM/IT needs.

IM&TS work to define IM/IT needs

• In 2018, IM&TS established an OPP Coordination and Engagement Manager to assist OPP initiatives in obtaining IM/IT support and to coordinate all OPP requests.
• This manager conducted site visits to the regions to gather information on the IM/IT needs of the OPP initiatives.
• As a result, IM&TS gained some understanding of the IM/IT requirements, which helped them to propose approaches to respond more efficiently to the demands. 

Strong governance within the initiatives

• Interviewees indicated that strong governance and frequent meetings to coordinate options and strategies between regions helped to minimize the impact of delays.
• For example, working together, scientists from the Maritimes were able to coordinate and conduct tests in the Pacific region where the season is longer.

Improved communication between program areas, IM&TS, and SSC

• As of March 2018, DFO started to provide funding to SSC for HPC project management.
• Frequent meetings with SSC, with the goal of mutually understanding each other’s work, have led to an improvement in providing HPC from the first to the second year of OPP.
• As a result, Modern Hydrography and Drift Prediction and Near-shore Modelling have signed an agreement with SSC to address the request for HPC and dynamic data storage (December 2018).

Good practices have increased the understanding of IM/IT requirements and timelines. Interviewees indicated that working together helps to coordinate options to minimize delays.

Mitigation efforts increased risks.

Alternative solutions for data storage and sharing

• Data acquisition has been a large focus in the first two years of OPP. The Department has limited data storage capacity. SSC estimates for data storage of $5,000 per terabyte (TB) for the first year and $1,000 /TB in subsequent years are not affordable for initiatives that are collecting, in some cases, more than 100 TB a year.
• As a result, some of the initiatives are using unsecured environments to store vast quantities of data. In addition, slow network and bandwidth capabilities within the department have led to mitigation solutions, including working from home to access quicker networks, and sharing data by courier or through university networks.
• These solutions are not sustainable and not without risks. Some of the risks cited by interviewees include:
• duplication of data as they are shared between unsecured environments and/or inadequately archived; and
• loss of data due to fire, flood, malfunction of data storage units or attrition of data holder.

Department and government-wide data management

• The DFO Internal Audit Review of Data Quality (2018) found that the Department has “no overall governance framework over data nor is there recognition of the importance of data as a key Departmental asset... the Department lacks an integrated approach to data management that recognizes the requirements for a coordinated approach to data needs identification, collection and usage as well as the benefits of information sharing across the sectors and regions”.
• The Report to the Clerk of the Privy Council: A Data Strategy Roadmap for the Federal Public Service (2018)^{Footnote 8}found that in the federal government “data are often collected in ways—based on informal principles and practices—that make it difficult to share with other departments or Canadians. Their use is inconsistent across the government and their value sub-optimized in the decision-making process and in day-to-day operations”.

Efficient data management practices are not always followed. In some cases, data are not effectively stored, accessed, or shared.

This may lead to challenges in providing evidence-based advice, delivering results, and being responsive and accountable to Canadians.

7.4 IM&TS is developing approaches to move OPP forward

Finding: IM&TS has recently proposed an approach to deliver all IM/IT needs for OPP with the objective of generating efficiencies within OPP, which could benefit the department as a whole. However, more departmental coordination is necessary to prioritize IM/IT elements that are essential for the achievement of OPP results, given the available financial resources.

IM&TS efforts to find a long-term solution

• IM&TS gathered information on the OPP IM/IT needs through consultations and site visits to the regions. Based on the analysis of outstanding needs and key IM/IT challenges, IM&TS has developed a coordinated approach for the delivery of IM/IT services for OPP.
• The coordinated approach could be an effective solution to IM/IT challenges. In particular:
• It groups IM/IT needs into bundles which could generate efficiencies by avoiding costly delays. Furthermore, linking the bundles to the OPP strategic results provides more clarity on potential risks and could inform priority decisions.
• Transitioning from on-premise data centres to cloud solutions for archiving data could facilitate access to scarce storage resources.
• IM&TS has presented this approach to DFO’s OPP senior management. There appears to be consensus that many of the identified IM/IT challenges, solutions, and potential benefits extend beyond OPP.
• The OPP budget is insufficient to fully support the implementation of the IM&TS proposed solutions. A more coordinated and holistic approach, involving all appropriate levels of governance at DFO/CCG, is needed to prioritize essential IM/IT requirements and make funding re-allocation decisions based on efficiency, feasibility, and the consideration of departmental and OPP objectives.

At the time of the evaluation, there was no decision on the feasibility of the approach proposed by IM&TS.

7.5 Department-wide efforts regarding data management and funding are in progress

Finding: OPP has highlighted existing department-wide IM/IT challenges and has brought attention to the need to coordinate efforts to find solutions across the organization. There are a number of actions in progress, which may address OPP challenges related to resource allocation processes and decision making, and to address data storage, data sharing, stewardship, and use. At the time of the evaluation, it was too early to accurately assess their effectiveness.

Actions led by the Strategic Policy

• In response to recommendations in DFO’s Internal Audit Review of Data Quality (2018) and the Report to the Clerk of the Privy Council: A Data Strategy Roadmap for the Federal Public Service (2018), a departmental data governance strategy is being developed.
• In September 2018, subject-matter experts from various DFO regions and sectors participated in a workshop which focused on assessing data governance capability. A broad range of data issues were identified in the following areas:
• Data ownership and stewardship;
• Data accessibility and availability;
• Data retention;
• Lack of a standard terminology; and
• Data quality.

• A number of recommended actions for improving data governance within the department are underway and are planned to be completed by September 2019 to meet federal government commitments.

Actions guided by the Chief Financial Officer

The Department is:

• Creating a team to provide guidance, advice, and coordination for the development of planning documents.
• Enhancing the costing capacity within DFO’s financial sector.
• Strengthening the finance governance:
• In January 2019, a new Financial and Investment Management Committee (Finance Committee) was formed and includes DFO and CCG senior executives. Its mandate is to promote sound stewardship of the Department’s financial resources and to support financial decision-making.

In response to a recommendation in DFO’s Internal Audit Review of OPP (2019), the OPP Implementation Team, with the support of the Chief Financial Officer, is developing:

• An OPP Project Cost Management Plan; and
• Standard Operating Procedures and Work Instructions for OPP financial monitoring.

At the time of the evaluation, it was not clear if potential improvements resulting from these efforts would be timely enough to address OPP-related challenges.

7.6 Beyond IM/IT, there were other findings related to the first two years of OPP

Finding: While the focus was on IM/IT, the evaluation highlighted other challenges and successes in the areas of reporting, gender-based analysis plus, and human resources.

8.0 Recommendations

1. Address data management risks

The initiatives examined as part of this evaluation showed that efficient data management practices are not always followed. In some cases, alternative solutions were used for data storage, access, and sharing. These solutions are not sustainable and they increase risks related to data security, stewardship, integrity, and use. Data are a key departmental asset and contribute to the delivery of OPP results. The ongoing department-wide efforts to improve data management should consider OPP needs as a high priority.

Recommendation: The Assistant Deputy Minister, Human Resources and Corporate Services, the Deputy Commissioner of Operations and the Senior Assistant Deputy Minister, Strategic Policy, should find short-term solutions to address immediate risks related to OPP data management to ensure that data are effectively stored, accessed, and shared within and among DFO/CCG sectors, regions, and with Canadians, in addition to and consistent with ongoing department-wide efforts to improve data management, where appropriate.

2. Review critical steps with a focus on results

The initiatives examined as part of this evaluation were self-reported as having achieved their milestones and key outputs during the first two years of OPP. Planned completion dates for some of the initiatives were adjusted and alternative IM/IT solutions were developed to achieve milestones. A number of good practices helped to minimize delays within individual initiatives; however, coordination with other initiatives could be improved, particularly with regards to priority setting, collaborating to achieve shared goals, and addressing common IM/IT challenges. As OPP moves into its third year, there is a need to reassess outstanding milestones, with a focus on results.

Recommendation: The Deputy Commissioner of Operations, with support from the Chief Financial Officer, should coordinate with Assistant Deputy Ministers, where relevant, to review and refine the outstanding critical steps that are necessary to achieve OPP’s expected results, taking into consideration existing resources, timelines, and the alignment of initiatives’ milestones and key outputs.

3. Improve planning processes

The initiatives examined as part of this evaluation are data-centric and rely on IM/IT resources for their activities. However, during the planning phase of OPP, the providers of IM/IT and the regional representatives who would be delivering much of the programming were not involved in the consultations. As a result, the IM/IT requirements were not well understood and costs for their inclusion in funding documents were not included or were underestimated. 

Recommendation: The Senior Assistant Deputy Minister, Strategic Policy, with support from the Chief Financial Officer and the Assistant Deputy Minister, Human Resources and Corporate Services, should improve the scoping, costing, and planning processes to encompass support for new programs, initiatives, and plans. This should include consultations with subject matter experts.

9.0 Appendices

Appendix A: Evaluation methodology

Lines of evidence

Case studies

• Conducted four case studies of IM/IT intensive initiatives to explore extent and implications of IM/IT challenges.
• Included:
• Modern Hydrography and Charting in Key Areas;
• Reducing the Threat of Vessel Traffic on Whales and Other Marine Mammals through Detection and Avoidance;
• Collaborative Situational Awareness Portal; and
• Improving Drift Prediction and Near-shore Modelling.
• Used triangulation of case study findings to make broader generalizations to OPP, where possible.

Interviews

• Conducted interviews with 84 individuals in the regions and National Headquarters, to discuss IM/IT related challenges. Interviewees included staff and stakeholders directly related to the case studies, as well as IM/IT stakeholders from within the department and Shared Services Canada.

Data analysis

• Gathered and examined data collected by the OPP Secretariat to assess the extent to which the initiatives had delivered on their commitments.

Document review

• Reviewed relevant documents related to OPP to understand the initiatives examined as part of the evaluation.

Site visits

• Four site visits were conducted (Dartmouth, Sidney-Victoria, Nanaimo, and Vancouver), to facilitate data collection. As part of each visit, interviews were conducted with key staff associated with the four case studies.

Appendix B: Limitations and mitigation strategies

Although the evaluation encountered some challenges, methodological limitations were mitigated, where possible, through the use of multiple lines of evidence and the triangulation of data. This approach was taken in order to establish the reliability and validity of the findings and to ensure that conclusions and recommendations were based on objective and documented evidence. Details on limitations and their associated mitigation strategies are described below.

Case studies

• A case study offers a means of investigating a single phenomenon and asking the question “what happened?” However, a single case study is limited in generalizing to the larger context. To mitigate this, the evaluation team conducted many scoping interviews with key stakeholders to identify four case studies that were identified as sharing similar IM/IT challenges.

Interviews and document review

• As the initiatives under investigation were in the planning and early implementation stages, changes and developments were frequent occurrences. To mitigate this, the evaluation team conducted follow-up interviews with key informants and requested revised plans to keep up-to-date on the progress of implementing each initiative.

Data analysis

• Analysis of performance data was based on self-reported information entered by initiatives in a shared performance tracking portal. Initiatives were provided with little guidance on how to enter information into the portal. As a result, there were differences in the level of detail of the data on milestones across initiatives’ reports. To mitigate this, the evaluation team did not attempt a comparative analysis and used information from interviews to validate the findings on the overall progress of the initiatives.

Appendix C: Gender-based analysis plus

As per the Treasury Board Directive on Results (2016), the evaluation aimed to include a gender-based analysis plus and incorporated indicators into the evaluation questions, where relevant.

What is gender-based analysis plus?

Gender-based analysis plus (GBA+) is an analytical tool, process, or product used to assess the potential impacts of policies, programs, services, and other initiatives on diverse groups of women and men, taking into account gender and other identity factors. The "plus" in the name highlights that gender-based analysis goes beyond gender, and includes the examination of a range of other intersecting identity factors (such as age, education, language, geography, culture, and income). 

How did the evaluation aim to conduct a gender-based analysis plus?

• Identify how women and under-represented populations have accessed and participated in these initiatives.
• Examine mitigating strategies to eliminate barriers, if any, to women and under-represented populations from accessing and participating in these initiatives.
• Identify factors affecting the availability of and access to self-identification data on GBA+ components.

Appendix D: Management action plan

Recommendation 1

The Assistant Deputy Minister, Human Resources and Corporate Services, the Deputy Commissioner, Operations and the Senior Assistant Deputy Minister, Strategic Policy, should find short-term solutions to address immediate risks related to OPP data management to ensure that data are effectively stored, accessed, and shared within and among DFO/CCG sectors, regions, and with Canadians, in addition to and consistent with ongoing department-wide efforts to improve data management, where appropriate.

Rationale

The initiatives examined as part of this evaluation showed that efficient data management practices are not always followed. In some cases, alternative solutions were used for data storage, access, and sharing. These solutions are not sustainable and they increase risks related to data security, stewardship, integrity, and use. Data are a key departmental asset and contribute to the delivery of OPP results. The ongoing department-wide efforts to improve data management should consider OPP needs as a high priority.

Response

IM&TS has been working with the OPP Implementation Team, sectors, and regions to develop a coordinated and holistic approach to prioritize essential IM/IT requirements and make funding re-allocation proposals based on efficiency, feasibility, and the consideration of departmental and OPP objectives. This will support the development and implementation of short-term solutions to address immediate risks. Program data management for OPP projects should be consistent with ongoing department-wide efforts and be governed and controlled under the DFO-CCG Data Governance Framework. The Data Governance Framework, already being implemented by Strategic Policy, is one component of the Data Strategy, some components of which are still under development. Roles and responsibilities for data stewardship, including data owners, data stewards and data custodians will be established to ensure good data management and control at the business level.

Link to larger program or departmental results (if applicable)

The results statement and milestones listed below are linked to Departmental Result 2.2: Scientific information on Canada’s oceans and other aquatic ecosystems is available to inform management decisions (2019-2020 Departmental Results Framework).

Recommendation 2

The Deputy Commissioner of Operations, with support from the Chief Financial Officer and Assistant Deputy Minister, Human Resources and Corporate Services, should coordinate with Assistant Deputy Ministers, where relevant, to review and refine the outstanding critical steps that are necessary to achieve OPP’s expected results, taking into consideration existing resources, timelines, and the alignment of initiatives’ milestones and key outputs.

Rationale

The initiatives examined as part of this evaluation were self-reported as having achieved their milestones and key outputs during the first two years of OPP. Planned completion dates for some of the initiatives were adjusted and alternative IM/IT solutions were developed to achieve milestones. A number of good practices helped to minimize delays within individual initiatives; however, coordination with other initiatives could be improved, particularly with regards to priority setting, collaborating to achieve shared goals, and addressing common IM/IT challenges. As OPP moves into its third year, there is a need to reassess outstanding milestones, with a focus on results.

Response

The OPP Implementation Team and CFO Sector launched an OPP multi-year spending plan review in November 2018. The objective was to assess whether existing and future funding levels were adequate to deliver on the key outputs and to ensure that spending plans were aligned with updated implementation plans. The benefits of the exercise were: reduction of significant in-year pressures and/or surpluses; provision of timely and accurate annual budget allocations; and, information for input to the 2020-21 Annual Reference Level Update and other budget adjustments. The review findings were presented to the 21 February 2019 meeting of OPPOC.

In addition, as described in response to Recommendation #1 above, IM&TS has been working with the OPP Implementation Team, sectors, and regions to review critical steps with respect to OPP IM/IT needs, with a focus on results. These collaborative efforts have resulted in the refinement of the outstanding critical steps necessary to achieve OPP’s expected results. The launch of a second multi-year spending plan, and the provision of a future revised mitigation approach to OPP IM/IT needs, based on an updated assessment will add to the refinement of the critical steps necessary to achieve OPP’s expected results.

Link to larger program or departmental results (if applicable)

The results statement and milestones listed below are linked to Departmental Result 2.2: Scientific information on Canada’s oceans and other aquatic ecosystems is available to inform management decisions (2019-2020 Departmental Results Framework).

Recommendation 3

The Senior Assistant Deputy Minister, Strategic Policy, with support from the Chief Financial Officer and the Assistant Deputy Minister, Human Resources and Corporate Services, should review the scoping, costing, and planning processes to encompass support for new programs, initiatives, and plans. This should include consultations with subject matter experts.

Rationale

The initiatives examined as part of this evaluation are data-centric and rely on IM/IT resources for their activities. However, during the planning phase of OPP, the providers of IM/IT and the regional representatives who would be delivering much of the programming were not involved in the consultations. As a result, the IM/IT requirements were not well understood and costs for their inclusion in funding documents were not included or were underestimated. 

Response

Through the development of Memoranda to Cabinet (MC), Strategic Policy provides advice to sectors to ensure that accurate and coherent policy proposals are brought forward for the Minister’s consideration and approval, and CFO provide attestation that the costings of the proposals are accurately reflected. The CFO sector is currently developing a proposal for a revised enabler formula which will include changes to the calculation of funding for enabler services when costing new programs, initiatives, and plans as well as a proposed checklist for programs that details the various entry points into the costing process. The revised formula and checklist are intended to lead to improved results from costing processes that better encompass support for new programs, initiatives, and plans. 

The development of a lessons-learned document with respect to OPP planning, and bundling strategy, and the review of the MC development process and the Treasury Board (TB) submission process, with the subsequent development of improvements to these processes, if required, based on the findings of the review, are expected to result in the department’s scoping, costing, and planning processes better encompassing support for new programs, initiatives, and plans. These efforts will include consultation with subject matter experts.

Link to larger program or departmental results (if applicable)

The results statement and milestones listed below are linked to Departmental Result 2.2: Scientific information on Canada’s oceans and other aquatic ecosystems is available to inform management decisions (2019-2020 Departmental Results Framework).