Survey Methodology and Data Quality
This section aims to illustrate the procedures used in the 2005 survey as well as the strengths and limitations of the data in order that they could be effectively used and analysed. Information on the methodology used and the data quality of survey results would of particular importance when making comparisons with data from other surveys and sources of information or when drawing conclusions regarding changes over time.
6.1 Sampling
Two types of survey questionnaires were mailed: one covering residents and another covering non-residents. Footnote 7 Resident and non-resident samples were selected using stratified, systematic random sampling of licence databases with stratification based on licence category. Resident samples were further stratified by area of residence.The sample sizes were determined from the reliability estimates for days fished for each stratum as calculated from the results of the 2000 Survey of Recreational Fishing in Canada.
6.2 Data Collection, Processing, Coding and Estimation
The survey was launched in January 2006 through a coordinated press release led by DFO Headquarters (Ottawa). The province of Ontario was responsible for survey mailout and coding of survey responses.
Upon receipt of the questionnaire from the respondent, the province established geographic codes for the survey in order to allow for sub-provincial/territorial data analysis. This was fielded by a trained contractor. The questionnaires were then given a thorough inspection to confirm the data contained within was acceptable. Most times all that was required was to ensure completeness of the questionnaire; verification that the information provided by anglers was both reasonable and sound for the region involved. This included assessing species caught by area as well as the availability of the specific species, determining whether or not the catch levels indicated were within acceptable limits, cross-checking administrative lists to eliminate substitutions (someone responding other than the designated respondent), etc. Once questionnaires had been completely assessed and coded, they were shipped to Ottawa for analysis.DFO Headquarters coordinated the questionnaire imaging, data capture and validation of survey responses in cooperation with the provincial coordinator. All documents were verified prior to data capture, and those with significant levels of incomplete coding, missing information, etc. were set aside. All data collection, coding, and data capture were completed in Fall 2006.
The required programming to produce survey estimates consisted of:
- systematic editing procedures;
- substitution algorithms for missing data;
- data output programs covering all data collected; and
- special output programs for publication purposes.
All estimates of effort and numbers are for adults (18 years of age and over). Averages for each lake are based on the numbers of participants determined for each, while those for the system as a whole are based on the total number of participants on any of the lakes in the system. Information in tables referring to totals for Ontario is based on the activities of Great Lakes anglers only. This information is presented in order to reflect the relative degree of importance associated with the Great Lakes when compared to all fishing activities of Great Lakes anglers in Ontario.
Although expenditures were collected on a household basis, allocation for the Great Lakes fishery, with the exception of package deals, was based on the fishing activity of individuals. If all days were spent on the Great Lakes system, all expenditures were assigned to the system. Expenditures for each water body within the system were allocated on the basis of the proportion of time spent on each water body. If the days spent on the Great Lakes system represented only a subset of all days in Ontario, the portion of the expenditures to be assigned to the system was estimated using the proportion of activity on the system. Estimates by water body within the system were then allocated on the basis of the amount of activity on each as a percentage of all days on the Great Lakes system. Expenses incurred for package deals have been allocated on the basis of only those packages purchased specifically on each of the Great Lakes.
6.3 Weighting Procedures
Weighted estimates were produced to ensure that the raw survey data reflect the estimates for the population. This procedure was a simple application of inverse weighting by stratum as licence sales were known (population of licence-holders divided by resultant sample). Where anglers held different types of licence, either by choice or due to regulations, adjustments to weighting were made to ensure that there was no double counting of anglers across licence types.
The estimates of population in each stratum of interest were then compared to official statistics on population available from Statistics Canada. Population data was based on the 2001 census with post-censal estimates as at July 1, 2005. The initial estimates of population and the respective estimates of anglers in each stratum were adjusted to reflect these statistics. Respondent weights for survey estimates were then derived using standard inverse weighting functions.
6.4 Post-Survey Adjustment Procedures
Preliminary survey results were distributed to the province in early 2007, providing them with the opportunity to review the preliminary estimates.
Adjustments were made provided that they were based on identifiable errors or changes required due to revised administrative information. Most errors were as a result of incorrect identification of species and the changes were relatively straightforward.
Revisions were completed by Spring 2007 and survey results were finalized.
6.5 Data Quality
6.5.1 Sampling Error
Information on reliability is presented in terms of the coefficient of variation of the mean, at one standard deviation. Because exact estimates of variance cannot be calculated, the coefficient of variation of the mean is used as a proxy for the assessing the variability of the data. Annex A.18 presents the statistical reliability of key variables for all anglers.
6.5.2 Non-Sampling Error
Many factors that are not related to sampling also affect the reliability of the data produced in the survey. For example, respondents may have made errors in interpreting questions, the answers may have been incorrectly entered on the questionnaires or errors may have been introduced during the data capture or tabulation process.During the data collection phase, efforts were made to reduce the occurrence of non-sampling errors in the survey.Footnote 8 These efforts included a complete verification of the reported data, validity and consistency edits and ongoing consultations with the coordinators in each jurisdiction.
6.5.3 Data Limitations
Despite all efforts to improve data accuracy, the survey results have limitations. Understanding these limitations will help the reader make informed decisions before conducting further research and analysis using the estimates in this report. For instance, the survey did not cover nor did it attempt to measure the illegal, unreported activities related to recreational fisheries in Ontario. Although the days fished and harvests of individuals who conducted recreational fishing activities without a licence where/when a licence or permit is required is not thought to be substantial, it is possible an impact would be observed if there was a way to measure such an activity.
6.6 Data comparability
The 2005 results are directly comparable with the 1995 and 2000 surveys. The core survey questions remained unchanged throughout these survey cycles. However, comparisons with reports prior to 1995 should not be attempted.
- Date modified: