Use of Crowdsourcing Platforms for Recruitment in Research
Guidance to support the implementation of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS)
Table of Contents
Part I: Context
Part II: Guidance
- Fairness and equity in research participation (TCPS core principle: Justice)
- Privacy and confidentiality (TCPS core principle: Concern for Welfare)
- Consent (TCPS core principle: Respect for Persons)
- Ethical considerations and questions related to the use of crowdsourcing platforms for recruitment in research
Part III: Practical Application
Part I: Context
1. Introduction
There is no agreement on a definition of crowdsourcing in the literature, but for the purpose of this document, crowdsourcing is the use of online services to recruit a large number of individuals (i.e., a crowd) to participate in research. Although crowdsourcing can be done in person, it is typically offered online, which creates greater flexibility and the ability to reach a larger number of individuals globally. Crowdsourcing has become a popular mechanism for recruitment in research as it offers different, convenient, and cost-effective approaches to select and engage research participants in a timely manner. Crowdsourcing platforms can be used for participant recruitment in any research discipline. However, the use of crowdsourcing as a recruitment strategy has limitations, challenges, and ethics issues that researchers and research ethics boards (REBs) should be aware of and consider in the conduct and review of research, as outlined throughout this guidance.
In general, crowdsourcing platforms are created and maintained by external entities or researchers. Some academic units/organizations also operate crowdsourcing platforms specifically geared toward research for academia. Crowdsourcing may include the use of external data collection services such as Amazon Mechanical Turk and SurveyMonkey Audience. Some crowdsourcing platforms were initially established for, and are still used for, the commercial purposes of the company that owns them. However, these platforms are increasingly used by academic researchers for non-commercial purposes as well. Some researchers may also develop home-grown crowdsourcing platforms, mainly for research and surveillance purposes, such as Flu Near You.
Typically, there are minimum prerequisites to register on crowdsourcing platforms. Registrants to such platforms are generally considered workers completing tasks but may also be considered participants taking part in research. Usually, they are offered incentives to participate in research. Incentives can be financial but can also be in the form of course credits for students, acknowledgement for participation or experience of participation.
2. Scope
This guidance addresses the use of crowdsourcing platforms offered online for the purpose of recruiting participants for research.
The guidance applies irrespective of where the crowdsourcing platform is hosted. The platform may not be subject to the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS), but the research itself may require REB review. Research initiated or conducted at or by members of a Canadian institution that is subject to the conditions of the TCPS and that plans to recruit human participants using crowdsourcing platforms must adhere to the TCPS. This research requires REB review unless it meets one of the exemptions from REB review outlined in the Policy (see definition of human participants in the Application of Article 2.1 and exemptions from REB review in Articles 2.2, 2.3 and 2.4). It should be noted however, that privately held crowdsourcing platforms unaffiliated with institutions eligible to receive and administer Agency funds are not subject to the TCPS.
While this guidance applies to research using crowdsourcing platforms for the recruitment of participants, it can also help inform other out-of-scope recruitment strategies such as the case of survey panels where researchers may target participants directly; thus, typically knowing their identity and establishing a longer-term relationship resulting from their ongoing involvement in the research. This guidance can also inform recruitment strategies using online platforms in other contexts such as the case of platforms developed by many Canadian institutions to recruit post-secondary students (mostly in first year psychology courses) to participate in research and receive bonus credits in academic courses (Consent #3 and 4).
Part II: Guidance
The TCPS does not discuss the merits of different strategies for participant recruitment because they are context specific. In the absence of specific guidance, the use and review of crowdsourcing platforms as a participant recruitment strategy in research should be guided by the core principles of the Policy: Justice, Concern for Welfare, and Respect for Persons (Article 1.1).
The following ethics considerations are intended as an educational resource to help guide REBs and researchers in the design, review and conduct of research using crowdsourcing platforms for participant recruitment. These include fairness and equity in research participation; privacy and confidentiality; and consent.
Recognizing that research using crowdsourcing platforms for recruitment has its limitations and challenges, it is still subject to the core principles and requirements outlined in the TCPS, including those related to the consent process, as well as privacy and confidentiality. This guidance is intended to be complementary to the Policy and does not preclude other obligations that researchers and REBs have in the conduct of research and its review.
3. Fairness and equity in research participation (TCPS core principle: Justice)
The following is an excerpt from a relevant TCPS interpretation available on the Panel on Research Ethics (PRE) website (Fairness and Equity #4). It addresses fairness and equity when participants are recruited for research purposes using crowdsourcing platforms:
“What should REBs consider when reviewing research that involves the use of crowdsourcing to recruit participants?
Following the principle of Justice, researchers and research ethics boards (REBs) should be concerned with the fair and equitable inclusion and exclusion criteria of using crowdsourcing to recruit participants in a research project. The research question should guide the recruitment process and the tools used to recruit groups/individuals targeted by the research. Researchers should satisfy their REB that using a specific participant pool is germane to answering the research question (Article 4.1). For example, if the research targets a specific socio-economic group, and the crowdsourcing pool is known for such socio-economic circumstances, this would justify the use of this recruitment tool. […]
Researchers should provide relevant information on their proposed crowdsourcing recruitment strategy to their REBs to consider in the review of the ethical acceptability of their research.”
This should be based on the researchers’ basic working knowledge of the crowdsourcing platform they select for recruitment in research, including its privacy and safety measures, as well as its incentive structure as outlined further in this guidance (see sections 4 and 6b).
4. Privacy and confidentiality (TCPS core principle: Concern for Welfare)
4a. Identifiability of the information
The TCPS defines information as non-identifiable “if it does not identify an individual, for all practical purposes, when used alone or combined with other available information … The assessment of whether information is identifiable is made in the context of a specific research project” (Chapter 5, Section A).
Researchers may be able to track registrants, and therefore the information is considered identifiable from the researchers’ perspectives. Researchers should follow disciplinary standards in the collection and protection of information, and should put in place necessary privacy and confidentiality safeguards (both physical and technological – encryption, access controls, workplace security, the use of passwords and multi-factor authentication) to protect both the registrants’ personal information and research data generated in the process of the research (Article 5.3). This is especially relevant when researchers create and maintain their own platforms.
In both cases where researchers use external crowdsourcing platforms to collect information that include registrants’ identifiers, or where the crowdsourcing platform policies and terms of use permit communication between the participants and the researchers, the researchers should put in place safeguards for protecting the identifiability of participants and their information. Researchers should take measures to de-identify the data as soon as possible using techniques and approaches appropriate to the researchers’ discipline and consistent with their institution's policies and procedures, as applicable (Article 5.3).
Where crowdsourcing platforms are managed by custodians independent of the researchers, and the researchers are not collecting identifiers, the information collected in response to research requests or tasks would be considered non-identifiable (coded or anonymized) from the researchers’ perspective. This decreases ethical concerns regarding participants’ privacy (Chapter 5, Section A). However, researchers and REBs should consider the likelihood of re-identification and mitigate it, especially where the research involves data linkage either within the collected data set or across multiple sources (Article 5.7).
4b. Data storage
Researchers should have basic working knowledge of the crowdsourcing platform they are planning to use for recruitment in their research, including for example whether the platform has existing privacy and data security standards and how incentives, where applicable, are managed. Where researchers create their own crowdsourcing platform, the creation and maintenance of the platform requires security measures to ensure safe storage of data, and appropriate training in proper safeguarding of information at all times (Article 5.3). Safeguarding the information also includes the researchers’ institutional responsibility to put in place security measures to provide necessary protections and mitigate privacy risks (Article 5.4).
Normally, collected data remains stored on crowdsourcing platforms– unless otherwise stipulated in the platform's policies – and may be stored on multiple servers. During the consent process, researchers should make prospective participants aware that their collected information could be accessed during its storage and advise them to verify the measures that the platform provider has put in place for safeguarding their information. Highlighting this information at the time of data collection becomes even more important when researchers are seeking to obtain sensitive information. Information about potential secondary use of data should also be described when relevant.
In addition to the TCPS, researchers and their institutions may be subject to other research ethics norms, laws, regulations, and policies, including but not limited to those concerning the protection of participants’ privacy and confidentiality (Chapter 1, Section C, Research Ethics and Law).
5. Consent (TCPS core principle: Respect for Persons)
Respect for Persons translates in part into providing participants with available information necessary to make an informed decision about voluntarily participating in research and about their right to withdraw at any time throughout the research. As part of the consent process, researchers should include an explanation of the responsibilities of participants, the researchers’ plan for managing participation and incentives - where applicable, and should clearly stipulate that they can withdraw from the research and stop participating at any time and for any reason (Articles 3.1 and 3.2). Where it would be impossible/impracticable to withdraw their data once collected, this information must be clearly communicated to prospective participants at the time of seeking their consent.
5a. Managing incentives
TCPS does not require researchers to provide incentives, and neither encourages nor discourages their use (Application of Article 3.1 and Consent #6). Where offered, incentives are an important consideration in assessing voluntary participation in research. A form of undue influence that would negate voluntariness of consent to participate is researchers’ refusal to provide the incentives promised to participants as part of the consent process, for example due to incomplete participation. The TCPS stipulates that “[t]he participant should not suffer any disadvantage or reprisal for withdrawing nor should any payment due prior to the point of withdrawal be withheld” (Application of Article 3.1).
Researchers should be clear in their expectations of participants’ contributions, including what incentive they will receive on the completion of what tasks (where applicable). Where researchers use incentives to encourage participation in their research, they must provide those incentives to participants based on what they communicated to participants during the consent process - lump sum or based on a schedule proportionate to the extent of participation. For example, in the case of offering incentives for a large task, researchers may consider breaking the task into shorter sessions and issuing incentives based on the completed sessions.
Incentives provided to participants may vary based on the platform used. Some platforms offer standard rates, and in some cases, researchers may not always have knowledge of the exact level of incentives the crowdsourcing platform offers to participants. The issue here is whether the level of incentives offered to encourage prospective participants to take part in specific research is reasonable, especially if there is any risk resulting from their participation in the research. In making this assessment, researchers and REBs should consider prospective participants’ circumstances to the extent possible, or the global nature of the participant pool.
Some platforms provide automatic incentives to participants, and researchers may be unaware of the level of incentives provided to those who took part in their research. Researchers should inform their REBs that they lack this information. Not knowing the exact level of individual incentives offered to participants creates challenges for researchers and REBs, and they need to consider it in making a reasoned assessment of the level of incentives provided to participants. For participants, the consent form should indicate that individual incentive amounts may not be fully known to the researcher, and will potentially vary by task, region, and level of participation in the research.
5b. Impact of unjustifiable exclusion of prospective participants
Individuals wishing to be added to a crowdsourcing platform, particularly a commercial crowdsourcing platform, are typically eligible if they meet initial enrollment criteria and subsequent minimum approval ratings which they need to maintain as part of the crowdsourcing platforms’ continuous evaluation of their scores. Researchers should not penalize prospective participants by:
- unjustifiably excluding them as prospective participants for reasons unrelated to the research; or
- disqualifying them from taking part in the research based on conditions that have not been communicated to the prospective participants.
Excluding prospective participants from the research for any of the above reasons can affect their approval ratings (where applicable), and this in turn affects their eligibility to participate in future research studies. This would therefore constitute a form of coercion (Application of Article 3.1).
As with any recruitment and consent process, researchers should make clear the eligibility criteria for inclusion in the research, and any conditions under which participants can be excluded, or disqualified from continuing to participate in the research, as well as their related consequences (Article 4.1).
5c. Perspectives on voluntary participation in research
There are different perspectives on the voluntary nature of participants’ consent within the context of their recruitment through crowdsourcing platforms.
One perspective is that participation through crowdsourcing platforms does not seem to increase the risk of undue influence or feeling pressured to participate in the research given that participants willingly agree to take part in research.
Another perspective is that this pool of participants may rely on their participation in research to earn a living, where financial incentives are offered, even though they are generally not paid well. This affects the voluntary nature of their participation in research by further encouraging their reliance on the research incentives for their livelihood. It is worth noting that, while the latter perspective mainly pertains to the offer of financial incentives to encourage participation in research, incentives can also take other forms such as acknowledgements of participation or the experience of participation.
6. Ethical considerations and questions related to the use of crowdsourcing platforms for recruitment in research
6a. Researchers' strategies to address challenges with participation that impact the integrity of their research
Researchers should anticipate, and incorporate in their research design, strategies to address challenges with participation and their effect on the integrity of their research. This should be balanced with concern for participants’ welfare. The following are examples of strategies to address challenges with participation:
-
Managing incomplete or nil responses
In the design of their survey tool, researchers should consider how to manage incomplete or nil responses and inform participants during the consent process of how such responses impact their offer of incentives. For example, offering incentives proportionate to their participation. If measures have not been incorporated into the design of the research or communicated during the consent process to clarify the responsibilities of participants and the conditions for receiving the incentives, then the researchers must provide participants the promised incentives. Where a crowdsourcing platform does not include incentives, the researcher should have a plan for managing incomplete responses. This may include exclusion of incomplete responses entirely using statistical methods of accounting for missing data. In either case, participants should be informed under what circumstances their data may not be utilized.
-
Managing inconsistent responses
Where crowdsourcing platforms allow it, the researchers can add validity check questions as part of their research design – for example, to detect inconsistencies in responses – that impact the reliability and validity of the research. Researchers should inform participants that the research design includes measures to detect inconsistencies, that major inconsistencies may lead to disqualifying them as participants in the specific research, and that there are consequences to disqualification.
-
Managing multiple attempts to participate
To manage challenges with participation via crowdsourcing platforms, the researchers may set clear parameters that limit it to single participation in the specific research; therefore, disqualifying multiple attempts to participate and multiple receipts of incentives. Researchers can inform participants that multiple attempts at participation will result in disqualification.
-
Managing improbably fast responses
With the evolution in artificial intelligence, there is an emergent threat that bots may be used to respond to research tasks, including in the context of research using crowdsourcing platforms for recruitment. This could undermine the integrity of the research and the reliability of the data collected. While various detection measures of fraudulent responses may be embedded within some platforms, researchers should also incorporate their own strategies to mitigate the risks of responses that are, for example, submitted too rapidly. Researchers may also consider adding questions that only humans can answer to assist in the detection of bot responses.
As part of the consent process, researchers should generally inform participants that validation checks are in place to detect suspicious and fraudulent responses, and to mitigate the risk of bot generated responses. To manage the quality and integrity of the collected data, as well as address potential fraudulent responses, researchers should also consider encouraging potential participants to take their time and complete tasks and responses to questions carefully. Researchers may inform participants that answering faster than a minimal response time threshold will result in disqualification.
In general, when researchers question the reliability of the data gathered through crowdsourcing platforms that could impact the validity of their research, they have the option to exclude that data.
6b. Selecting crowdsourcing platforms
The onus is on the researchers to provide a rationale for selecting the specific crowdsourcing platform as opposed to other recruitment tools and strategies in their application for REB review. Researchers should consider the following in the selection of a crowdsourcing platform(s):
-
Limitations on the use of crowdsourcing platforms
Some institutions put limitations on the use of crowdsourcing platforms for recruitment in research unless they meet basic criteria, for example limiting it to crowdsourcing platforms with privacy and security standards or limited to their use in recruitment in minimal risk research. When deciding which crowdsourcing platform to use, researchers should consider their institutional policies and requirements, and whether those are consistent with the proposed crowdsourcing platforms’ terms of service and other legal requirements, where applicable.
-
Appropriateness of the crowdsourcing platform for recruitment in the specific research
Some but not all types of research or research designs can benefit from recruiting participants via crowdsourcing platforms. This can be determined by factors such as the nature of the research, its design and methodology, the level of risk, the timelines for completing the research tasks, and the selection and appropriateness of the target group. Depending on the prospective participants to be recruited for the specific research, some crowdsourcing platforms may not offer the required representativeness which can result in a sample that is skewed towards a certain demographic group.
Some crowdsourcing platform policies do not permit direct communication with participants or cannot guarantee that the same participants will continue to participate in research that is conducted in stages. Recruiting participants via certain crowdsourcing platforms may therefore not be a good recruitment tool for some research designs that require follow-up with the same participants.
Researchers should consider the appropriateness of their proposed recruitment strategy through crowdsourcing platforms at the planning and design stage of their research (Application of Article 3.1).
-
Knowledge of the crowdsourcing platform
Researchers should have a basic working knowledge of relevant details about the platform to be able to answer basic questions from prospective participants and their REB. Researchers may wish to inform themselves of the crowdsourcing platform’s terms of use and policies, including the process the platform uses for its selection and engagement in research, the screening tools available for targeting participants, whether and how the platform verifies the participants’ identities and their personal data, and the management of incentives for participation in research.
These factors ultimately impact the reliability and validity of the research and will help inform the decision on whether the selected platform is appropriate in the context of their research.
6c. Considerations and questions during the REB review
The following is a list of questions that REBs may consider during the ethics review of research that intends to use a crowdsourcing platform as part of their recruitment strategy:
- Does the researcher demonstrate basic working knowledge of the crowdsourcing platform selected for recruitment in their research? REBs may consider requesting confirmation from the researcher that the selected platform(s) have known policies for data security, and that those policies do not conflict with their institutional policies.
- Does the research clearly describe that participation is for “research purposes” and not for “paid work”?
- How will the researcher manage the inclusion/exclusion criteria of participants for the specific research? Does the researcher describe how prospective participants will be identified as eligible to participate in the research? For example, is there a pre-screening test or clear criteria?
- Where generalization is the goal, are the platform participants sufficiently representative to allow for generalization of the findings to the population of interest?
- Are there conditions under which the participants will be disqualified or unjustifiably excluded from continuing to participate in the research? Are there consequences to disqualification? Are these conditions and consequences clearly outlined for prospective participants in the consent form? For example, if they fail attention checks within a survey tool, or do not complete a minimum expectation of the research tasks.
- Does the research include an incentive plan? Are incentives prorated based on participation? Will the incentive plan be communicated to prospective participants during the consent process?
- Does the researcher know how the platform will manage the incentives relative to participation? For example, some platforms may have standard rates.
- Has the researcher considered strategies to address challenges with participation in research that have the potential to undermine the validity of the research findings? This includes the emergent threat of using artificial intelligence to replace human responses, and the need to employ appropriate strategies to prevent fraudulent responses, answers provided faster than the minimum time threshold, and multiple attempts at participation by the same individual in the research.
- Does the researcher have a plan to ensure that participants receive the promised incentive via the crowdsourcing platform administration, including conditions under which participants will not receive the incentives?
- What is the researcher’s plan to inform participants of how to access the results of the research (Article 4.8)? For example, the researcher may consider reposting the results of the research on the same platform or provide a link to participants of where the research results will be posted.
Part III: Practical Application
7. Scenarios
Scenario 1: Scope of REB review
Context
A researcher uses a crowdsourcing platform to recruit individuals who previously participated in other research on vaccines through the same platform. The current research asks participants whether their experiences taking part in the previous research impacted their views and trust in vaccines, and whether it influenced their willingness to get vaccinated.
Does the research require REB review?
The focus of this research is on the personal experiences of those who took part in the previous research on vaccines. As participants’ responses are relevant to answering the research question, they meet the definition of “human participants” as per the TCPS (Application of Article 2.1). This research falls within the scope of the TCPS and is subject to REB review, unless it meets exemptions outlined in the Policy (Articles 2.2, 2.3 and 2.4). When in doubt about the requirement for REB review, researchers should consult their REB.
Scenario 2: Scope of the TCPS
Context
A researcher is using a crowdsourcing platform to invite individuals to contribute to the development of open-source software as a solution to a research problem.
Does the research fall within the scope of the TCPS?
The focus of the research is on software development rather than the individuals or their personal data. As the individuals contributing ideas/solutions are not themselves the focus of the research, and if no personal data will be collected, they do not meet the definition of “human participant” in the TCPS (Application of Article 2.1).
The research does not fall within the scope of the TCPS, but can still be guided by its core principles: Respect for Persons, Concern for Welfare, and Justice (Scope #10 and #15). When in doubt about the requirement for REB review, researchers should consult their REB.
Scenario 3: Managing incentives
Context
A researcher will recruit prospective participants through a publicly available crowdsourcing platform. The researcher encourages prospective participants to take part in completing a survey as part of their research by offering them monetary incentives. The researcher has limited resources and wants to avoid giving out unnecessary incentives to participants.
How should the researcher manage their incentives plan?
The onus is on the researcher to justify to the REB the use of a particular incentive plan and the level of incentives proposed (Application of Article 3.1). The researcher may determine the level of incentives based on participation, or the host platform may have standard rates. The researcher needs to manage the incentives process as part of the design of their research and budget.
The researcher must give participants the promised incentive as communicated to them during the consent process. If prospective participants were informed of a lump sum incentive, then they are entitled to the lump sum incentive. If the incentive is based on a schedule where participants are offered an incentive proportionate to their participation, then the participants are entitled to the amount in proportion to the extent of their participation up to the point of their withdrawal. Also refer to Section 6a. of this document for more information on incorporating strategies to address challenges with participation.
Scenario 4: Balancing participants’ expectations with the integrity of research
Context
An REB receives complaints from research participants who took part in research on a crowdsourcing platform. They were denied the promised incentives by the researchers because a) they completed their responses to the survey too quickly, b) the same participant completed the survey multiple times, or c) the researchers have noted inconsistencies with responses in the Likert scale direction in the survey tool. The researchers are concerned about protecting the integrity of their research data and would like to avoid giving out unnecessary incentives. However, the researchers had not anticipated problematic responses and did not inform participants in the consent process that there were criteria that would limit the offering of incentives.
How can the researchers balance the integrity of the research with participants’ expectations?
The researchers should give participants the promised incentives based on the information communicated to them during the consent process, which should have included an explanation of their responsibilities and how they are related to the offer of incentives (e.g., prorated based on completion of certain tasks).
Researchers have the option of excluding data where they question its reliability as it impacts the validity of their research. The consent process should include an explanation of the circumstances in which the researchers might exclude a participant's responses. This should be separate from the promise of incentives for participants. Researchers can also clearly state in their research recruitment material that they will only give out the promised incentive for a single response. Refer to Section 6a of this document for more information on incorporating strategies to address challenges with participation.
To address the reliability or consistency of the data, researchers should have included in their research design measures for internal reliability used in self-reporting. For example, where one item or question is used to measure a construct and the extent to which the scores or items correlate, or by adding questions that validate responses to other questions. Other mechanisms to uncover the likelihood of unreliable answers include pre-screening or passing a test before doing a task.
Scenario 5: Privacy considerations
Context
A researcher affiliated with a Canadian institution plans to use a mobile Health application (mHealth app) to crowdsource data for their research. They would like to recruit participants to ask them about their lifestyle choices for a study on chronic diseases. The research is a US-Canadian collaboration, and the Canadian researcher is a co-investigator. As such, the app was developed in the US by an application development firm. The app is available on all major app stores. It is a commercial app that was designed solely for crowdsourcing data for research. While there is a user agreement to use the app, there is no privacy policy or formal consent form. Furthermore, there is no institutional review board (IRB) approval associated with the app in the US. In order to recruit research participants, the researcher wants to place an advertisement on social media.
Does the researcher need to apply for REB approval in Canada and if so, what should the REB consider?
The research involves personal data collected from users of the mHealth app. As such, the research involves “human participants” as defined in the TCPS and falls within its scope. The research is subject to REB review irrespective of whether IRB approval has been obtained in the US (Article 2.1).
When the REB reviews this research, it should give special attention to obtaining proper consent, data use, data security, and data sharing, commercialization, as well as the lack of a privacy policy associated with the app. As the researcher did not develop the app and since the app is accompanied by a sparse user agreement, there is a risk that the research participants will not have the necessary information to make an informed decision about their participation in the research (Article 3.2). Furthermore, there is no explicit consent form, so documentation of consent cannot be satisfied (Article 3.12).
As the app collects personal data and has no privacy policy, there are privacy and data security concerns as well. The researcher may not be able to speak to these concerns as there is little to no information available as to how the data is stored, if it is shared, and if it is coded or anonymized (Articles 5.2, 5.3 and 5.4). Finally, as the app is run by a third-party app developer, there is a risk of the crowdsourced data being commercialized. The research participants would need to be made aware of this risk (Application of Article 3.2).
- Date modified: