TCPS 2 (2018) – Chapter 11: Clinical Trials
- A. Key Concepts
- B. Ethical Issues for Clinical Trial Design and Review
- C. Safety Monitoring and Reporting New Information
- D. Registration of Clinical Trials
This chapter focuses on the ethical issues involved in the design, review and conduct of clinical trials. In particular, it addresses ethical issues associated with clinical trial design, therapeutic misconception, safety, reporting new information, and registration. Clinical trials are, perhaps, the most regulated type of research and are subject to provincial, national and international regulatory bodies. However, the emphasis in this chapter is on ethics guidance, grounded in the core principles of this Policy: Respect for Persons, Concern for Welfare, and Justice. As is the case throughout this Policy, the welfare of participants takes precedence over the interests of researchers, institutions, and sponsors.
For the purposes of this Policy, a clinical trial is any investigation involving participants that evaluates the effects of one or more health-related interventions on health outcomes. Interventions include, but are not restricted to, drugs, radiopharmaceuticals, cells and other biological products, surgical procedures, radiologic procedures, devices, genetic therapies, natural health products (NHPs), process-of-care changes, preventive care, manual therapies, and psychotherapies. Clinical trials may also include questions that are not directly related to therapeutic goals (e.g., drug metabolism) in addition to those that directly evaluate the treatment of participants. The terms “clinical trial” and “study” are used interchangeably in this chapter.
Clinical trials are most frequently undertaken in biomedical research, although research that evaluates interventions, usually by comparing two or more approaches, is also conducted in related disciplines such as psychology. The researcher leading a clinical trial is often, but not always, a clinician, that is, a health care provider (e.g., physician, dentist, naturopath, nurse, physiotherapist). The ethics principles articulated in this chapter apply to all clinical trials, irrespective of their type, form, or methodology.
Other types of research may benefit from the guidance in this chapter, particularly when they involve greater than minimal risk. Prospectively assigning participants to receive a potentially harmful intervention carries with it a heightened responsibility to protect participant welfare because the risks to which participants are exposed are caused by the research. While this chapter is intended to guide clinical trials, its discussion of justifying control groups and monitoring participant safety may be an important source of guidance for other types of research.
A. Key Concepts
Clinical trials, like other research covered in this Policy, are subject to a proportionate approach to research ethics review; trials that pose greater foreseeable risk to participants should receive proportionately greater scrutiny (Article 2.9). Not all clinical trials are high risk, and care should be taken to avoid an automatic classification of this nature. However, some clinical trials may involve the risk of serious harm or death and may involve large numbers of people or participants whose circumstances may make them vulnerable in the context of research.
As discussed in Chapter 2, Section B, the evaluation of possible harms that participants may experience due to their involvement in research is of primary importance. Clinical trials should not be conducted unnecessarily on questions that have already been definitively answered. For this reason, the researcher has a responsibility to present the proposed research in the context of a systematic review of the literature on that topic. The systematic review should be carried out according to professional standards of the relevant disciplines(s) or field(s) of research. In all clinical trials, the research ethics board (REB) should carefully evaluate previous relevant research summarized by the researcher (e.g., laboratory, animal and human research with a drug or other therapy), to ensure that the foreseeable risk is justified by the potential benefits to be gained and appropriately minimized. REBs may consult ad hoc advisors as needed (Article 6.5).
An intervention is the planned imposition of a condition or set of conditions on participants for the purposes of research. The conditions may be such things as a task, an activity, a treatment, exposure to stimuli, or a change to environment. The purpose of the research may be to describe, measure, evaluate, explain, or observe participants’ reactions or responses to one or more of the imposed conditions.
Prospective Assignment of Participants
When a study involves one or more interventions and the study design or the principal investigator (PI) determines which intervention each participant will experience, this is known as “prospective assignment.” Prospective assignment may be randomized or based on specific criteria relevant to the study conditions. This assignment normally takes place before data collection, though it may be preceded by a screening process to help researchers determine whether prospective participants meet the inclusion criteria for the study. This screening process may require participant consent.
Prospective assignment may also take place at different stages of a study. For example, a study may begin with all participants in the same group (or assigned to multiple groups), and after a period of observation or testing, the participants may be re-assigned to different groups.
Prospective assignment may be conducted at the level of individuals, groups or populations. For example, a comparison of the effects of vitamin D on the general health and mood of seniors could be carried out by randomizing individual participants to receive either vitamin D supplements or placebos. Or, it could be carried out by assigning some clinics to give high dose vitamin D injections to their senior patients and other clinics to give lower doses.
Alterations to consent requirements that may be necessary for studies conducted at the group and population level are discussed in Article 3.7A.
Clinical equipoise means a genuine uncertainty exists on the part of the relevant expert community about what interventions are most effective for a given condition. This uncertainty necessitates the conduct of research to determine the comparative therapeutic merits of different interventions (not all of which may be represented in a given clinical trial). Clinical equipoise provides a link between the duty of care of a clinician and the need to do research to ensure that the therapies or interventions offered are demonstrably safe and effective.
In trials where participants are prospectively assigned to different interventions (e.g., treatment A; treatment B; no treatment), ethical issues relevant to the principle of Justice arise when one group may fare better or worse than another (see Article 11.4 on use of placebos). For this reason, clinical equipoise should be considered as a starting point for the design and review of clinical trials.
Duty of Care
The duty of care in a medical context is the obligation of clinicians to act in the best interests of patients. In the context of clinical trials, researchers are concerned with the welfare of individual participants, but are also focused on the generation of new knowledge that may or may not confer direct benefits on participants. Nevertheless, researchers have a duty of care to ensure that the foreseeable risks to participants are justified by the potential benefits, and that the safety of participants is an integral part of the research design and conduct. Duty of care may also include the researchers’ responsibility to have a plan to communicate any information relevant to individual participants’ health to their primary clinician. Clinician-researchers (clinicians who also conduct research) need to manage any conflict that may arise from their dual role and they must also be particularly sensitive to the issue of therapeutic misconception (Articles 7.4 and 11.5).
Although clinical trials may provide benefits to some participants, the purpose of a clinical trial is to evaluate an experimental therapy or intervention, not to provide therapy. Therapeutic misconception occurs when trial participants do not understand that research is aimed primarily at producing knowledge and may not provide any therapeutic benefit to them. It also occurs when participants enter trials without understanding the ways in which elements of a clinical trial design may interfere with their own health care objectives.
Clinical trials often involve individuals in need of treatment, for whom the experimental therapy is hoped to be effective. Even when foreseeable risks, potential benefits and treatment alternatives are explained to them, it is common that clinical trial patient-participants do not fully appreciate the differences between clinical care and research participation. As a result, some patient-participants may assume that there must be therapeutic value in the research procedures they are undergoing, or that they have been invited to participate because their clinicians believe it would contribute to their health (Article 11.5).
Dual Roles of Clinician-Researchers
Research has shown that clinician-researchers may conflate their clinical practice with their clinical trial research. Some may be overly optimistic about the prospects of an experimental intervention and overstate potential benefits or understate foreseeable risks to prospective participants. This can foster therapeutic misconception among patients and influence the recruitment and consent process (Articles 11.5 and 3.1). Clinicians must take care not to create unrealistic expectations among participants with respect to the potential benefits of the research.
To preserve the trust on which their professional relationships with patients and colleagues reside, researchers should take all necessary measures to separate their role as researcher from their role as clinician (e.g., enlist associates to recruit participants, rely on colleagues to determine when a patient should be withdrawn). It is important that REBs appreciate the potential conflicts between these roles and the possible impact on the welfare of participants (Article 7.4).
In studies involving more than one researcher, a principal investigator (PI) is the researcher who has responsibility for the ethical conduct of the study, and for the actions of any member of the research team at a local site. In a multi-site study, a lead principal investigator (lead PI) is a designated PI who is responsible for the ethical conduct of the study for all sites. The lead PI is responsible for communicating any changes to the study, new information, and/or unanticipated events to the REB, to the sponsor (if any), and to local site PIs. PIs must inform their local REBs in situations where no alternative review model for research involving multiple institutions has been established.
In the context of balancing foreseeable risks and potential benefits, researchers and REBs must consider the need for mechanisms to:
- stop all or part of the study due to evidence of greater than expected harms or greater than expected benefits in any of the study conditions;
- remove individual participants from a study for their own safety.
These mechanisms are most commonly referred to as “stopping rules.” A study condition could be an experimental intervention, a standard of care, or a control condition.
Stopping rules are pre-determined rules that consist of one or more safety and efficacy criteria (end points) that, if met, warrant a temporary or permanent stop to all or part of the study or a participant’s involvement in the study. Study-wide stopping rules identify when a study should be stopped due to compelling evidence that:
- a study condition is more or less efficacious or effective than another;
- a study condition is more or less safe than another; or
- the study is futile because:
- the data are invalid or unreliable (e.g., due to insufficient numbers of participants);
- it is unlikely that the findings will be able to address the research question based on the collected data.
For example, in a study comparing an experimental drug for a disease with the standard treatment, an end point may be the onset of remission in a pre-set percentage of participants in any arm of the study. If the experimental intervention is achieving higher rates of remission than the standard of care, then this may trigger a decision to pause or stop the trial for reasons of superior efficacy. If the severity of side effects is greater than expected among participants receiving the experimental intervention, then the study may be stopped, and then either amended or closed, for reasons of inferior safety. If a study cannot attract enough participants to yield valid results or if the design of the study was compromised in some way (e.g., repeated equipment malfunctions), then the researcher may decide to end the study for reasons of futility.
Stopping rules may specify what actions should be taken and must be disclosed to participants (Article 3.2). To avoid conflicts of interest, researchers may have an independent Data and Safety Monitoring Board (DSMB) assess whether the interim analyses of a study meet the criteria that might trigger a stopping rule. Stopping rules may be applied to one or more arms of the study.
Participants may be removed from a study for efficacy and safety reasons. For example, a study comparing an experimental drug with a standard treatment for participants with weakened immune systems may have an end point of the onset of an infection. This end point may be used to remove participants from the study for their own safety, but may not warrant stopping the entire study. Sometimes participants may stop receiving the experimental intervention but may continue to be involved in the study. It may also be a precondition of the study that participants who are not complying with the study requirements can be removed from the study by the researcher.
To distinguish the effect of an intervention, researchers may assemble participants into a group that receives the intervention and a group that does not receive the intervention. The group that does not receive the intervention of interest is called the control group, or control arm. The choice of control arm may range from currently approved treatments to placebo, placebo add-on, or no treatment.
B. Ethical Issues for Clinical Trial Design and Review
This section discusses ethical issues associated with the design and review of specific types of clinical trials. Though not all possible clinical trial designs are represented in this section, the guidance provided can be applied and adapted as needed. Researchers and REBs should also consider how applicable regulations affect the design and conduct of clinical trials.
Guidance regarding a proportionate approach to research ethics review, consent, privacy, confidentiality, dissemination of findings, conflicts of interest and research involving human biological materials and other ethical guidance described in other chapters of this Policy apply equally to clinical trials.
In developing and reviewing proposals involving clinical trials, researchers and REBs should refer to other chapters in this Policy. This chapter does not reiterate guidance set out in other chapters. Rather, it focuses on issues that arise specifically in the context of clinical trials and provides guidance for ethics issues associated with control groups, the use of placebos, safety monitoring, reporting new information, and trial registration.
In the design and review of a clinical trial, researchers and REBs shall consider the type of trial, its phase (if appropriate), and the corresponding particular ethical issues associated with it, in light of the core principles of this Policy.
Each type of clinical trial has specific ethical issues that correspond to the risks faced by the participants. In a proposal submitted for research ethics review, the researcher shall clearly specify the type of trial proposed and, where relevant, its phase. REBs reviewing clinical trials need to be familiar with the ethical issues raised by different phases, and by different types of clinical trials.
Clinical trials involving pharmaceutical products are commonly categorized into four phases, each of which gives rise to particular ethical issues. Detailed descriptions of the phases of clinical trials are provided in other guidance documents, for example Health Canada’s guidance documents. The ethical concerns described are most likely to arise in a specific phase of a clinical trial. Some issues may arise at any phase of a clinical trial. Ethical issues raised by the different phases of pharmaceutical trials may also arise in other types of clinical trials.
Safety concerns are particularly acute in phase I research because it may be the first time participants are exposed to the new drug (“first-in-human” trials), and there may be little or no experience with the drug. Phase I trials often depend on healthy participants who are offered incentives for their participation, or they may include participants with specific diseases for whom conventional therapy has failed. The combination of clinical risk with uncertain or no likelihood of clinical benefit, and the often substantial incentives offered to participants, raises ethical concerns about safety, the selection and recruitment of participants, and the consent process. For safety, it is important to ensure that the drug is initially given to a small number of participants and that dosing is increased in clearly defined increments only after participants’ responses to the initial dose is known. Recruitment and consent procedures shall ensure that participants are aware of the untested nature of the therapy and that participants do not accept, because of the incentives being offered, risks they would otherwise refuse. Consideration should be given to minimizing the possibility of therapeutic misconception.
Phase II, or combined phase I/II clinical trials, raise particular ethical concerns, because they are often conducted with populations whose therapeutic options have been exhausted. Examples include patients with cancer that is incurable by standard therapies, or people with conditions that cause them acute or chronic pain. These circumstances may affect the perceptions of patients and their families as to the balance between the risks and potential benefits of the trial and thus may affect their decision whether to participate. Consideration should be given to minimizing the possibility of therapeutic misconception. Participants in phase II trials may include patients who are unwell. Participants and their families may be financially impacted. The REB should ensure that incentives for research participation are not coercive and that patients or authorized third parties do not accept risks they would otherwise refuse because of the incentives being offered. Researchers are encouraged to consult informally with the REB about any recruiting, consent or safety issues that arise.
During the course of a phase II clinical trial, participants will have access to a new drug that may or may not provide clinical benefit. Researchers shall: (a) provide details on access to the new drug upon trial completion as part of the consent process; and (b) make reasonable efforts to secure continued access to the drug following the phase II trial for those patient-participants for whom the drugs appear to be beneficial.
REBs should carefully examine phase III clinical trials to ensure that the care of patient-participants is not compromised in the assignment to any arm of the trial. Researchers should provide a plan for interim analysis of data, early unblinding of clinicians and/or participants, and/or ending the trial if the drug should prove efficacious or harmful. The REB should evaluate such plans with due consideration for the welfare of the participants and the group that is the focus of the research (Article 3.2[l]).
Phase III trials are normally conducted with participants in need of treatment and may involve clinicians in dual roles as researchers. Researchers in dual roles should explain how they will eliminate, minimize or manage their involvement with any of their patients recruited to the trial (Article 7.4, Chapter 11, Section A).
Researchers and the REB should address the issue of continuing access to the experimental therapy after the trial closes. If the treatment benefits participants and is safe, the proposal should state whether it will continue to be provided and under what conditions. REBs should be concerned about what provisions are possible to ensure that participants continue to receive adequate treatment.
Phase IV trials are conducted after a drug has been approved by the regulator for the market. They are conducted to assess the long-term safety and effectiveness of marketed drugs and devices through the identification of side effects, toxicities, drug interactions and overall tolerance that may only emerge over time. Phase IV trials may be surveillance studies or they may involve interventions (e.g., a comparison of two approved drugs).
In some cases, phase IV trials may be designed to serve primarily as marketing initiatives to encourage the prescription and continued use of an approved drug. For example, a clinician may be paid a per capita fee by a sponsor to collect data on the side effects and acceptance by patients of a drug being marketed by that drug’s sponsor. REBs should carefully consider any financial terms between sponsors and investigators (Articles 6.24 and 7.4) that may create problems such as inappropriate prescription practices, billing practices and/or inappropriate utilization of public resources (e.g., diagnostic services and medical imaging). Researchers and REBs must ensure that trials are undertaken for a bona fide scientific purpose. This includes ensuring that the design and objectives are scientifically, and not only commercially, driven.
Natural Health Product Trials
A common public misconception about natural health products (NHPs) is that they are safe simply because they are natural. Some NHPs, however, can pose serious health risks. NHPs may also be part of a multi-treatment therapeutic approach (e.g., an herbal medicine added to a conventional medicine or to a complementary alternative therapy). A research proposal for an NHP clinical trial shall clearly identify the known effects of the product under investigation and its possible contraindications. REBs should ensure that NHP clinical trial proposals are reviewed with the appropriate level of scrutiny as indicated by the foreseeable risks to the participants.
In evaluating the research design, REBs should consider the history of the NHP as provided in the literature review contained in the researcher’s brochure and/or in a monograph, such as those published by Health Canada setting out approved uses and cautionary information. For NHPs with an established safe history of human use, the researcher does not have to present the findings of prior testing with animals if the proposed conditions of use in the trial do not differ from approved uses. However, if the NHP is a new product without an established safe history of human use, prior animal testing may be necessary before it can be approved for first-in-human trials.
Some NHPs do not fall under the jurisdiction of Health Canada, and their efficacy may not have been rigorously tested. Researchers and REB members should know how applicable legal and regulatory requirements affect the design and conduct of NHP clinical trials.
Medical Device Trials
Medical devices may take many forms (e.g., magnetic resonance imaging machine, cardiac pacemaker, hip implant). The term “medical device” covers a wide range of instruments used in the prevention, diagnosis, mitigation, or treatment of a disease or abnormal physical condition, or in the restoration, correction or modification of body function or structure.
Researchers are responsible for seeking appropriate bioengineering input and providing up-to-date information about the device, such as any feasibility studies the device has been subject to in Canada or in other countries, and its risk classification. If an REB does not have enough safety information about the device to consider in its review of the trial, the researcher should be advised to work with the manufacturer of the device to provide appropriate risk information in the research proposal.
Some of the issues surrounding the comparison of different surgical techniques are: whether the technique is appropriate for the participants; whether the technique has been validated; whether the tools required have been approved for use in Canada; whether it is appropriate to employ a control group that undergoes sham surgeries; and how well the experimental procedures will have been explained to prospective participants.
When there is a crossover from non-surgical to surgical treatment, it can be difficult to assess whether participants’ health outcomes were due to the surgical intervention. In these situations, it may be appropriate to use a placebo surgical comparator. The risk of subjecting participants to a potentially scientifically inconclusive trial needs to be weighed against the risk of subjecting them to a potentially harmful placebo intervention. REBs should be satisfied that the research question cannot be addressed in any other way. To ensure participants are fully aware that they may be undergoing unnecessary surgery, REBs should examine the consent process for clear explanations of the experimental procedures, rationale, risks and potential benefits in language that is appropriate for the participant group (Article 3.2).
REBs should be aware that it is possible that the principal investigators of surgical clinical trials need not, themselves, be surgeons or technicians trained in the procedure. For example, a biomechanical engineer who has developed a new type of skin graft material to aid in surgical repair may conduct a surgical clinical trial, with the assistance of a surgical team, to compare the new material with an existing material.
A psychotherapy trial tests a psychotherapeutic approach in populations with the same psychological diagnosis. It may compare the outcomes of those receiving the therapy to those on a wait list. Often, a trial will compare a psychotherapeutic approach to a pharmaceutical approach or to some combination of both.
REBs should be aware that trials involving psychotherapy may be more focused on effectiveness in practice than on efficacy under tightly controlled conditions. For example, the research question may be how participants undergoing a particular therapy are functioning in their daily lives. The duration of these trials may be longer as a function of the therapeutic approach and the characteristics of the condition to which it is applied. Researchers must clearly identify any risk of a negative impact on participants’ mental health and how they intend to minimize and/or manage these risks.
Issues of participant privacy and confidentiality may receive closer scrutiny in cases where people with specific psychological profiles are being recruited from the same institution as the researchers. Researchers shall indicate how recruitment, data collection and management, and compensation procedures have been designed to protect participant confidentiality (Chapter 5).
Pilot trials, also known as “feasibility trials” or “vanguard trials,” are smaller versions of a main trial (Articles 2.1 and 6.11).
Cluster Randomized Trials
Cluster randomized trials (CRTs) involve the prospective assignment to one or more interventions at the level of a group or population (e.g., hospital wards, schools, communities) rather than at the level of individual participants. CRTs may involve outcome evaluation at the level of group or “cluster,” or the individual cluster members.
CRTs raise issues of participant autonomy depending on whether the design is based on an exception to the requirement to seek prior consent from individual participants. Researchers and REBs need to consider whether:
- the randomization of clusters will take place before it is possible to identify participants and seek consent;
- individuals who are in the cluster but who are not the primary focus of the trial may be directly or indirectly affected by any intervention applied to the cluster; and
- cluster-level trial interventions may be difficult or impossible for individuals to avoid, removing the possibility of refusing to participate.
Where individual consent is possible in CRTs, it shall be sought, and the process of prospective assignment shall be explained so that individual participants are aware that they may or may not receive an intervention. When this approach would render the trial impossible or impracticable to carry out, the researcher must be able to justify an alteration to, or a waiver of, the consent requirements to the satisfaction of the REB (Article 3.7A). It may be feasible to seek consent for some aspects of the study but not for others. For example, a cluster-level intervention may make it impossible to seek participant consent for the intervention, but it may still be possible to seek consent for data collection.
Researchers should seek engagement to establish an interaction between the research team and the community that is relevant to the trial. They must use a reasonable process to identify individuals who legitimately represent the community and provide details of this process to the REB. In their research proposals, researchers must also clearly identify risks to individuals, to the cluster as a whole, and to any sub-groups within the cluster.
Adaptive Design Trials
Adaptive design trials are also known as “multi-stage design,” “flexible/dynamic design,” or “data-driven design.” They typically include a prospective adaptation of some design features according to decision points based on accrued data. For example, analysis of trial arm outcomes after one month will determine which participants stay in their assigned arm and which will be moved to another arm. They may include changes to sample size, allocation ratio, stopping rules, the trial objective or hypothesis (e.g., from superiority to non-inferiority), the primary outcome, the population, or dropping or adding an intervention or control arm. Adaptations to trial design may affect informed consent, clinical equipoise, and the fair distribution of risks and benefits throughout the trial. Researchers should set criteria for adaptation decision points regarding exposure of greater numbers of participants to possible undetected risks of an intervention. These criteria should be justified. There should be a plan to manage or minimize the involvement of clinician-researchers in research decisions that may affect their patients (e.g., prospective assignment, data-driven changes to trial design).
The possible advantages of adaptive design trials are shorter trials with fewer participants and earlier identification of the most and least promising interventions, particularly for interventions that cannot be tested with a conventional approach. For example, to test the safety and efficacy of an experimental intervention for a rare disease in a conventional clinical trial, researchers would have to recruit far more participants than the number of people who actually have the disease. Using an adaptive design, it could be possible to achieve the same statistical power with fewer participants participating in a sequence of interventions. An advantage of adaptive design trials is that they provide opportunities for participants to affirm their ongoing consent in a formal manner.
However, adaptive designs also raise particular ethical issues. Participants joining a trial in a later phase might experience fewer risks and greater benefits than those who were involved in the earlier phases of the same trial. The re-assignment of more participants to an intervention that looks more promising could inadvertently expose more participants to side effects that take longer to emerge. Smaller numbers of participants may reduce the ability to do more, focused analyses (e.g., subgroups). This may increase the possibility of false negative or false positive results.
Registry-based trials use health registries as platforms for different aspects of trial operations such as recruitment, data collection, randomization, and/or follow-up. Health registries are collections of patient health information. In registry-based trials, researchers seek access to registries in order to recruit participants and prospectively assign interventions, which may include the standard of care. The intervention, measurement and/or analysis may be at the group or individual level. Different levels of intervention, measurement, and analysis can present different issues of consent and privacy. This type of research may benefit from guidance regarding secondary use of identifiable information for research purposes, including issues related to linkage of records in different registries (Chapter 5, Section D). Although registry-based trials are relatively recent, it appears that the ethics issues they present are not novel and are currently addressed by the principles of this Policy.
Registry-based trials are efficient because they offer researchers access to a pool of participants within an existing infrastructure for recruitment, data collection or follow-up. Such trials require coordination with registry administrators, as aspects of their roles may overlap with the roles of the researchers (e.g., safeguarding privacy or monitoring safety).
Justification for Control Groups
As with other aspects of the trial design, the choice of including a control arm must be justified based on scientific, ethical, medical and methodological reasons and must meet an acceptable risk/benefit ratio. Risks to the safety of participants can come from lack of efficacy or from undesirable side effects. These risks must be assessed for each treatment arm, including the experimental and control arm(s).
It is important to avoid the error of equating the absence of a potential benefit conferred by the intervention to participants who are assigned to a control group – to a risk. For example, in a study comparing the addition of a new treatment for heroin addiction to the standard treatment, some participants will be prospectively assigned to receive this intervention, and some will not. Those who are assigned to the control group are no better or worse off than if no research were taking place. It is the intervention group that is exposed to any harms or benefits of the intervention. For example, the intervention may or may not work, or it may have unforeseen side effects. The role of the researcher is to address the research question in the context of clinical equipoise - to discover if a particular intervention is beneficial or less harmful than the status quo – and to disseminate the findings. It is the role of advocates, policy makers, and service providers to use the findings of research to ensure the equitable distribution of potential benefits for the welfare of society.
Article 11.3 addresses the considerations required of researchers and REBs to assess whether a particular control group design is ethically acceptable. Article 11.4 deals specifically with the ethical issues pertaining to the use of placebos.
In clinical trials, researchers shall justify their choice of control group(s) to the REB by demonstrating that the choice is:
- relevant to the research question;
- appropriate for the population of interest; and
- consistent with the criteria for clinical equipoise.
The use of prospective assignment of participants to different groups may result in one group of participants experiencing greater benefits or greater harms than another. Prospective assignment must be justified by the research question. It is important that the researcher take into account relevant characteristics of the participant population when choosing the type of control group. For example, a wait-list control group may not be appropriate for investigation of a behavioural approach to anxiety due to the increased anxiety participants in the wait-list condition may experience.
In the context of studies that involve interventions intended to have a beneficial effect upon participants, there must exist a genuine uncertainty (i.e., clinical equipoise) on the part of the relevant expert community about which interventions are most effective.
Use of Placebos
The term “placebo” traditionally refers to an inactive substance or treatment (e.g., a pill, an injection) given to participants to simulate an active substance or treatment. Often, the purpose of using a placebo as a comparator in a clinical trial is to control for the reaction participants may have to any kind of intervention and their beliefs about its possible effects. This may be done in order to distinguish the effects of the intervention of interest from the effects caused by participant belief. It may also be used as a control to distinguish effects of the intervention of interest from the natural background rate of symptoms or variability in a disease that occurs in a population.
A clinical trial in which one or more intervention arms are compared with a placebo control group raises specific ethical issues. Where there is an established effective treatment, use of a placebo may deprive participants of needed therapy. It is the responsibility of the researcher or sponsor to provide justification to the REB for the choice of a placebo control group, as opposed to the other possible choices of control group (e.g., active control, wait-list control, dose-response, and combination therapies). The following article sets out criteria for the use of a placebo control group to ensure that this type of clinical trial design is used only in situations that do not compromise the safety and welfare of participants.
A new therapy or intervention should generally be tested against an established effective therapy. A placebo control is ethically acceptable in a randomized controlled clinical trial only if:
- its use is scientifically and methodologically sound in establishing the efficacy or safety of the test therapy or intervention;
- it does not compromise the safety or health of participants;
- the researcher articulates to the REB a compelling scientific justification for the use of the placebo control; and
- the general principles of consent are respected and participants or their authorized third parties are specifically informed (Article 3.2) about any:
- intervention or therapy that will be withdrawn or withheld for purposes of the research; and
- of the anticipated consequences of withdrawing or withholding the intervention or therapy.
Researchers must justify the decision to use placebo control instead of other possible options. The design must be methodologically sound to be ethically acceptable research. The use of placebos must be necessary to address the research question. For example, in a study comparing a combination of two therapies against two therapies normally administered singly, the research design would call for an intervention group to receive the combined therapy. One control group would receive one therapy plus a placebo. A second control group would receive the other therapy plus a placebo. This design preserves the integrity of blinding and protects the scientific validity of the study without posing unnecessary additional risks to participants.
The use of an active treatment comparator in a clinical trial of a new therapy is generally the appropriate trial design when an established effective therapy exists for the population and clinical indication under study.
Great care should be taken to avoid abuse of placebo comparators. However, they are acceptable in any of the following situations:
- There are no established effective therapies for the population or for the indication under study.
- Existing evidence raises substantial doubt within the relevant expert community regarding the net therapeutic benefit of available therapies.
- Available therapies are known to be ineffective for patients by virtue of their past treatment history or known medical history.
- The trial involves adding a new investigational therapy to established effective therapies: established effective therapy plus new therapy vs. established effective therapy plus placebo.
- Patients with decision-making capacity have provided an informed refusal of established effective therapy, and withholding such therapy will not cause serious or irreversible harm.
The determination of response satisfaction and refusal of treatment must take place outside the context of recruitment for the clinical trial and prior to offering trial participation to the prospective participant, and both must be documented.Footnote 1
The use of a placebo comparator in situation (5) is permitted because prospective trial participants are not using established therapies and therefore are not benefiting from therapy. For that reason, such participants would not be further disadvantaged if enrolled in a placebo-controlled trial than participants in a trial for whom there are no established effective therapies for the indication under study. This justification would be included in the research proposal submitted to the REB.
Use of Placebos in Superiority and Non-Inferiority Studies
A superiority study is one in which researchers empirically test whether an experimental intervention is more efficacious or beneficial than a control intervention. A superiority study may be placebo-controlled when the control intervention is a placebo or sham intervention, or it may be actively controlled when the control intervention is an established effective intervention.
A non-inferiority trial compares a control intervention with an experimental intervention that offers some additional advantage (e.g., easier to administer, fewer side effects). The goal of the trial is to demonstrate that the experimental intervention is not inferior to the control intervention with regard to a specified end point. From an ethical perspective, a critical component in such trials is defining, in advance of the trial, the maximal lowering of efficacy that would be acceptable in order to justify use of the experimental treatment. An advantage of a non-inferiority trial is that placebos are not used and both groups receive an active intervention.
For example, consider a trial of an experimental medication for cancer. The experimental medication offers one or more advantages over the established treatment, such as fewer side effects, lower cost, or easier administration. Physicians and patients would like to capitalize on these advantages and use the experimental medication, but only if this does not come at the cost of significantly lowering survival. In such an instance, one would conduct a non-inferiority trial to show that the experimental medication is not inferior to the established treatment with respect to survival.
To properly assess the ethics of placebo-controlled superiority design vs. active controlled non-inferiority design, an appreciation of the interplay of ethics and science is required (Article 2.7). There may also be regulatory considerations. A low and/or variable response to the existing treatment can make it difficult to carry out a successful non-inferiority trial. The researcher must provide adequate justification for the use of a non-inferiority design.
REBs and clinical trial researchers should be conscious of the phenomenon of therapeutic misconception. They should ensure that procedures for recruitment and consent emphasize which specific elements of a clinical trial are required for research purposes, as well as the differences between research and the standard clinical care patients might otherwise receive.
When treating clinicians conduct research with their patients, special efforts may be required, as part of the consent process, to distinguish between their dual role as clinician and researcher – and to ensure that patients who become participants understand the differences between the goals of health care and the goals of research.
It is important that clinician-researchers take care not to overplay the benefits of research participation to patients in vulnerable circumstances, who may be misled to enter trials with false hopes. Research has shown that clinicians can affect how well their patients appreciate the uncertainty of research, the seriousness and magnitude of risks, and the possibility that participation may not result in any direct benefits to their own health status.
Article 3.2 describes the requirements for consent to research participation. It indicates that participants shall be provided with relevant information, including a clear description of those elements of participation that are experimental in nature and those not primarily intended to benefit the participant directly.
In general, therapeutic misconception can be minimized by ensuring that the clinicians who provide the patient’s regular care are involved as little as possible in the recruitment and the consent process. Ideally, treatment and research functions should be performed by different people. However, there may be instances in which participants’ best interests are served by having their primary care clinicians involved in recruitment and consent, for example, because of the degree of expertise needed to compare the standard of care with that of the clinical trial. In these cases, the research proposal shall indicate what other measures will be taken to minimize therapeutic misconception. For example, a consent discussion may be conducted by the clinician-researcher, and the agreement to participate in research may be sought by an individual who is not involved in the clinical care of the patient.
C. Safety Monitoring and Reporting New Information
In accordance with the core principle of Concern for Welfare, it is a key responsibility of researchers to ensure that, as clinical trials proceed, the risks to participants remain within the acceptable range, and the safety of participants is monitored. Articles 11.6 and 11.7 address researchers’ responsibility to include a safety monitoring plan in their research proposals for REB review, and their responsibility to ensure that any new information that may affect participant welfare or consent is shared with the REB and participants. See also Articles 6.15 and 6.16. Article 11.9 addresses the REB’s responsibility to have procedures in place to receive and respond to reports of new information, including, but not limited to, safety data, unanticipated issues and newly discovered risks.
In the case of clinical trials, there are provincial, national and international guidelines that govern safety monitoring and reporting of new information. It is the responsibility of researchers to be aware of the guidelines that apply to their research and to adhere to them for the safety and benefit of participants.
Researchers shall provide the REB with an acceptable plan for monitoring safety, efficacy/effectiveness (where feasible) and validity. This plan shall describe:
- how participant safety will be monitored and what actions will be taken in the event of a threat to participant safety;
- how intervention efficacy will be monitored (where feasible) and what actions will be taken if efficacy is found to be greater than expected;
- the criteria by which participants may be removed from a study for safety reasons;
- the study-wide stopping rules (if any) by which studies may be stopped or amended due to evidence of inferior safety, superior efficacy or futility; and
- the reporting procedure that will be followed to ensure any information relevant to participant welfare or consent is reported clearly and in a timely fashion to the REB.
A data and safety monitoring plan may (but need not) include the establishment of an independent DSMB (Article 11.7).
The responsibility for establishing a data and safety monitoring plan lies with the researcher or the research sponsor. The REB must assess whether the plan sufficiently addresses the foreseeable risks and potential benefits for study participants.
Paragraphs (c) and (d) require that the data and safety monitoring plan describe any mechanisms for removing participants for safety reasons and/or for stopping a study for reasons of safety, efficacy, or futility. Stopping rules are the most common mechanism for making these decisions. All clinical trials that pose more than minimal risk to participants should have criteria by which participants who are experiencing harm can be removed from the study.
Paragraph (e) requires the data and safety monitoring plan to describe the reporting procedure that will be followed. In order to fulfill their mandate to safeguard the interests of participants, REBs require sufficient information about the study design, stopping rules, and monitoring mechanisms at the point of initial review, as well as reports of unanticipated events and any proposed changes to approved study design (Articles 6.16 and 11.9).
Researchers are responsible for ensuring that the results of any interim analyses that affect participant welfare or consent are provided to REBs. This summary report should be provided promptly and should include information about the context and significance of reported data to permit a fair interpretation and meaningful review by the REB. When the REB requires additional information, the researcher shall provide it. If necessary, the REB may require that an evaluation be conducted by a qualified source who has no conflict of interest and who is independent of any sponsor, such as an independent DSMB.
Independent Data and Safety Monitoring Boards
Independent Data and Safety Monitoring Boards (DSMBs) are known by a variety of names, including “data monitoring committees,” “data safety committees,” and “data and safety monitoring boards.” They are comprised of people with the relevant scientific, ethical and/or community expertise to oversee the data and procedures of one or more ongoing trials with respect to participant safety, intervention effects and data validity. The duties of independent monitoring boards are typically described in a charter based on a trial design approved by an REB. These duties may include applying stopping rules and recommending changes to the trial design.
An independent DSMB normally conducts regular monitoring of the data from all sites (in the case of multi-site studies) at pre-set intervals. In the case of blinded studies, independent DSMBs can unblind data for the purpose of making recommendations based on the results of interim analyses. Information pertaining to the welfare or consent of participants must be reported by the researcher to any REBs that have approved the study (Articles 6.15 and 11.8). Not every study is required to have an independent DSMB. Factors to consider when making this decision appear in Article 11.7.
The appointment of an independent DSMB does not alter the responsibility of researchers to monitor safety, efficacy, and validity throughout the conduct of the study. In the context of multi-site research, when new information at one site could be relevant to participant welfare and consent at other sites, principal investigators must ensure that this information is shared with researchers at each site and with REBs (Article 11.8). The REB must be prepared to act upon these reports, especially where urgent action is required (Article 11.9).
The following factors should be considered to determine whether a study, with or without stopping rules, should have an independent DSMB:
- the magnitude of foreseeable research-attributable harms to participants;
- whether the circumstances of the participants make them vulnerable in the context of research;
- the feasibility of interim data analysis;
- the complexity of the study; and
- conflicts of interest.
Not all clinical trials require an independent DSMB, but it is important to consider the factors listed in this article before making this decision. As participants in a clinical trial may be prospectively assigned to an intervention that poses more than minimal risk, the magnitude of foreseeable harm is a primary consideration even if the probability is low. If the possible outcomes for participants in a clinical trial are severe (e.g., irreversible harm or death), the study is more likely to need an independent DSMB and stopping rules than a study in which the possible outcomes are moderate (temporary, non-life-threatening conditions). For example, a study assessing the efficacy of a non-toxic remedy for the relief of cold symptoms may be less likely to need stopping rules than a study assessing the effect of a highly toxic drug on rates of lung cancer mortality. Researchers and REBs should also consider the invasiveness of the intervention involved and whether there is prior evidence of high risk to participant safety.
Clinical trials in which the intended participant population is already at risk due to their existing circumstances (e.g., weakened mental state, weakened immunity, lack of access to support) may need stopping rules to ensure that researchers and/or independent DSMBs can recognize when incidents of harm to participants require a reconsideration of the study design. It is important to distinguish between idiosyncratic incidents of harm that are limited to a small number of participants and incidents of harm that indicate a general problem with the study design or procedures.
For a stopping rule to be effective, there must be sufficient data available for analysis prior to the conclusion of the study. Studies of short-term interventions or studies of small participant groups may not generate enough data to permit the use of a study-wide stopping rule. However, in short-term studies in which participants are at high risk of severe outcomes, researchers and REBs should consider whether a staged enrolment design, with pauses to permit interim analyses and application of stopping rules, is warranted. In studies of interventions that may not reveal their effects on participants until long after the intervention has ended, interim analyses are not practicable. If the study design does not permit timely interim data analyses that could have an impact on participants’ safety or well-being, there may be no use in establishing study-wide stopping rules.
Studies that involve multiple data collection sites, blinded data, long-term data collection and/or large numbers of participants may need an independent DSMB regardless of whether study-wide stopping rules are indicated. The interim analysis of all available data is necessary to determine whether measures of harm or benefit to participants meet the study-wide stopping rules. Individual site researchers without the same level of access to data and the expertise to conduct the necessary analyses may not be able to make informed decisions in the best interests of participants. Conversely, a single-site study with unblinded data collection and clear stopping rules may not require an independent DSMB.
The use of independent DSMBs is one way to manage studies in which conflicts of interest within the research team or among research partners are a concern. Safety can be maximized and futility minimized by having an independent DSMB conduct interim analyses of data. The DSMB can also make recommendations about the study design and conflicts of interest that could affect decisions about the validity of data or evidence for efficacy/effectiveness. The membership of the DSMB should be free of inappropriate influence. Conflicts of interest should be resolved in accordance with the DSMB charter.
Reporting New Information
Researchers shall promptly report new information revealed during the conduct of the trial that might affect the welfare or consent of participants to the REB (Article 6.15), to a publicly accessible registry (Article 11.10) and to other appropriate regulatory or advisory bodies. When new information is relevant to participants’ welfare, researchers shall promptly inform all participants to whom the information applies (including former participants). Researchers shall work with their REBs to determine which participants must be informed, and how the information should be conveyed.
In the course of any type of clinical trial, new information may arise that is relevant to participants’ welfare or their ongoing consent to participate (Articles 2.8, 3.3, 6.15 and 6.16). This new information might arise from unanticipated issues (e.g., adverse reactions to interventions) or from routine evaluations of participant health that occur in the context of the trial. It might pertain to all participants, to those in one arm of an intervention, or only to one participant with a particular issue. It might be information that arises from other related research that has repercussions for ongoing trials.
New information thus covers a range of matters that includes, but is not limited to, the following:
- changes to the research design;
- evidence of any new risks;
- unanticipated issues that have possible health or safety consequences for participants;
- new information discovered in the course of the trial before the end-of-study date that decisively shows that the benefits of one intervention exceed those of another;
- new research findings, including relevant non-trial findings discovered before the end-of-study date;
- unanticipated problems involving lack of efficacy/effectiveness, recruitment issues or other matters determined to be serious enough to warrant disclosure; or
- closure of trials at other sites for reasons that may be relevant to the welfare or consent of participants in the ongoing trial.
Article 11.8 outlines the continuing duty of researchers to share new and relevant information regarding clinical trials with the REB, the publicly accessible registry, and other appropriate bodies. This information may also need to be shared with participants and, possibly, other relevant third parties (e.g., circle of care), as indicated by the nature of the information. The more relevant, serious and urgent the information, the more promptly it should be disclosed. To understand the particular relevance of new information, it should be considered from the perspective of the participant.
Researchers should also promptly share new information about an intervention with other researchers or health professionals administering it to participants or patients, and with the scientific community – to the extent that it may be relevant to the general public’s welfare.
The duty to report new information to the REB, along with the necessary analysis and evaluation to make the new information interpretable, lies with the researcher. It is incumbent upon the researcher to keep abreast of reports of findings from studies investigating similar interventions (e.g., through professional journals, online reports, conferences, contact with colleagues). New findings that are reported during the conduct of a trial may affect the assumption of equipoise between the interventions/control groups made during the design phase of that trial. For example, if a trial reported that intervention A was conclusively safer or more effective than intervention B, any researchers testing either of these two interventions in a clinical trial would need to report this finding to their REBs (and DSMBs, where in use). Researchers should also consider the impact of this finding on equipoise, participant welfare, and participants’ ongoing consent.
In the case of newly discovered risks or unanticipated issues, the researcher’s report shall include a plan to eliminate or mitigate any increased risks to participants. Researchers should raise potentially relevant developments with the REB at an early stage to better determine the appropriate scope and timing of information sharing with participants and regulatory authorities.
The welfare of participants must also be considered when a trial is unexpectedly discontinued. When a researcher, a sponsor or other body stops or unblinds all or part of a clinical trial, the principal investigator has an ethical and a regulatory responsibility to inform both study participants and the REB of the discontinuance or unblinding and the reasons for it. Researchers must ensure that the publicly accessible trial registry is updated with any changes to the trial that require REB review and approval, adverse events that occur during a trial, and decisions to end a trial early. Any risks to participants that may arise from the unexpected closing of the study must be communicated to the REB and the participants. The researcher shall indicate any measures that will be taken to mitigate these risks.
Former participants are those who have withdrawn from the study, those who have been removed from the study and those whose participation in the study has ended. When new information is relevant to the welfare of all participants, researchers and REBs have a duty to ensure that all participants are informed, including former participants. Where new information affects only participants actively involved in the trial, the REB may decide that former participants need not be informed. However, researchers may choose to voluntarily share this information with all participants.
Completion of study
The obligation to report new information ends with the completion of the study. The conditions for study completion are set out in the research protocol approved by the REB. Typically, the study is complete when the final data analysis is complete. Study completion may also be defined as the last contact with the last participant for the purposes of collecting data or human biological materials, or for the purposes of follow-up monitoring.
REBs shall develop procedures to review and respond appropriately to reports that arise during the trial concerning safety, efficacy and/or validity and other new information arising from clinical trials that may affect the welfare or consent of participants.
As noted in the preceding articles and elsewhere in the Policy (e.g., Articles 6.15, 6.16), REBs can expect to receive reports of safety, efficacy, and new information, including but not limited to, unanticipated issues, proposed changes to the research design, and newly discovered benefits or risks. It is the REB’s responsibility to establish procedures for reviewing these reports that will be used to determine how to respond to evidence of increased risks or benefits to participants, and to be ready to implement these responses as needed. Responses shall be relative to the magnitude and likelihood of the risk or benefit to the welfare of participants. REBs may advise researchers as to the steps to take to eliminate or mitigate newly reported risks, or to equitably distribute benefits, and how this information should be shared with participants. In exceptional cases, REBs may decide to suspend recruitment, or to suspend all participant involvement in a study pending further investigation.
D. Registration of Clinical Trials
There are compelling ethical reasons for the registration of all clinical trials. Registration improves researchers’ awareness of similar studies so that they may avoid unnecessary duplication and thereby reduce the burden on participants. Registration also improves researchers’ ability to identify potential collaborators and/or gaps in research so that they may pursue new avenues of inquiry with potential benefits to participants and to society. Another reason to register is to prevent researchers or sponsors from reporting only those studies with favourable outcomes. When all studies must be registered, it is easier to identify those studies where outcomes have not been reported or findings have been withheld.
The registration of clinical trials upholds the principles of Respect for Persons, Concern for Welfare, and Justice, by ensuring that the efforts of all participants in clinical trials are acknowledged, and by reducing the potential for endangerment of others through publication bias.
All clinical trials shall be registered before recruitment of the first trial participant in a publicly accessible registry that is acceptable to the World Health Organization (WHO) or the International Committee of Medical Journal Editors (ICMJE).
Clinical trial registries are intended to increase transparency and accountability by providing a record of clinical trials at the recruitment stage that can be used to locate publication of trial results. This helps prevent publication bias, that is, the selective publication of only those trials that yield results in support of an intervention. These registries, in addition to agency policies, editorial policies, and national and institutional ethics policies and results disclosure requirements, contribute to a multi-faceted approach to eliminate non-disclosure. The collective goal is to reduce publication bias and prevent the suppression of data in clinical research.
All fields outlined in the WHO Trial Registration Data Set (TRDS) must be completed in order for a trial to be considered fully registered. Researchers shall provide the REB with evidence of registration (e.g., registration number).
Following registration of their study in accordance with Article 11.10, researchers are responsible for ensuring that the registry is updated in a timely manner with:
- new information (Article 11.8);
- safety and, where feasible, efficacy reports (Article 11.6);
- reasons for stopping a trial early; and
- the location of findings.
Researchers shall promptly update the study registry with any new information that may affect the welfare or consent of participants and with reports of findings or information about where to access findings (e.g., lists of publications, links to publications or to the trial website).
Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada, Access to Research Results: Guiding Principles, Modified 2016-12-21. Retrieved on July 11, 2018.
Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada, Tri-Agency Open Access Policy on Publications, Modified 2016-12-21. Retrieved on July 11, 2018.
Council for International Organizations of Medical Sciences (CIOMS), International Ethical Guidelines for Health-Related Research Involving Human, 2016. Retrieved on August 2, 2018.
Council of Europe, Convention for the Protection of Human Rights and Dignity of the Human Being with regard to the Application of Biology and Medicine: Convention on Human Rights and Biomedicine [ PDF (144 KB) - external link ], 1997. Retrieved on July 11, 2018.
Food and Drugs Act. Natural Health Products Regulations, Part 4: Clinical Trials Involving Human Subjects, SOR/2003-196, 2008. Retrieved on July 11, 2018.
Health Canada, Guidance document. Good Clinical Practice: Integrated Addendum to E6(R1) ICH Topic E6(R2), Adopted November 9, 2016, Effective May 25, 2017. Retrieved on June 29, 2018.
Health Canada. Guidance documents – Medical devices. Retrieved on July 11, 2018.
International Committee of Medical Journal Editors, Sponsorship, Authorship, and Accountability, Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals [ PDF (246 KB) - external link ], 2007. Retrieved on August 7, 2018.
Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials [ PDF (488 KB) - external link ]. BMJ 2010; 340:c869. Retrieved on August 7, 2018.
National Institutes of Health, Further Guidance on a Data and Safety Monitoring for Phase I and Phase II Trials, 2000. Retrieved on June 29, 2018.
National Institutes of Health, NIH Policy for Data and Safety Monitoring, 1998. Retrieved on June 29, 2018.
Ottawa Group, Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1), BMJ 2005; 330(7497):956. Retrieved on June 29, 2018.
Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials [ PDF (218 KB) - external link ]. Retrieved on August 7, 2018.
United Nations, Convention on the Rights of the Child, 1990. Retrieved on June 28, 2018.
United States Food and Drug Administration. Food and Drug Administration Amendments Act of 2007.
Weijer C, Grimshaw JM, Eccles MP, McRae AD, White A, Brehaut JC, Taljaard M and the Ottawa Ethics of Cluster Randomized Trials Consensus Group. The Ottawa Statement on the Ethical Design and Conduct of Cluster Randomized Trials [ PDF (173 KB) - external link ]. PLoS Med 2012; 9(11):e1001346. Retrieved on August 7, 2018.
World Medical Association, Declaration of Helsinki – Ethical Principles for Medical Research Involving Human Subjects, 2013. Retrieved on July 11, 2018.
- Date modified: