Lewis, M, Bromley, K, Sutton, CJ, McCray, G, Myers, HL and Lancaster, GA (2021) Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! Pilot and Feasibility Studies, 7 (1). 40 - ?. ISSN 2055-5784

[thumbnail of Determining sample size for progression criteria for pragmatic pilot RCTs the hypothesis test strikes back!.pdf]
Determining sample size for progression criteria for pragmatic pilot RCTs the hypothesis test strikes back!.pdf - Published Version
Available under License Creative Commons Attribution.

Download (935kB) | Preview


BACKGROUND: The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. METHODS: Many pilot trials adopt a 'traffic light' system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant. RESULTS: For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0-17 participants (0-50%) will fall into the RED zone and be statistically non-significant, 18-25 (51-74%) fall into AMBER and may or may not be significant and 26-34 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity. DISCUSSION: In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.

Item Type: Article
Additional Information: This is the final published version (version of record). It was first published online via BioMed Central at https://doi.org/10.1186/s40814-021-00770-x - please refer to any applicable terms of use of the publisher.
Uncontrolled Keywords: Outcome and process assessment, Pilots, Sample size, Statistics
Subjects: H Social Sciences > HA Statistics
R Medicine > R Medicine (General)
R Medicine > R Medicine (General) > R735 Medical education. Medical schools. Research
Divisions: Faculty of Medicine and Health Sciences > School of Medicine
Related URLs:
Depositing User: Symplectic
Date Deposited: 12 Feb 2021 15:38
Last Modified: 21 Oct 2021 13:12
URI: https://eprints.keele.ac.uk/id/eprint/9148

Actions (login required)

View Item
View Item