The best analysis is like a treasure hunt, searching through data to find the information hidden within. The exciting field of experimental design is all about finding ways to make the most of data before it's even collected. It's a fascinating area of research that unlocks the potential to optimize the insights from the data, even before it's gathered. This is great news for you because it means you can get more "bang for your buck" – more results (and more papers) at lower cost and with fewer resources spent in your study.
The best study design starts with identifying the model (or models) you plan to use to analyze the data—and it's an exciting step! If you can estimate a plausible range of the parameters of the model—both the outcome effect and the remaining parameters—you can estimate how much entropy the posterior distribution will have. This is great because it means you can then decide whether this is sufficient to be able to reject or accept a hypothesis about the target parameters with a given degree of certainty. There are so many fascinating methods for this! They range from simple power estimation that provides sample size requirements to complex comparisons of study designs in design parameters such as the number of subjects, the number of measurement items, the number of measurement time points, the number of multi-level or mixed-effects units (e.g., schools, sites, clinicians, etc.), the measurement accuracy, the study length, or the assumptions.
Here's some great news! Two designs that provide the same statistical power, that is, the same expected entropy for the target parameters, are power equivalent. For example, if you want to investigate the effect of a training program, we can get the same amount of information gain with fewer participants if we extend the duration of the training program—it's a win-win! Since adding participants and increasing the duration of the training program each come with different costs (e.g., financial costs, workload, participant burden, etc.), it is an opportunity to compare different power-equivalent designs in order to find the one that provides the best outcomes at a given cost! The great news is that the transformation between power-equivalent models can be done quickly and efficiently using mathematical methods developed by researchers at the Thomas Bayes Institute.
Here at the Thomas Bayes Institute, we are researchers, too!
We:
(1) research efficient mathematical methods to search the space
of possible study designs to find the optimal design,
(2) explore ways to transfer the concept of power-equivalence
to Bayesian descriptions of power, and
(3) develop software that allows empirical researchers to
compare designs and find the most resource-efficient study
design for their specific research question.