Lab#
Learning Objectives#
At the end of this learning activity you will be able to:
Estimate the effect size given a set of confidence intervals.
Calculate the
effect_size,alpha,power, andsample_sizewhen given 3 of the 4.Interpret a power-plot of multiple experimental choices.
Calculate how changes in estimates of the experimental error impact sample size requirements.
Rigorously choose the appropriate experimental design for the best chance of success.
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import pingouin as pg
sns.set_style('whitegrid')
Step 1: Define the hypothesis#
For this lab we are going to investigate a similar metric. We will imagine replicating the analysis considered in Figure 3C. This analysis considers the different sub-values of the vigalence index. It shows that SK609 is improving attention by reducing the number of misses.
Copying the relevant part of the caption:
“Paired t-tests revealed that SK609 (4mg/kg; i.p.) specifically affected the selection of incorrect answers, significantly reducing the average number of executed misses compared to vehicle conditions (t(6))=3.27, p=0.017; 95% CI[1.02, 7.11]).”
Since this is a paired t-test we’ll use the same strategy as the walkthrough.
Step 2: Define success#
Q1: What is the average difference in misses between vehicle control and SK609 rodents?#
Hint: Calculate the center (average) of the confidence interval; the CI is bolded in the caption above.
Total Points |
5 |
|---|---|
Included Checks |
1 |
Points: 5
q1_change = ...
print(f'On average, during an SK609 trial the rodent missed {q1_change} fewer prompts than vehicle controls.')
grader.check("q1_change")
Q2: Calculate the effect size.#
Hint: Use the change just defined in Q1.
Assume from our domain knowledge and inspection of the figure that there is an error of 3.5 misses.
Total Points |
5 |
|---|---|
Included Checks |
1 |
Points: 5
error = 3.5
q2_effect_size = ...
print(f'The normalized effect_size of SK609 is {q2_effect_size:0.3f}')
grader.check("q2_effect_size")
Step 3: Define your tolerance for risk#
For this assignment consider that we want to have 80% chance of detecting a true effect and a 1% chance of falsely accepting an effect.
Total Points |
5 |
|---|---|
Included Checks |
2 |
Points: 5
power = ...
alpha = ...
grader.check("q3_tolerance")
Step 4: Define a budget#
In the figure caption we see that the paper used a nobs of 16 mice:
“Difference in VI measurements calculated against previous day vehicle performance in rats (n=16) showed SK609 improved sustained attention performance …”
Step 5: Calculate#
Q4: Calculate the minimum change detectable with 16 animals.#
Use alternative='two-sided' as we do not know whether the number of misses is always increasing.
Hint: Use the power-calculator, and then use that effect size to calculate the min_change.
Total Points |
5 |
|---|---|
Included Checks |
2 |
Points: 5
q4_effect_size = ...
print('The effect size is:', q4_effect_size)
# What is the minimum change that we can detect at this power?
q4_min_change = ...
print(f'with 16 animals, one could have detected as few as {q4_min_change:0.2f} min change.')
grader.check("q4_min_effect")
Step 6: Summarize#
Let’s propose a handful of different considerations for our experiment. As before, we’ll keep the power and alpha the same, but we’ll add the following experimental changes:
A grant reviewer has commented on the proposal and believes that your estimate of the error is too optimistic. They would like you to consider a scenario in which your error is 50% larger than the current estimate.
A new post-doc has come from another lab that has a different attention assay. Their studies show that it has 25% less error than the current one.
Consider these two experimental changes and how they effect sample size choices.
Q5: Calculate new effect sizes for these conditions.#
Hint: Refer to the bolded experimental changes above and adjust the errors then the effect sizes, keeping in mind the q1_change variable.
This can be done in two steps if needed.
Points: 5
q5_high_noise_effect_size = ...
q5_new_assay_effect_size = ...
print(f'Expected effect_size {q2_effect_size:0.2f}')
print(f'High noise effect_size {q5_high_noise_effect_size:0.2f}')
print(f'New assay effect_size {q5_new_assay_effect_size:0.2f}')
grader.check("q5_multiple_choices")
Use the power-plot below to answer the next question.
# Check many different nobs sizes
nobs_sizes = np.arange(1, 31)
names = ['Expected', 'High-Noise', 'New-Assay']
colors = 'krb'
effect_sizes = [q2_effect_size, q5_high_noise_effect_size, q5_new_assay_effect_size]
fig, ax = plt.subplots(1,1)
# Loop through each observation size
for name, color, effect in zip(names, colors, effect_sizes):
# Calculate the power across the range
powers = pg.power_ttest(d = effect,
n = nobs_sizes,
power = None,
alpha = alpha,
contrast = 'paired')
ax.plot(nobs_sizes, powers, label = name, color = color)
ax.legend(loc = 'lower right')
ax.set_ylabel('Power')
ax.set_xlabel('Sample Size')
Q6 Summary Questions#
Hint: Remember, the power level is 80%, so examine the nobs at 0.8 at the specified effect size to determine sufficient power or question being asked.
# Would an experiment that had nobs=15 be sufficiently powered
# to detect an effect under the expected assumption?
# 'yes' or 'no'
q6a = ...
# Would an experiment that had nobs=15 be sufficiently powered
# to detect an effect under the high-noise assumption?
# 'yes' or 'no'
q6b = ...
# How many fewer animals could be used if the new experiment was implemented
# vs. the expected/current one (using 80% power)?
# Hint: Use the power calculator. Round up.
q6c = ...
grader.check("q6")
grader.check_all()
Submission#
Check:
That all tables and graphs are rendered properly.
Code completes without errors by using
Restart & Run All.All checks pass.
Then save the notebook and the File -> Download -> Download .ipynb. Upload this file to BBLearn.