Lab

Lab#

Learning Objectives#

At the end of this learning activity you will be able to:

Estimate the effect size given a set of confidence intervals.
Calculate the effect_size, alpha, power, and sample_size when given 3 of the 4.
Interpret a power-plot of multiple experimental choices.
Calculate how changes in estimates of the experimental error impact sample size requirements.
Rigorously choose the appropriate experimental design for the best chance of success.

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import pingouin as pg
sns.set_style('whitegrid')

Step 1: Define the hypothesis#

For this lab we are going to investigate a similar metric. We will imagine replicating the analysis considered in Figure 3C. This analysis considers the different sub-values of the vigalence index. It shows that SK609 is improving attention by reducing the number of misses.

Copying the relevant part of the caption:

“Paired t-tests revealed that SK609 (4mg/kg; i.p.) specifically affected the selection of incorrect answers, significantly reducing the average number of executed misses compared to vehicle conditions (t(6))=3.27, p=0.017; 95% CI[1.02, 7.11]).”

Since this is a paired t-test we’ll use the same strategy as the walkthrough.

Step 2: Define success#

Q1: What is the average difference in misses between vehicle control and SK609 rodents?#

Hint: Calculate the center (average) of the confidence interval; the CI is bolded in the caption above.

Checked variables:

q1_change - A number indicating the average number of fewer missed prompts between treated and vehicle controls.

Total Points	5
Public Checks	1

Points: 5

q1_change = ...

print(f'On average, during an SK609 trial the rodent missed {q1_change} fewer prompts than vehicle controls.')

grader.check("q1_change")

Q2: Calculate the effect size.#

Hint: Use the change just defined in Q1.

Assume from our domain knowledge and inspection of the figure that there is an error of 3.5 misses.

Checked variables:

q2_effect_size - The cohen’s-D effect size for this experiment.

Total Points	5
Public Checks	1

Points: 5

error = 3.5

q2_effect_size = ...

print(f'The normalized effect_size of SK609 is {q2_effect_size:0.3f}')

grader.check("q2_effect_size")

Step 3: Define your tolerance for risk#

For this assignment consider that we want to have 80% chance of detecting a true effect and a 1% chance of falsely accepting an effect.

Checked variables:

power - The desired power.
alpha - The desired alpha threshold.

Total Points	5
Public Checks	2

Points: 5

power = ...
alpha = ...

grader.check("q3_tolerance")

Step 4: Define a budget#

In the figure caption we see that the paper used a nobs of 16 mice:

“Difference in VI measurements calculated against previous day vehicle performance in rats (n=16) showed SK609 improved sustained attention performance …”

Step 5: Calculate#

Q4: Calculate the minimum change detectable with 16 animals.#

Use alternative='two-sided' and as we do not know whether the number of misses is always increasing. Also, use contrast='paired' as we are doing a paired test we are compairing the same mice across each category.

Hint: Use the power-calculator, and then use that effect size to calculate the min_change.

Checked variables:

q4_effect_size - The minimum detectable effect size given N, alpha, and power.
q4_min_change - The minimum change in misses that we can detect.

Total Points	5
Public Checks	2

Points: 5

q4_effect_size = ...

print('The effect size is:', q4_effect_size)

# What is the minimum change that we can detect at this power?

q4_min_change = ...

print(f'with 16 animals, one could have detected as few as {q4_min_change:0.2f} min change.')

grader.check("q4_min_effect")

Step 6: Summarize#

Let’s propose a handful of different considerations for our experiment. As before, we’ll keep the power and alpha the same, but we’ll add the following experimental changes:

A grant reviewer has commented on the proposal and believes that your estimate of the error is too optimistic. They would like you to consider a scenario in which your error is 50% larger than the current estimate.
A new post-doc has come from another lab that has a different attention assay. Their studies show that it has 25% less error than the current one.

Consider these two experimental changes and how they effect sample size choices.

Q5: Calculate new effect sizes for these conditions.#

Hint: Refer to the bolded experimental changes above and adjust the errors then the effect sizes, keeping in mind the q1_change variable.

This can be done in two steps if needed.

Checked variables:

q5_high_noise_effect_size - The minimum detectable effect size in a high error consideration.
q5_new_assay_effect_size - The minimum detectable effect size in a new assay configuration.

Total Points	5
Public Checks	2

Points: 5

q5_high_noise_effect_size = ...
q5_new_assay_effect_size = ...

print(f'Expected effect_size {q2_effect_size:0.2f}')
print(f'High noise effect_size {q5_high_noise_effect_size:0.2f}')
print(f'New assay effect_size {q5_new_assay_effect_size:0.2f}')

grader.check("q5_multiple_choices")

# DO NOT EDIT


# Check many different sample sizes
sample_sizes = np.arange(1, 31)


names = ['Expected', 'High-Noise', 'New-Assay']
colors = 'krb'
effect_sizes = [q2_effect_size, q5_high_noise_effect_size, q5_new_assay_effect_size]

fig, ax = plt.subplots(1,1)

# Loop through each observation size
for name, color, effect in zip(names, colors, effect_sizes):
    # Calculate the power across the range
    powers = pg.power_ttest(d = effect,
                            n = sample_sizes,
                            power = None,
                            alpha = alpha,
                            contrast = 'paired')

    ax.plot(sample_sizes, powers, label = name, color = color)



ax.legend(loc = 'lower right')

ax.set_ylabel('Power')
ax.set_xlabel('Sample Size')

The plot above shows the expected power, the likelihood of detecting an effect, as a function of the sample size of the experiment. The black line represents the current experimental system. The red line indicates the high noise worst-case scenario presented by the grant reviewer. And the blue line shows the effect of the newly designed system. Use this to answer the questions below.

Q6 Summary Questions#

Assume an 80% power threshold for the following questions.

Checked variables:

q6a - Would an experiment that had sample size of 15 be sufficiently powered to detect an effect under the expected assumption? yes/no
q6b - Would an experiment that had sample size of 15 be sufficiently powered to detect an effect under the high-noise assumption? yes/no
q6c - How many fewer animals could be used if the new experiment was implemented vs. the expected/current one (using 80% power)? (number)

Total Points	5
Included Checks	3
Hidden Tests	3

Points: 5

# Would an experiment that had nobs=15 be sufficiently powered
# to detect an effect under the expected assumption?
# 'yes' or 'no'
q6a = ...

# Would an experiment that had nobs=15 be sufficiently powered
# to detect an effect under the high-noise assumption?
# 'yes' or 'no'
q6b = ...

# How many fewer animals could be used if the new experiment was implemented
# vs. the expected/current one (using 80% power)?


q6c = ...

grader.check("q6")

grader.check_all()

Submission#

Check:

That all tables and graphs are rendered properly.
Code completes without errors by using Restart & Run All.
All checks pass.

Then save the notebook and the File -> Download -> Download .ipynb. Upload this file to BBLearn.

Lab

Contents

Lab#

Learning Objectives#

Step 1: Define the hypothesis#

Step 2: Define success#

Q1: What is the average difference in misses between vehicle control and SK609 rodents?#

Q2: Calculate the effect size.#

Step 3: Define your tolerance for risk#

Step 4: Define a budget#

Step 5: Calculate#

Q4: Calculate the minimum change detectable with 16 animals.#

Step 6: Summarize#

Q5: Calculate new effect sizes for these conditions.#

Q6 Summary Questions#

Submission#