Skip to content

Instantly share code, notes, and snippets.

@eustin
Created May 15, 2023 22:55
Show Gist options
  • Save eustin/538acc7a908171112525b80fb53a735d to your computer and use it in GitHub Desktop.
Save eustin/538acc7a908171112525b80fb53a735d to your computer and use it in GitHub Desktop.
stat_sig_random_14
NUM_CONTROL_USERS = 1_000_000
NUM_CONVERTING_CONTROL_USERS = 26_000
NUM_VARIANT_USERS = 1_000_000
NUM_CONVERTING_VARIANT_USERS = 26_400
# create our arrays of users
control_users = np.zeros(NUM_CONTROL_USERS)
control_users[:NUM_CONVERTING_CONTROL_USERS] = 1.0
control_conversion_rate = control_users.mean()
variant_users = np.zeros(NUM_VARIANT_USERS)
variant_users[:NUM_CONVERTING_VARIANT_USERS] = 1.0
variant_conversion_rate = variant_users.mean()
# calculate our experiment result
observed_diff_in_rates = variant_conversion_rate - control_conversion_rate
# chuck our users into a bag
all_users = np.hstack([control_users, variant_users])
print(f"control conversion rate = {control_conversion_rate:.2%}")
print(f"variant conversion rate = {variant_conversion_rate:.2%}")
print(f"observed absolute difference in conversion rates = {observed_diff_in_rates:.2%}")
print(f"observed relative difference in conversion rates = {observed_diff_in_rates / control_conversion_rate:.2%}")
#> control conversion rate = 2.60%
#> variant conversion rate = 2.64%
#> observed absolute difference in conversion rates = 0.04%
#> observed relative difference in conversion rates = 1.54%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment