Too Long; Didn't Read
P-Hacking is a term used to describe the scientific manipulation of data to get the desired P value. The more comparisons you make the more likely you are to see a false positive. You can avoid that by limiting the comparisons before running the experiment and limiting them. Power analysis is used to estimate the minimum sample size (number of users per cohort) needed for the experiment to get to the desired power. Confidence intervals (CI) is the range the metric value lies in. Usually we use the 95% CI. A CI of -10 to +10 indicates that the mean of the metric.