A and B Won’t Always be Different

Effect size is a thing right?

Sometimes, there actually isn’t a difference worth anything…

Unfortunately, like so many other things that reach buzzword status, the use of testing in business seems to be almost symbolic in application rather than practical. As long as you can put the words “A/B test” or “Multivariate test” on a slide associated with whatever was done you can label yourself as data driven. Planning, actually being able to link what you are testing to a business outcome, follow some kind of a process, no matter how simple or even understanding some basic concepts like p-values, effect size and confidence intervals, that hardly matters.