A/B testing is the gold standard of causal inference because it allows us to make valid causal statements under minimal assumptions, thanks to randomization. In fact, by randomly assigning a treatment (a medicine, advertisement, product,…), we can compare the result of interest (an illness, company income, customer satisfaction,…) in subjects (patients, users, clients,…) and attribute the average difference in the results to the causal effect of the treatment.
However, in many environments, it is it is not possible to randomize treatment, whether for ethical, legal or practical reasons. A common online setup is on-demand features, such as subscriptions or premium memberships. Other configurations include features by which we cannot discriminate customers, such as insurance contracts, or features that are so deeply encoded that an experiment might not be worthwhile. Can we still make valid causal inferences in those settings?
The answer is yes, thanks to instrumental variables and the corresponding experimental design called stimulus design. In many of the environments mentioned above, we cannot assign treatment, but we can encourage customers to take it. For example, we may offer a subscription discount or we may change the order in which options are presented. While clients have the final say on how to take the treatment, we can still estimate a causal effect of the treatment. Let’s see how.
In the rest of the article we will use a toy example. Suppose we were a product company starting a week Newsletter to promote product and feature updates. We would like to know if the newsletter is worth the effort and if it ultimately manages to increase sales. Unfortunately, we cannot run a standard A/B test as we cannot force customers to subscribe to the newsletter. Does this mean that we cannot evaluate the newsletter? Not quite.
Suppose we have also performed an A/B test on a new notification in our mobile application that promotes the newsletter. A random sample of our clients has…
