Biostatistics Seminar Series, November 1, 2023

Evaluating the treatment effects has become an important topic for many applications. However, most existing literature focuses mainly on average treatment effects. When the individual effects are heavy-tailed or have outlier values, not only may the average effect not be appropriate for summarizing treatment effects, but also the conventional inference for it can be sensitive and possibly invalid due to poor large-sample approximations. In this paper we focus on quantiles of individual treatment effects, which can be more robust in the presence of extreme individual effects. Moreover, our inference for them is purely randomization-based, avoiding any distributional assumptions on the units. We first consider completely randomized experiments, where we propose valid p-values for testing null hypothesis on effect quantiles. These p-values can be efficiently computed by utilizing distribution-free rank-based statistics. We then extend to stratified randomized experiments, where we show the computation of valid p-values for null hypotheses on effect quantiles can be transformed into instances of the multiple-choice knapsack problem, which can be efficiently solved exactly or slightly conservatively. We finally consider matched observational studies and propose sensitivity analysis to investigate to what extent our inference on effect quantiles is robust to unmeasured confounding. The proposed randomization inference and sensitivity analysis are simultaneously valid for all quantiles of individual effects, noting that the analysis for the maximum (or minimum) individual effect coincides with the conventional analysis assuming constant treatment effects.