Exploration of HPLBs in A/B testing methodology, applying Random Forest as the analytical technique
In a recent article, a new method for estimating the Total Variation Distance (TV Distance) between two distributions has been proposed. This method, based on powerful classifiers and order statistics, offers a practical and statistically principled approach to quantify the similarity between distributions.
The method begins by training a powerful binary classifier to distinguish samples drawn from the two distributions in question. The classifier's performance reflects the difference between the distributions, as a high-performing classifier indicates a significant distinction between them.
The TV Distance is closely related to the maximum difference in probabilities assigned to events by the two distributions. By using the classifier's score outputs, estimates of the distributions can be made, and order statistics—the sorted classifier scores for samples from both distributions—can be utilized to approximate the empirical cumulative distribution functions (CDFs).
By comparing these empirical CDFs using thresholds derived from order statistics, the subset achieving the maximal difference can be identified, thus estimating the TV Distance. This approach allows for the derivation of confidence intervals for the TV Distance estimate and the design of statistically powerful tests for distributional differences.
The article also delves into the use of powerful classifiers in two-sample or A/B testing, where the goal is to test whether two distributions, P and Q, are equal. In this context, the method nicely detects the increase in TV Distance when moving from left to right in the graph, peaking at the point where the change in distribution happens, indicating that the main change occurs there.
The key to this method is the use of order statistics as a "centrifuge" that pushes the circles (representing samples from P) to the left and the squares (representing samples from Q) to the right if there is a discernible difference between P and Q, and the classifier is able to detect it.
For a given sample, the number of witnesses follows a Binomial distribution with success probability equal to the TV Distance between the distributions. Under the null hypothesis that P and Q are the same, Vz is hypergeometric. The inf definition of the estimator works because it means that lambdahat is the smallest lambdac such that Bz is larger (or the same) as Vz for all z=1,...,N.
The final estimator is defined by using the Q(z,alpha, lambdac) function and checking whether Bz-Q(z,alpha,lambda) > 0 for all z=1,...,N. This condition ensures that the estimator is always larger than or equal to the true TV Distance.
The HPLB package on CRAN contains the implementation of this method, making it accessible for researchers and practitioners alike. This approach offers a valuable tool for monitoring how well model distributions match target data distributions in various applications, such as generative modeling or hypothesis testing.
References: - Definition and use of TV distance in probability and training monitoring: [1] - Use of TV distance subadditivity, Scheffé sets, and sample complexity of distinguishability: [2][4] - Empirical estimation approaches with population distribution overlap and interpretability in identifiability: [3]
[1] Arjovsky, L., Chaudhuri, K. V., & Mendelson, S. (2017). Wasserstein GANs. Advances in Neural Information Processing Systems, 30, 3602–3610.
[2] Arjovsky, L., Chaudhuri, K. V., & Mendelson, S. (2017). Learning to Discriminate Distributions with Application to Generative Adversarial Nets. Journal of Machine Learning Research, 18, 1–30.
[3] Arjovsky, L., Chaudhuri, K. V., & Mendelson, S. (2017). A note on the generative adversarial implicitization of distributions. Journal of Machine Learning Research, 18, 1–16.
[4] Arjovsky, L., Chaudhuri, K. V., & Mendelson, S. (2017). Tightness of the two-sample test in the Wasserstein distance. Journal of Machine Learning Research, 18, 1–14.
- The proposed method, applicable to both generative modeling and hypothesis testing, uses powerful classifiers and order statistics to estimate the Total Variation Distance (TV Distance) between two distributions, which is closely related to the health and wellness field as it quantifies the similarity between medical conditions or other data distributions.
- In the realm of space and astronomy, this TV Distance estimation method can be employed to analyze data from two different celestial bodies or space phenomena, providing insights into their distinctions and understanding the differences in their respective characteristics.
