Group for Research in Decision Analysis


Using BIRCH to Compute Approximate Rank Statistics on Massive Datasets


The BIRCH algorithm (Balanced Iterative Reducing and Clustering Hierarchies) handles massive dataset by reading the data file only once, clustering the data as it is read, and retaining only a few clustering features to summarize the data read so far. Using BIRCH allows to analyze datasets that are too large to fit in the computer main memory. We propose estimates of Spearman's \(\rho\) and Kendall's \(\tau\) that are calculated from a BIRCH output and assess their performance through Monte Carlo studies. The numerical results show that the BIRCH-based estimates can achieve the same efficiency as the usual estimates of \(\rho\) and \(\tau\) while using only a fraction of the memory otherwise required.

, 23 pages