Big Data and the Divide & Recombine (D&R) Statistical Methodology in AutoRegression Integrated Moving Average (ARIMA) Modeling
Jeremy Troisi '08
Big Data is highly touted by big industry, but the skill set required to handle such complicated problems is both very advanced and diverse. My research group has been investigating optimal Divide & Recombine (D&R) statistical methods for various types of data. Personally, I am seeking to find the optimal application of D&R methods to massive univariate time series data, such as a stock price recorded over time, in AutoRegressive Integrated Moving Average (ARIMA) Model estimation. In this talk, I will discuss the long standing current ARIMA model estimation method, how it works, and why it is intractable for Big Data, i.e. the purpose of my research. I will then discuss how our method both succeeds in ARIMA model estimation in the Big Data framework, while the current method does not, and how our method is vastly more efficient computationally.