You are here

Managing the Forecast Factory

TitleManaging the Forecast Factory
Publication TypePeer-reviewed Conference Paper
Year of Publication2006
Conference Name22nd International Conference on Data Engineering Workshops (ICDEW'06)
AuthorsBright L, Maier D, Howe B
Conference LocationAtlanta, GA, USA

The CORIE forecast factory consists of a set of data product generation runs that are executed daily on dedicated local resources. The goal is to maximize productivity and resource utilization while still ensuring timely completion of all forecasts. Many existing workflow management systems address low-level workflow specification and execution challenges, but do not directly address the high-level challenges posed by large-scale data product factories. In this paper we discuss several specific challenges to managing the CORIE forecast factory including planning and scheduling, improving data flow, and analyzing log data, and point out their analogs in the "physical" manufacturing world. We present solutions we have implemented to address these challenges, and present experimental results that show the benefits of these solutions.