关键词：优化数据并行计算；查询优化器；高效执行
摘 要：Performant execution of data-parallel jobs needs good executionplans. Certain properties of the code, the data, and the interaction between them are crucial to generate these plans. Yet, these propertiesare dicult to estimate due to the highly distributed nature of theseframeworks, the freedom that allows users to specify arbitrary code，as operations on the data, and since jobs in modern clusters haveevolved beyond single map and reduce phases to logical graphs ofoperations. Using xed apriori estimates of these properties to chooseexecution plans, as modern systems do, leads to poor performance inseveral instances. We present RoPE, a rst step towards re-optimizingdata-parallel jobs. RoPE collects certain code and data properties bypiggybacking on job execution. It adapts execution plans by feedingthese properties to a query optimizer. We show how this improves thefuture invocations of the same (and similar) jobs and characterize thescenarios of benet. Experiments on Bing's production clusters showup to 2 improvement across response time for production jobs at the75th percentile while using 1:5 fewer resources.