Graduate School and Research Center in Digital Sciences

Cache-based multi-query optimization for data-intensive scalable computing frameworks

Michiardi, Pietro; Carra, Damiano; Migliorini, Sara

Submitted to ArXiv on May 22nd, 2018

In modern large-scale distributed systems, analytics jobs submitted by various users often share similar work, for example scanning and processing the same subset of data. Instead of optimizing jobs independently, which may result in redundant and wasteful processing, multi-query optimization techniques can be employed to save a considerable amount of cluster resources. In this work, we introduce a novel method combining in-memory cache primitives and multi-query optimization, to improve the efficiency of data-intensive, scalable computing frameworks. By careful selection and exploitation of common (sub)expressions, while satisfying memory constraints, our method transforms a batch of queries into a new, more efficient one which avoids unnecessary recomputations. To find feasible and efficient execution plans, our method uses a cost-based optimization formulation akin to the multiple-choice knapsack problem. Extensive experiments on a prototype implementation of our system show significant benefits of worksharing for both TPC-DS workloads and detailed micro-benchmarks.

Arxiv Bibtex

Title:Cache-based multi-query optimization for data-intensive scalable computing frameworks
Type:Conference
Language:English
City:
Date:
Department:Data Science
Eurecom ref:5560
Copyright: © EURECOM. Personal use of this material is permitted. The definitive version of this paper was published in Submitted to ArXiv on May 22nd, 2018 and is available at :
Bibtex: @inproceedings{EURECOM+5560, year = {2018}, title = {{C}ache-based multi-query optimization for data-intensive scalable computing frameworks}, author = {{M}ichiardi, {P}ietro and {C}arra, {D}amiano and {M}igliorini, {S}ara}, booktitle = {{S}ubmitted to {A}r{X}iv on {M}ay 22nd, 2018}, address = {}, month = {05}, url = {http://www.eurecom.fr/publication/5560} }
See also: