Graduate School and Research Center in Digital Sciences

In-memory caching for multi-query optimization of data-intensive scalable computing workloads

Michiardi, Pietro; Carra, Damiano; Migliorini, Sara

DARLI-AP: 3rd International workshop on Data Analytics solutions for Real-LIfe APplications, in conjunction with EDBT/ICDT 2019, March 26-29, 2019, Libson, Portugal

In modern large-scale distributed systems, analytics jobs submitted by various users often share similar work. Instead of optimizing jobs independently, multi-query optimization techniques can be employed to save a considerable amount of cluster resources. In this work, we introduce a novel method combining inmemory cache primitives and multi-query optimization, to improve the efficiency of data-intensive, scalable computing frameworks. By careful selection and exploitation of common (sub) expressions, while satisfying memory constraints, our method transforms a batch of queries into a new, more efficient one which avoids unnecessary recomputations. To find feasible and efficient execution plans, our method uses a cost-based optimization formulation akin to the multiple-choice knapsack problem. Experiments on a prototype implementation of our system show significant benefits of worksharing for TPC-DS workloads.

Document Bibtex

Title:In-memory caching for multi-query optimization of data-intensive scalable computing workloads
Type:Conference
Language:English
City:Lisbon
Country:PORTUGAL
Date:
Department:Data Science
Eurecom ref:5830
Copyright: CEUR
Bibtex: @inproceedings{EURECOM+5830, year = {2019}, title = {{I}n-memory caching for multi-query optimization of data-intensive scalable computing workloads}, author = {{M}ichiardi, {P}ietro and {C}arra, {D}amiano and {M}igliorini, {S}ara}, booktitle = {{DARLI}-{AP}: 3rd {I}nternational workshop on {D}ata {A}nalytics solutions for {R}eal-{LI}fe {AP}plications, in conjunction with {EDBT}/{ICDT} 2019, {M}arch 26-29, 2019, {L}ibson, {P}ortugal}, address = {{L}isbon, {PORTUGAL}}, month = {03}, url = {http://www.eurecom.fr/publication/5830} }
See also: