Graduate School and Research Center in Digital Sciences

A data-driven approach to dynamically adjust resource alocation for compute clusters

Pace, Francesco; Milios, Dimitrios; Carra, Damiano; Venzano, Daniele; Michiardi, Pietro

Submitted to ArXiV on July, 1st, 2018

Nowadays, data-centers are largely under-utilized because resource allocation is based on reservation mechanisms which ignore actual resource utilization. Indeed, it is common to reserve resources for peak demand, which may occur only for a small portion of the application life time. As a consequence, cluster resources often go under-utilized.  In this work, we propose a mechanism that improves cluster utilization, thus decreasing the average turnaround time, while preventing application failures due to contention in accessing finite resources such as RAM. Our approach monitors resource utilization and employs a data-driven approach to resource demand forecasting, featuring quantification of uncertainty in the predictions. Using demand forecast and its confidence, our mechanism modulates cluster resources assigned to running applications, and reduces the turnaround time by more than one order of magnitude while keeping application failures under control. Thus, tenants enjoy a responsive system and providers benefit from an efficient cluster utilization.

Arxiv Bibtex

Title:A data-driven approach to dynamically adjust resource alocation for compute clusters
Type:Conference
Language:English
City:
Date:
Department:Data Science
Eurecom ref:5684
Copyright: © EURECOM. Personal use of this material is permitted. The definitive version of this paper was published in Submitted to ArXiV on July, 1st, 2018 and is available at :
Bibtex: @inproceedings{EURECOM+5684, year = {2018}, title = {{A} data-driven approach to dynamically adjust resource alocation for compute clusters}, author = {{P}ace, {F}rancesco and {M}ilios, {D}imitrios and {C}arra, {D}amiano and {V}enzano, {D}aniele and {M}ichiardi, {P}ietro}, booktitle = {{S}ubmitted to {A}r{X}i{V} on {J}uly, 1st, 2018}, address = {}, month = {07}, url = {http://www.eurecom.fr/publication/5684} }
See also: