Graduate School and Research Center in Digital Sciences

HFSP: Bringing size-based scheduling to Hadoop

Pastorelli, Mario; Carra, Damiano; Dell'Amico, Matteo; Michiardi, Pietro

IEEE Transactions on Cloud Computing, Vol.5, N°1, January-March 2017; ISSN: 2168-7161

Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and nearoptimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors.

Document Doi Bibtex

Title:HFSP: Bringing size-based scheduling to Hadoop
Keywords:MapReduce, Performance, Data Analysis, Scheduling
Department:Data Science
Eurecom ref:4489
Copyright: © 2017 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Bibtex: @article{EURECOM+4489, doi = {}, year = {2017}, month = {01}, title = {{HFSP}: {B}ringing size-based scheduling to {H}adoop}, author = {{P}astorelli, {M}ario and {C}arra, {D}amiano and {D}ell'{A}mico, {M}atteo and {M}ichiardi, {P}ietro}, journal = {{IEEE} {T}ransactions on {C}loud {C}omputing, {V}ol.5, {N}°1, {J}anuary-{M}arch 2017; {ISSN}: 2168-7161}, url = {} }
See also: