Graduate School and Research Center in Digital Sciences

Sequeval: An offline evaluation framework for sequence-based recommender systems

Monti, Diego; Palumbo, Enrico; Rizzo, Giuseppe; Morisio, Maurizio

Information 2019, 10(5), 174, Special Issue Modern Recommender Systems: Approaches, Challenges and Applications

Recommender systems have gained a lot of popularity due to their large adoption in various industries such as entertainment and tourism. Numerous research efforts have focused on formulating and advancing state-of-the-art of systems that recommend the right set of items to the right person. However, these recommender systems are hard to compare since the published evaluation results are computed on diverse datasets and obtained using different methodologies. In this paper, we researched and prototyped an offline evaluation framework called Sequeval that is designed to evaluate recommender systems capable of suggesting sequences of items. We provide a mathematical definition of such sequence-based recommenders, a methodology for performing their evaluation, and the implementation details of eight metrics. We report the lessons learned using this framework for assessing the performance of four baselines and two recommender systems based on Conditional Random Fields (CRF) and Recurrent Neural Networks (RNN), considering two different datasets. Sequeval is publicly available and it aims to become a focal point for researchers and practitioners when experimenting with sequence-based recommender systems, providing comparable and objective evaluation results.

Document Doi Bibtex

Title:Sequeval: An offline evaluation framework for sequence-based recommender systems
Keywords:evaluation framework; offline evaluation; sequence; sequence-based recommender systems; recommender systems; metrics
Type:Journal
Language:English
City:
Date:
Department:Data Science
Eurecom ref:5876
Copyright: MDPI
Bibtex: @article{EURECOM+5876, doi = {http://doi.org/10.3390/info10050174}, year = {2019}, month = {05}, title = {{S}equeval: {A}n offline evaluation framework for sequence-based recommender systems}, author = {{M}onti, {D}iego and {P}alumbo, {E}nrico and {R}izzo, {G}iuseppe and {M}orisio, {M}aurizio}, journal = {{I}nformation 2019, 10(5), 174, {S}pecial {I}ssue {M}odern {R}ecommender {S}ystems: {A}pproaches, {C}hallenges and {A}pplications}, url = {http://www.eurecom.fr/publication/5876} }
See also: