BEERR: Bench of embedded system experiments for reproducible research

Reproducing experiments is a key component to further research and knowledge. Testbeds provide a controlled and configurable environment in which experiments can be conducted in a repeatable and observable manner. In the field of system security, and binary analysis, several challenges hinder reproducible research, in particular when code is interacting tightly with low level hardware and physical devices. In those conditions, dynamic analysis techniques often require the physical device to correctly complete (hardware-in-the-loop). In recent years many rehosting techniques have been developed and evaluating their respective performance requires to compare them with an hardware-in-the-loop evaluation. However, it is challenging to share, acquire or maintain the original devices. In this paper, we tackle this problem by proposing a new infrastructure, and online service called “Bench of Embedded system Experiments for Reproducible Research” (BEERR). It aims to both make physical devices available remotely and facilitate the setup and reproduction of published experiments.