Automated identification of flaky builds using knowledge graphs

DevOps’ mindset encourages the use of Continuous Integration to quickly identify regressions introduced on code changes by running a set of checks (i.e., a build) on each commit or pull-request. This approach leads to a very large number of builds every day: in the order of tens of thousands at Amadeus. Builds can either fail because of a regression which has been introduced (a legitimate failure) or because of infrastructure instabilities (an illegitimate failure). Investigating the cause of failure is generally a manual process done by developers. If the failure is due to the infrastructure, the developers have wasted time investigating a failure they cannot directly fix. Therefore, there is a strong interest in automating identification of builds broken by infrastructure instabilities. Considering the large amount of heterogeneous data available about a build, we propose a method based on knowledge graph embeddings to automatically identify whether the failure is legitimate or not. We first propose an ontology to model the continuous integration process widely used in software engineering. To this end, we follow the best practices from the knowledge engineering community using competency questions and we also re-use well known ontologies. Next, we propose a method to build a labelled dataset and we experiment with knowledge graph embedding techniques. We empirically demonstrate that we can predict whether a failure is due to the infrastructure or not with an accuracy of 94% on a balanced validation dataset.