Graduate School and Research Center in Digital Sciences

Static and dynamic autopsy of deep networks

Lorieul, Titouan; Ghorra, Antoine; Merialdo, Bernard

CBMI 2016, 14th International Workshop on Content-based Multimedia Indexing, June 15-17, 2016, Bucharest, Romania

Although deep learning has been a major breakthrough in the recent years, Deep Neural Networks (DNNs) are still the subject of intense research, and many issues remain on how to use them efficiently. In particular, training a Deep Network remains a difficult process, which requires extensive computation, and for which very precise care has to be taken to avoid overfitting, a high risk because of the extremely large number of parameters. The purpose of our work is to perform an autopsy of pre-trained Deep Networks, with the objective of collecting information about the values of the various parameters, and their possible relations and correlations. The motivation is that some of these observations could be later used as a priori knowledge to facilitate the training of new networks, by guiding the exploration of the parameter space into more probable areas. In this paper, we first present a static analysis of the AlexNet Deep Network by computing various statistics on the existing parameter values. Then, we perform a dynamic analysis by measuring the effect of certain modifications of those values on the performance of the network. For example, we show that quantizing the values of the parameters to a small adequate set of values leads to similar performance as the original network. These results suggest that pursuing such studies could lead to the design of improved training procedures for Deep Networks.

Document Doi Bibtex

Title:Static and dynamic autopsy of deep networks
Department:Data Science
Eurecom ref:4918
Copyright: © 2016 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Bibtex: @inproceedings{EURECOM+4918, doi = {}, year = {2016}, title = {{S}tatic and dynamic autopsy of deep networks}, author = {{L}orieul, {T}itouan and {G}horra, {A}ntoine and {M}erialdo, {B}ernard}, booktitle = {{CBMI} 2016, 14th {I}nternational {W}orkshop on {C}ontent-based {M}ultimedia {I}ndexing, {J}une 15-17, 2016, {B}ucharest, {R}omania}, address = {{B}ucharest, {ROMANIA}}, month = {06}, url = {} }
See also: