Information Theory

  • Since 1948, the year of publication of Shannon's landmark paper "A mathematical theory of communications", Information theory has paved the ground for the most important developments of today's information/communication world making it perhaps the most important theoretical tool to understand the fundamentals of information technologies.
  • Information theory studies the ultimate theoretical limits of source coding and data compression, of channel coding and reliable communications via channels, and provides the guidelines for the development of practical signal-processing and coding algorithms.
  • This course covers Information theory at an introductory level.
  • The practical implications of theoretical results presented are put in evidence through examples.
  • Various perspectives are given to understand every single theoretical results from a intuitive point of view, regardless of your background or study track.

Teaching and Learning Methods : Lectures, Exercise and  Lab sessions (group of 2 students)

Course Policies : Attendance to Lab session is mandatory (25% of final grade).


We follow the famous book by Cover and Thomas "Elements of Information Theory" which is available at the library.


Basic knowledge in statistics and probabilities. Basic MATLAB knowledge for the lab session - No C programming needed.


We follow the famous book by Cover and Thomas "Elements of Information Theory" which is available at the library.

  • The "toolbox" of the information theorist! Entropy, divergence and mutual information : Definitions, elementary relations, inequalities.
  • Lossless Source Compression : Source coding theorems, Huffman codes, universal data compression.
  • Channel coding ("How to communicate quickly and without error") : the channel coding theorem.
  • Gaussian channels : Capacity of discrete-time Gaussian channels, correlated noise, intersymbol interference.
  • Rate-distortion theory : Compression of Gaussian sources, vector quantization.
  • Topics in network  information theory:   The multi-access, the broadcast channel, the interference channel, MIMO (multiple input Multiple output) networks.

Learning outcomes:

The fundamental notions: What is information? How does one quantify it? How does one communicate it and how fast can it be? How does one compress it and how space (memory)  does that require? How can a network communicate or compress information efficiently? 

Nb hours: 42.00, 6hr exercise session, 6hr lab session.

Grading Policy: Lab reports (25%), Final Exam (75%.) 2 hour written exam - all documents authorized.