Syllabus

• The course is a basic course in Information theory and Machine Learning.
  

• The course is theoretic based on probability, optimization and algebra.
  

• It has a major impact in communication, signal processing and computer engineering. Here is a movie on Shannon, who invented the mathematical tool information diagram learned in the class.

Prerequisite

You should have seen some probability at the level of introduction to stochastic processes or equivalent. For instance, you should be familiar with terms such as i.i.d. random variables, expectation and Gaussian random variables.

Grades

90 - Final exam. 10 - Homework. Homework will be given throughout the semester.

Outline

• Information measures: entropy, mutual information, Kullback-Libler divergence, data processing inequality.

• Basic concepts in convex optimization: convex set, convex function, Jensen’s inequality.

• Variable lenth source coding: Huffman code, Kraft inequality.

• Typical sequences: Weak and strong typicality, joint typicality

• Lossless source coding (data compression): block coding, data compression using typical sets.

• Random codes for transmission via noisy medium: channel capacity, capacity computation, Achieving capacity through random coding.shannon diagram

• Joint source-channel coding: data processing, separation theorem.

• *Basic principles in Machine learning

• *(Deep) Neural networks

• *usefulness of information measures in Machine learning

• Optional: Introduction to Random coding: LDPC and Turbo codes.

• Optional: Introduction to network coding.

Textbook

The course will follow only one textbook, which is fun and easy to read:

• Textbook: “Elements of Information Theory”, by Cover and Thomas, 1st or 2nd Edition, New York: Wiley, 2006.