Spring 2019: Deep Learning and Its Applications to Signal and Image Processing and Analysis, 361-21120

Tammy Riklin Raviv

Time Wednesday, 14:00-17:00
Location Building 90, Room 224

About the course

The class will cover a diverse set of topics in signal processing, image analysis and computer vision and various Neural Network architectures. It will be an interactive course where we will discuss interesting topics on demand and latest research buzz. The goal of the class is to learn about different domains of the field, understand, identify and analyze the main challenges, what works and what doesn't, as well as to identify interesting new directions for future research.
Starting from the last week of November (the second month of the semester) we will read 1 to 4 papers every week. The papers will be presented by the students in a seminar format. The success of the discussion in class will thus be due to how prepared the students come to class. Each student is expected to read at least one paper for each class (and prepare a question or a comment for a discussion). In addition, each student will present a paper in one of the classes. There will be also talks by invited speakers and a lab. class (April 27, room 330, building 33) on Tensor Flow. The Last class will be dedicated to final project presentation (about ten minutes for each project). Details on the final project will be published soon. The final project does not have to be necessarily related to the paper presented in class.

Schedule, links and lecture notes

February 27, 2019   Course overview

March 6, 2019     Optimination, Stochastic gradient descent, Backpropagation, Regularization, Loss functions

March 13, 2019     Regularization for Deep Learning, CNN

March 20, 2019     Happy Purim - No Classes

March 27, 2019   Tensor Flow (by Assaf Arbelle)
                         location: Electrical Engineering Computer Laboratory, room 330, building 33

April 3, 2019   Guest Lecture; Object detection and classification

                       Learning to Sample
                       Authors: Oren Dovrat, Itai Lang, Shai Avidan
                       Guest lecture: Itai Lang, Tel-Aviv University

                       Deep Residual Learning for Image Recognition
                       Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
                       Pesenters: Natalie Lang and Tomer Malach

                       You Only Look Once: Unified, Real-Time Object Detection
                       Authors: Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad
                       Pesenter: Senyang Zhang

April 10, 2019   Guest Lecture: Roy Fahn, MATLAB and Simulink
                       Deep Learning with Matlab

May 1, 2019   Face Recognition and Semantic Segmentation

                       U-Net: Convolutional Networks for Biomedical Image Segmentation
                       Authors: Olaf Ronneberger, Philipp Fischer, Thomas Brox
                       Presenters: Ofir Krauz and Gal Morgenshtern

                       FaceNet: A Unified Embedding for Face Recognition and Clustering
                       Florian Schroff, Dmitry Kalenichenko, James Philbin
                       Presenters: Marina Eni and Matan Sivan

                       Fully Convolutional Networks for Semantic Segmentation
                       Authors: Jonathan Long, Evan Shelhamer, Trevor Darrell
                       Presenters: Yahel Salomon and Netanel Biton

May 8, 2019   Memorial day, no classes

May 15, 2019   Style transfer, GANs

                       A Neural Algorithm of Artistic Style
                       Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
                       Presenters: Ido Michealovich and Ido Moskovich

                       Generative Adversarial Networks
                       Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio
                       Presenters: Mor Elmakies and Adam Sofer

                       Image-to-Image Translation with Conditional Adversarial Networks
                         P. Isola, JY. Zhu, T. Zhou, A.A. Efros
                         Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
                         Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros
                         Presenters: Odded Geffen and David Koblev

May 22, 2019 (Wed. schedule)
                       Guest Lecture - Dr. Aharon Bar-Hillel

                       Playing Atari with Deep Reinforcement Learning
                       Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
                       Presenters: Teodor Linnik and Yoav Amiel

                       Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
                       Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, Dhruv Batra
                       Presenters: Mor Avi-Aharon and Yael Ben-Gigi

May 22, 2019   AE and more

                       Semantic Image Synthesis with Spatially-Adaptive Normalization
                        Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
                       Presenters: Mira Barshai and Valeria Mordoh

                       Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
                        Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

                       Auto-Encoding Variational Bayes
                       Diederik P Kingma, Max Welling
                       Presenters: Roy Shaul, Itamar David and Tal Goldfryd

                       Adversarial Autoencoders
                        Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey
                       Presenters: Tom Dadon, Uria Levi, Matan Hofman


May 29, 2019   Student Day - No Classes

June 5, 2019   
                       Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
                      Jure Zbontar, Yann LeCun
                       Presenter: Wenpeng Fu

                       Spatial Transformer Networks
                       Authors: Max Jaderberg, Karen Simonyan, Andrew Zisserman and Koray Kavukcuoglu
                       Presenter: Shaul Shmilovich and Yuval Zedek

                       Optical Flow Estimation using a Spatial Pyramid Network
                       Anurag Ranjan and Michael J. Black
                       Presenters: Omer Amar and Tidhar Lambez

June 12, 2019   RNN

                       Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks
                       Presenters: Moti-Ben-Laish and Yoram Furth

                       Unsupervised Learning of Video Representations using LSTMs
                       Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov
                       Presenters: Dor Livne and Moshe Koziashvili

                       Object segmentation in videos via LSTM and adversarial networks
                       Guest Lecture: Ran Shadmi

                       Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
                       Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, Anton van den Hengel

June 19, 2019   ICA and Information Maximization Approach to Unsupervised Learning in Feedforward and Recurrent Neural Networks
                       Guest Lecture , Prof. Oren Shriki, Cognitive and Brain Sciences at Ben-Gurion University

                       Guest Lecture: Assaf Arbelle

June 26, 2019 - Semester break   Final project presentation day
Links

Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press Online

Convolutional Neural Networks for Visual Recognition, Stanford Course cs231n, see also Lectures

Neural Networks for Machine Learning, Coursera course by Jeff Hinton

ConvNet Arithmetic

ConvNet Demo by Stanford

Tensor flow course

Tensor flow official site

Tensor flow code

Tensor flow playground

MATLAB for Deep Learning

Free Training – Deep Learning with MATLAB – No MATLAB license required

Student Presentations

Each student will need to present a paper in class. Please refer to the list of papers below and note time schedule to be presented. Ideas for papers not in the list need approval. The presentation should be clear and practiced and the student should read the assigned paper and related work in enough detail to be able to lead a discussion and answer questions. A presentation should be roughly 30 minutes long including time for questions and a discussion. Typically this is about 25 to 30 slides. In the presentation provide the citation to the paper you present and to any other related work you reference.
Deadline:The presentation should be handed in one day before the class (or before if you want feedback).
Structure of presentation: High-level overview with contributions, Main motivation, Clear statement of the problem, Overview of the technical approach, Strengths/weaknesses of the approach, Overview of the experimental evaluation, Strengths/weaknesses of evaluation, Discussion: future directions, links to other works.

Further reading

Spiking-YOLO: Spiking Neural Network for Real-time Object Detection

Authors: Seijoon Kim,Seongsik Park,Byunggook Na,Sungroh Yoon

Simultaneous Deep Transfer Across Domains and Tasks
Authors:Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko

Densely Connected Convolutional Networks
Authors:Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy

Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal and Zoubin Ghahramani

Rich feature hierarchies for accurate object detection and semantic segmentation
L-C. Chen, G. Papandreou, I. Kokkinos, Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik

U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., Brox, T.

Spectral Representations for Convolutional Neural Networks
Oren Rippel, Jasper Snoek, Ryan P. Adams

You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Very Deep Convolutional Networks for large-scale image recognition
Karen Simonyan & Andrew Zisserman

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A. Alemi

Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra

A First Look at Music Composition using LSTM Recurrent Neural Networks
Douglas Eck and Jurgen Schmidhuber

Generative Adversarial Networks
Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Multimodal learning with Deep Boltzman Machines
Nitish Srivastava and Ruslan Salakhutdinov

Auto-Encoding Variational Bayes
Diederik P Kingma,Max Welling

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Building High-level Features Using Large Scale Unsupervised Learning
Quoc V. Le, Marc Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado ,Jeff Dean, Andrew Y. Ng

Where Are They Looking?
Adria Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Learning to Compare Image Patches via Convolutional Neural Networks
Sergey Zagoruyko, Nikos Komodakis

Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Jure Zbontar, Yann LeCun

Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Pim Moeskops, Max A. Viergever, Adrienne M. Mendrik, Linda S. de Vries, Manon J. N. L. Benders, and Ivana Isgum

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero,Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao,Stuart Cook, Declan ORegan
and Daniel Rueckert


Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton


Monocular Object Instance Segmentation and Depth Ordering with CNNs
Ziyu Zhang, Alex Schwing, Sanja Fidler, Raquel Urtasun

Instance-Level Segmentation with Deep Densely Connected MRFs
Ziyu Zhang, Sanja Fidler, Raquel Urtasun

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor,Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala


The Multiverse Loss for Robust Transfer Learning
Etai Littwin, Lior Wolf

Simultaneous Deep Transfer Across Domains and Tasks
Eric Tzeng, Judy Hoffman, Trevor Darrell

Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

Tutorial: Bayesian Reasoning and Deep Learning
Shakir Mohamed

Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

Explaining and Harnessing Adversarial Examples
Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

Recurrent Models of Visual Attention
Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu

Generative Adversarial Nets
Ian J. Goodfellow,Jean Pouget-Abadie,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair,Aaron Courville, Yoshua Bengio

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, Geoffrey E. Hinton

DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, Lior Wolf

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
Oren Z. Kraus, Lei Jimmy Ba, Brendan Frey

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook, Declan ORegan, and Daniel Rueckert We Are Humor Beings: Understanding and Predicting Visual Humor
Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

A First Look at Music Composition using LSTM Recurrent Neural Networks
Douglas Eck, Jurgen Schmidhuber


Homework assignments

Homework assignments should be submitted via moodle using a single zip file: "YourName-XX.zip" where XX is the assignment number.

Assignment #1 due to April 28, 2019