Spring 2020: Deep Learning and Its Applications to Signal and Image Processing and Analysis, 361-21120

Tammy Riklin Raviv

Time Wednesday, 14:00-17:00
Location Building 90, Room 135


About the course

The class will cover a diverse set of topics in signal processing, image analysis and computer vision and various Neural Network architectures. It will be an interactive course where we will discuss interesting topics on demand and latest research buzz. The goal of the class is to learn about different domains of the field, understand, identify and analyze the main challenges, what works and what doesn't, as well as to identify interesting new directions for future research.
Right after Passover, we will read about three papers every week. The papers will be presented by the students in a seminar format. The success of the discussion in class will thus be due to how prepared the students come to class. Each student is expected to read at least one paper for each class (and prepare a question or a comment for a discussion). In addition, each student will present a paper in one of the classes. There will be also talks by invited speakers and a lab. class on Tensor Flow. The Last class will be dedicated to final project presentation (about ten minutes for each project). Details on the final project will be published soon. The final project does not have to be necessarily related to the paper presented in class.

Schedule, links and lecture notes

March 11, 2020   Course overview

March 18, 2020     Optimination, Stochastic gradient descent, Backpropagation, Regularization, Loss functions

March 25, 2020     Regularization for Deep Learning, CNN

April 1, 2020     Deep Learning tools in Matlab - Roy Fahn

April 2, 2020   Tensor Flow - Roy Shaul
                        

April 22, 2020   ImageNet Classification with Deep Convolutional Neural Networks
                       Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton
                       Pesenters: Michael Sidorov and Lial Sharon

                       Deep Residual Learning for Image Recognition
                       Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
                       Pesenters: Moshe Shalom and Ariel Cohen

                       You Only Look Once: Unified, Real-Time Object Detection
                       Authors: Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad
                       Presenters: Moshe Shabat and Matan Idan

May 6, 2020   Image Segmentation

                       FaceNet: A Unified Embedding for Face Recognition and Clustering
                       Florian Schroff, Dmitry Kalenichenko, James Philbin
                       Presenters: Itay Hazan

                       U-Net: Convolutional Networks for Biomedical Image Segmentation
                       Authors: Olaf Ronneberger, Philipp Fischer, Thomas Brox
                       Presenters: Guy Samuels and Noam Bergmen

                       Fully Convolutional Networks for Semantic Segmentation
                       Authors: Jonathan Long, Evan Shelhamer, Trevor Darrell
                       Presenters:

                       Mask R-CNN
                       Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick

                      YOLACT Real-time Instance Segmentation
                       Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee
                       Presenters: Shai elkayam, Eitamar tripto, Matan arosh

May 13, 2020   Style transfer, GANs

                       A Neural Algorithm of Artistic Style
                       Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
                       Presenter: Meitar Ronen

                       Generative Adversarial Networks

                       Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio
                       Presenter: Tom Danino

                       Image-to-Image Translation with Conditional Adversarial Networks

                         P. Isola, JY. Zhu, T. Zhou, A.A. Efros
                         Presenter: Oron Barazani


                         Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

                         Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros
                         Presenters:

May 20, 2020   AE and more

                       Semantic Image Synthesis with Spatially-Adaptive Normalization
                        Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu
                       Presenters: Tal Ben-Haim and Ron Sofer

                       Adversarial Autoencoders
                        Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey
                       Presenter: Maor Asif

                       Subsampled MRI Reconstruction with Adversarial Neural Networks
                        Roy Shaul, Itamar David, Ohad Shitrit and Tammy Riklin Raviv
                       Guest Speakers: Roy Shaul and Itamar David

May 27, 2020 3D and multimodal

                       Mesh R-CNN
                         Authors: Georgia Gkioxari, Jitendra Malik, Justin Johnson

                       PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud
                         Authors: Shaoshuai Shi, Xiaogang Wang, Hongsheng Li
                       Presenters: Avigayel Schwartz and Yoav Noach



                      Objects that Sound
                        Relja Arandjelovi, Andrew Zisserman
                       Presenters: Niv Beeri and Eyar Ben-Tolila

June 3, 2020   Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
                      Jure Zbontar, Yann LeCun
                       Presenter:

                       Spatial Transformer Networks
                       Authors: Max Jaderberg, Karen Simonyan, Andrew Zisserman and Koray Kavukcuoglu
                       Presenter: Gidon Levakov and Shmuel Horowitz

                       Optical Flow Estimation using a Spatial Pyramid Network
                       Anurag Ranjan and Michael J. Black
                       Presenters:

June 10, 2020   RNN

                       Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks
                       Presenters:

                       A First Look at Music Composition using LSTM Recurrent Neural Networks
                       Douglas Eck and Jurgen Schmidhuber
                       Presenters: Ben wiesel and Aviv zadok

                       Unsupervised Learning of Video Representations using LSTMs
                       Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov
                       Presenters:

                       Object segmentation in videos via LSTM and adversarial networks

                       Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
                       Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, Anton van den Hengel

June 17, 2020  

                      
Playing Atari with Deep Reinforcement Learning
                       Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
                       Presenters: Yuval Haitman and Asaf Lavi
June 24, 2020   Final project presentation day
Links

Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press Online

Convolutional Neural Networks for Visual Recognition, Stanford Course cs231n, see also Lectures

Neural Networks for Machine Learning, Coursera course by Jeff Hinton

ConvNet Arithmetic

ConvNet Demo by Stanford

Tensor flow course

Tensor flow official site

Tensor flow code

Tensor flow playground

MATLAB for Deep Learning

Free Training – Deep Learning with MATLAB – No MATLAB license required

Student Presentations

Each student will need to present a paper in class. Please refer to the list of papers below and note time schedule to be presented. Ideas for papers not in the list need approval. The presentation should be clear and practiced and the student should read the assigned paper and related work in enough detail to be able to lead a discussion and answer questions. A presentation should be roughly 30 minutes long including time for questions and a discussion. Typically this is about 25 to 30 slides. In the presentation provide the citation to the paper you present and to any other related work you reference.
Deadline:The presentation should be handed in one day before the class (or before if you want feedback).
Structure of presentation: High-level overview with contributions, Main motivation, Clear statement of the problem, Overview of the technical approach, Strengths/weaknesses of the approach, Overview of the experimental evaluation, Strengths/weaknesses of evaluation, Discussion: future directions, links to other works.

Further reading

Progressive Color Transfer with Dense Semantic Correspondences
Mingming He, Jing Liao, Dongdong Chen, Lu Yuan, Pedro V. Sander,

Learning Sparse Neural Networks through L0 regularization
Christos Louizos, Max Welling, Diederik P. Kingma

Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
Shi Hu, Daniel Worrall, Stefan Knegt, Bas Veeling, Henkjan Huisman, Max Welling

Learning Sparse Networks Using Targeted Dropout
Aidan N. Gomez,Ivan Zhang, Siddhartha Rao Kamalakara, Divyam Madaan,Kevin Swersky, Yarin Gal, Geoffrey E. Hinton

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal, Zoubin Ghahramani

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

YOLACT Real-time Instance Segmentation
Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee

Deep Image Prior
Dmitry Ulyanov, Andrea Vedaldi,Victor Lempitsky

Objects that Sound
Relja Arandjelovi, Andrew Zisserman

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Andrew Owens, Alexei A. Efros

Learning Correspondence from the Cycle-consistency of Time
Xiaolong Wang, Allan Jabri, Alexei A. Efros

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, Dhruv Batra

Mesh R-CNN
Authors: Georgia Gkioxari Jitendra Malik Justin Johnson

Mask R-CNN
Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick

Spiking-YOLO: Spiking Neural Network for Real-time Object Detection
Authors: Seijoon Kim,Seongsik Park,Byunggook Na,Sungroh Yoon

Simultaneous Deep Transfer Across Domains and Tasks
Authors:Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko

Densely Connected Convolutional Networks
Authors:Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy

Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal and Zoubin Ghahramani

Rich feature hierarchies for accurate object detection and semantic segmentation
L-C. Chen, G. Papandreou, I. Kokkinos, Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik

U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., Brox, T.

Spectral Representations for Convolutional Neural Networks
Oren Rippel, Jasper Snoek, Ryan P. Adams

You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Very Deep Convolutional Networks for large-scale image recognition
Karen Simonyan & Andrew Zisserman

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A. Alemi

Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra

Generative Adversarial Networks
Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Multimodal learning with Deep Boltzman Machines
Nitish Srivastava and Ruslan Salakhutdinov

Auto-Encoding Variational Bayes
Diederik P Kingma,Max Welling

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Building High-level Features Using Large Scale Unsupervised Learning
Quoc V. Le, Marc Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado ,Jeff Dean, Andrew Y. Ng

Where Are They Looking?
Adria Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Learning to Compare Image Patches via Convolutional Neural Networks
Sergey Zagoruyko, Nikos Komodakis

Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Jure Zbontar, Yann LeCun

Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Pim Moeskops, Max A. Viergever, Adrienne M. Mendrik, Linda S. de Vries, Manon J. N. L. Benders, and Ivana Isgum

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero,Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao,Stuart Cook, Declan ORegan
and Daniel Rueckert



Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton


Monocular Object Instance Segmentation and Depth Ordering with CNNs
Ziyu Zhang, Alex Schwing, Sanja Fidler, Raquel Urtasun

Instance-Level Segmentation with Deep Densely Connected MRFs
Ziyu Zhang, Sanja Fidler, Raquel Urtasun

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor,Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala


The Multiverse Loss for Robust Transfer Learning
Etai Littwin, Lior Wolf

Simultaneous Deep Transfer Across Domains and Tasks
Eric Tzeng, Judy Hoffman, Trevor Darrell

Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

Tutorial: Bayesian Reasoning and Deep Learning
Shakir Mohamed

Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

Explaining and Harnessing Adversarial Examples
Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

Recurrent Models of Visual Attention
Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu

Generative Adversarial Nets
Ian J. Goodfellow,Jean Pouget-Abadie,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair,Aaron Courville, Yoshua Bengio

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, Geoffrey E. Hinton

DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, Lior Wolf

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
Oren Z. Kraus, Lei Jimmy Ba, Brendan Frey

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook, Declan ORegan, and Daniel Rueckert We Are Humor Beings: Understanding and Predicting Visual Humor
Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

A First Look at Music Composition using LSTM Recurrent Neural Networks
Douglas Eck, Jurgen Schmidhuber


Homework assignments

Homework assignments should be submitted via moodle using a single zip file: "YourName-XX.zip" where XX is the assignment number.

Assignment #0 due to April 14, 2020


Assignment #1 due to April 23, 2020