Spring 2021: Deep Learning and Its Applications to Signal and Image Processing and Analysis, 361-21120

Tammy Riklin Raviv
Teaching assistants: Ron Sofer and Tal Ben-Hayim

Time Monday, 14:00-17:00
Location Zoom, Online course


About the course

The class will cover a diverse set of topics in signal processing, image analysis and computer vision and various Neural Network architectures. It will be an interactive course where we will discuss interesting topics on demand and latest research buzz. The goal of the class is to learn about different domains of the field, understand, identify and analyze the main challenges, what works and what doesn't, as well as to identify interesting new directions for future research.
Right after Passover, we will read about three papers every week. The papers will be presented by the students in a seminar format. The success of the discussion in class will thus be due to how prepared the students come to class. Each student is expected to read at least one paper for each class (and prepare a question or a comment for a discussion). In addition, each student will present a paper in one of the classes. There will be also talks by invited speakers and a lab. class on Tensor Flow. The Last class will be dedicated to final project presentation (about ten minutes for each project). Details on the final project will be published soon. The final project does not have to be necessarily related to the paper presented in class.

Schedule, links and lecture notes

March 1, 2021   Course overview, the biological and the artificial neurons, feature representation, common architectures

March 8, 2021   Optimination, Stochastic gradient descent, Backpropagation, Regularization, Loss functions

March 15, 2021 Capacity, Overfitting, Underfitting, Regularization, Optimization

                        Tensor Flow - Tal Ben-Hayim and Ron Sofer

March 22, 2021 Guest Lecture: Roy Fahn: Deep Learning tools in Matlab

                        More on Optimization for Deep Learning

April 5, 2021   Convolutional Neural Networks (CNNs), Receptive Fields Arithmetic and Batch Normalization

April 12, 2021   Neural Network Analysis

                       Stochastic Weight Pruning and the Role of Regularization in Shaping Network Structure
                        Yael Ben-Gigi, Jacob Goldberger and Tammy Riklin Raviv
                       Guest Speaker: Yael Ben-Gigi

April 19, 2021   Deep Residual Learning for Image Recognition
                       Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
                       Presenters:

                       You Only Look Once: Unified, Real-Time Object Detection
                       Authors: Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad
                       Presenters:

                       FaceNet: A Unified Embedding for Face Recognition and Clustering
                       Florian Schroff, Dmitry Kalenichenko, James Philbin
                       Presenters:

April 26, 2021   Image Segmentation

                       U-Net: Convolutional Networks for Biomedical Image Segmentation
                       Authors: Olaf Ronneberger, Philipp Fischer, Thomas Brox
                       Presenters:

                      YOLACT Real-time Instance Segmentation
                       Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee
                       Presenters:

                       Mask R-CNN
                       Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick
                       Presenters:

                       Panoptic segmentation
                       Authors: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollar
                       Presenters:

May 3, 2021   GANs

                       Generative Adversarial Networks

                       Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio
                       Presenter:

                       Image-to-Image Translation with Conditional Adversarial Networks

                         P. Isola, JY. Zhu, T. Zhou, A.A. Efros
                         Presenter:

                       Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

                         J-Y Zhu, T. Park, P. Isola, A. A. Efros
                         Presenter:                       

                       SinGAN: Learning a Generative Model from a Single Natural Image
                       T. Rott Shaham, T. Dekel, T. Michaeli
                        Presenter:

May 10, 2021   AE and more

                       Auto-Encoding Variational Bayes
                      Diederik P Kingma, Max Welling
                       Presenters:

                       Adversarial Autoencoders
                        Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, Brendan Frey
                       Presenter:

                       Old Photo Restoration via Deep Latent Space Translation
                        Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, Fang Wen
                       Presenter:

                       Subsampled MRI Reconstruction with Adversarial Neural Networks
                        Roy Shaul, Itamar David, Ohad Shitrit and Tammy Riklin Raviv
                       Guest Speakers:

May 24 , 2021   RNNs and Transformer

                     A First Look at Music Composition using LSTM Recurrent Neural Networks
                       Douglas Eck and Jurgen Schmidhuber
                     Presenters:

                     Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
                       Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio
                     Presenters:

                     RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
                       Zachary Teed and Jia Deng
                     Presenters:

                     Attention Is All You Need
                       Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
                       Presenters:

                     DeepTract: A Probabilistic Deep Learning Framework for White Matter Fiber Tractography
                       Authors: Itay Benou and Tammy Riklin Raviv
                       Guest lecture: Itay Benou

May 31, 2021 Spatial Transformer Networks
                       Authors: Max Jaderberg, Karen Simonyan, Andrew Zisserman and Koray Kavukcuoglu
                       Presenter:

                      
Playing Atari with Deep Reinforcement Learning
                       Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
                       Presenters:

                       Bootstrap your own latent: A new approach to self-supervised Learning
                       Authors: Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo,                       Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko
                       Presenter:

                      Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
                       Yarin Gal, Zoubin Ghahramani
                       Presenter:

                       Zero-Shot Text-to-Image Generation
                       Authors:Aditya Ramesh,Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss , Alec Radford, Mark Chen, Ilya Sutskever
                       Presenter:

June 7, 2021 3D and GraphCNN

                       Geometric Deep Learning: going beyond Euclidean Data
                         Authors: Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, Pierre Vandergheynst
                       Presenters:

                       Dynamic Graph CNN for Learning on Point Clouds
                         Authors: Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, Justin M. Solomon
                       Presenters:

                       PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud
                         Authors: Shaoshuai Shi, Xiaogang Wang, Hongsheng Li
                       Presenters:

                       Mesh R-CNN
                       Authors: Georgia Gkioxari Jitendra Malik Justin Johnson
                       Presenters:

                      NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
                        Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
                       Presenters:

June 14, 2021   Final project presentation day
Links

Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press Online

Convolutional Neural Networks for Visual Recognition, Stanford Course cs231n, see also Lectures

Neural Networks for Machine Learning, Coursera course by Jeff Hinton

ConvNet Arithmetic

ConvNet Demo by Stanford

Tensor flow course

Tensor flow official site

Tensor flow code

Tensor flow playground

MATLAB for Deep Learning

Free Training – Deep Learning with MATLAB – No MATLAB license required

Student Presentations

Each student will need to present a paper in class. Please refer to the list of papers below and note time schedule to be presented. Ideas for papers not in the list need approval. The presentation should be clear and practiced and the student should read the assigned paper and related work in enough detail to be able to lead a discussion and answer questions. A presentation should be roughly 30 minutes long including time for questions and a discussion. Typically this is about 25 to 30 slides. In the presentation provide the citation to the paper you present and to any other related work you reference.
Deadline:The presentation should be handed in one day before the class (or before if you want feedback).
Structure of presentation: High-level overview with contributions, Main motivation, Clear statement of the problem, Overview of the technical approach, Strengths/weaknesses of the approach, Overview of the experimental evaluation, Strengths/weaknesses of evaluation, Discussion: future directions, links to other works.

Further reading

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton

Progressive Color Transfer with Dense Semantic Correspondences
Mingming He, Jing Liao, Dongdong Chen, Lu Yuan, Pedro V. Sander,

Learning Sparse Neural Networks through L0 regularization
Christos Louizos, Max Welling, Diederik P. Kingma

Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
Shi Hu, Daniel Worrall, Stefan Knegt, Bas Veeling, Henkjan Huisman, Max Welling

Learning Sparse Networks Using Targeted Dropout
Aidan N. Gomez,Ivan Zhang, Siddhartha Rao Kamalakara, Divyam Madaan,Kevin Swersky, Yarin Gal, Geoffrey E. Hinton

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal, Zoubin Ghahramani

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

YOLACT Real-time Instance Segmentation
Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee

Deep Image Prior
Dmitry Ulyanov, Andrea Vedaldi,Victor Lempitsky

Objects that Sound
Relja Arandjelovi, Andrew Zisserman

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Andrew Owens, Alexei A. Efros

Learning Correspondence from the Cycle-consistency of Time
Xiaolong Wang, Allan Jabri, Alexei A. Efros

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, Dhruv Batra

Mesh R-CNN
Authors: Georgia Gkioxari Jitendra Malik Justin Johnson

Mask R-CNN
Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick

Spiking-YOLO: Spiking Neural Network for Real-time Object Detection
Authors: Seijoon Kim,Seongsik Park,Byunggook Na,Sungroh Yoon

Simultaneous Deep Transfer Across Domains and Tasks
Authors:Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko

Densely Connected Convolutional Networks
Authors:Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy

Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal and Zoubin Ghahramani

Rich feature hierarchies for accurate object detection and semantic segmentation
L-C. Chen, G. Papandreou, I. Kokkinos, Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik

U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., Brox, T.

Spectral Representations for Convolutional Neural Networks
Oren Rippel, Jasper Snoek, Ryan P. Adams

You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Very Deep Convolutional Networks for large-scale image recognition
Karen Simonyan & Andrew Zisserman

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A. Alemi

Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra

Generative Adversarial Networks
Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Multimodal learning with Deep Boltzman Machines
Nitish Srivastava and Ruslan Salakhutdinov

Auto-Encoding Variational Bayes
Diederik P Kingma,Max Welling

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Building High-level Features Using Large Scale Unsupervised Learning
Quoc V. Le, Marc Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado ,Jeff Dean, Andrew Y. Ng

Where Are They Looking?
Adria Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Learning to Compare Image Patches via Convolutional Neural Networks
Sergey Zagoruyko, Nikos Komodakis

Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Jure Zbontar, Yann LeCun

Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Pim Moeskops, Max A. Viergever, Adrienne M. Mendrik, Linda S. de Vries, Manon J. N. L. Benders, and Ivana Isgum

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero,Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao,Stuart Cook, Declan ORegan
and Daniel Rueckert



Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton


Monocular Object Instance Segmentation and Depth Ordering with CNNs
Ziyu Zhang, Alex Schwing, Sanja Fidler, Raquel Urtasun

Instance-Level Segmentation with Deep Densely Connected MRFs
Ziyu Zhang, Sanja Fidler, Raquel Urtasun

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor,Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala


The Multiverse Loss for Robust Transfer Learning
Etai Littwin, Lior Wolf

Simultaneous Deep Transfer Across Domains and Tasks
Eric Tzeng, Judy Hoffman, Trevor Darrell

Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

Tutorial: Bayesian Reasoning and Deep Learning
Shakir Mohamed

Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

Explaining and Harnessing Adversarial Examples
Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

Recurrent Models of Visual Attention
Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu

Generative Adversarial Nets
Ian J. Goodfellow,Jean Pouget-Abadie,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair,Aaron Courville, Yoshua Bengio

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, Geoffrey E. Hinton

DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, Lior Wolf

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
Oren Z. Kraus, Lei Jimmy Ba, Brendan Frey

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook, Declan ORegan, and Daniel Rueckert We Are Humor Beings: Understanding and Predicting Visual Humor
Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

A First Look at Music Composition using LSTM Recurrent Neural Networks
Douglas Eck, Jurgen Schmidhuber


Homework assignments

Homework assignments should be submitted via moodle using a single zip file: "YourName-XX.zip" where XX is the assignment number.

Assignment #0 due to April 14, 2020


Assignment #1 due to April 23, 2020