Spring 2022: Deep Learning and Its Applications to Signal and Image Processing and Analysis, 361-21120

Tammy Riklin Raviv
Teaching assistants: Shaked Cohen

Time Wednesday, 14:00-17:00
Location Zoom, Online course

About the course

The class will cover a diverse set of topics in signal processing, image analysis and computer vision and various Neural Network architectures. It will be an interactive course where we will discuss interesting topics on demand and latest research buzz. The goal of the class is to learn about different domains of the field, understand, identify and analyse the main challenges, what works and what doesn't, as well as to identify interesting new directions for future research.
Right after Passover, we will read about three papers every week. The papers will be presented by the students in a seminar format. The success of the discussion in class will thus be due to how prepared the students come to class. Each student is expected to read at least one paper for each class (and prepare a question or a comment for a discussion). In addition, each student will present a paper in one of the classes. There will be also talks by invited speakers and a lab. class on Tensor Flow. The Last class will be dedicated to final project presentation (about ten minutes for each project). Details on the final project will be published soon. The final project does not have to be necessarily related to the paper presented in class.

Schedule, links and lecture notes

March 23, 2022   Course overview, the biological and the artificial neurons, feature representation, common architectures

March 30, 2022   Loss functions, Optimization, Regularization

April 6, 2022 Tensor Flow - Shaked Cohen

April 13, 2022   BackProp, Optimization and Batch Normalization

April 27, 2022   Deep Residual Learning for Image Recognition
                       Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
                       Presenters: Shai Aharon and Maxim Levit

                       You Only Look Once: Unified, Real-Time Object Detection
                       Authors: Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad
                       Presenters: Yoram Segal and Elad Sofer

                       FaceNet: A Unified Embedding for Face Recognition and Clustering
                       Florian Schroff, Dmitry Kalenichenko, James Philbin
                       Presenters: Tomer Shaked and Ofek Finkelstein

May 11, 2022   Image Segmentation

                       U-Net: Convolutional Networks for Biomedical Image Segmentation
                       Authors: Olaf Ronneberger, Philipp Fischer, Thomas Brox
                       Presenters: Maor Turner and Ori Feldman

                      YOLACT Real-time Instance Segmentation
                       Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee
                       Presenters: Itamar Elmakias and Jameel Nassar

                       Mask R-CNN
                       Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick
                       Presenters: Yoel Bokobza, Oded George and Ziv Shahar

May 18, 2022   GANs

                       Generative Adversarial Networks
                       Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio
                       Presenters: Yael Elkin and Ophir Gruteke

                       Image-to-Image Translation with Conditional Adversarial Networks
                         P. Isola, JY. Zhu, T. Zhou, A.A. Efros
                         Presenters: Ohad Shapira and Ran Greidi

                       Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
                         J-Y Zhu, T. Park, P. Isola, A. A. Efros
                         Presenter:                       

                       SinGAN: Learning a Generative Model from a Single Natural Image
                       T. Rott Shaham, T. Dekel, T. Michaeli
                        Presenters: Itay Buchnik and Or Berebi

May 25, 2022   AE and more

                       Auto-Encoding Variational Bayes
                      Diederik P Kingma, Max Welling
                       Presenters: Raviv Ilani & Lotem Drori

                       Old Photo Restoration via Deep Latent Space Translation
                        Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, Fang Wen
                       Presenters: Elor and Michael

                       A Style-Based Generator Architecture for Generative Adversarial Networks
                        Tero Karras, Samuli Laine, Timo Aila
                       Speakers:Priel Nissim and Carmi Shimon

June 1 , 2022   RNNs and Reinforcement Learning

                     A First Look at Music Composition using LSTM Recurrent Neural Networks
                       Douglas Eck and Jurgen Schmidhuber
                     Presenters: Tal Itschakian & Rom Schilman

                     DeepTract: A Probabilistic Deep Learning Framework for White Matter Fiber Tractography
                       Authors: Itay Benou and Tammy Riklin Raviv
                       Guest lecture: Itay Benou

                    Dual-Task ConvLSTM-UNet for Instance Segmentation of Weakly Annotated Microscopy Videos
                       Authors: Assaf Arbelle, Shaked Cohen and Tammy Riklin Raviv
                       Guest lecture: Shaked Cohen

                       Playing Atari with Deep Reinforcement Learning
                       Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
                       Presenters: Eliraz and Amichai

June 8, 2022   Transformers and Explainable AI

                      Attention Is All You Need
                       Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
                       Presenters:Inbal Rimon, Camron, Guy

                      Attention on attention for image captioning
                       Authors: Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei
                       Presenters:Hanna Kossowsky

                      Grad-CAM: Why did you say that?
                       Authors:Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh and Dhruv Batra
                       Presenters:Reut Moshe and Daniel

June 15, 2022   Multiple Topics

                       "Zero Shot" Super-Resolution using Deep Internal Learning
                         Authors: Assaf Shocher, Nadav Cohen and Michal Irani
                       Presenter: Yakov Geltser

                       Adversarial domain adaptation for stable brain-machine interfaces
                         Authors: Ali Farshchian, Juan A. Gallego, Joseph P. Cohen, Yoshua Bengio, Lee E. Miller, Sara A. Solla
                       Presenter:Ophir Almagor

                       Mesh R-CNN
                       Authors: Georgia Gkioxari Jitendra Malik Justin Johnson
                       Presenters: Orel Ben Zaken and Omer Luxembourg

June 22, 2022   Offline class


June 29, 2022   Final project presentation day
Links

Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016). Deep Learning. MIT Press Online

Convolutional Neural Networks for Visual Recognition, Stanford Course cs231n, see also Lectures

Neural Networks for Machine Learning, Coursera course by Jeff Hinton

ConvNet Arithmetic

ConvNet Demo by Stanford

Tensor flow course

Tensor flow official site

Tensor flow code

Tensor flow playground

MATLAB for Deep Learning

Free Training – Deep Learning with MATLAB – No MATLAB license required

Student Presentations

Each student will need to present a paper in class. Please refer to the list of papers below and note time schedule to be presented. Ideas for papers not in the list need approval. The presentation should be clear and practiced and the student should read the assigned paper and related work in enough detail to be able to lead a discussion and answer questions. A presentation should be roughly 30 minutes long including time for questions and a discussion. Typically this is about 25 to 30 slides. In the presentation provide the citation to the paper you present and to any other related work you reference.
Deadline:The presentation should be handed in one day before the class (or before if you want feedback).
Structure of presentation: High-level overview with contributions, Main motivation, Clear statement of the problem, Overview of the technical approach, Strengths/weaknesses of the approach, Overview of the experimental evaluation, Strengths/weaknesses of evaluation, Discussion: future directions, links to other works.

Further reading

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton

Progressive Color Transfer with Dense Semantic Correspondences
Mingming He, Jing Liao, Dongdong Chen, Lu Yuan, Pedro V. Sander,

Learning Sparse Neural Networks through L0 regularization
Christos Louizos, Max Welling, Diederik P. Kingma

Supervised Uncertainty Quantification for Segmentation with Multiple Annotations
Shi Hu, Daniel Worrall, Stefan Knegt, Bas Veeling, Henkjan Huisman, Max Welling

Learning Sparse Networks Using Targeted Dropout
Aidan N. Gomez,Ivan Zhang, Siddhartha Rao Kamalakara, Divyam Madaan,Kevin Swersky, Yarin Gal, Geoffrey E. Hinton

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal, Zoubin Ghahramani

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

YOLACT Real-time Instance Segmentation
Daniel Bolya Chong Zhou Fanyi Xiao Yong Jae Lee

Deep Image Prior
Dmitry Ulyanov, Andrea Vedaldi,Victor Lempitsky

Objects that Sound
Relja Arandjelovi, Andrew Zisserman

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Andrew Owens, Alexei A. Efros

Learning Correspondence from the Cycle-consistency of Time
Xiaolong Wang, Allan Jabri, Alexei A. Efros

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Abhishek Das, Satwik Kottur, Jose M.F. Moura, Stefan Lee, Dhruv Batra

Mesh R-CNN
Authors: Georgia Gkioxari Jitendra Malik Justin Johnson

Mask R-CNN
Authors: Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick

Spiking-YOLO: Spiking Neural Network for Real-time Object Detection
Authors: Seijoon Kim,Seongsik Park,Byunggook Na,Sungroh Yoon

Simultaneous Deep Transfer Across Domains and Tasks
Authors:Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko

Densely Connected Convolutional Networks
Authors:Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe and Christian Szegedy

Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal and Zoubin Ghahramani

Rich feature hierarchies for accurate object detection and semantic segmentation
L-C. Chen, G. Papandreou, I. Kokkinos, Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik

U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., Brox, T.

Spectral Representations for Convolutional Neural Networks
Oren Rippel, Jasper Snoek, Ryan P. Adams

You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon , Santosh Divvala, Ross Girshick , Ali Farhad

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Very Deep Convolutional Networks for large-scale image recognition
Karen Simonyan & Andrew Zisserman

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A. Alemi

Recurrent Neural Networks and Long Short Term Memory (LSTM) Networks

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra

Generative Adversarial Networks
Ian J. Goodfellow, Jean Pouget-Abadie ,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, Yoshua Bengio

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

Multimodal learning with Deep Boltzman Machines
Nitish Srivastava and Ruslan Salakhutdinov

Auto-Encoding Variational Bayes
Diederik P Kingma,Max Welling

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
Pascal Vincent, Hugo Larochelle , Isabelle Lajoie , Yoshua Bengio , Pierre-Antoine Manzagol

Building High-level Features Using Large Scale Unsupervised Learning
Quoc V. Le, Marc Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado ,Jeff Dean, Andrew Y. Ng

Where Are They Looking?
Adria Recasens, Aditya Khosla, Carl Vondrick, Antonio Torralba

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Learning to Compare Image Patches via Convolutional Neural Networks
Sergey Zagoruyko, Nikos Komodakis

Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Jure Zbontar, Yann LeCun

Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Pim Moeskops, Max A. Viergever, Adrienne M. Mendrik, Linda S. de Vries, Manon J. N. L. Benders, and Ivana Isgum

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero,Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao,Stuart Cook, Declan ORegan
and Daniel Rueckert


Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich

ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever,Geoffrey E. Hinton


Monocular Object Instance Segmentation and Depth Ordering with CNNs
Ziyu Zhang, Alex Schwing, Sanja Fidler, Raquel Urtasun

Instance-Level Segmentation with Deep Densely Connected MRFs
Ziyu Zhang, Sanja Fidler, Raquel Urtasun

DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor,Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, Soumith Chintala


The Multiverse Loss for Robust Transfer Learning
Etai Littwin, Lior Wolf

Simultaneous Deep Transfer Across Domains and Tasks
Eric Tzeng, Judy Hoffman, Trevor Darrell

Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
Mateusz Malinowski, Marcus Rohrbach, Mario Fritz

Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling

Tutorial: Bayesian Reasoning and Deep Learning
Shakir Mohamed

Deep learning and the information bottleneck principle
Naftali Tishby,Noga Zaslavsky

Unsupervised Learning of Video Representations using LSTMs
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

Explaining and Harnessing Adversarial Examples
Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy

Recurrent Models of Visual Attention
Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu

Generative Adversarial Nets
Ian J. Goodfellow,Jean Pouget-Abadie,Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair,Aaron Courville, Yoshua Bengio

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, Geoffrey E. Hinton

DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, Lior Wolf

FaceNet: A Unified Embedding for Face Recognition and Clustering
Florian Schroff, Dmitry Kalenichenko, James Philbin

Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
Oren Z. Kraus, Lei Jimmy Ba, Brendan Frey

Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Ozan Oktay, Wenjia Bai, Matthew Lee, Ricardo Guerrero, Konstantinos Kamnitsas, Jose Caballero, Antonio de Marvao, Stuart Cook, Declan ORegan, and Daniel Rueckert We Are Humor Beings: Understanding and Predicting Visual Humor
Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanislaw Antol, Mohit Bansal, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh

A Neural Algorithm of Artistic Style
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge

A First Look at Music Composition using LSTM Recurrent Neural Networks
Douglas Eck, Jurgen Schmidhuber


Homework assignments

Homework assignments should be submitted via moodle using a single zip file: "YourName-XX.zip" where XX is the assignment number.

Assignment #0 due to April 14, 2020


Assignment #1 due to April 23, 2020