Welcome to the course homepage of CS699  Representation Learning: Theory and Practice.
The course is designed for PhD students wishing to gain theoretical and applied skills in Inference, Machine Learning, Information Theory, and Deep Learning. The goal of this course is to bring students uptospeed on skills required for publishing Machine Learning papers in toptier venues (ICML, NIPS, KDD, CVPR, etc). We will cover the necessary background in Mathematics (Linear Algebra, Matrix Calculus, Information Theory) and Programming (Numpy, Computational Graphs, TensorFlow). Course includes a variety of interrelated theoretical topics including Deep Learning, Graphical Models, Variational Methods, Embeddings, and others. We extensively apply the theoretical concepts on applications (Natural Language Processing, Computer Vision, Graph Theory). The majority of the grade is handson: specifically, by implementing programs which require the student to thoroughly understand both the theory and the application domain. Our goal is to show students that most of the applications require a similar set of theoretical skills, which we will teach in the course.
There are no official prerequisits for the course. The unofficial requirements are good math skills and good programming skills.
Class will be held Mondays and Wednesdays, 10AMnoon, at VKC 102.
Sami's office hours are held Mondays and Wednesdays, 23:20 PM, at Basement of Leavey Library (at the open discussion tables).
This course is delivered to you by:
We will follow the following grading scheme:
Component  % of Grade 
Assignment 1 (due Sunday Sept 15)  8 
Assignment 2 (due Saturday Oct 5)  16 
Assignment 3  14 
Assignment 4  14 
Beyond Assignment  14 
Test [last class; multiple choice]  14 
Participation [inclass, piazza]  10 
Total  90 
The purpose of the assignment is give you sufficient experience in deriving mathematical expressions for models, implementing them (in TensorFlow), and understanding the models e.g. through visualizations. All assignments must be completed individually. You can ask your classmates for help, but following rules: Assignment discussions must without a computer (e.g. using whiteboard). No one is allowed to take physical/electronic notes from the discussion. After the discussion, students must take an hour gap before returning to the assignment. The only electronic thing, that can be shared, are links of thirdparty information (e.g. paper links)
All assignments must be completed on Vocareum. If you have not received access, please contact Sami.
Each student has a total of 7 late days for the semester, that can be used on all the assignments combined. If a student uses more than 7 late days, they will get 20% deduction for every additional late day for the assignment they are submitting.
Students are to form groups (of 2to4 students each) where they will extend one of the assignments in some direction. The direction is completely optional and there are no guidelines except for: be creative. Some suggestions include: try the same problem on a different dataset (preferrably, from a different domain e.g. NLP if the assignment was on Vision), or extend the model in some novel way which should hopefully improve some metrics of your choice. The instructors will give a list of ideas on each of the assignments. You can choose any of them or come up with your own. We will be sending more information about this during the course.
There is no one golden textbook in Machine Learning or Representation Learning.
The reason is obvious: If one starts drafting a textbook, the field would
significantly change by the time the writing is done. Nonetheless, we will use
the Deep Learning book (though we wont cover the whole thing) and we will
supplement the book with published papers and (free) online notes, as linked on
the syllabus.
In addition, we are currently developing a supplement handout, to be completed in class,
for the Deep Learning material.
The course outline is a moving target. We will be changing the schedule aswego. If you have any suggestions (E.g. topics you really like or do not like, even they are not listed on the syllabus), please talk to us, either inperson (preferred) or via email.
Week  Monday  Wednesday 
Refresher  
Aug/26 


Sept/2  Labor Day Holiday [no class] 

Supervised Representation Learning: P(YX)  
Sept/9 

[by Sami; on slides] 
Sept/16 

[by Sami, slides] 
Unsupervised (Representation) Learning: P(X)  
Sept/23 
[by Aram, slides] 
[by Aram, slides] 
Sept/30 
[Logistics slides] 
[Logistic Slide] 
Oct/7 
[by Sami, slides] 
Review of derivation of Cross Entropy. Deeper dive on the Computational Graphs of:
[by Sami, on whiteboard] 
Oct/14 
Deeper dive on the Computational Graphs of:
[by Sami, on whiteboard] 
Guest Speaker: Alessandro Achille

Representations for variablesized data  
Oct/21 
Introduction to Embedding Learning.

Beyond Sequences and Euclidean Structures: Intro to Machine Learning on Graphs (part 1).

Oct/28 
Intro to ML on Graphs (part 2):

Sequences & Recurrent Neural Networks (RNNs)

Nov/4 
Convolution on Graphstructured Data.

Application: Computational Graphs for Implementing Graph Convolutional Networks

Misc Topics  
Nov/11  Normalizing Flows; Generative Adversarial Networks (GANs) (DL#20)  Guest Speaker: Bryan Perozzi (Senior Research Scientist @ Google AI) 
Nov/18  SequencetoSequence and Transformer Models; on Videos and NLP.  Gradient estimators for nondifferentiable computation graphs 
Nov/25  MetaLearning  Thanksgiving Holiday. No class 
Dec/2 

In class test (multiple choice) on Misc topics; Farewell; Advice for the future. 
The End 