COMP 646: Deep Learning for Vision and Language | Spring 2024

Instructor: Vicente Ordóñez-Román (vicenteor at, Office Hours: Wed 10am to noon (Duncan 2080).
TA: Ayush Sachdeva (as126 at, Office Hours: Mondays 10am to 11am (Brochestein Pavilion).
TA: Jaywon Koo (jk125 at, Office Hours: Tuesdays 10 to 11am (Duncan 3037).
TA: Jefferson Hernandez (jeh16 at, Office Hours: Wednesdays 1 to 2pm (Duncan 2088).
TA: Zilin Xiao (zx51 at, Office Hours: Thursdays 10am to 11am (Duncan 3108).
Class Time: Tuesdays and Thursdays from 4pm to 5:15pm Central Time (Location: Keck Hall 100).

Course Description: Visual recognition and language understanding are two challenging tasks in AI. In this course we will study and acquire the skills to build machine learning and deep learning models that can reason about images and text for generating image descriptions, find objects in images, generating images from text, image retrieval, and other general tasks involving both text and images. On the technical side we will leverage models such as convolutional neural networks (CNNs), Transformer networks (e.g. BERT, T5, ViTs), Diffusion Models (e.g Latent Diffusion), among others. Emphasis will be place also on re-using large scale pre-trained models such as CLIP, Stable Diffusion, BLIP-2, LLaMA-2, etc.

Learning Objectives: (a) Develop intuitions about the connections between language and vision, (b) Understand concepts in representation learning for both images and text, (c) Become familiar with state-of-the-art models for tasks in vision and language, (d) Obtain practical experience in the implementation and adaptation of these models.

Prerrequisites: There are no formal pre-requisities for this class. However a basic command of machine learning, deep learning or computer vision will be useful when taking this class. Students should have knowledge of linear algebra, differential calculus, and basic statistics and probability. Moreover students are expected to have attained some level of proficiency in Python programming or be willing to learn Python programming. Students are encouraged to complete the following activity before the first lecture: [Primer on Image Processing].


Date Topic  
Tue, Jan 9 Introduction to Vision and Language [pptx] [pdf]
Thu, Jan 11 Machine Learning I: Supervised vs Unsupervised Learning, Linear Classifiers [pptx] [pdf]
Tue, Jan 16 Classes cancelled due to weather
Assignment on Pytorch + Image Classification [colab]
Due Monday January 29th, 11:59pm (CT).
Thu, Jan 18 Machine Learning II: Stochastic Gradient Descent / Regularization / Softmax [pptx] [pdf]
Tue, Jan 23 Neural Networks: Multi-layer Perceptrons and Backpropagation [pptx] [pdf]
Thu, Jan 25 Computer Vision I: The Convolutional Operator, Image Filtering and Convolutional Neural Networks [pptx] [pdf]
Assignment on Movie Posters and Plots [colab]
Due Monday February 12th, 11:59pm (CT).
Tue, Jan 30 Computer Vision II: Convolutional Neural Network Architectures: LeNet, AlexNet, VGG, InceptionNets, ResNets [pptx] [pdf]
Thu, Feb 1 Natural Language Processsing I: Introduction: Bag of Words, Language Models, Word Embeddings [pptx] [pdf]
Tue, Feb 6 Natural Language Processsing II: RNNs and Transformer Models [pptx] [pdf]
Thu, Feb 8 Monday Schedule.
Tue, Feb 13 Transformer Models: BERT, GPT-2, ViT, CLIP [pptx] [pdf]
Assignment on Generative AI [colab]
Due Wednesday February 28th, 11:59pm (CT).
Thu, Feb 15 Transformers Models: Instruction Tuning, Learning from Human Feedback, Efficient Finetuning [pptx] [pdf]
Tue, Feb 20 Computer Vision IV: Convolutional Neural Networks for Object Detection and Segmentation [pptx] [pdf]
Thu, Feb 22 Quiz
Tue, Feb 27 AutoEncoders, Introduction to Diffusion Models [pptx] [pdf]
Thu, Feb 29 Quiz Recap and Review Session
Tue, Mar 5 Adversarial Examples + Generative Adversarial Networks (GANs) [pptx] [pdf]
Thu, Mar 7 Text-to-Image Networks: Diffusion Models I
Tue, Mar 12 Spring Break (No Scheduled Classes)
Thu, Mar 14 Spring Break (No Scheduled Classes)
Tue, Mar 19 Text-to-Image Networks: Diffusion Models II [pptx] [pdf]
Thu, Mar 21 Working with training large scale jobs in practice (Wandb, Containers, SLURM)
Tue, Mar 26 Self-supervised Models for Computer Vision [pptx] [pdf]
Thu, Mar 28 Project Progress Report Working Session
Tue, Apr 2 In Class Activity: Feature Inversion
Thu, Apr 4 In Class Activity: Working Session on Projects
Tue, Apr 9 Spotlight: Mobile UI Navigation with Vision and Language Models
Thu, Apr 11 Other Topics [pptx] [pdf]
Tue, Apr 16 What could be the Next Generation of LLMs: Guest Lecture by Zilin Xiao [pptx] [pdf]
Thu, Apr 18 Course Recap: Summar of the Class: Take away messages [pptx] [pdf]

Disclaimer: The professor reserves to right to make changes to the syllabus, including assignment due dates. These changes will be announced as early as possible.

Grading: Assignments: 30% (3 assignments), Class Project: 60%, Quiz: 10%. Grade cutoffs -- no stricter than the following: A [between 90% and 100%], B [between 80% and 90%), C [between 70% and 80%), D [between 55% and 70%), F [less than 55%)

COVID-19 Notice: If you have any flu-like symptoms you should stay home. There is no grade for attendance in this class.

Late Submission Policy: No late assignments will be accepted in this class. Unless the student has procured special accommodations for warranted circumstances -- or due to exceptional personal situations. If you consider this might be your case please contact the instructor directly as early as possible. If you contact the instructor on the day of the deadline but before the deadline is past due, then you are required to also submit a copy of your notebook with the progress you have made so far in order to make your request. In general, unless a medical condition or other serious situation is affecting you, please do not email the instructor or TAs requesting to have special considerations. If you need special accommodations regarding a disability, then contact the Disability Resource Center at Rice, please follow the advice on that section of this syllabus.

Honor Code and Academic Integrity: "In this course, all students will be held to the standards of the Rice Honor Code, a code that you pledged to honor when you matriculated at this institution. If you are unfamiliar with the details of this code and how it is administered, you should consult the Honor System Handbook at This handbook outlines the University's expectations for the integrity of your academic work, the procedures for resolving alleged violations of those expectations, and the rights and responsibilities of students and faculty members throughout the process." For this class: If assignments are individual then no collaboration is expected, no two students should submit the same source code. Regardless of circumstances I will assume that any source code, text, or images submitted alongside reports or projects are of the authorship of the students unless otherwise explicitly stated through appropriate means. Any missing information regarding sources will be regarded potentially as a failure to abide by the academic integrity statement even if that was not the intent. Please be careful about citing sources and clearly stating what is your original work and what is not in all assignments and projects. Especially avoid vague statements such as "we built our model based on X", instead be explicit e.g. "we downloaded X and modified the encoder so that it can work with videos instead of images by adding three more layers". Avoid vague statements that make it difficult to understand what you did from what was done by others. Sometimes great projects consist in simply putting together two existing components that someone else developed, however this has to be clearly acknowledged as such.

Title IX Support: Rice University cares about your wellbeing and safety. Rice encourages any student who has experienced an incident of harassment, pregnancy discrimination or gender discrimination or relationship, sexual, or other forms interpersonal violence to seek support through The SAFE Office. Students should be aware when seeking support on campus that most employees, including myself, as the instructor/TA, are required by Title IX to disclose all incidents of non-consensual interpersonal behaviors to Title IX professionals on campus who can act to support that student and meet their needs. For more information, please visit or email

Disability Resource Center: "If you have a documented disability or other condition that may affect academic performance you should: 1) make sure this documentation is on file with the Disability Resource Center (Allen Center, Room 111 / / x5841) to determine the accommodations you need; and 2) talk with me to discuss your accommodation needs."