MIT 6.S191 (2023): Deep Generative Modeling

The lecturer expresses excitement about the current age of generative AI, explaining that it involves building systems capable of not only recognizing patterns in data but also generating new data instances based on those patterns. The concept is complex and powerful, and the field has seen significant growth in recent years. To demonstrate the power of generative models, the lecturer presents three synthesized faces, revealing that all are fake but appear real, thus showcasing the potential of deep generative models in creating new data.

The lecture distinguishes between supervised learning, where a function is learned to map data to labels, and unsupervised learning, which involves understanding the hidden structure of unlabeled data. Generative modeling, a subset of unsupervised learning, aims to learn the distribution of data samples. It has two main forms: density estimation, which learns the underlying probability distribution, and sample generation, which uses the learned model to create new instances resembling the training data. The focus is on real-world applications of generative modeling, such as uncovering biases in facial detection models and outlier detection in autonomous vehicles.

The concept of latent variable models is introduced, using Plato's 'Allegory of the Cave' as an analogy to explain latent variables as underlying, unobservable factors that affect observable data. Autoencoders are presented as a simple generative model that encodes data into a low-dimensional latent space and decodes it back to the original data space, aiming to reconstruct the input data. The process involves neural network layers and does not require labels, fitting the unsupervised learning paradigm.

The training process of autoencoders is described, emphasizing the importance of the dimensionality of the latent space. A lower dimensionality results in less accurate reconstructions but a more efficient encoding, while a higher dimensionality offers better reconstructions at the cost of efficiency. The goal is to find a balance that allows the network to learn a compact representation of the data without losing essential information.

Variational autoencoders (VAEs) are introduced as an extension of autoencoders that incorporate randomness and probability to generate new data instances. Unlike traditional autoencoders, VAEs use a probabilistic approach to encode and decode data, defining a distribution over the latent variables. The VAE loss function consists of a reconstruction loss and a regularization term, with the latter encouraging the latent space to follow a prior distribution, often a standard normal distribution. This regularization helps achieve continuity and completeness in the latent space.

The lecture delves into the intuition behind VAEs, focusing on the benefits of regularization in achieving a smooth and continuous latent space. Without regularization, the latent space might have discontinuities or gaps, leading to poor quality reconstructions. The use of a normal prior on the latent space, enforced through KL Divergence, encourages the encoder to distribute the encoding smoothly. The concept of reparametrization is introduced to allow backpropagation through the stochastic sampling process in VAEs.

The importance of disentangled latent spaces in VAEs is discussed, where each latent variable captures a semantically meaningful and independent feature. The concept of beta-VAEs is introduced, which uses a weighting constant (beta) to control the strength of the regularization term in the loss function, encouraging greater disentanglement of the latent variables. The lecture demonstrates how perturbing individual latent variables can lead to interpretable changes in the decoded output.

The lecture transitions to generative adversarial networks (GANs), which consist of two competing neural networks: a generator that creates synthetic examples and a discriminator that classifies examples as real or fake. Through adversarial training, the generator aims to produce data that the discriminator cannot distinguish from the true data distribution. The process is illustrated with a 1D example, showing how the discriminator learns to separate real and fake data, while the generator improves its output to become increasingly indistinguishable.

The lecture covers advanced GAN architectures and their applications, such as iteratively growing GANs for detailed image generation and conditional GANs for paired translations between different types of data. It also discusses unpaired translations using cycle-consistent adversarial networks (CycleGANs), which can transform images between different domains without paired examples. The potential of GANs for audio-to-audio translation is mentioned, highlighting their versatility in generative modeling across various data types.

The lecture concludes with an introduction to diffusion models, a new approach in generative AI that has driven significant advances in the field. Unlike GANs, which are limited to generating examples similar to the training data, diffusion models can imagine and create completely new objects and instances. The lecture teasers the upcoming discussion on diffusion models in the context of new frontiers in deep learning, noting their potential to transform various fields and their position at the cutting edge of AI research.