Math 785R: Deep Generative Modeling

Authors: admin
Date: January 21, 2025

Overview

This graduate course explores the mathematical foundations of deep generative models, emphasizing theoretical principles and their connections to fields such as optimal transport, high-dimensional probability, and dynamical systems. Rather than focusing on applications to specific datasets, the course examines the underlying mathematical structures that power modern generative AI.

Topics Covered

The course provides a comprehensive mathematical treatment of:

  • Probabilistic Modeling - Foundations, mixture models, and probabilistic circuits
  • Autoregressive Models - Mathematical theory behind neural networks and transformers for sequence generation
  • Flow-Based Models - Normalizing flows and connections to optimal transport theory
  • Latent Variable Models - Variational autoencoders (VAEs) and hierarchical models
  • Hybrid Models - Combining different generative approaches
  • Energy-Based Models - Boltzmann machines and Markov Chain Monte Carlo methods
  • Generative Adversarial Networks (GANs) - Game-theoretic foundations of adversarial training
  • Score-Based Generative Models - Diffusion models, stochastic differential equations, and flow matching
  • Neural Compression - Information-theoretic perspectives on generative models
  • Large Language Models - Mathematical foundations of transformer architectures and scaling laws

Learning Approach

Through guided readings, active learning sessions, and mathematical problem solving, students develop a deep understanding of how generative models work, their computational implementations, and their theoretical properties. The course emphasizes connections between mathematical principles and computational methods, bridging theory and practice in machine learning.

Course Materials

Primary Text: Deep Generative Modeling by Jakub Tomczak
Supplementary Text: Probabilistic Machine Learning: Advanced Topics by Kevin Murphy

Prerequisites

Strong background in probability theory, linear algebra, numerical analysis, calculus, and programming experience.