Ph.D., Computer Science
Chief Scientist, Luma AI
Curriculum Vitae (outdated)
I am Chief Scientist in Luma AI, where we are working on next-generation multimodal foundation models.
During Luma AI, I led the training efforts on Dream Machine, a large-scale video generation model, and Genie, a text-to-3d generation model.
Prior to joining Luma AI, I was:
- A research scientist in the Deep Imagination Research (DIR) group of NVIDIA Research. I worked on NVIDIA AI Foundations, specifically the Picasso project.
- A research scientist in the Learning and Perception Research (LPR) group of NVIDIA Research. I was the first author of an ICLR 2023 and an ICML 2023 paper and also collaborated on various other papers.
- A postdoc in Computer Science in Stanford University working with Stefano Ermon.
- Ph.D. in Computer Science at Stanford University advised by Stefano as well.
- Undergrad in the Department of Computer Science and Technology, Tsinghua University, where I worked with Jun Zhu and Lawrence Carin.
My “generative AI” qualifications:
- I created the earliest accelerated algorithm for diffusion models that is widely used in recent generative AI systems including DALL-E 2, Imagen, Stable Diffusion, and ERNIE-ViLG 2.0.
- I co-authored the paper that is the foundation of Stable Diffusion’s img2img method.
- I also democratized the use of diffusion models for general applications, such as super-resolution, inpainting, and JPEG restoration.
Email: jiaming [dot] tsong [atgmaildotcom]
News
- Organized the CVPR 2023 diffusion model tutorial. Slides are available.
- Organizing the SIGGRAPH 2023 Course on Diffusion Models.
- Talks at University of Edinburgh, DeepMind, Imperial College London, and Oxford. [slides]
Publications
2023
- [58]Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation
ICML 2023, In International Conference on Machine Learning. - [59]DiffCollage: Parallel Generation of Large Content with Diffusion Models
CVPR 2023, In The IEEE/CVF Conference on Computer Vision and Pattern Recognition. - [57]Pseudoinverse-Guided Diffusion Models for Inverse Problems
ICLR 2023, In International Conference on Learning Representations. - [56]Dual Diffusion Implicit Bridges for Image-to-Image Translation
ICLR 2023, In International Conference on Learning Representations.[website] [code] [colab] - [55]Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
AAAI 2023, In AAAI Conference on Artificial Intelligence.
2022
- [54]eDiff-I: Text-to-Image Diffusion Models with Ensemble of Expert Denoisers
Preprint, arXiv preprint arXiv:2211.01324.[website] - [53]JPEG Artifact Correction using Denoising Diffusion Restoration Models
NeurIPS 2022 SBM Workshop, In Neural Information Processing Systems (NeurIPS) Workshop on Score-Based Methods.[website] [code] - [52]Denoising Diffusion Restoration Models
NeurIPS 2022, In Neural Information Processing Systems.[website] [code] Short version in ICLR 2022 Workshop on Deep Generative Models for Highly Structured Data (Oral presentation). - [51]Concrete Score Matching: Generalized Score Matching for Discrete Data
NeurIPS 2022, In Neural Information Processing Systems. - [50]LISA: Learning Interpretable Skill Abstractions from Language
NeurIPS 2022, In Neural Information Processing Systems. - [49]A General Recipe for Likelihood-free Bayesian Optimization
ICML 2022, In International Conference on Machine Learning, Long oral presentation (Top 2.2%).[website] [code] - [48]Experience Replay with Likelihood-free Importance Weights
L4DC 2022, In 4th Annual Conference on Learning for Dynamics and Control, Best paper award finalist. - [47]SDEdit: Image Synthesis and Editing with Stochastic Differential Equations
ICLR 2022, In International Conference on Learning Representations.[website] [code] [colab] - [46]Comparing Distributions by Measuring Differences that Affect Decision Making
ICLR 2022, In International Conference on Learning Representations, ICLR 2022 Outstanding Paper Award. - [45]IS-COUNT: Large-scale Object Counting from Satellite Images with Covariate-based Importance Sampling
AAAI 2022, In AAAI Conference on Artificial Intelligence.[website] [code] [colab] - [44]Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models
Preprint, arXiv preprint arXiv:2206.04615.
2021
- [41]D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
NeurIPS 2021, In Neural Information Processing Systems.[website] [code] [colab] - [40]IQ-Learn: Inverse soft-Q Learning for Imitation
NeurIPS 2021, In Neural Information Processing Systems, Spotlight presentation.[website] [code] - [39]Pseudo-Spherical Contrastive Divergence
NeurIPS 2021, In Neural Information Processing Systems. - [38]CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation
NeurIPS 2021, In Neural Information Processing Systems.[code] - [37]Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
NeurIPS 2021, In Neural Information Processing Systems.[website] [code] - [36]Imitation with Neural Density Models
NeurIPS 2021, In Neural Information Processing Systems. - [35]Denoising Diffusion Implicit Models
ICLR 2021, In International Conference on Learning Representations.[code] - [34]Negative Data Augmentation
ICLR 2021, In International Conference on Learning Representations.[code] - [33]Improved Autoregressive Modeling with Distribution Smoothing
ICLR 2021, In International Conference on Learning Representations, Oral presentation.[code]
2020
- [32]Multi-label Contrastive Predictive Coding
NeurIPS 2020, In Neural Information Processing Systems, Oral presentation. - [31]Autoregressive Score Matching
NeurIPS 2020, In Neural Information Processing Systems. - [30]Belief Propagation Neural Networks
NeurIPS 2020, In Neural Information Processing Systems. - [29]Robust and On-the-fly Dataset Denoising for Image Classification
ECCV 2020, In European Conference on Computer Vision.[slides] - [28]Permutation Invariant Graph Generation via Score-Based Generative Modeling
AISTATS 2020, In International Conference on Artificial Intelligence and Statistics.[code] - [27]Gaussianization Flows
AISTATS 2020, In International Conference on Artificial Intelligence and Statistics.[code] - [26]Training Deep Energy-Based Models with f-Divergence Minimization
ICML 2020, In International Conference on Machine Learning. - [25]Bridging the Gap Between f-GANs and Wasserstein GANs
ICML 2020, In International Conference on Machine Learning.[slides] [code] - [24]Domain Adaptive Imitation Learning
ICML 2020, In International Conference on Machine Learning. - [23]Understanding the Limitations of Variational Mutual Information Estimators
ICLR 2020, In International Conference on Learning Representations.[slides] [code] - [22]A Theory of Usable Information under Computational Constraints
ICLR 2020, In International Conference on Learning Representations, Oral presentation. - [21]Multi-agent Adversarial Inverse Reinforcement Learning with Latent Variables
AAMAS 2020, In International Conference on Autonomous Agents and MultiAgent Systems (extended abstract). - [20]Privacy Preserving Recalibration under Domain Shift
Preprint, arXiv:2008.09643.
2019
- [19]Bias Correction of Learned Generative Models using Likelihood-free Importance Weighting
NeurIPS 2019, In Advances in Neural Information Processing Systems. - [18]Calibrated Model-based Deep Reinforcement Learning
ICML 2019, In International Conference on Machine Learning.[code] - [17]Multi-agent Adversarial Inverse Reinforcement Learning
ICML 2019, In International Conference on Machine Learning.[code] - [16]InfoVAE: Balancing Learning and Inference in Variational Autoencoders
AAAI 2019, In AAAI Conference on Artificial Intelligence. - [15]Learning Controllable Fair Representations
AISTATS 2019, In International Conference on Artificial Intelligence and Statistics.[code] - [14]Unsupervised Out-of-Distribution Detection with Batch Normalization
Preprint, arXiv:1910.09115.
2018
- [13]Multi-Agent Generative Adversarial Imitation Learning
NeurIPS 2018, In Advances in Neural Information Processing Systems.[code] - [12]Bias and Generalization in Deep Generative Models: An Empirical Study
NeurIPS 2018, In Advances in Neural Information Processing Systems, Spotlight presentation.[code] - [11]The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
UAI 2018, In Conference on Uncertainty in Artificial Intelligence, Oral presentation.[code] - [10]Accelerating Natural Gradient with Higher-Order Invariance
ICML 2018, In International Conference on Machine Learning.[code] - [9]Adversarial Constraint Learning for Structured Prediction
IJCAI 2018, In International Joint Conference on Artificial Intelligence.[code] - [8]Learning with weak supervision from physics and data-driven constraints
AI Magazine, AI Magazine.
2017
- [7]A-NICE-MC: Adversarial training for MCMC
NeurIPS 2017, In Advances in Neural Information Processing Systems.[slides] [code] [blog] - [5]InfoGAIL: Interpretable imitation learning from visual demonstrations
NeurIPS 2017, In Advances in Neural Information Processing Systems.[code] - [6]Learning Hierarchical Features from Deep Generative Models
ICML 2017, In International Conference on Machine Learning.[code] - [4]Towards deeper understanding of variational autoencoding models
Preprint, arXiv:1702.08658.
2016
- [3]Factored Temporal Sigmoid Belief Networks for Sequence Learning
ICML 2016, In International Conference on Machine Learning. - [2]Discriminative nonparametric latent feature relational models with data augmentation
AAAI 2016, In AAAI Conference on Artificial Intelligence. - [1]Max-margin Nonparametric Latent Feature Models for Link Prediction
Preprint, arXiv:1602.07428.
Professional Services
Journal reviewer: IEEE TPAMI, JAIR, IEEE TIT, ACM TIST, JMLR
Conference reviewer / Program committee: ICML (2019, 2020, 2021), NeurIPS (2019, 2020, 2021), ICLR (2018, 2019, 2020, 2021), COLT (2019), UAI (2019, 2020, 2021), CVPR (2020, 2021), ECCV (2020), ICCV (2019, 2021), AAAI (2021), ACML (2018, 2019), WACV (2020)
Workshop organization:
- NeurIPS 2019 Workshop on Information Theory and Machine Learning (chair)
- DALI 2018 Workshop on Generative Models and Reinforcement Learning (chair)
Awards and Fellowships
- ICLR 2022 Outstanding Paper Award
- Qualcomm Innovation Fellowship (QInF 2018, 4.6%)
- Stanford School of Engineering Fellowship (2016)
- Google Excellence Scholarship (2015)
- Outstanding Undergraduate, China Computer Federation (2015)
- Outstanding Winner, Interdisciplinary Contest in Modeling (2015, 0.4%)
- Zhong Shimo Scholarship (2013, 0.75%)
Teaching
- CS228, Probablistic Graphical Models (Winter 2020, Head TA)
- CS236, Deep Generative Models (Fall 2018, TA)
Acknowledgements: based on the al-folio template by Maruan Al-Shedivat.