Metadata-Version: 2.1
Name: tglcourse
Version: 0.0.2
Summary: work-in-progress course
Home-page: https://github.com/johnowhitaker/tglcourse/
Author: Jonathan Whitaker
Author-email: johnowhitaker@gmail.com
License: Apache Software License 2.0
Keywords: generative
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
License-File: LICENSE

The Generative Landscape
================

<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

Work has just started on this - for now, if you’re interested in these
topics you might want to check out the previous course I ran on AI ART:
[AIAIART](https://github.com/johnowhitaker/aiaiart)

If you want to be notified when the course goes ‘live’ you can sign up
for an email at <http://thegenerativelandscape.com/>

If you want to support this effort, I now have a patreon:
https://www.patreon.com/johnowhitaker

The material will show in github pages at
<https://johnowhitaker.github.io/tglcourse/> (and the
http://thegenerativelandscape.com will redirect there once we launch).
Hooray for the magic of nbdev.

<div>

[![](https://github.com/johnowhitaker/tglcourse/actions/workflows/test.yaml/badge.svg)](https://github.com/johnowhitaker/tglcourse/actions/workflows/test.yaml)

</div>

![](index_files/figure-gfm/cell-2-output-1.png)

## Curriculum plan / TODO

### Topics:

- Intro: The Generative Landscape
- PyTorch, AutoGrad and Gradient Descent
- Constructing ANNs, Loss Functions
- Representations: What do ANNs Learn? Style Transfer
- CLIP/CLOOB: Multiple Modalities in a Shared Latent Space. Use as a
  loss function
- Generative modelling: Noise -\> Data (Auto-Encoders and VAEs)
- GANs 1 - GAN training, a simple DC-GAN
- GANs 2 - Conditioning, Improvements, Modern GANs, CLIP guidance
- Sequence Modelling 1 - Ideas, RNNs, Language Models as Representation
  Learners
- Sequence Modelling 2 - Transformers
- Sequence Modelling 3 - Everything is a sequence (VIT, VQGANs,
  Dalle-mini, Parti, flamingo, music generation, other applications)
- Diffusion Models 1 - A New Type of Generative Model
- Diffusion Models 2 - Conditioning, Guiding, Improvements
- Diffusion Models 3 - The Current DM Landscape, Using Diffusion Models
  Creatively (inpainting, guiding, animation)
- Spotlight: Audio
- Spotlight: Video

### Extra Skills / Bonus Material:

- Datasets and Dataloaders Intro (also defines data util funcs for the
  rest of the course)
- Experiment Tracking (eg W&B)
- Pretrained networks and transfer learning
- **Ethics in generative modelling**
- Fine-tuning existing models
- Working with GPUs?
- Multi-GPU or TPU training?
- Sharing demos w/ Gradio
- Managing cloud machines
- Dataloaders deep dive, streaming data
- Navigating other codebases
- Version control and CI (+ NBDev)
- “Text inversion” (https://arxiv.org/pdf/2208.01618.pdf)
- “prompt to prompt editing” https://arxiv.org/pdf/2208.01626.pdf
- Lots of paper readings / sumaries

### Projects:

- Train a GAN, explore hyperparameters
- Fine-tune a diffusion model on a custom dataset
- Create and share a final project, including a report and demo

Guest discussions: as many as we can :)

Paper explainers: as many as we can :)

### Status

| Lesson                              | Description                                                                     | TODOs                                                                          | Status           |
|-------------------------------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------|
| Lesson 1: PyTorch Basics            | Intro to PT, tensor manipulation, images as tensors, LeastAverageImage exercise | Video, tidy                                                                    | Rough Draft Done |
| Lesson 2: Optimization              | Intro to GD, optimization examples exercise                                     | Video, Optimize with gen+loss without explaining                               | WIP              |
| Lesson 3: Building NNs              | nn.Module, building blocks, CNNs                                                | Video, project, export functions? optimization/training tricks?                | WIP              |
| Lesson 4: Learning Representations  | What do networks learn, style transfer                                          | Video, tidy, …                                                                 | WIP              |
| Lesson 5: CLIP                      | Contrastive loss, CLIP, as a loss function                                      | everything (pull from AIAIART)                                                 | Not Started      |
| Lesson 6: Generative Modelling      | Intro to generative models, AEs and VAEs, issues                                | everything (pull from AIAIART)                                                 | Not Started      |
| Lesson 7: GANs 1                    | Intro to GANs, DC GAN, Conditioning                                             | everything                                                                     | Not Started      |
| Lesson 8: GANs 2                    | GAN training tricks, NOGAN, using modern GANs, VQGAN                            | everything                                                                     | Not Started      |
| Lesson 9: Sequence Modelling Intro  | idea, language modelling concept, transformer demo                              | everything                                                                     | Not Started      |
| Lesson 10: Transformers             | Intro to transformes, attention, from scratch or dissect existing ones? ViT?    | everything, check Niels’ repo and other resources                              | Not Started      |
| Lesson 11: Everythign is a sequence | Show whistlegen, protein, VQGAN, parti…                                         | everything                                                                     | Not Started      |
| Lesson 12: DM 1                     | Intro to diffusion models, toy example, comparison to DDPM                      | Video, tidy, second half, logs of my version training, better unet replacement | Rough Draft      |
| Lesson 13: DM2                      | Conditioning, CFG, guiding, sampling, better training                           | everything                                                                     | Not Started      |
| Lesson 14: DM3                      | SD deep dive                                                                    | combine grokking 1 and 2, polish                                               | Rough Draft Done |
| Lesson 15: DM4                      | Other modalities - ideally demo class-conditioned audio generation              | wavenet style diffusion for birdcalls, everything                              | Not Started      |

General TODOs - Create a page for project explanations + figure out
submission info. - Figure out what the discussions page (pages) looks
like - Ethics page but also callouts in each lesson - Pick a cloud for
cloud stuff (sponsor?) - Figure out plan for datasets - Figure out
losses+generators notebook/library bit and include in the plan
somewhere - Figure out where smaller ‘bonus’ things like paper readings
or topical deep-dives go. Do we add a blog? - Make the website pretty -
Generate a bunch of images - Get Dave to make video assets + logo -
Streamline skipping tests / avoiding anything long-running during CI -
Set up org on HF for sharing models - Set up W&B projects for sharing
training runs/reports - Notebook to document the library
