GPU Acceleration¶

Learning objectives:

• Understand the GPU parallelisation: data parallel, model parallel, pipeline parallel, etc.
• How to use GPU acceleration to train a model.
• How to use FSDP and DeepSpeed in Accelerate to train a model on multiple GPUs.

Resources:

• Make LLM training possible across multi-gpus using FSDP and DeepSpeed in Accelerate