Welcome to my LLM lab!

This is a repo of my experiments and notes while learning about LLMs. I'm starting with a decent theoretical understanding of neural networks, and hands on experience training large models on distributed systems. I'm very comfortable with data and ML engineering.

What's done

I've completed:

• Andrej Karpathy's Neural Networks: Zero to Hero guide.
• Thoroughly read The Annotated Transformers paper and run the code side-by-side.

I've read:

• Efficiency:
  • Flash Attention, Flash Attention 2, Flash Attention 3
  • Paged Attention
  • Online Softmax
  • Self-attention Does Not Need O(n2) Memory
• Synthetic Data
  • GenAI for Synthetic Data Gen: Methods, Challenges and the Future
  • A survey of GenAI for Synthetic Data
  • Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning
  • Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
• Modelling
  • LLaMA

What's up next

Here are all the things I'd like to do:

Implementations:

• Implement FlashAttention myself (in Cuda maybe?)
• Implement FSDP myself (no idea how!?)

Experiments

• Model efficiency experiments. Try out the following and benchmark performance changes:
  • Speculative decoding
  • Knowledge distillation
  • Quantization
  • Pruning
  • Sparsity low ran compression
  • etc
• Play around with LLAMA models locally

Readings:

• Depthwise Seperable Convolutions for NMT
• One Model To Learn Them All
• Self-Attention with Relative Position Representations
• GANs
• Stable Diffusion
• KANs
• Explore LLM evaluation
• Explore LLM interpretability