Practical course about Large Language Models.

[SIGIR'24] The official implementation code of MOELoRA.

Official implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation"

An Efficient LLM Fine-Tuning Factory Optimized for MoE PEFT

Code for NOLA, an implementation of "nola: Compressing LoRA using Linear Combination of Random Basis"

Official repository of my book "A Hands-On Guide to Fine-Tuning LLMs with PyTorch and Hugging Face"

[ICML'24 Oral] APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

CRE-LLM: A Domain-Specific Chinese Relation Extraction Framework with Fine-tuned Large Language Model

Official code implemtation of paper AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

memory-efficient fine-tuning; support 24G GPU memory fine-tuning 7B

AI Community Tutorial, including: LoRA/Qlora LLM fine-tuning, Training GPT-2 from scratch, Generative Model Architecture, Content safety and control implementation, Model distillation techniques, Dreambooth techniques, Transfer learning, etc for practice with real project!

High Quality Image Generation Model - Powered with NVIDIA A100

A Python library for efficient and flexible cycle-consistency training of transformer models via iteratie back-translation. Memory and compute efficient techniques such as PEFT adapter switching allow for 7.5x larger models to be trained on the same hardware.

[ICML 2025] Fast and Low-Cost Genomic Foundation Models via Outlier Removal.

Mistral and Mixtral (MoE) from scratch

【AIGC 实战入门笔记 —— AIGC 摩天大楼】分享 大语言模型（LLMs），大模型高效微调（SFT）,检索增强生成（RAG），智能体（Agent），PPT自动生成, 角色扮演，文生图（Stable Diffusion） ，图像文字识别（OCR），语音识别（ASR），语音合成（TTS），人像分割（SA），多模态（VLM），Ai 换脸(Face Swapping), 文生视频(VD)，图生视频（SVD），Ai 动作迁移，Ai 虚拟试衣，数字人，全模态理解（Omni），Ai音乐生成 干货学习 等 实战与经验。

EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

Fine-tune StarCoder2-3b for SQL tasks on limited resources with LORA. LORA reduces model size for faster training on smaller datasets. StarCoder2 is a family of code generation models (3B, 7B, and 15B), trained on 600+ programming languages from The Stack v2 and some natural language text such as Wikipedia, Arxiv, and GitHub issues.

Fine-tuning Llama3 8b to generate JSON formats for arithmetic questions and process the output to perform calculations.

PEFT is a wonderful tool that enables training a very large model in a low resource environment. Quantization and PEFT will enable widespread adoption of LLM.