Parameter-Efficient Fine-Tuning (PEFT)
Adapting models by training a small subset of parameters or a low-rank update.
Tier Grades
Tier A: Frontier AI labs
feasible
high confidence
Broadly used by industry, and a 65B fine-tune fits one 48GB GPU.
Tier B: Indonesian AI labs
feasible
high confidence
QLoRA fits a 65B model on one 48GB GPU in 24 hours.
Tier C: Institution units such as ITB
feasible
high confidence
One 48GB GPU for 24 hours falls inside the academic A6000 or A100 range of 1 to 8 GPUs.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
high confidence
NVIDIA states SLM fine-tuning needs a few GPU-hours, within a proof-of-concept budget.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeLoRA: about 10,000x fewer trainable parameters and about 3x less memory than full fine-tuning of GPT-3 175B; QLoRA: 65B on one 48GB GPU, Guanaco 99.3% of ChatGPT on Vicuna after 24 hours on one GPU.
- DataTask-specific fine-tuning data; volume not given.
- Team size (proxy)Proxy only: LoRA and QLoRA papers have small author lists (counts not given).
- Time horizonDays.
- Shelf-life signalsnot in the notes
- Pipeline roleMain adaptation path: Tier A's open weights are tuned by Tier B and Tier C with LoRA or QLoRA, and Tier D uses the same path for client-specific proofs of concept.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Links
R&D TopicParameter-Efficient Fine-Tuning (PEFT)
Use Casesnone recorded
Who Runs Itnone recorded
Papers
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu et al., arXiv preprint, 2021 founding
- Parameter-Efficient Transfer Learning for NLPNeil Houlsby et al., arXiv preprint, 2019 founding
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers et al., arXiv preprint, 2023 state of the art
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive SurveyZeyu Han et al., arXiv preprint, 2024 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- $100K or 100 Days:Trade-offs when Pre-Training with Academic Resources arxiv.org https://arxiv.org/html/2410.23261v1
- Small Language Models are the Future of Agentic AI arxiv.org https://arxiv.org/html/2506.02153v1
- LoRA: Low-Rank Adaptation of Large Language Models arxiv.org https://arxiv.org/abs/2106.09685
- QLoRA: Efficient Finetuning of Quantized LLMs arxiv.org https://arxiv.org/abs/2305.14314
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Parameter-Efficient Fine-Tuning (PEFT)", https://aurion.semai.id/topics/parameter-efficient-fine-tuning/, records of October 2026.