Quantization and Edge Inference
Lower-bit weights and on-device serving.
Tier Grades
Tier A: Frontier AI labs
feasible
high confidence
OpenAI ships gpt-oss sized for 16GB and single 80GB GPU deployment.
Tier B: Indonesian AI labs
feasible
high confidence
AWQ needs only a small calibration set and no backpropagation.
Tier C: Institution units such as ITB
feasible
high confidence
Academic groups produced AWQ, and the method needs no training compute.
Tier D: Organisations such as SEMAI (PoT, PoC, PoV)
feasible
high confidence
gpt-oss-20b in 16GB and over 3x speedups on desktop and mobile GPUs fit proof-of-technology work on modest hardware.
Where this comes from
Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.
Anchors
- ComputeAWQ: weight-only low-bit quantization with small offline calibration and no backpropagation, over 3x faster than FP16 on desktop and mobile GPUs (TinyChat); gpt-oss-20b runs in 16GB, gpt-oss-120b on one 80GB GPU.
- DataSmall offline calibration set.
- Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): not sourced in the notes.
- Time horizonDays to weeks.
- Shelf-life signalsFixed-capability inference price falls 9x to 900x per year
- Pipeline roleTakes Tier A's open weights and makes them deployable on the hardware Tier B, Tier C and Tier D actually have; Tier D relies on it for on-premise and edge proofs of concept.
Where this comes from
Team size is a proxy (author counts of exemplar papers), not a headcount.
Who and What
Exemplars
Observed actors
Academic groups
OpenAI
OpenAILinked models
gpt-ossLLMexemplar
LFM2.5-8B-A1Bedge LLMproduced by
MiniCPM5 1Bedge LLMproduced by
Gemma-SEA-LION-v4.5-E2B-ITedge LLMproduced by
Gemma 4 E2Bedge LLMproduced byGemma 4 E4Bedge LLMproduced by
MobileCLIP2-S0embeddingused byGemma 3 270Medge LLMproduced byApple on-device foundation model (2025)edge LLMproduced byEmbeddingGemma 300Membeddingused by
Gemma 3n E2Bedge LLMproduced byMiniCPM4 / 4.1edge LLMproduced byMiniCPM4 0.5Bedge LLMproduced byGemma 3n E4Bedge LLMproduced byGemma 3edge LLMproduced by
Phi-4-mini-instructedge LLMproduced by
MobileLLM 1Bedge LLMproduced byMobileLLM 125Medge LLMproduced byLlama 3.2 1Bedge LLMused byLlama 3.2 3Bedge LLMused byPhi-3-miniedge LLMproduced byMobileCLIP-S0embeddingused byMobileLLMedge LLMproduced byMobileCLIPembeddingused by+26
faster-whisper-smallspeech recognitionproduced byfaster-whisper-tinyspeech recognitionproduced by
all-MiniLM-L6-v2 (Xenova, Transformers.js ONNX)embeddingused byEfficientDet-Litevisionproduced byEfficientNet-Litevisionproduced byMobileNet v5visionproduced byMobileNet-EdgeTPUvisionproduced byAutoSeg-EdgeTPUvisionproduced by
Intel binary, sparse and retrieval modelsvisionproduced byFunctionGemma 270Medge LLMproduced byGemini Nanoedge LLMproduced byGemma 3 1Bedge LLMproduced byGemma 3nedge LLMproduced by
SmolLM2edge LLMproduced by
Granite 4 350Medge LLMproduced byLFM2.5-Thinking 1.2Bedge LLMproduced byLlama 3.2 1B quantized (QLoRA)edge LLMproduced byLlama 3.2 1B quantized (SpinQuant)edge LLMproduced byLlama 3.2 3B quantized (QLoRA)edge LLMproduced byLlama 3.2 3B quantized (SpinQuant)edge LLMproduced byMoondream 1.8Bedge LLMproduced by
Ministral 3B (24.10)edge LLMproduced byBitCPM4 0.5Bedge LLMproduced byBitCPM4 1Bedge LLMproduced byMiniCPM3-1Bedge LLMproduced by
TinyLlama 1.1Bedge LLMproduced byLinks
R&D TopicQuantization and Edge Inference
Modelsgpt-ossOpenAI, LLMMobileNet v5Google, visionMobileNet-EdgeTPUGoogle, visionEfficientNet-LiteGoogle, vision+46 more
gpt-ossOpenAI, LLMMobileNet v5Google, visionMobileNet-EdgeTPUGoogle, visionEfficientNet-LiteGoogle, vision+46 more
Models 50
Gemma-SEA-LION-v4.5-E2B-ITAI Singapore, edge LLMproduced byGemma 3Google, edge LLMproduced bygpt-ossOpenAI, LLMexemplarGemma 4 E2BGoogle, Google DeepMind, edge LLMproduced byMiniCPM4 / 4.1OpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byGemma 3nGoogle DeepMind, edge LLMproduced byLlama 3.2 1BMeta, edge LLMused byGemma 3n E2BGoogle DeepMind, edge LLMproduced byGemma 3n E4BGoogle DeepMind, edge LLMproduced byLlama 3.2 3BMeta, edge LLMused byall-MiniLM-L6-v2 (Xenova, Transformers.js ONNX)Xenova (Transformers.js ports), embeddingused byAutoSeg-EdgeTPUGoogle, visionproduced by+38
BitCPM4 1BOpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byEfficientDet-LiteGoogle, visionproduced byfaster-whisper-smallSYSTRAN, speech recognitionproduced byfaster-whisper-tinySYSTRAN, speech recognitionproduced byGemini NanoGoogle, edge LLMproduced byGemma 4 E4BGoogle, Google DeepMind, edge LLMproduced byLFM2.5-8B-A1BLiquid AI, edge LLMproduced byLlama 3.2 1B quantized (QLoRA)Meta, edge LLMproduced byLlama 3.2 1B quantized (SpinQuant)Meta, edge LLMproduced byLlama 3.2 3B quantized (QLoRA)Meta, edge LLMproduced byLlama 3.2 3B quantized (SpinQuant)Meta, edge LLMproduced byMobileCLIPApple, embeddingused byMobileLLMMeta, edge LLMproduced byPhi-4-mini-instructMicrosoft AI, edge LLMproduced byBitCPM4 0.5BOpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byEfficientNet-LiteGoogle, visionproduced byEmbeddingGemma 300MGoogle, embeddingused byGemma 3 1BGoogle, edge LLMproduced byGemma 3 270MGoogle, edge LLMproduced byLFM2.5-Thinking 1.2BLiquid AI, edge LLMproduced byMiniCPM3-1BOpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byMiniCPM4 0.5BOpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byMinistral 3B (24.10)Mistral AI, edge LLMproduced byMobileCLIP-S0Apple, embeddingused byMobileCLIP2-S0Apple, embeddingused byMobileLLM 125MMeta, edge LLMproduced byMobileLLM 1BMeta, edge LLMproduced byMobileNet v5Google, visionproduced byMobileNet-EdgeTPUGoogle, visionproduced byPhi-3-miniMicrosoft AI, edge LLMproduced byApple on-device foundation model (2025)Apple, edge LLMproduced byFunctionGemma 270MGoogle, edge LLMproduced byGranite 4 350MIBM, edge LLMproduced byIntel binary, sparse and retrieval modelsIntel, visionproduced byMiniCPM5 1BOpenBMB (Tsinghua NLP Lab and ModelBest), edge LLMproduced byMoondream 1.8Bedge LLMproduced bySmolLM2Hugging Face, edge LLMproduced byTinyLlama 1.1BSingapore University of Technology and Design, edge LLMproduced byPapers
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleTim Dettmers et al., NeurIPS 2022 (per arXiv comment), 2022 founding
- AWQ: Activation-aware Weight Quantization for LLM Compression and AccelerationJi Lin et al., MLSys 2024 (per arXiv comment), 2023 state of the art
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersElias Frantar et al., ICLR 2023 (per arXiv comment), 2022 state of the art
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao et al., ICML 2023 (per arXiv comment), 2022 state of the art
- A Survey of Quantization Methods for Efficient Neural Network InferenceAmir Gholami et al., arXiv preprint, 2021 survey
Where this comes from
The papers to read first: where the direction began, its current best, and a survey when one exists.
Sources
- LLM inference prices have fallen rapidly but unequally across tasks epoch.ai https://epoch.ai/data-insights/llm-inference-price-trends
- openai.com/index/introducing-gpt-oss https://openai.com/index/introducing-gpt-oss/
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration arxiv.org https://arxiv.org/abs/2306.00978
To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Quantization and Edge Inference", https://aurion.semai.id/topics/quantization-and-edge-inference/, records of October 2026.