Zero-Shot Extraction and Classification with Small Encoders

Using compact encoders to extract entities or assign labels named at run time, as a cheaper alternative to large language models.

Tier Grades

Tier A: Frontier AI labs

feasible

Microsoft distilled knowledge from an LLM into an XLM-RoBERTa-large encoder for LLMLingua-2.

Tier B: Indonesian AI labs

feasible

GLiNER beats ChatGPT zero-shot on NER with a compact encoder, cheap for a national lab to adapt to Indonesian.

Tier C: Institution units such as ITB

feasible

GLiNER came from a 4-author team.

Tier D: Organisations such as SEMAI (PoT, PoC, PoV)

feasible

Zero-shot extraction lets a proof of concept pull entities from client documents without labelled data.

Where this comes from

Tiers: a) frontier AI labs, b) Indonesian AI labs, c) institution units such as ITB, d) organisations such as SEMAI delivering proof of technology, concept and value. Each rationale cites the anchors below.

Anchors

ComputeLLMLingua-2 uses XLM-RoBERTa-large and mBERT and is 3x to 6x faster than earlier prompt compression; GLiNER gives no compute.
DataYin et al. unify zero-shot text classification as textual entailment; LLMLingua-2 builds an extractive compression dataset distilled from an LLM.
Team size (proxy)Proxy only (author count of an exemplar paper, not a headcount): 4 authors on GLiNER; 3 on the zero-shot classification benchmark.
Time horizonWeeks to months.
Shelf-life signalsSmall LLMs becoming as cheap as encoders for extraction
Pipeline roleTier C and Tier A publish compact extractors; Tier B adapts them to Indonesian; Tier D uses them for document triage and extraction.

Where this comes from

Team size is a proxy (author counts of exemplar papers), not a headcount.

Who and What

Exemplars

GLiNER

BART large MNLI Microsoft AI

LLMLingua-2

Observed actors

Urchade Zaratiana et al. (GLiNER authors) Meta

Links

R&D TopicZero-Shot Extraction and Classification with Small Encoders

ModelsGLiNERUrchade Zaratiana et al. (GLiNER authors), language encoder

BART large MNLIMeta, language encoder

LLMLingua-2Microsoft AI, language encoder GLiNER multi v2.1Urchade Zaratiana et al. (GLiNER authors), language encoder+2 more

Use Casesnone recorded

Who Runs Itnone recorded

Graph

Models 6

GLiNERUrchade Zaratiana et al. (GLiNER authors), language encoderexemplar

BART large MNLIMeta, language encoderexemplar GLiNER medium v2.1Urchade Zaratiana et al. (GLiNER authors), language encoderproduced by GLiNER multi v2.1Urchade Zaratiana et al. (GLiNER authors), language encoderproduced by bert-base-NERdslim (David S. Lim), language encoderproduced by

LLMLingua-2Microsoft AI, language encoderexemplar

Papers

Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment ApproachWenpeng Yin et al., EMNLP2019 (per arXiv comment), 2019 founding
GLiREL -- Generalist Model for Zero-Shot Relation ExtractionJack Boylan et al., NAACL 2025 (per arXiv comment), 2025 state of the art
NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated DataSergei Bogdanov et al., arXiv preprint, 2024 state of the art
UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity RecognitionWenxuan Zhou et al., ICLR 2024 (per arXiv comment), 2023 state of the art
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional TransformerUrchade Zaratiana et al., arXiv preprint, 2023 state of the art

Where this comes from

The papers to read first: where the direction began, its current best, and a survey when one exists.

Sources

arxiv.org/abs/2311.08526 https://arxiv.org/abs/2311.08526
arxiv.org/abs/2403.12968 https://arxiv.org/abs/2403.12968
arxiv.org/abs/1909.00161 https://arxiv.org/abs/1909.00161

To cite this page: Project AURION, AI Use-Cases Research & Innovation, "Zero-Shot Extraction and Classification with Small Encoders", https://aurion.semai.id/topics/zero-shot-extraction-small-encoders/, records of October 2026.