HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation

Teaser

Hierarchies in text and Ricci-curvature evidence for tree-like structure in document embeddings

Documents naturally organize into branching hierarchies, from general topics to specific entities (A). Euclidean spaces distort these hierarchies through crowding, whereas hyperbolic geometry preserves them via exponential volume growth. Ollivier–Ricci curvature of document-embedding graphs from strong retrievers (Linq Embed Mistral, LLaMA Nemotron 8B, Qwen3 Embedding 4B) is predominantly negative (B) — empirical evidence of intrinsic tree-like structure.

TL;DR

Dense retrievers for RAG are almost universally Euclidean, yet language is hierarchical — and Euclidean geometry cannot represent branching hierarchies without crowding, which makes semantically distant documents look spuriously similar and increases hallucination. We introduce hyperbolic dense retrieval, with two encoders in the Lorentz model: HyTE-FH (fully hyperbolic transformer) and HyTE-H (hybrid, projecting pretrained Euclidean embeddings into hyperbolic space). To stop sequence pooling from collapsing hierarchical structure, we propose the Outward Einstein Midpoint (OEM), a geometry-aware pooling operator that provably preserves radial depth. HyTE-FH beats matched Euclidean baselines on MTEB, hyperbolic retrieval delivers up to 29% gains over Euclidean baselines in context and answer relevance on RAGBench, HyTE-H stays competitive with SOTA retrievers several times its size, and hyperbolic embeddings encode document specificity through norm-based separation (a +20.2% radial increase from general to specific concepts) — a property absent in Euclidean embeddings.

Motivation

Retrieval quality is the bottleneck of RAG: if the top-$k$ documents are wrong, generation is ungrounded no matter how strong the LLM. Natural language is strongly hierarchical — broad topics branch into increasingly specific subtopics, inducing locally tree-like neighborhoods. Euclidean spaces struggle to represent such branching structure because their volume grows only polynomially, forcing a trade-off between keeping general concepts close and separating fine-grained descendants. The result is neighborhood crowding: a query about a specific subtopic retrieves sibling or parent documents that are similar but lack the required specificity.

This is not just theory. Building $k$-NN graphs on MS MARCO document embeddings and measuring Ollivier–Ricci curvature — where negative values indicate tree-like branching — reveals predominantly negative curvature across several strong retrievers (Linq Embed Mistral, LLaMA Nemotron 8B, Qwen3 Embedding 4B). Retrieval-relevant embeddings already carry intrinsic hyperbolic structure, motivating hyperbolic geometry as a natural inductive bias for dense retrieval.

Prior hyperbolic work does not close this gap. HELM operates entirely in hyperbolic space but targets generation, not retrieval. HyperbolicRAG projects embeddings into the Poincaré ball to encode hierarchy in a static knowledge graph — but relies on Euclidean encoders for the initial embeddings, leaving the geometric mismatch in place. Critically, aggregating tokens into document embeddings is itself an unsolved problem in hyperbolic space: naive averaging collapses representations toward the origin, destroying the very hierarchy that is meant to be preserved.

Key Ideas

1. Hyperbolic dense retrieval, two ways

We frame embedding geometry as a design choice for grounding and study two complementary instantiations:

• HyTE-FH (Hyperbolic Text Encoder, Fully Hyperbolic) — operates entirely in the Lorentz model, enabling end-to-end representation learning where semantic abstraction and specificity are encoded along the radial axis.
• HyTE-H (Hybrid) — projects embeddings from off-the-shelf Euclidean encoders into hyperbolic space, leveraging strong pretrained initializations. Its intrinsic geometry enables parameter-efficient scaling: HyTE-H outperforms Euclidean baselines several times its size.

2. The Outward Einstein Midpoint (OEM)

Both variants need to pool variable-length token sequences into a fixed document vector. We prove standard strategies fail, and introduce a pooling operator that provably preserves hierarchical structure by amplifying tokens farther from the origin.

3. Drop-in with existing retrieval stacks

Hyperbolic nearest-neighbor search in the Lorentz model reduces exactly to maximum inner product search (MIPS) via a simple sign-flip of the time coordinate — so HyTE runs unmodified through FAISS IndexFlatIP and HNSW, making it a drop-in replacement for Euclidean encoders.

Method

Lorentz model preliminaries

All embeddings live in $d$-dimensional hyperbolic space $\mathbb{H}^d_K$ with constant negative curvature $K < 0$, realized as the upper sheet of a hyperboloid in $\mathbb{R}^{d+1}$:

The geodesic distance is a monotone function of the Lorentzian inner product:

HyTE-FH replaces every transformer operation with its Lorentzian counterpart — the Lorentz linear layer, hyperbolic layer normalization, Lorentz residual connection, and hyperbolic self-attention (where attention weights use squared geodesic distances and values are aggregated via a Lorentzian weighted midpoint). All operations keep representations on the manifold, preserving the radial structure induced by negative curvature.

Architecture

HyTE architectures: HyTE-FH block, fully hyperbolic HyTE-FH, and hybrid HyTE-H

HyTE architecture. (A) The HyTE-FH block — multi-head latent attention and a Lorentz FFN, each wrapped in Lorentz LayerNorm and Lorentz residual connections. (B) HyTE-FH stacks $L$ such blocks over hyperbolic token embeddings, then pools with OEM into a document embedding. (C) HyTE-H feeds text through a pretrained Euclidean encoder, projects the token embeddings into hyperbolic space, and pools with OEM.

Why naive hyperbolic pooling fails

Standard practice maps to tangent space, averages in Euclidean space, then maps back — but this systematically contracts representations toward the origin, erasing the radial separation that encodes specificity. We define the radial depth of $\mathbf{x} \in \mathbb{H}^d_K$ as $r(\mathbf{x}) = x_0$ (equivalent to ordering by hyperbolic distance from the origin), and formalize the failure:

Proposition (Euclidean mean contracts). For $\{\mathbf{x}_i\}_{i=1}^n \subset \mathbb{H}^d_K$ with $n \geq 2$, the projected Euclidean mean $\mathbf{m}^{\mathrm{Euc}} = \Pi_K(\bar{\mathbf{x}})$ satisfies $r(\mathbf{m}^{\mathrm{Euc}}) \leq \tfrac{1}{n}\sum_i r(\mathbf{x}_i)$, with equality iff all $\mathbf{x}_i$ are identical.

The standard Einstein midpoint weights points by the Lorentz factor $\lambda_i = x_{i,0}$, but this is still not enough:

Proposition (implicit radial weighting is insufficient). The Lorentz factor grows as $\exp(\sqrt{-K}\,\rho)$ while hyperbolic volume grows as $\exp((d-1)\sqrt{-K}\,\rho)$, so Einstein weighting undercompensates by a factor of $\exp((d-2)\sqrt{-K}\,\rho)$ for $d \geq 3$.

A correct pooling operator must be outward-biased: $r(\mathcal{P}(\{\mathbf{x}_i\})) \geq \bar{r}$, the (weighted) mean radius.

Outward Einstein Midpoint pooling

Naive pooling collapses toward the origin while the Outward Einstein Midpoint pushes the document embedding outward

Outward Einstein Midpoint. Naive pooling drags the document embedding (red) toward the origin; OEM up-weights tokens at larger radius so the aggregate is pushed outward, preserving specificity. Marker size reflects each token's contribution.

OEM introduces a radius-dependent weighting $\phi_p(\mathbf{x}_i) = x_{i,0}^{\,p}$ ($p > 0$) on top of attention weights $w_i$ and Lorentz factors $\lambda_i$:

followed by reprojection onto $\mathbb{H}^d_K$. We prove OEM corrects the contraction:

Theorem (OEM pre-projection bound). For $p \geq 0$, the OEM weighted average has time component $\tilde{v}_0 = \frac{\sum_i w_i x_{i,0}^{p+2}}{\sum_i w_i x_{i,0}^{p+1}} \geq \bar{r}_w$ (proved via Chebyshev's sum inequality on the co-monotonic sequences $x_{i,0}^{p+1}$ and $x_{i,0}$).

Theorem (OEM outward bias). For all $p \geq 1$, $r(\mathbf{m}^{\mathrm{OEM}}_{K,p}) \geq r(\mathbf{m}^{\mathrm{Ein}}_K)$ — OEM provably dominates the standard Einstein midpoint in preserving radial structure.

Training

HypRAG training pipeline: hyperbolic MLM, then contrastive pre-training and fine-tuning with OEM pooling and geodesic similarity

Training pipeline. Stage 1 initializes the encoder with hyperbolic masked language modeling using a Lorentzian MLR head. Stages 2 and 3 add the OEM pooler and train contrastively with geodesic similarity.

We follow a three-stage recipe (after nomic-embed), with all objectives operating directly on the manifold via geodesic similarity:

1. Hyperbolic Masked Language Modeling — BERT-style MLM in hyperbolic space with a Lorentzian multinomial logistic regression (LorentzMLR) head. Only HyTE-FH is trained here; HyTE-H starts from a pretrained Euclidean MLM base.
2. Contrastive pre-training — InfoNCE over ~235M general-domain query–document pairs with a large batch size (16,384), using negative geodesic distance as similarity.
3. Contrastive fine-tuning — refinement on curated query–document data with mined hard negatives:

Hyperbolic retrieval is just MIPS

Because $d_K$ is monotone in $\langle \mathbf{x}, \mathbf{y}\rangle_L = -x_0 y_0 + \mathbf{x}^\top \mathbf{y}$, negating each document's time coordinate ($x_0 \to -x_0$) turns hyperbolic ranking into a standard dot product. Exact retrieval runs through FAISS IndexFlatIP unmodified, and HNSW works out of the box — HyTE is a drop-in replacement for Euclidean encoders in any retrieval stack.

Experiments

All three core models — HyTE-FH, HyTE-H, and the Euclidean control EucBERT — share the same architecture: 149M parameters, 12 transformer layers, 768-dimensional embeddings, trained on NVIDIA H100 GPUs with HyperCore. RAG generation and judging use Llama-3.1-8B-Instruct, RAG metrics use RAGAS, and the retrieval context window is fixed at 5 documents unless noted.

MTEB

HyTE-FH achieves the best mean score across tasks, with both hyperbolic variants beating the Euclidean control.

RAGBench — controlled comparison

Against the matched Euclidean control, both hyperbolic variants substantially improve faithfulness (F), context relevance (CR), and answer relevance (AR), averaged across five datasets.

Full per-dataset breakdown:

RAGBench — vs. state-of-the-art retrievers

HyTE-H, initialized from ModernBERT-base or Qwen-Embedding, is competitive with or beats much larger SOTA retrievers, especially in context and answer relevance.

Full per-dataset breakdown:

Concept-level hierarchy analysis

Embedding norms across five hierarchical specificity levels: Euclidean models flatten or invert, hyperbolic models increase

Empirical validation of hierarchical encoding. Across five levels of increasing specificity (Science → Mathematics → Algebra → Linear Algebra → Linear Transformations), Euclidean models show flat or decreasing norms (left), while HyTE-H shows increasing norms that fine-tuning sharpens (middle). HyTE-FH increases monotonically from 2.902 (L1) to 3.488 (L5), a +20.2% total increase (right). Euclidean baselines instead drop by up to ~30%, reflecting inverted structure.

Additional Results (Appendix)

Full MTEB breakdown

Per-task-type MTEB scores (all models share 149M parameters, 768 dims, 12 layers). Naming follows the paper; * denotes the Euclidean equivalent of HyTE.

MeSH hierarchy-sensitive retrieval

A zero-shot diagnostic built from the MeSH taxonomy with PubMed abstracts: each query must be matched to an abstract at a specific depth against parent, grandparent, sibling, and unrelated distractors. SpecHR (Specificity Hit Rate) is the fraction of queries where the correct-depth target outranks every ancestor-level document.

Top-1 prediction breakdown by distractor type — HyTE-FH eliminates grandparent confusion entirely and roughly quarters parent/sibling errors:

ANN retrieval on MS MARCO

Indexing HyTE-FH and EucBERT on MS MARCO (100k passages, 6,980 dev queries) with FAISS IndexHNSWFlat, applying the sign-flip reduction to HyTE-FH. It achieves higher recall at every $k$ and 3× lower latency (0.09 ms vs. 0.29 ms; identical ~10s build time) — OEM's outward bias yields a more tree-like HNSW graph with fewer spurious shortcuts.

Pooling ablation

OEM outperforms every alternative by a wide margin; naive mean pooling is worst (consistent with the contraction result), and the standard Einstein midpoint lags well behind OEM — confirming the outward bias is essential.

OEM sensitivity to $p$

Performance peaks at $p = 1.0$ (used throughout) and degrades gracefully on both sides: $p = 0$ recovers the standard Einstein midpoint, while large $p$ over-amplifies peripheral tokens.

Similarity function

Geodesic distance in the contrastive objective outperforms the Lorentz inner product, indicating better alignment of representations on the manifold.

Robustness to retrieval budget

Hyperbolic models stay strong with a small context window; the Euclidean baseline (Gemma) needs a larger window and still falls short.

Hubness

A well-known pathology of dense retrieval is hubness — a few documents dominate nearest-neighbor lists while most of the corpus is never retrieved. By preserving radial spread, OEM reduces skewness ~4× over Euclidean mean pooling and retrieves far more of the corpus.

Runtime

End-to-end latency (encoding + retrieval) for 32 queries over 100k documents on a single H100. HyTE-H adds negligible overhead; HyTE-FH's larger gap is entirely in encoding and reflects the current absence of optimized hyperbolic CUDA kernels rather than asymptotic complexity — encoding and retrieval share the same $\mathcal{O}(L(n^2 d + n d^2))$ and $\mathcal{O}(|\mathcal{D}|\,d)$ complexity as a Euclidean transformer.

Case study: geometry decides answer quality

On the query "What is the feature of Bixby guide?" (Emanual), only the hyperbolic model retrieves the correct passage and produces a faithful answer. Euclidean retrievers either collapse to generic "hub" documents (honest refusal) or drift to tangential content that triggers a confident hallucination — high answer relevance masking low faithfulness.

Citation

@inproceedings{madhu2026hyprag,
  title={HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation},
  author={Madhu, Hiren and Bui, Ngoc and Maatouk, Ali and Tassiulas, Leandros and Krishnaswamy, Smita and Yang, Menglin and Ganguly, Sukanta and Srinivasan, Kiran and Ying, Rex},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}