Linkup releases the 149-million-parameter SPARSEUP retrieval model

Introducing SPARSEUP: An Open‑Source Sparse Retrieval Model

Linkup Research has made SPARSEUP publicly available under the Apache 2.0 licence, and the model weights have been uploaded to the Hugging Face platform. This release marks a notable step toward transparent, community‑driven development of sparse retrieval techniques, as the source code and parameters can be inspected, modified, and redistributed without restrictive licensing constraints.

The architecture of SPARSEUP builds on a ModernBERT backbone that contains 149 million parameters. Unlike conventional dense retrievers that generate a single dense vector per document, SPARSEUP produces a sparse vector in which each dimension is directly tied to a token in the model's vocabulary. Consequently, the representation preserves a clear linguistic interpretation, as each non‑zero entry signals the presence or relevance of a specific token.

The initialisation strategy follows a two‑stage process. First, the model inherits weights from the LateOn‑unsupervised checkpoint, a predecessor that already incorporates unsupervised pre‑training on large corpora. Second, SPARSEUP undergoes contrastive fine‑tuning using the open‑source LightOn data mixture. During this phase, each query is paired with one positive document and seven hard negatives that are selected from a pool of fifty candidates, a design intended to sharpen the model's discriminative capacity.

Mechanisms Controlling Sparsity

Three explicit mechanisms regulate the sparsity of the output vectors. The first mechanism adds a constant offset of 15 to the logits before applying the softmax, effectively pushing many token scores below the activation threshold. The second mechanism caps the number of active dimensions per token position at twelve, ensuring that no single position contributes an excessive number of non‑zero entries. The third mechanism merges variants of case and whitespace, collapsing tokens that differ only in capitalization or spacing into a single dimension. Together, these controls shape a representation that is both interpretable and computationally efficient.

The vocabulary size undergoes a reduction as part of the case‑and‑spacing fusion. Starting from an approximate token set of 50,000 entries, the merging process yields a final dimensionality of about 34,000. This reduction directly influences the storage and indexing cost of the sparse vectors, while preserving the essential lexical distinctions required for retrieval.

Performance on Standard Benchmarks

Linkup reports a normalized Discounted Cumulative Gain at rank 10 (nDCG@10) of 56.4 on the BEIR‑13 collection, excluding the MS MARCO subset. According to the authors, this figure constitutes the best publicly disclosed result for any model with fewer than 150 million parameters within this benchmark category. The metric is presented as a direct outcome of the evaluation protocol applied to the standard BEIR test queries.

When comparing SPARSEUP to dense and late‑interaction baselines that share a similar data foundation and backbone size, the reported scores indicate that SPARSEUP does not surpass the leading dense approaches. Specifically, the DenseOn configuration achieves 57.9 nDCG@10, while the LateOn configuration reaches 58.9 on the same benchmark. These figures, cited by Linkup, illustrate a performance gap that persists despite the sparsity‑focused design.

The gap suggests that, under comparable training conditions, dense representations may retain an advantage in capturing nuanced semantic similarity. However, the sparse model offers distinct trade‑offs, such as reduced index size and potentially faster retrieval, which could be valuable in scenarios where resource constraints dominate the design choices.

Speed and Recall with the Seismic Index

Linkup introduces an indexing structure named Seismic, optimized for sparse vectors. Using this index, the company claims that SPARSEUP attains a recall exceeding 97 % relative to exact search on the MS MARCO dataset, while processing each query in roughly 380 microseconds. These numbers are presented as empirical measurements obtained on the MS MARCO test collection.

The authors explicitly note that the reported latency and recall figures must be reproduced by end‑users on their own data. This disclaimer acknowledges that the observed performance may depend on factors such as hardware configuration, query distribution, and dataset characteristics, and therefore cannot be assumed to generalise without verification.

• Apache 2.0 licence ensures free redistribution
• ModernBERT backbone of 149 M parameters
• Contrastive training with seven hard negatives from fifty candidates
• Three sparsity controls: logit offset, per‑position cap, case‑spacing fusion

The list above summarises the core technical choices that differentiate SPARSEUP from other retrieval models. Each element reflects a design decision that directly influences either the legal accessibility, the model capacity, the training dynamics, or the sparsity characteristics of the final representation.

From a methodological perspective, the reliance on a fixed number of hard negatives introduces a controlled level of difficulty during contrastive learning. Yet, because the negatives are drawn from a limited pool of fifty, the diversity of challenging examples may be constrained, potentially limiting the model's ability to generalise to unseen query‑document pairs.

The sparsity mechanisms, while effective at reducing dimensionality, also impose hard thresholds that could discard informative signals. The logit offset of 15, for instance, pushes many token scores below activation, which may improve efficiency but at the risk of suppressing subtle lexical cues that are relevant for certain queries.

The reduction from roughly 50 k to 34 k dimensions through case‑and‑spacing fusion simplifies the index but also merges tokens that could carry distinct semantic weight in languages with case‑sensitive meanings. This trade‑off illustrates the tension between compression and linguistic fidelity inherent in sparse retrieval designs.

Open questions remain regarding the scalability of SPARSEUP to larger vocabularies or multilingual settings. The current configuration operates on a monolingual token set; extending the approach would require reassessing the sparsity controls to avoid excessive dimensionality growth.

Another area for future investigation concerns the interaction between sparse representations and hybrid retrieval pipelines. It is conceivable that combining SPARSEUP vectors with dense embeddings could capture complementary aspects of relevance, though the precise integration strategy would need empirical validation.

In summary, SPARSEUP contributes a transparent, open‑source sparse retrieval model that achieves competitive recall and low latency under the Seismic index, while delivering a benchmark score that is respectable but not leading within the sub‑150 M parameter regime. The model’s design choices highlight the balance between interpretability, efficiency, and retrieval effectiveness, and they open several avenues for further research on sparsity‑driven information retrieval.