Large Concept Models (LCMs)

Large Language Models (LLMs) process and generate text mainly at the token level, making them effective for tasks such as text generation, translation and question answering. To explore a higher-level approach, Meta introduced Large Concept Models (LCMs), which predict the next concept instead of the next token. In the original LCM architecture, a concept is represented as a sentence-level embedding in the SONAR semantic space.

For example

While an LLM predicts the next token in a sentence, an LCM represents the current sentence as an embedding and predicts the embedding of the next sentence. This allows language to be modeled as a sequence of higher-level concepts rather than individual tokens.

Working

The basic working of an LCM can be divided into three steps:

1. Convert sentences into embeddings: Input sentences are represented as semantic embeddings using the SONAR embedding space.
2. Predict the next concept: The LCM predicts the embedding corresponding to the next sentence or concept.
3. Generate text: The predicted embedding is decoded into text.

Therefore, instead of directly learning:

Token -> Token -> Token

LCM models a higher-level sequence:

Concept -> Concept -> Concept

The original LCM research explored different approaches for predicting these concept embeddings, including regression, diffusion-based generation and quantized representations.

Features

1. Concept-Level Processing

• LCMs model language using higher-level semantic representations instead of directly predicting individual tokens. In the original LCM implementation, each concept corresponds to a sentence embedding.
• This allows the model to learn relationships between sentence-level ideas while reducing the number of prediction steps compared with token-level modeling.

2. Multilingual Representation

• The original LCM uses SONAR, a multilingual and multimodal sentence-embedding space. The original SONAR system supported text in 200 languages and provided a shared representation space for text and speech.
• This shared space allows related meanings to be represented across languages, making it useful for multilingual modeling and zero-shot generalization.

3. Higher-Level Sequence Modeling

• LCMs model sequences using sentence-level representations, reducing the number of autoregressive prediction steps compared with token-level modeling.
• Meta reported that this approach becomes more computationally efficient as input context grows.

Advantages over LLMs

Current Developments

• 2024: Meta introduced LCMs as an experimental approach to language modeling in a sentence representation space.
• 2026: Meta introduced v-Sonar and v-LCM, extending concept-space modeling to vision-language tasks such as image/video captioning and question answering.
• 2026: OmniSONAR extended the shared semantic representation approach to text, speech, code and mathematical expressions across thousands of languages.
• OmniSONAR: Extended the shared semantic representation approach to broader multilingual and multimodal applications.

Applications

• Text Summarization: Generate summaries using sentence-level concepts.
• Summary Expansion: Expand summaries into detailed text.
• Multilingual Generation: Model concepts across languages.
• Multimodal Tasks: Extend concept-level modeling to vision-language tasks.

Limitations

• Information Loss: Sentence embeddings may lose fine-grained details.
• Additional Decoding: Concept embeddings must be decoded into text.
• Limited Evaluation: Research has focused on selected tasks rather than the broad range of LLM applications.
• Emerging Technology: LCMs are still experimental and are not a general replacement for LLMs.