LAION-AUDIO-XL & CLAP2
Collection
Release of the LAION-Audio-XL dataset and the CLAP2 model achitectures • 7 items • Updated • 2
You need to agree to share your Hugging Face information to access this dataset
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
The embeddings are computed from the audio clips of LAION-Audio-XL, which carry their own licences (see the audio and links repositories of LAION-Audio-XL). By requesting access you agree to use the embeddings in line with the licences of the clips they were computed from; embeddings of non-commercially licensed clips are for non-commercial use.
Log in or Sign Up to review the conditions and access this dataset content.
LAION-Audio-XL Embeddings
Audio and caption embeddings of 364,985,867 clips of LAION-Audio-XL from the five LAION CLAP2 models (CLAPNaFlex-v3 L, M, S, XS
and XXS, each at its 1B-samples checkpoint), one config per model. Every clip gets an audio vector and, where it has them,
vectors of its short and long caption; all live in the model's shared audio-text space, so audio-text, text-text and
audio-audio similarity are plain dot products.
LAION-Audio-XL is the corpus of 366,007,518 audio clips of at most 30 s with captions that the LAION CLAP2 models were trained on. It is published in these repositories:
| repository | content | clips | access |
|---|---|---|---|
| laion/laion-audio-xl | audio (configs open, sharealike, noncommercial) and links (config links) | 365,984,635 | automatic approval |
| laion/laion-audio-xl-embeddings | CLAP2 audio and caption embeddings (models L, M, S, XS, XXS) | 364,985,867 | automatic approval |
Clips whose licence forbids derivatives (CC ... ND, 1,021,651 clips) have no embeddings.
Loading
from datasets import load_dataset
emb = load_dataset("laion/laion-audio-xl-embeddings", "L", streaming=True) # or "M", "S", "XS", "XXS"
The rows of a file are the same, in the same order, in all five configs, so files of different models can be zipped
row by row. audio_id is the clip key of every LAION-Audio-XL repository.
Columns
| column | type | content |
|---|---|---|
audio_id |
int64 | the clip's id in LAION-Audio-XL |
audio_emb |
list of float16 | audio embedding, never null |
short_caption_emb |
list of float16 | embedding of the clip's short_caption; null if it has none |
long_caption_emb |
list of float16 | embedding of the clip's long_caption; null if it has none |
text_emb |
list of float16 | embedding of the clip's text, only for clips with neither caption, else null |
Vectors are L2-normalised (computed in fp32, stored as fp16).
| model | embedding dim | audio tower | text tower | parameters |
|---|---|---|---|---|
| L | 960 | 1024 x 12 (projected to 960) | 960 x 11 | 340.1 M |
| M | 640 | 640 x 12 | 640 x 11 | 152.8 M |
| S | 512 | 512 x 12 | 512 x 11 | 103.9 M |
| XS | 320 | 320 x 12 | 320 x 12 | 48.3 M |
| XXS | 224 | 224 x 12 | 224 x 12 | 27.3 M |
L is best on every benchmark suite; S is about as good as M at a smaller dimension; XS and XXS are the cheapest to store, index and run.
Licences
The embeddings are derived from clips that carry their own licences (see the audio repositories and the links repository). Use them in line with those licences; embeddings of non-commercially licensed clips are for non-commercial use.
- Downloads last month
- 174
