ALIGN: A Large-scale ImaGe and Noisy-text Model

Last Updated : 21 Aug, 2025

ALIGN (A Large-scale Image and Noisy-text) is a vision-language model developed to align images with their associated textual descriptions, even when those texts are noisy or sourced directly from the web. It uses a massive minimally-filtered dataset and a dual-encoder architecture to embed both images and texts in a shared semantic space, enabling robust cross-modal understanding and retrieval at unprecedented scale. Dual-Encoder Structure is adapted in the ALIGN Model,

  • Image Encoder: EfficientNet-based convolutional neural network transforms images into vector embeddings.
  • Text Encoder: BERT-large model converts the textual descriptions into dense vector representations.
  • Projection Layer: A fully connected layer harmonizes output dimensions for both encoders, so embeddings can be compared in the same space.

Step-by-Step Implementation

Let's see the step-by-step implementation of ALIGN,

Step 1: Import Libraries

Let's import all the required libraries,

  • PIL.Image: For reading and converting images.
  • AlignProcessor: Preprocesses (tokenizes, tensorizes) images and text.
  • AlignModel: Pretrained model to embed images and texts together.
  • torch: For tensor computation.
  • cosine_similarity (torch): Measures vector similarity.
  • requests: Downloads images from URLs.
  • BytesIO: Handles in-memory image data for PIL.
Python 
from PIL import Image
from transformers import AlignProcessor, AlignModel
import torch
from torch.nn.functional import cosine_similarity
import requests
from io import BytesIO

Step 2: Model and Processor Initialization

Let's initialize the model,

  • AlignProcessor: Preps images/text for model.
  • AlignModel: Loads pretrained weights/config for ALIGN.
Python 
processor = AlignProcessor.from_pretrained("kakaobrain/align-base")
model = AlignModel.from_pretrained("kakaobrain/align-base")

Step 3: Text and Image Inputs

Input the text and image URL's,

  • texts: List of captions.
  • image_urls: List of image links.
Python 
texts = ["A cat sitting on a mat", "A dog playing with a ball"]
image_urls = [
    "https://i.postimg.cc/7hJL9KK4/fluffy-siberian-cat-sitting-on-600nw-2150187551.png",
    "https://i.postimg.cc/4xgzKc3y/images.jpg"
]

Step 4: Download and Prepare Images

The model downloads the image from the URL for further processing by converting to RGB,

Python 
images = []
for url in image_urls:
    resp = requests.get(url)
    if resp.status_code == 200:
        img = Image.open(BytesIO(resp.content)).convert("RGB")
        images.append(img)

Step 5: Preprocess Images and Texts along with Device Setup

The images are converted to padded PyTorch tensors since the neural networks require uniform and correctly typed batch input. Use GPU if available otherwise CPU.

Python 
inputs = processor(
    images=images, text=texts, return_tensors="pt",
    padding='max_length', max_length=128, truncation=True
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
inputs = {k: v.to(device) for k, v in inputs.items()}

Step 6: Compute Model Outputs and Extract Embeddings

  • Passes inputs through ALIGN, creates modality embeddings.
  • image_embeds: Vector per input image.
  • text_embeds: Vector per caption.
Python 
outputs = model(**inputs)

image_embeds = outputs.image_embeds
text_embeds = outputs.text_embeds

Step 7: Calculate Cosine Similarity and Display Result

Compute the similarity for each image-text pair and display the results.

Python 
similarity = cosine_similarity(image_embeds, text_embeds)

for i, score in enumerate(similarity):
    print(f"Similarity score for pair {i+1}: {score.item():.4f}")

Output:

Similarity score for pair 1: 0.2410

Similarity score for pair 2: 0.2758

Application of ALIGN

  • Zero-shot Image Classification: Classifies images using only text descriptions of categories, with no retraining for new classes.
  • Image-Text Retrieval: Finds relevant images for text queries (text-to-image search) or retrieves text based on an image (image-to-text search).
  • Multimodal Search: Supports complex queries involving both images and texts together, such as searching for “red electric car” with an example photo and description.
  • Cross-modal Embedding: Enables downstream tasks that require both visual and linguistic understanding, such as visual question answering or captioning.

Advantages

  • Massive Scale: Trained on over 1.8 billion image-text pairs, making it robust and generalizable to real-world data.
  • Minimal Data Cleaning: Handles noisy, web-sourced data with little filtering, enabling adaptability to uncontrolled environments.
  • Dual-Encoder Efficiency: Architected for fast similarity computation, making large-scale retrieval tasks practical.
  • Strong Zero-shot Capability: Performs well on unseen classes and tasks without the need for task-specific training.

Limitations

  • Sensitive to Noisy Data: While robust, excessive noise or irrelevant text-image pairings can still degrade performance.
  • Language and Domain Bias: May reflect biases present in large-scale, web-sourced data, impacting fairness and reliability.
  • Quality vs. Quantity Trade-off: Minimal filtering allows more data, but risks including more irrelevant or low-quality pairs.
  • Limited Fine-grained Localization: Focuses on global image-text correspondence; less effective at tasks needing detailed localization or fine object matching.
Comment

Explore