Skip to content

opendatalab/CLIP-Parrot-Bias

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

alt_text Parrot Captions Teach CLIP to Spot Text

[ Paper ] [ Website ] [ Dataset (OpenDataLab)] [ Dataset (Hugging face) ] [Demo]

overview

TL;DR

  1. Captions in LAION-2B have a significant bias towards describing visual text content embedded in the images.
  2. Released CLIP models have strong text spotting bias in almost every style of web image, resulting in the CLIP-filtering datasets being inherently biased towards visual text-dominant data.
  3. CLIP models easily learn text-spotting capacity from parrot captions while failing to connect the vision-language semantics, just like a text-spotting parrot.
  4. We provide an alternative solution by releasing a less biased filtered LAION-2B 100M subset and pre-trained CLIP models.

News and Updates


Kmeans Model from LAION-400M

We trained the Kmeans model from the LAION-400M dataset CLIP ViT-B-32 features using fassi. We first used PCA to reduce the feature dimension. The training and inference code in kmeans.py.

PCA weigths Kmeans centrios
Download Download

Generating Synthetic Images from N-gram Vocabulary

The generation pipeline of synthetic images (sys_benchmark.py and Arial.ttf) and the N-gram Vocabulary we built from the dataset.

LAION-2B Caption 1-gram LAION-2B Caption 2-gram LAION-2B Co-Emb Text 1-gram
Download Download Download

A Less Text-biased LAION-100M Subset and CLIP Model

Data Cruation Pipeline

  • Selecting all the images without any embedded text using the text spotting model DeepSolo.
  • Filtering samples with CLIP score > 0.3 and Aesthetics score > 4.5
  • Deduplication using CLIP features similarity based cluster labels.
  • Finally, we got 107,166,507 (100M) LAION-2B subset.

Training Details

our training code is based on OpenCLIP

  • batch size 32k
  • lr 5e-4
  • epochs 32
  • local loss
  • precision amp

Note that the OCR model is not perfect. The images in our filtered subset still contain some text content. Therefore, we also benchmark our trained model on the synthetic images benchmark.

100M subset ViT-B Models
Download Download
1-gram Synthetic Benchmark Ours
(100M)
CLIP
(WIT-400M)
OpenCLIP
(LAION-2B)
DC medium
128M (DC)
DC large
1.28B (DC)
Sync. Score (mean) $\downarrow$ 0.163 0.317 0.368 0.268 0.338
Sync. Score (std) 0.0659 0.0305 0.0427 0.0247 0.0341
DataComp benchmark Ours
(100M)
CLIP
(WIT-400M)
OpenCLIP
(LAION-2B)
DC medium
128M (DC)
DC large
1.28B (DC)
ImageNet 0.526 0.633 0.666 0.176 0.459
ImageNet dist. shifts 0.404 0.485 0.522 0.152 0.378
VTAB 0.481 0.526 0.565 0.259 0.426
Retrieval 0.421 0.501 0.560 0.219 0.419
Average 0.443 0.525 0.565 0.258 0.437

Acknowledgement

Thanks for these good works:

  • faiss A library for efficient similarity search and clustering for building the Kmeans model.
  • DeepSolo A strong transformer-based text spotting model for profiling the LAION dataset.
  • CLIP Pre-trained CLIP models on WIT-400M.
  • OpenCLIP An open-source CLIP implementation of training codes and pre-trained models on the LIAON dataset.
  • DataComp A comprehensive evaluation benchmark for CLIP models' downstream performance.
  • Aesthetic Score Predictor An aesthetic score predictor ( how much people like on average an image ) based on a simple neural net that takes CLIP embeddings as inputs.

Reference

@article{lin2023parrot,
    title={Parrot Captions Teach CLIP to Spot Text}, 
    author={Yiqi Lin and Conghui He and Alex Jinpeng Wang and Bin Wang and Weijia Li and Mike Zheng Shou},
    journal={arXiv preprint arXiv:2312.14232},
    year={2023}
}

@article{he2024opendatalab,
  title={Opendatalab: Empowering general artificial intelligence with open datasets},
  author={He, Conghui and Li, Wei and Jin, Zhenjiang and Xu, Chao and Wang, Bin and Lin, Dahua},
  journal={arXiv preprint arXiv:2407.13773},
  year={2024}
}

License

Apache 2.0 License

Releases

No releases published

Packages

No packages published

Languages