Add SiglipForImageClassification and CLIPForImageClassification

[NielsRogge]

What does this PR do?

This PR adds 2 new classes to the library, SiglipForImageClassification and CLIPForImageClassification.

This makes it easier to fine-tune SigLIP/CLIP for image classification, as otherwise people had to manually write a class based on SiglipVisionModel/CLIPVisionModel with a head on top. Given that SigLIP and CLIP are among the best vision encoders out there, it makes sense to add them.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Looks great - thanks for adding!

[NielsRogge]

@amyeroberts I assume the "build PR documentation" check is failing but unrelated to this PR, so I can merge?

[amyeroberts]

@NielsRogge Yes, if you look on main you can see that the build is failing for lots of (unrelated) PRs at the moment. I'm happy for you to merge as all the other CIs are passing.

[zhjohnchan]

Hi @NielsRogge and @amyeroberts,

Thanks for your contribution to the community @NielsRogge! I am wondering if "do_rescale": true is necessary for SiglipImageProcessor? Let me know if I miss anything!

Also, did you encounter the same problem as in #28968?

Thanks!

Best,

[amyeroberts]

@zhjohnchan do_rescale=True is necessary if the input images are in the range [0, 255]. If they've already been rescaled to [0, 1] you can set it to False.

[zhjohnchan]

@amyeroberts Thank you so much! Just realized it should be rescaled for all Transformers' preprocessors.

[NielsRogge]

@zhjohnchan I haven't encountered the same problem, made a fine-tuning notebook for SigLIP here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SigLIP/Fine_tuning_SigLIP_and_friends_for_multi_label_image_classification.ipynb.

[zhjohnchan]

@NielsRogge Got it! Thank you so much!

[Andron00e]

Hi, I am the one who proposed to add a CLIPForImageClassification. Now I am back to contributing with it.
I see "Build documentation" failure. How can it be fixed?

[NielsRogge]

HI @Andron00e, thanks for suggesting it, it's now available here: https://huggingface.co/docs/transformers/en/model_doc/siglip#transformers.SiglipForImageClassification

[Andron00e]

Thank you! I'll try to fix minor bugs with CLIPForImageClassification and place its code into SigLIP's directory.