Can promptbench be used to attack samples instead of prompts?

[WanliYoung]

Hi~
Thanks to your great contribution to the community.
I would like to try attacking the samples of datasets using promptbench instead of attacking the prompt. Is this feasible? Do you have any suggestions? Thank you so much!

[Immortalise]

Hi, if you would like to attack samples, you could refer to the textattack repo, which is the base of prompt attacks.

[WanliYoung]

Thank you very much for your reply. I am referring to TextAttack. I found TextAttack has a parameter like --pct-words-to-swap to limit the range of perturbations to the text. May I ask, in promptbench, what is the default magnitude of perturbation? Are there any constraints?

[Immortalise]

Yes, please refer to this file where we defines the constraints of different attacks.

Also, we have the LabelConstraint where we prevent the attack from modifying certain words (such as the label 'positive' for sentiment analysis).

[WanliYoung]

Thank you very much for solving my problem~