-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add mixtral support #12
Conversation
main.py
Outdated
for sublayer_name in aq_handlers.keys(): | ||
print(f"Quantizing module {sublayer_name} of layer {layer_index}") | ||
if "mixtral" in model.config.model_type.lower() and args.mix_compression: | ||
print("mix_compression") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
print("mix_compression") |
main.py
Outdated
for sublayer_name in aq_handlers.keys(): | ||
print(f"Quantizing module {sublayer_name} of layer {layer_index}") | ||
if "mixtral" in model.config.model_type.lower() and args.mix_compression: | ||
print("mix_compression") | ||
assert args.nbits_per_codebook == 16 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpicking: this looks like a very specific configuration for a general option (--mix_compression)
if you have time, can you please
- assert that model type is mixtral
- double the number of codebooks for self-attn layers
- use standard number of codebooks for non self-attn layers
If you don't have time, please
- merge this PR
- create an issue, saying
PR https://github.com/Vahe1994/AQLM/pull/12/ adds a very specific configuration for--mix_compression.
It would be great to generalize it to something more generic.
For instance, we can
- assert that model type is mixtral
- double the number of codebooks for self-attn layers
- use standard number of codebooks for non self-attn layers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will do, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (with optional requsts in the comments)
Added Mixtral support