Can i use 65b model and how?

[LightVolk]

I download this files:

ggml-model-q4_0.bin
ggml-model-q4_0.bin.1
ggml-model-q4_0.bin.2
ggml-model-q4_0.bin.3
ggml-model-q4_0.bin.4
ggml-model-q4_0.bin.5
ggml-model-q4_0.bin.6
ggml-model-q4_0.bin.7

Can i run some command like ./chat -m ggml-model-q4_0.bin and load all of this models to use 65b power?

[Degot]

To run this model, I modified chat.cpp and changed in LLAMA_N_PARTS { 8192, 1 } to { 8192, 8 }... Then recompiled and used -m ggml-model-q4_0.bin . Worked smoothly. ~42GB RAM used

[LightVolk]

To run this model, I modified chat.cpp and changed in LLAMA_N_PARTS { 8192, 1 } to { 8192, 8 }... Then recompiled and used -m ggml-model-q4_0.bin . Worked smoothly. ~42GB RAM used

Yes it works for me , but in Windows chat.exe version. Somehow it works for me.
But it generate words so slow.. and yes - use 40 gb of RAM

Thank you!

[clover1980]

I would like to experiment with this. I have Asrock board with 128Gb, that's the only manufacturer as i think which still announcing new motherboards with multi-slots huge RAM on consumers market (the other choice is only Microstar server boards). I kinda anticipated all this in 2020th, but usable Ai's surfaced only now.

To run this model, I modified chat.cpp and changed in LLAMA_N_PARTS { 8192, 1 } to { 8192, 8 }... Then recompiled and used -m ggml-model-q4_0.bin . Worked smoothly. ~42GB RAM used

Have you posted your compiled file anywhere? I'm using Win 10 x64.

[Degot]

Have you posted your compiled file anywhere? I'm using Win 10 x64.

No, I didn't.. I used standard windows instructions... cmake-based... to build binary

[flarens]

No, I didn't.. I used standard windows instructions... cmake-based... to build binary

I spent several days on unsuccessful attempts to compile chat.cpp. I beg you, can you share a ready -made application for launching 65B model?