[FEATURE] ADD VPTQ

[Qubitium]

@YangWang92 I very much would like to add support VPTQ quantization into GPTQModel for both quantization and inference. As GPTQModel is currently a modular codebase after recent refractor we are now able to add new quantization methods like QQQ via a plugin (process) structure. We are also part of the upstream HF Transformers, Optimum, Peft, vLLM, SGLang eco-system when it comes to quantization support.

I see there are some samples to quantize but not sure how to first generate the hessians. GPTQ also use hessians based on calibration data fed to module forwards so not sure if this is the same thing.

Ref: https://github.com/microsoft/VPTQ/issues/126

The above guide requires hessian_path and inv_ hessian_path matrix but I don't see a hessian processor? Or are the hessian paths optional and if not exists, it will generate it on the fly (slow)?

Thanks!

[YangWang92]

Hi @Qubitium , sorry about the late reply, thanks for ping me and letting me quickly check your requirements.

[YangWang92]

BTW, my colleague @wejoncy integrated VPTQ in his qllm. Here is an example of how qllm processes the hessian matrix. https://github.com/wejoncy/QLLM/blob/main/qllm/quantization/vptq/qllm_hessian.py

[Qubitium]

BTW, my colleague @wejoncy integrated VPTQ in his qllm. Here is an example of how qllm processes the hessian matrix. https://github.com/wejoncy/QLLM/blob/main/qllm/quantization/vptq/qllm_hessian.py

@YangWang92 I think this is exactly what I was looking for!