Repository navigation
support qwen3 awq - #1951
Merged
Merged
support qwen3 awq#1951
Conversation
Merged
Collaborator
|
Have you test it with https://huggingface.co/Qwen/Qwen3-32B-AWQ? The CI/CD builds wheels for PRs. For this PR: https://github.com/OpenNMT/CTranslate2/actions/runs/20275888121 I will expect this official model to work. |
Contributor
Author
|
An AWQ model has to be converted to a ctranslate2 model I thought. But yes, I converted that model to ctranslate2 format and it works. Only takes up about 18.6 GB of VRAM as well...unless there's a really long context in which case it slowly climbs. Might be a good time to bring back flash attention or I might just need to learn how to compile the library with it...All the models here work: https://huggingface.co/collections/ctranslate2-4you/ctranslate2-qwen3 |
|
I am surprised AWQ is able to convert to CT2 with no issues. Did you notice any artefacts in the output in your testing? Do the throughput improvements also carry over? |
Contributor
Author
|
@winstxnhdw I had to modify the source code for autoawq because the repository was archived in May, but after that it worked perfectly - no artifacts. |
Collaborator
|
@BBC-Esq It will be great if you can publish your autoawq with the changes the I can test this PR end to end. Thanks |
Contributor
Author
|
@jordimas Do you mean publish the pypi? I mean...here is my fork of autoawq if you wanted to pip install from the fork? I'm not very familiar with publishing to pypi... |
Collaborator
|
Hello @BBC-Esq I can install from the repo, no need to have the package publish. If you can share the exact instructions that you are following to convert the model to ct2 then I can produce them and test the PR. Thanks |
Contributor
Author
|
I'm not using a specific command per se...Rather, I'm using this script because I hate constructing commands. 😄 Enjoy! CONVERSION SCRIPT WITH GUIimport sys
import os
from pathlib import Path
def set_cuda_paths():
venv_base = Path(sys.executable).parent.parent
nvidia_base_path = venv_base / 'Lib' / 'site-packages' / 'nvidia'
cuda_path_runtime = nvidia_base_path / 'cuda_runtime' / 'bin'
cuda_path_runtime_lib = nvidia_base_path / 'cuda_runtime' / 'bin' / 'lib' / 'x64'
cuda_path_runtime_include = nvidia_base_path / 'cuda_runtime' / 'include'
cublas_path = nvidia_base_path / 'cublas' / 'bin'
cudnn_path = nvidia_base_path / 'cudnn' / 'bin'
nvrtc_path = nvidia_base_path / 'cuda_nvrtc' / 'bin'
nvcc_path = nvidia_base_path / 'cuda_nvcc' / 'bin'
paths_to_add = [
str(cuda_path_runtime),
str(cuda_path_runtime_lib),
str(cuda_path_runtime_include),
str(cublas_path),
str(cudnn_path),
str(nvrtc_path),
str(nvcc_path),
]
current_value = os.environ.get('PATH', '')
new_value = os.pathsep.join(paths_to_add + [current_value] if current_value else paths_to_add)
os.environ['PATH'] = new_value
triton_cuda_path = nvidia_base_path / 'cuda_runtime'
os.environ['CUDA_PATH'] = str(triton_cuda_path)
set_cuda_paths()
import traceback
import subprocess
from PySide6.QtWidgets import (
QApplication, QWidget, QPushButton, QLabel, QFileDialog,
QMessageBox, QVBoxLayout, QHBoxLayout, QCheckBox,
QTextEdit, QStyleFactory, QComboBox,
)
from PySide6.QtCore import QThread, Signal
class ConversionThread(QThread):
conversion_started = Signal(str, str)
conversion_finished = Signal(str, str)
conversion_error = Signal(str)
def __init__(self, model_dir, selected_file, command, quantization, needs_renaming):
super().__init__()
self.model_dir = model_dir
self.selected_file = selected_file
self.command = command
self.quantization = quantization
self.needs_renaming = needs_renaming
def run(self):
self.conversion_started.emit(self.quantization, self.command)
if not self.needs_renaming:
self._run_conversion()
else:
self._run_with_renaming()
def _run_conversion(self):
try:
proc = subprocess.run(
self.command, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, env=os.environ.copy()
)
if proc.returncode == 0:
self.conversion_finished.emit(self.quantization, proc.stdout)
else:
self.conversion_error.emit(
f"Conversion failed (code {proc.returncode}):\n{proc.stderr}"
)
except Exception:
self.conversion_error.emit(
"Exception during conversion:\n" + traceback.format_exc()
)
def _run_with_renaming(self):
if self.selected_file.endswith('.bin'):
expected_name = "pytorch_model.bin"
temp_name = "temp_pytorch_model.bin"
else:
expected_name = "model.safetensors"
temp_name = "temp_model.safetensors"
original_path = os.path.join(self.model_dir, expected_name)
temp_path = os.path.join(self.model_dir, temp_name)
selected_path = os.path.join(self.model_dir, self.selected_file)
try:
if os.path.exists(original_path):
os.rename(original_path, temp_path)
os.rename(selected_path, original_path)
self._run_conversion()
except Exception:
self.conversion_error.emit(
"Exception during renaming/conversion:\n" + traceback.format_exc()
)
finally:
try:
if os.path.exists(original_path):
os.rename(original_path, selected_path)
if os.path.exists(temp_path):
os.rename(temp_path, original_path)
except Exception:
pass
class App(QWidget):
def __init__(self):
super().__init__()
self.setWindowTitle("chintellalaw.com - for non-commercial use")
self.resize(800, 500)
self.model_path = ""
self.output_dir = ""
self.current_thread = None
self.conversion_queue = []
self.needs_file_renaming = False
layout = QVBoxLayout(self)
row = QHBoxLayout()
self.browse_btn = QPushButton("Browse Model Directory")
self.browse_btn.clicked.connect(self.browse)
row.addWidget(self.browse_btn)
self.file_combo = QComboBox()
self.file_combo.setEnabled(False)
row.addWidget(self.file_combo)
layout.addLayout(row)
self.path_label = QLabel("")
layout.addWidget(self.path_label)
row = QHBoxLayout()
self.output_btn = QPushButton("Select Output Directory")
self.output_btn.clicked.connect(self.browse_output)
row.addWidget(self.output_btn)
self.output_label = QLabel("")
row.addWidget(self.output_label)
layout.addLayout(row)
self.quant_opts = [
"float32", "float16", "bfloat16",
"int8_float32", "int8_float16", "int8_bfloat16", "int8"
]
self.quant_boxes = {opt: QCheckBox(opt) for opt in self.quant_opts}
row = QHBoxLayout()
for cb in self.quant_boxes.values():
row.addWidget(cb)
self.awq_cb = QCheckBox("AWQ")
row.addWidget(self.awq_cb)
layout.addLayout(row)
self.run_btn = QPushButton("Run Conversion")
self.run_btn.clicked.connect(self.run_conversion)
layout.addWidget(self.run_btn)
self.log = QTextEdit()
self.log.setReadOnly(True)
layout.addWidget(self.log)
def analyze_model_directory(self, path):
try:
files = os.listdir(path)
except OSError:
return "error", []
sharded_pattern = [f for f in files if
('-of-' in f and f.endswith(('.bin', '.safetensors')))]
single_model_files = [f for f in files if
f.endswith(('.bin', '.safetensors')) and
'-of-' not in f]
if sharded_pattern:
return "sharded", sharded_pattern
elif len(single_model_files) > 1:
return "multiple_variants", single_model_files
elif len(single_model_files) == 1:
return "single", single_model_files
else:
return "none", []
def browse(self):
path = QFileDialog.getExistingDirectory(self, "Select Model Directory")
if not path:
return
self.model_path = path
self.path_label.setText(f"Model: {path}")
parent_path = str(Path(path).parent)
self.output_dir = parent_path
self.output_label.setText(f"Output: {parent_path}")
model_type, files = self.analyze_model_directory(path)
if model_type == "sharded":
self.file_combo.clear()
self.file_combo.addItem(f"Sharded model ({len(files)} files)")
self.file_combo.setEnabled(False)
self.needs_file_renaming = False
self.append_log(f"✅ Detected sharded model with {len(files)} files")
elif model_type == "multiple_variants":
self.file_combo.clear()
self.file_combo.addItems(files)
if "pytorch_model.bin" in files:
self.file_combo.setCurrentText("pytorch_model.bin")
elif "model.safetensors" in files:
self.file_combo.setCurrentText("model.safetensors")
self.file_combo.setEnabled(True)
self.needs_file_renaming = True
self.append_log(f"🔍 Multiple model variants found - please select one")
elif model_type == "single":
self.file_combo.clear()
self.file_combo.addItem(files[0])
self.file_combo.setEnabled(False)
self.needs_file_renaming = False
self.append_log(f"✅ Single model file detected: {files[0]}")
else:
self.file_combo.clear()
self.file_combo.setEnabled(False)
self.needs_file_renaming = False
self.append_log("❌ No model files found in directory")
def browse_output(self):
path = QFileDialog.getExistingDirectory(self, "Select Output Directory")
if path:
self.output_dir = path
self.output_label.setText(f"Output: {path}")
def append_log(self, msg: str):
self.log.append(msg)
def run_conversion(self):
self.log.clear()
self.conversion_queue.clear()
if not self.model_path:
QMessageBox.critical(self, "Error", "Please select a model directory.")
return
if not self.output_dir:
QMessageBox.critical(self, "Error", "Please select an output directory.")
return
if not self.file_combo.count():
QMessageBox.critical(self, "Error", "No model files detected.")
return
if self.awq_cb.isChecked():
self.conversion_queue.append("awq")
for opt, cb in self.quant_boxes.items():
if cb.isChecked():
self.conversion_queue.append(opt)
if not self.conversion_queue:
QMessageBox.information(self, "Info", "No quantization options selected.")
return
self._process_next()
def _process_next(self):
if self.current_thread and self.current_thread.isRunning():
return
if not self.conversion_queue:
self.append_log("🎉 All conversions completed!")
return
option = self.conversion_queue.pop(0)
selected_file = self.file_combo.currentText() if self.file_combo.isEnabled() else None
model_name = os.path.basename(self.model_path)
out_dir = os.path.join(self.output_dir, f"{model_name}-ct2-{option}")
try:
extras = [
fn for fn in os.listdir(self.model_path)
if not fn.endswith((".bin", ".safetensors", "onnx", "Pooling"))
and fn not in [
"config.json", ".git", "coreml", "configs", "runs", ".idea",
".cache",
"__pycache__",
".gitignore",
".gitattributes",
"flax_model.msgpack",
"tf_model.h5",
]
and not fn.startswith('.')
and os.path.isfile(os.path.join(self.model_path, fn))
]
copy_part = "--copy_files " + " ".join(f'"{e}"' for e in extras) + " " if extras else ""
except:
copy_part = ""
if option == "awq":
cmd = (
f'ct2-transformers-converter --model "{self.model_path}" '
f'--output_dir "{out_dir}" '
f'--low_cpu_mem_usage --trust_remote_code {copy_part.strip()}'
)
else:
cmd = (
f'ct2-transformers-converter --model "{self.model_path}" '
f'--output_dir "{out_dir}" '
f'--quantization {option} '
f'--low_cpu_mem_usage --trust_remote_code {copy_part.strip()}'
)
thread = ConversionThread(
self.model_path, selected_file, cmd, option, self.needs_file_renaming
)
self.current_thread = thread
# Only log on these signals - don't chain to _process_next here
thread.conversion_started.connect(
lambda q, c: self.append_log(f"🚀 Starting {q} conversion...")
)
thread.conversion_finished.connect(
lambda q, out: self.append_log(f"✅ {q} conversion completed")
)
thread.conversion_error.connect(
lambda err: self.append_log(f"❌ Error: {err}")
)
# Chain to next conversion only after thread truly finishes
thread.finished.connect(self._on_thread_finished)
thread.start()
def _on_thread_finished(self):
if self.current_thread:
self.current_thread.deleteLater()
self.current_thread = None
self._process_next()
def _cleanup_thread(self):
# Can remove this method now, or keep for other uses
pass
def closeEvent(self, event):
if self.current_thread and self.current_thread.isRunning():
self.current_thread.wait()
event.accept()
if __name__ == "__main__":
app = QApplication([])
app.setStyle(QStyleFactory.create("Fusion"))
window = App()
window.show()
app.exec()You can remove the |
Contributor
Author
|
Please let me know if the script doesn't save to the new ctranslate2 model's directory certain required files...I was always wondering about if I did it right... |
Collaborator
|
I need the exact steps to reproduce for example https://huggingface.co/CTranslate2HQ/Qwen3-0.6B-ct2-AWQ, this includes
Thanks |
Contributor
Author
|
Geesh! You want all my hard work don't ya! 😉 Well here ya go sir: ANOTHER BADASS SCRIPT TO CONVERT TO AWQimport multiprocessing
if __name__ == '__main__':
multiprocessing.set_start_method('spawn', force=True)
import sys
import os
from pathlib import Path
def set_cuda_paths():
venv_base = Path(sys.executable).parent.parent
nvidia_base_path = venv_base / 'Lib' / 'site-packages' / 'nvidia'
cuda_path_runtime = nvidia_base_path / 'cuda_runtime' / 'bin'
cuda_path_runtime_lib = nvidia_base_path / 'cuda_runtime' / 'bin' / 'lib' / 'x64'
cuda_path_runtime_include = nvidia_base_path / 'cuda_runtime' / 'include'
cublas_path = nvidia_base_path / 'cublas' / 'bin'
cudnn_path = nvidia_base_path / 'cudnn' / 'bin'
nvrtc_path = nvidia_base_path / 'cuda_nvrtc' / 'bin'
nvcc_path = nvidia_base_path / 'cuda_nvcc' / 'bin'
paths_to_add = [
str(cuda_path_runtime),
str(cuda_path_runtime_lib),
str(cuda_path_runtime_include),
str(cublas_path),
str(cudnn_path),
str(nvrtc_path),
str(nvcc_path),
]
current_value = os.environ.get('PATH', '')
new_value = os.pathsep.join(paths_to_add + [current_value] if current_value else paths_to_add)
os.environ['PATH'] = new_value
triton_cuda_path = nvidia_base_path / 'cuda_runtime'
os.environ['CUDA_PATH'] = str(triton_cuda_path)
set_cuda_paths()
import logging
import traceback
from PySide6.QtWidgets import (QApplication, QWidget, QPushButton, QLabel, QFileDialog,
QMessageBox, QVBoxLayout, QHBoxLayout, QCheckBox, QTextEdit,
QComboBox, QStyleFactory, QGroupBox, QProgressBar, QSpinBox, QSizePolicy)
from PySide6.QtCore import QThread, Signal, QObject, QTimer
from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
class QuantizationProcess(multiprocessing.Process):
def __init__(self, model_dir, output_dir, quant_config, calib_config, pipe):
super().__init__()
self.model_dir = model_dir
self.output_dir = output_dir
self.quant_config = quant_config
self.calib_config = calib_config
self.pipe = pipe
def load_cosmopedia_simple(self):
data = load_dataset('HuggingFaceTB/cosmopedia-100k', split="train")
data = data.filter(lambda x: x["text_token_length"] >= 1024)
return [text for text in data["text"]]
def load_cosmopedia_tokenized(self):
data = load_dataset('HuggingFaceTB/cosmopedia-100k', split="train")
data = data.filter(lambda x: x["text_token_length"] >= 1024)
texts = [text for text in data["text"]]
tokenized_inputs = self.tokenizer(texts, truncation=True, max_length=self.calib_config['seq_len'])
return tokenized_inputs['input_ids']
def run(self):
try:
self.pipe.send("Starting model loading...")
model = AutoAWQForCausalLM.from_pretrained(
self.model_dir,
**{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(self.model_dir, trust_remote_code=True)
self.tokenizer = tokenizer
self.pipe.send("Loading calibration data...")
try:
calib_data = self.load_cosmopedia_simple()
self.pipe.send("Starting quantization...")
model.quantize(
tokenizer,
quant_config=self.quant_config,
calib_data=calib_data,
n_parallel_calib_samples=self.calib_config['n_samples'],
max_calib_samples=self.calib_config['max_samples'],
max_calib_seq_len=self.calib_config['seq_len']
)
except TypeError as e:
if "missing 1 required positional argument: 'attention_mask'" in str(e):
self.pipe.send("Model requires attention masks, switching to tokenized format...")
calib_data = self.load_cosmopedia_tokenized()
self.pipe.send("Restarting quantization...")
try:
model.quantize(
tokenizer,
quant_config=self.quant_config,
calib_data=calib_data,
n_parallel_calib_samples=self.calib_config['n_samples'],
max_calib_samples=self.calib_config['max_samples'],
max_calib_seq_len=self.calib_config['seq_len']
)
except Exception as inner_e:
tb = traceback.format_exc()
self.pipe.send(f"Detailed error during tokenized quantization:\n{str(inner_e)}\n\nTraceback:\n{tb}")
self.pipe.send(f"Model class: {model.__class__.__name__}")
self.pipe.send(f"Model type: {model.config.model_type if hasattr(model, 'config') else 'unknown'}")
self.pipe.send(f"AutoAWQ version: {awq.__version__ if 'awq' in globals() else 'unknown'}")
self.pipe.send(f"Transformers version: {transformers.__version__ if 'transformers' in globals() else 'unknown'}")
self.pipe.send(f"Calibration data type: {type(calib_data)}")
if isinstance(calib_data, list):
self.pipe.send(f"First calib item type: {type(calib_data[0]) if calib_data else 'empty list'}")
raise
else:
raise e
self.pipe.send("Saving quantized model...")
model.save_quantized(self.output_dir)
tokenizer.save_pretrained(self.output_dir)
self.pipe.send(f"Quantization complete. Model saved to {self.output_dir}")
except Exception as e:
tb = traceback.format_exc()
self.pipe.send(f"Error during quantization: {str(e)}\n\nTraceback:\n{tb}")
logging.error(f"Error during quantization: {str(e)}")
logging.error(f"Traceback:\n{tb}")
finally:
self.pipe.close()
class ProcessCommunicator(QObject):
message_received = Signal(str)
def __init__(self, pipe):
super().__init__()
self.pipe = pipe
def check_pipe(self):
if self.pipe.poll():
message = self.pipe.recv()
self.message_received.emit(message)
class App(QWidget):
def __init__(self):
super().__init__()
self.setWindowTitle("AWQ Quantization GUI")
self.model_path = ""
self.output_dir = ""
self.resize(800, 600)
self.quantization_process = None
self.process_communicator = None
layout = QVBoxLayout(self)
browse_group = QGroupBox("Model Selection")
browse_layout = QVBoxLayout()
model_layout = QHBoxLayout()
self.browse_btn = QPushButton("Browse Model Directory")
self.browse_btn.setToolTip("Select the directory containing the model files to be quantized")
self.browse_btn.clicked.connect(self.browse_model)
model_layout.addWidget(self.browse_btn)
self.path_label = QLabel("")
model_layout.addWidget(self.path_label)
browse_layout.addLayout(model_layout)
output_layout = QHBoxLayout()
self.output_browse_btn = QPushButton("Select Output Directory")
self.output_browse_btn.setToolTip("Choose the directory where the quantized model will be saved")
self.output_browse_btn.clicked.connect(self.browse_output)
output_layout.addWidget(self.output_browse_btn)
self.output_path_label = QLabel("")
output_layout.addWidget(self.output_path_label)
browse_layout.addLayout(output_layout)
browse_group.setLayout(browse_layout)
layout.addWidget(browse_group)
quant_group = QGroupBox("Quantization Settings")
quant_layout = QHBoxLayout()
version_layout = QVBoxLayout()
version_label = QLabel("Version:")
self.version_combo = QComboBox()
self.version_combo.addItems(["gemm", "gemv", "gemv_fast", "marlin"])
self.version_combo.setToolTip(
"Select quantization algorithm version:\n\n"
"gemm: General Matrix Multiplication algorithm\n"
"- Optimized for general matrix operations\n"
"- Suitable for batch sizes below 8\n"
"- Performs well with large contexts\n"
"- Faster than FP16 for small to medium batch sizes\n"
"- Requires zero_point=True\n\n"
"gemv: General Matrix-Vector algorithm\n"
"- 20% faster than GEMM for batch size 1\n"
"- Not recommended for large contexts\n"
"- Best suited for small batch sizes and low-latency tasks\n"
"- Requires zero_point=True\n\n"
"gemv_fast: Optimized GEMV implementation\n"
"- Specifically designed for single-token and small-batch inference\n"
"- 20% faster than standard GEMV for batch size 1\n"
"- Implements advanced packing and decoding techniques to reduce overhead\n"
"- Best for low-latency real-time tasks\n"
"- Uses efficient interleaving and dequantization to minimize memory bottlenecks\n"
"- Capable of efficient weight packing to enhance performance in memory-bound environments\n"
"- Requires zero_point=True\n\n"
"marlin: Advanced quantization algorithm\n"
"- Specialized for large-scale inference tasks, supporting batch sizes from 16 to 32 tokens\n"
"- Optimized for high-throughput environments with complex partitioning of inputs\n"
"- Delivers near-ideal 4x speedups compared to FP16, even for larger batch sizes\n"
"- Implements a unique memory layout and partitioning strategy to fully utilize GPU resources (L2 cache, tensor cores)\n"
"- Designed to handle high-demand tasks such as speculative decoding or multi-inference scenarios\n"
"- Features multiple optimizations like conflict-free memory operations, ideal access patterns, and reduced memory offset calculations\n"
"- Best suited for high-throughput and large-model scenarios\n"
"- Requires zero_point=False"
)
self.version_combo.currentTextChanged.connect(self.on_version_changed)
version_layout.addWidget(version_label)
version_layout.addWidget(self.version_combo)
quant_layout.addLayout(version_layout)
zero_point_layout = QVBoxLayout()
self.zero_point_chk = QCheckBox("Enable Zero Point")
self.zero_point_chk.setChecked(True)
self.zero_point_chk.setToolTip(
"Zero point setting for quantization:\n\n"
"- Must be True for gemm/gemv/gemv_fast\n"
"- Must be False for marlin\n"
"- This setting will update automatically based on version selection"
)
self.zero_point_chk.setEnabled(False)
zero_point_layout.addWidget(QLabel("Zero Point:"))
zero_point_layout.addWidget(self.zero_point_chk)
quant_layout.addLayout(zero_point_layout)
group_size_layout = QVBoxLayout()
group_size_label = QLabel("Group Size:")
self.group_size_spin = QSpinBox()
self.group_size_spin.setRange(8, 256)
self.group_size_spin.setValue(128)
self.group_size_spin.setToolTip(
"Quantization group size parameter:\n\n"
"Controls the granularity of weight quantization by defining the number of weights that share a scale factor.\n"
"- Default value is 128, meaning each group of 128 weights shares the same scale.\n"
"- Smaller group sizes (e.g., 32) increase precision but can use more memory.\n"
"- Larger group sizes (e.g., 128) optimize memory efficiency at the cost of some precision.\n"
"- Applies to all quantization algorithms like GEMM, GEMV, GEMVFast, and Marlin."
)
group_size_layout.addWidget(group_size_label)
group_size_layout.addWidget(self.group_size_spin)
quant_layout.addLayout(group_size_layout)
bit_width_layout = QVBoxLayout()
bit_width_label = QLabel("Bit Width:")
self.bit_width_spin = QSpinBox()
self.bit_width_spin.setRange(2, 8)
self.bit_width_spin.setValue(4)
self.bit_width_spin.setEnabled(False)
self.bit_width_spin.setToolTip(
"Number of bits to use for weight quantization:\n\n"
"Controls the bit width for quantizing weights. This affects the balance between memory efficiency and precision:\n"
"- Default is 4 bits, which significantly reduces memory usage while maintaining acceptable precision.\n"
"- Lower bit widths are more memory-efficient but less precise.\n"
"- This parameter is used across all quantization algorithms, including GEMM, GEMV, GEMVFast, and Marlin.\n"
"- Currently, AutoAWQ only supports 4-bit quantizations."
)
bit_width_layout.addWidget(bit_width_label)
bit_width_layout.addWidget(self.bit_width_spin)
quant_layout.addLayout(bit_width_layout)
quant_group.setLayout(quant_layout)
layout.addWidget(quant_group)
calib_group = QGroupBox("Calibration Settings")
calib_layout = QHBoxLayout()
parallel_samples_layout = QVBoxLayout()
parallel_samples_label = QLabel("Parallel Calibration Samples:")
self.parallel_samples_spin = QSpinBox()
self.parallel_samples_spin.setRange(1, 32)
self.parallel_samples_spin.setValue(1)
self.parallel_samples_spin.setToolTip(
"Number of calibration samples to process simultaneously during quantization:\n\n"
"- Controls how many samples from the dataset are processed in parallel\n"
"- Higher values can speed up calibration but require more GPU memory\n"
"- If set too high combined with a large max_calib_samples, may cause out-of-memory (OOM) errors\n"
"- Set to 1 for minimum memory usage but slower processing\n"
"- If set to None (not available in GUI), runs all samples simultaneously\n"
"- Adjust based on your GPU memory capacity and model size"
)
parallel_samples_layout.addWidget(parallel_samples_label)
parallel_samples_layout.addWidget(self.parallel_samples_spin)
calib_layout.addLayout(parallel_samples_layout)
max_samples_layout = QVBoxLayout()
max_samples_label = QLabel("Max Calibration Samples:")
self.max_samples_spin = QSpinBox()
self.max_samples_spin.setRange(1, 512)
self.max_samples_spin.setValue(128)
self.max_samples_spin.setToolTip(
"Maximum total number of samples to use for calibration:\n\n"
"Defines the total number of samples from the dataset that will be used to calibrate the model.\n"
"- The AWQ algorithm is sample efficient, so 128-256 samples is typically sufficient\n"
"- A higher number of samples may provide better calibration but will take longer\n"
"- Combined with parallel samples setting, affects total memory usage during quantization\n"
"- Adjust based on desired calibration quality and available compute resources"
)
max_samples_layout.addWidget(max_samples_label)
max_samples_layout.addWidget(self.max_samples_spin)
calib_layout.addLayout(max_samples_layout)
seq_len_layout = QVBoxLayout()
seq_len_label = QLabel("Max Sequence Length:")
self.seq_len_spin = QSpinBox()
self.seq_len_spin.setRange(4096, 32768)
self.seq_len_spin.setValue(8192)
self.seq_len_spin.setToolTip(
"Maximum sequence length for calibration samples:\n\n"
"Defines the maximum sequence length for samples during calibration.\n"
"- The max_calib_seq_len parameter ensures that any sample exceeding this length will be truncated to fit within the specified limit.\n"
"- Setting an appropriate value is important to ensure memory efficiency and maintain accuracy during calibration.\n"
"- Larger sequence lengths will require more memory, but may provide better calibration for models handling long contexts.\n"
"- Choose this value based on the expected input sequence length for your model and available memory resources."
)
seq_len_layout.addWidget(seq_len_label)
seq_len_layout.addWidget(self.seq_len_spin)
calib_layout.addLayout(seq_len_layout)
calib_group.setLayout(calib_layout)
layout.addWidget(calib_group)
button_layout = QHBoxLayout()
self.reset_btn = QPushButton("Reset to Defaults")
self.reset_btn.setToolTip("Reset all settings to their default values")
self.reset_btn.clicked.connect(self.reset_to_defaults)
button_layout.addWidget(self.reset_btn)
self.run_btn = QPushButton("Run Quantization")
self.run_btn.setToolTip("Start the quantization process with current settings")
self.run_btn.clicked.connect(self.run_quantization)
button_layout.addWidget(self.run_btn)
layout.addLayout(button_layout)
self.progress_bar = QProgressBar()
self.progress_bar.setRange(0, 0)
self.progress_bar.hide()
layout.addWidget(self.progress_bar)
self.output_text = QTextEdit()
self.output_text.setReadOnly(True)
self.output_text.setToolTip("Status messages and error output will appear here")
self.output_text.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Expanding)
layout.addWidget(self.output_text)
def on_version_changed(self, version):
if version == "marlin":
self.zero_point_chk.setChecked(False)
else:
self.zero_point_chk.setChecked(True)
def browse_model(self):
path = QFileDialog.getExistingDirectory(self, "Select Model Directory")
if path:
self.model_path = path
self.path_label.setText(f"Model Directory: {path}")
parent_path = str(Path(path).parent)
self.output_dir = parent_path
self.output_path_label.setText(f"Output Directory: {parent_path}")
def browse_output(self):
path = QFileDialog.getExistingDirectory(self, "Select Output Directory")
if path:
self.output_dir = path
self.output_path_label.setText(f"Output Directory: {path}")
def append_to_text_widget(self, content):
self.output_text.append(content)
def run_quantization(self):
if not self.model_path:
QMessageBox.critical(self, "Error", "Please select a model directory.")
return
if not self.output_dir:
QMessageBox.critical(self, "Error", "Please select an output directory.")
return
quant_config = {
"zero_point": self.zero_point_chk.isChecked(),
"q_group_size": self.group_size_spin.value(),
"w_bit": self.bit_width_spin.value(),
"version": self.version_combo.currentText()
}
calib_config = {
"n_samples": self.parallel_samples_spin.value(),
"max_samples": self.max_samples_spin.value(),
"seq_len": self.seq_len_spin.value()
}
parent_conn, child_conn = multiprocessing.Pipe()
self.quantization_process = QuantizationProcess(
self.model_path,
self.output_dir,
quant_config,
calib_config,
child_conn
)
self.quantization_process.start()
self.process_communicator = ProcessCommunicator(parent_conn)
self.process_communicator.message_received.connect(self.on_process_message)
self.timer = QTimer(self)
self.timer.timeout.connect(self.process_communicator.check_pipe)
self.timer.start(100)
self.progress_bar.show()
self.run_btn.setEnabled(False)
def on_process_message(self, message):
self.append_to_text_widget(message)
if "complete" in message.lower() or "error" in message.lower():
self.progress_bar.hide()
self.timer.stop()
self.run_btn.setEnabled(True)
if self.quantization_process:
self.quantization_process.join()
self.quantization_process = None
def reset_to_defaults(self):
self.version_combo.setCurrentText("gemm")
self.zero_point_chk.setChecked(True)
self.group_size_spin.setValue(128)
self.bit_width_spin.setValue(4)
self.parallel_samples_spin.setValue(1)
self.max_samples_spin.setValue(128)
self.seq_len_spin.setValue(8192)
self.append_to_text_widget("Reset all parameters to defaults.")
if __name__ == "__main__":
multiprocessing.freeze_support()
app = QApplication([])
app.setStyle(QStyleFactory.create("Fusion"))
widget = App()
widget.show()
sys.exit(app.exec())It uses the Steps
I also tested it starting with AWQ models provided by Qwen3 themselves...no difference, both work. The only reason I created a custom autoawq library was because Qwen3 didn't provide AWQ version of the 1.7 nor .6B versions. |
Contributor
Author
|
If you only get this error when starting the program don't worry, it's normal but will still work: 
|
Collaborator
|
Thanks so much, great work |
Lanurence666
pushed a commit
to Lanurence666/CTranslate2_backport_py38
that referenced
this pull request
May 23, 2026
veilnetwork
pushed a commit
to veilnetwork/CTranslate2
that referenced
this pull request
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will support running a Qwen3 AWQ model that has been converted into the ctranslate2 format.
Just FYI, currently the
autoawqGithub repository is archived as of May 11, 2025. The "vllm" project has expanded upon it in theirllm-compressorlibrary, however, it doesn't output an AWQ model in a format that can be converted to the ctranslate2 format. Thus, I was forced to patch the originalautoawqrepository to perform the conversions of qwen3.TLDR - only the models here https://huggingface.co/collections/ctranslate2-4you/qwen3 will work unless/until someone continues with the original
autoawqpackage.Alternate uploads here:
https://huggingface.co/collections/CTranslate2HQ/ctranslate2-qwen3
The best way to test this before approving the PR would be to simply copy and paste my modified Qwen3Loader into
transformers.pyin an already pip installed version of ctranslate2, but here it is again for convenience:MODIFIED QWEN3LOADER HERE