A Coding Implementation to Run Qwen3.5 Reasoning Models Distilled with Claude-Style Thinking Using GGUF and 4-Bit Quantization

Code implementation to run Qwen3.5 reasoning models distilled with Claude-style thinking using GGUF and 4-bit quantization.

A single flag switches between a 27B GGUF variant and a lightweight 2B 4-bit version; MODEL_PATH is set to "2B_HF" and the code checks for "27B_GGUF" to decide installation steps.

A Colab pipeline is provided; GPU availability is validated at start, the GPU name and total VRAM (GB) are printed, and a RuntimeError is raised instructing to change the runtime type to a T4 GPU if no GPU is present.

Dependency setup installs either llama.cpp or transformers with bitsandbytes depending on the chosen path; torch is used to check CUDA availability and device info.

A ChatSession class implements multi-turn interaction and utilities parse <think> traces to separate reasoning from final outputs.