Sharing Two Open-Source Projects for Local AI & Secure LLM Access 🚀

Quansloth is an air-gapped private AI server that brings advanced KV-cache compression to consumer hardware.

It connects a Gradio Python frontend with a highly optimized llama.cpp CUDA backend.

Quansloth prevents GPU crashes and enables very large contexts, handling 32k+ token contexts on a 6GB RTX 3060 for workloads that typically demand a 24GB RTX 4090.

Using Google’s TurboQuant (ICLR 2026) approach, it compresses 16-bit to 4-bit for about 75% VRAM savings and includes live analytics that reports exact VRAM allocation by intercepting C++ engine logs.

Context Injector uploads long PDFs into the chat stream; API2CHAT is an ultra-lightweight (<9KB) client-side GUI that runs entirely in browser memory, needs no PHP/Node/Python host, can be served from GitHub Pages/S3/or locally, and reads local files without uploading.