prism-ml/Ternary-Bonsai-27B-gguf
text-generation · llama.cpp, gguf, conversational
Home / ⚡ Inference / Serving
⚡ Inference / Serving
77 View all · Daily curated AI / LLM open-source intelligence, with plain-language notes and license checks.
empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF
image-text-to-text · gguf, llama.cpp, quantized
deepreinforce-ai/Ornith-1.0-35B-GGUF
text-generation · transformers, gguf, text-generation
DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF
image-text-to-text · gguf, unsloth, fine tune
unsloth/gemma-4-12b-it-GGUF
image-text-to-text · gguf, gemma4, unsloth
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF
image-text-to-text · transformers, gguf, llama.cpp
GnLOLot/MiniCPM5-1B-Claude-Opus-Fable5-Thinking-GGUF
text-generation · gguf, llama.cpp, quantized
unsloth/gemma-4-12B-it-qat-GGUF
any-to-any · transformers, gguf, gemma4
prism-ml/Bonsai-27B-mlx-1bit
text-generation · mlx, safetensors, qwen3_5
yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF
text-generation · gguf, gemma4, coding
unsloth/DeepSeek-V4-Flash-GGUF
· gguf, deepseek_v4, unsloth
huihui-ai/Huihui-GLM-5.2-abliterated-GGUF
text-generation · transformers, gguf, glm_moe_dsa
OBLITERATUS/Gemma-4-12B-OBLITERATED
text-generation · transformers, safetensors, gguf
bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF
text-generation · gguf, llama.cpp, qwen
LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF
image-text-to-text · hermes, gguf, uncensored
Mia-AiLab/Qwable-3.6-27b
· transformers, gguf, qwen
mistralai/Leanstral-1.5-119B-A6B
· vllm, base_model:mistralai/Leanstral-2603, base_model:finetune:mistralai/Leanstral-2603
poolside/Laguna-M.1
text-generation · vllm, safetensors, laguna
mradermacher/Mythos-nano-i1-GGUF
· transformers, gguf, reasoning
hotdogs/qwen3.6-27b-fable5-lora
text-generation · peft, safetensors, gguf
unsloth/Qwen3.6-27B-MTP-GGUF
image-text-to-text · transformers, gguf, unsloth
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive
image-text-to-text · gguf, uncensored, qwen3.6
DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF
image-text-to-text · gguf, unsloth, fine tune
BugTraceAI/BugTraceAI-CORE-Ultra-27B-Q6
· gguf, qwen3, security
build-small-hackathon/llm-cinema
· gradio, build-small-hackathon, thousand-token-wood
Jiunsong/supergemma4-26b-uncensored-gguf-v2
text-generation · gguf, gemma4, uncensored
zeraix/zeraix
Open-source local AI workspace — advancing on-device inference.
Jia-Ethan/codex-keysmith
Version-independent Codex instruction deployment with dry-run, backups, hook isolation, and recovery.
avifenesh/bw24
From-scratch Rust+CUDA inference engine, bit-exact by construction — NVFP4, MoE, MTP speculative decoding, tuned against measured limits of one RTX 5090 Laptop (sm_120a).
RavichandranJ/Dolphin3-Cyber-8B-GGUF
text-generation · transformers, gguf, llama
Cactus-Compute/needle
· jax, safetensors, needle
re4/LibreCode
LibreCode - A Ollama cursor like coding / Reversing Interface
seuksaedu-beep/Persona-Flow-Chat
Simulate AI Personas in Social Feeds with Real-Time Chat Bots 2026
OpenCPIL/prima.cpp
[Official] prima.cpp: Fast 30-70B LLM inference on heterogeneous and everyday home devices
ggml-org/gguf-my-repo
· docker, region:us
0xSero/glm-5.2-sm120
GLM-5.2-NVFP4-REAP-469B serving on SM120 (4× RTX PRO 6000 Blackwell) — one-command vLLM launch recipe, 250K context, DeepSeek Sparse Attention + MTP speculative decode
cyyself/OpenTihui
on-device LLM for iOS with keyboard shortcuts
marchinthesun/EPYC-Max-Inference-LLM-Optimizer
LLM Optimizer for NUMA, and monitor LLM system
chiennv2000/orthrus
Fast, lossless LLM inference via dual-view diffusion decoding.
walkinglabs/modern-llm-notebook
A hands-on course for building modern LLMs from scratch in PyTorch, with 26 runnable Jupyter Notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation, and distillation.
Avarok-Cybersecurity/atlas
Pure Rust Inference Engine
hec-ovi/vllm-awq4-qwen
vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA.
AlexsJones/llmfit
Hundreds of models & providers. One command to find what runs on your hardware.
ahammadmejbah/Awesome-Datasets-Hub
A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
amitshekhariitbhu/llm-internals
Learn LLM internals step by step - from tokenization to attention to inference optimization.
devnen/qwen3.6-windows-server
One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.
0xNyk/council-of-high-intelligence
18 AI personas deliberate your hardest decisions across multiple LLM providers. Aristotle, Feynman, Kahneman, Torvalds & more — structured multi-round deliberation with genuine model diversity. One command: /council
Arthur-Ficial/apfel
The free AI already on your Mac. CLI tool, OpenAI-compatible server, and interactive chat — all on-device via Apple Intelligence. No API keys, no cloud, no downloads.
mnfst/awesome-free-llm-apis
List of Permanent Free LLM API (API Keys)
Andyyyy64/whichllm
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
generative-computing/granite-switch
Granite Switch — Build AI models like you build software
PMZFX/intel-arc-pro-b70-benchmarks
Benchmark results and performance data for the Intel Arc Pro B70 GPU (Xe2/Battlemage) - LLM inference, video generation, dual-GPU scaling.
jmaczan/tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Dynamis-Labs/spectralquant
SpectralQuant: Calibrated Eigenbasis Rotation and Water-Filled Bit Allocation for KV-Cache Compression
quantumaikr/quant.cpp
LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.
tinyBigGAMES/VindexLLM
VindexLLM is a pure Delphi, GPU-powered LLM inference engine that uses Vulkan compute shaders to run GGUF models entirely on the GPU. It performs full transformer inference without relying on Python, CUDA, or other external runtimes, requiring only vulkan-1.dll, which is typically included with modern GPU drivers.
NightMean/OlliteRT
Turn your Android phone into an OpenAI-compatible LLM inference server — Fully local, private and Open Source
brontoguana/krasis
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
Kaden-Schutt/hipfire
RDNA-native LLM inference engine in Rust.
lambda-calculus-LLM/lambda-RLM
Method for Long Context RLMs using verifiable Lambda Calculus
AnkitNayak-eth/llmBench
llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing deep hardware intelligence.
mukel/gemma4.java
Fast Gemma 4 inference in pure Java
psmarter/CUDA-Practice
CUDA编程练习项目-Hands-on CUDA kernels and performance optimization, covering GEMM, FlashAttention, Tensor Cores, CUTLASS, quantization, KV cache, NCCL, and profiling.
Zyora-Dev/zse
The inference engine the open-source world built for itself.
Mcourtyard/m-courtyard
M-Courtyard: Local AI Model Fine-tuning Assistant for Apple Silicon. Zero-code, zero-cloud, privacy-first desktop app powered by Tauri + React + mlx-lm.
matt-k-wong/mlx-flash
Flash weight streaming for MLX: run massive models larger than your RAM on Apple Silicon.
OnlyTerp/turboquant
First open-source implementation of Google TurboQuant (ICLR 2026) -- near-optimal KV cache compression for LLM inference. 5x compression with near-zero quality loss.
BenChaliah/Tensa-Lang
TensaLang is a Tensor-first programming language, compiler, and runtime that let you write the Model’s inference engine (e.g. LLMs) and sampling in high level language, then compile it through MLIR to Multiple targets (e.g. CPU, CUDA, ROCm)
TheToughCrane/nano-kvllm
This project aims to provide a high effective KV cache manage framework for llm inference and improve memory utilization and inference speed.
OpenDCAI/Flash-MinerU
Ray-powered accelerator for MinerU, turning PDF → Markdown into a scalable, cluster-ready data infrastructure. 基于 Ray 的 MinerU 加速层，将 PDF → Markdown 构建为可扩展、面向集群的数据基础设施。
unixsysdev/llama-turboquant
TurboQuant 3-bit KV-cache quantization for llama.cpp
Aaryan-Kapoor/ModelGate-Hackathon
🏆Winning Project | ModelGate is a contract-aware AI control plane that ingests customer contracts, extracts SLA/privacy/routing constraints, and generates an OpenAI-compatible endpoint that automatically routes every request to the optimal model. Simple queries go to cheap models. Complex queries go to premium ones.
AbdelStark/attnres
Rust implementation of Attention Residuals from MoonshotAI/Kimi
openmed-labs/synthvision
Synthetic medical VQA pipeline: 119K images annotated by frontier VLMs, cross-validated at 93% agreement, fine-tuned on 3 model families (2-3B params)
Yog-Sotho/LLM-fine-tuner
Powerful no-code LLM fine-tuner: upload data → train → deploy in minutes. Unsloth 2-5× acceleration · QLoRA/DPO/RLHF/PPO/ORPO · Reward Model training · GGUF export · vLLM inference · BLEU/ROUGE/BERTScore · full CLI · Heretic Mode to unlock full model potential
eullm/eullm
Open-source platform for creating, distributing and running sovereign EU-compliant LLMs. Verticalize any model for your domain, language and brand. AI Act ready.