Home / State of AI / Inference & Efficiency

Inference & Efficiency

Inference optimization, quantization, serving infrastructure, and hardware

Updated 2026-W39 #state-of-ai #state-of-ai-inference-and-efficiency

Living document tracking inference optimization, quantization, serving infrastructure, and hardware. Newest entries appear at the top.

Key Areas to Watch

Quantization techniques (GPTQ, AWQ, GGUF)
Serving frameworks (vLLM, TGI, Ollama)
Hardware advances (GPUs, TPUs, custom silicon)
Edge and on-device inference
Cost optimization and throughput improvements

2026-W39

NVIDIA Vera Rubin NVL72 in MLPerf Inference v6.1 — Benchmark results provide evidence about inference throughput and infrastructure scaling. [NVIDIA]
How GLM built its own inference infrastructure — Production account of the serving stack behind a large model family. [HN]
RheoSampling — Targets acceptance-rate collapse in stochastic dynamic-tree speculative decoding. [arXiv]
Retrieve-for-Train — Google Research moves repeated search work into training to reduce inference-time bottlenecks. [Google Research]

2026-W38

Simple Attention Sparsification — Gated sparse attention optimized by language-modeling loss targets long-context cost. [x.com]
BeaconKV — Training-free KV-cache compression for long reasoning traces and repeated context access. [x.com]
Uno — Diffusion head on autoregressive LLMs for parallel token generation and faster inference. [x.com]
Qwen3.8-27B quantized for vLLM/SGLang — Deployable 4-bit NVFP4/FP8 artifacts keep open multimodal models practical. [x.com]

2026-W37

Qwen 3.8 27B on Cerebras — High-throughput serving option for latency-sensitive agent sub-tasks. [HN]
Hugging Face WebGPU kernels — 200+ local/browser AI kernels expand practical WebGPU deployment paths. [Hugging Face]
Puro-2B — Small open recipe trained on consumer GPUs, relevant for narrow helper-model routing. [x.com]

2026-W36

OpenAI API pricing — GPT-5.6 Sol price changes affect model-routing economics for agent loops. [HN]
Jalapeno inference results — Inference speed and efficiency claims are relevant to deployment cost and latency tradeoffs. [OpenAI]
Learning a Size-Weight Frontier for Synthetic-Augmented Inference — Research on inference efficiency tradeoffs for synthetic-augmented setups. [arXiv]

2026-W32

GPT-5.6 price-performance frontier — Model cost reductions are now an agent-system design constraint. [OpenAI]
Show HN: Gemma 4 26B in 2 GB RAM on M-series Macs — Local inference experiments keep squeezing larger models into developer hardware. [HN]
DeepSeek V4 Flash on a single AMD MI300X — Deployment work around new open models continued alongside hosted API price moves. [HN]

2026-W28

WattGPU — Power and latency prediction for unseen GPU/LLM pairings supports deployment planning. [arXiv]
NVIDIA inference software stack — Token cost stayed a primary production-inference metric. [NVIDIA]
Hugging Face Kernels — Kernel distribution improvements keep open inference stacks moving. [Hugging Face]

2026-W27

OpenAI and Broadcom unveil LLM-optimized inference chip — Jalapeño targets LLM inference throughput and efficiency. [OpenAI]
DSpark: Speculative decoding accelerates LLM inference — DeepSeek published speculative decoding work for faster serving. [HN]
Run a vLLM Server on HF Jobs in One Command — Hugging Face reduced friction for quick vLLM deployment experiments. [Hugging Face]

2026-W24

Online Pandora’s Box for Contextual LLM Cascading — Adaptive API-query selection formalizes model cascading for cost-aware production inference. [arXiv]
Sparsely gated tiny linear experts — Explores smaller sparse experts as a way to scale parameter count without dense compute growth. [arXiv]
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling — Reports a sparse MoE training path on a single eight-GPU node, relevant to smaller-lab scaling strategies. [arXiv]
NVIDIA Jetson Brings Agentic AI to the Physical World — JetPack 7.2 and Jetson runtime updates matter for edge inference and robotics agents. [NVIDIA]
Uber’s $1,500/month AI limit is a useful signal for AI tool pricing — Quota caps make cost-aware routing and loop bounding part of agent-inference architecture. [Simon Willison]

2026-W23

NVIDIA Vera CPU benchmark notes — Initial benchmarks emphasize high-core sustained performance and memory bandwidth for AI-factory workloads. [NVIDIA]
Shipping a Trillion Parameters With a Hub Bucket — Delta Weight Sync in TRL targets practical distribution of very large model updates. [Hugging Face]
GPU Forecasters — Explores language models as selective surrogates for expensive GPU-kernel runtime measurements during optimization. [arXiv]
On Efficient Scaling of GNNs via IO-Aware Layers Implementations — Analyzes sparse and irregular memory-access bottlenecks in graph neural network layers with an I/O-aware lens. [arXiv]
NVIDIA quantized Qwen3.6 MoE on Hugging Face — A 35B-total, 3B-active NVFP4 release highlighted continued momentum in lower-memory open-model serving. [x.com]

2026-W22

Mistral 3 NVFP4 checkpoint — Mistral targets practical vLLM serving paths on Blackwell NVL72 and single-node 8xA100/8xH100 systems. [Mistral]
MagenticLite, MagenticBrain, Fara 1.5 — Small-model agent orchestration highlights efficiency as an agent-design constraint, not just an inference benchmark. [Microsoft Research]

2026-W19

DeepSeek 4 Flash local inference engine for Metal — Salvatore Sanfilippo ships a Metal-native inference engine for DeepSeek 4 Flash; opens low-latency local-inference on Apple Silicon for the DS4 model family. [HN]
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge by Wenbo Zhang et al. — Routes only ambiguous judgments to expensive reasoning models; 47% cost reduction at near-identical accuracy for evaluation pipelines. [arXiv]
Zed Zeta2.1: 3x fewer tokens, 50ms faster — Inline code completion model drops token count 3x and latency 50ms versus Zeta2.0; shows per-task model specialization continues to outperform general-purpose scaling for low-latency editor use. [Zed Blog]
Grok voice models and audio APIs — xAI releases grok-voice-think-fast-1.0 plus standalone Grok STT and TTS APIs; first real-time audio inference stack from xAI that’s separate from the Grok chat product. [xAI]

2026-W18

2.5x faster inference with Qwen 3.6 27B using MTP — Multi-token prediction landing in open inference stacks; first credible local-agent latency profile for the Qwen 3.6 line. [Reddit]
Accelerating Gemma 4: faster inference with multi-token prediction drafters — Google ships MTP drafters for Gemma 4. [Google]
NVIDIA Spectrum-X — the Open, AI-Native Ethernet Fabric — NVIDIA’s open Ethernet alternative for hyperscale AI clusters at gigascale. [NVIDIA]
Unlocking large scale AI training networks with MRC (Multipath Reliable Connection) — OpenAI infrastructure post on supercomputer networking for distributed training (May 5 — early W19). [OpenAI]
vLLM V0 to V1: Correctness Before Corrections in RL — Hugging Face technical write-up on the vLLM rewrite for RL training correctness. [Hugging Face]
Grok 4.3 ~40% input price reduction — xAI rebases input pricing for the 1M-context tier; reshapes cost-per-token math for agentic workloads. [xAI]
Mistral Medium 3.5 runs on as few as four GPUs — Self-hosting story for the new flagship; notable inference-cost lever for enterprise deployments. [Mistral]

2026-W17

DeepSeek V4 — 1M-token context for agents — DeepSeek V4-Pro and V4-Flash ship with 1M-token MoE contexts at a fraction of frontier price; first open-weight 1M model with the latency profile to be useful inside agents. [Hugging Face]
Google’s eighth-generation TPUs: two chips for the agentic era — 452 HN points; Google’s silicon answer to mixed agent workloads. [HN]
GPT-5.5 on NVIDIA GB200 NVL72 — GPT-5.5 deployed on rack-scale GB200 NVL72; over 10,000 reference systems already running. [NVIDIA]
Anthropic and Amazon expand to 5 GW of Trainium — New Trainium2 capacity in H1 2026 and ~1 GW Trainium2 + Trainium3 by year-end. [Anthropic]
Decoupled DiLoCo: distributed AI training — DeepMind’s next-generation distributed training algorithm with stronger fault tolerance. [DeepMind]
Decoupled DiLoCo (paper) — Companion arXiv discussion of agentic world modeling under distributed-training constraints. [arXiv]
QuantClaw: Precision Where It Matters for OpenClaw — Quantization sensitivity in autonomous-agent workloads; precision needed only at specific layers, not uniformly. [arXiv]
Running Qwen3.6-27B locally via unsloth GGUF — GGUF quants available within hours of release. [Hugging Face]
xAI Colossus 2 expands to 1.5 GW — Confirmed expansion from 1 GW to 1.5 GW supercluster capacity. [xAI]

2026-W16

Anthropic invests $50B in American AI infrastructure — Texas and New York as initial sites; sizes Anthropic’s 2026–2027 training and serving capacity. [Anthropic]
The RAM shortage could last years — Hardware bottleneck that is now directly shaping model-release roadmaps. [Verge / HN]
Rethinking AI TCO: Why Cost per Token Is the Only Metric That Matters — NVIDIA reframes datacenters as “AI token factories” and argues cost-per-token is the central economic metric. [NVIDIA]
xAI $20B Series E earmarked for compute buildout — NVIDIA and Cisco among strategic investors in the compute-focused upsize. [xAI]
Running Qwen3.6-35B-A3B at 79 t/s on RTX 5070 Ti + 9800X3D — Practical consumer-GPU deployment notes, including the --n-cpu-moe flag as the key unlock. [Reddit]
Qwen3.6-35B-A3B 8-bit on an M5 Max 128GB, ‘as good as Claude’ — Apple Silicon inference report. [Reddit]
1-bit Bonsai 1.7B running locally in-browser on WebGPU — 290MB 1-bit LLM entirely in the browser; 1,110 Reddit points. [Reddit]
Ternary Bonsai: Top intelligence at 1.58 bits — Further 1.58-bit quantization progress for small models. [Reddit]
The local LLM ecosystem doesn’t need Ollama — Argues the local-LLM community has outgrown Ollama’s abstraction layer; 642 HN points. [HN]
Measuring Claude 4.7’s tokenizer costs — Independent cost-impact study of the Opus 4.7 tokenizer change. [HN]

2026-W15

Anthropic’s multi-gigawatt compute partnership with Google and Broadcom — Scale-up at the inference/training infrastructure layer; signals how frontier labs are locking in silicon capacity. [Anthropic]
Audio processing landed in llama-server with Gemma-4 — Multimodal audio inference now native in the reference open-source serving stack. [Reddit]
Gemma 4 audio with MLX — Apple-silicon audio pipeline for Gemma 4. [Simon Willison]
Waypoint-1.5: Higher-Fidelity Interactive Worlds for Everyday GPUs — Consumer-GPU-first interactive world model update; continues the on-device inference push. [Hugging Face]

2026-W14

Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs — PrismML claims first commercially viable 1-bit LLMs; 430 HN points, direct extension of BitNet research into production-ready packaging. [HN]
Gemma 4 on iPhone via Google AI Edge Gallery — Google ships Gemma 4 running on-device on iOS; 868 HN points, 234 comments — the strongest open-model on-device story since M5 Apple Silicon. [HN / App Store]
Lemonade by AMD: a fast and open source local LLM server using GPU and NPU — AMD’s first serious developer-focused inference server routes across GPU and NPU; 572 HN points, counters NVIDIA’s dominance in local inference tooling. [HN]
From RTX to Spark: NVIDIA Accelerates Gemma 4 for Local Agentic AI — NVIDIA details running Gemma 4 locally on RTX and DGX Spark for agentic workflows. [NVIDIA]

2026-W13

iPhone 17 Pro Demonstrated Running a 400B LLM — On-device 400B inference demo sparks debate about quantization depth and measurement methodology; 713 HN points, 326 comments. [Twitter / HN]
TurboQuant: Redefining AI efficiency with extreme compression — Google Research quantization technique achieves compression beyond INT4 with competitive accuracy; 576 HN points, one of the most-discussed efficiency papers of the month. [Google Research / HN]
CERN uses ultra-compact AI models on FPGAs for real-time LHC data filtering — CERN burns tiny AI models into silicon for nanosecond-level trigger decisions; extreme edge inference in a scientific production context; 334 HN points. [The Open Reader / HN]

2026-W12

Show HN: Sub-millisecond VM sandboxes using CoW memory forking — Sub-ms VM boot via copy-on-write memory forking; directly relevant to fast isolated agent code execution at scale; 311 HN points. [GitHub / HN]
Introducing GPT-5.4 mini and nano — Smaller, faster GPT-5.4 variants optimized for tool use, multimodal reasoning, and high-volume sub-agent pipelines; 248 HN points. [OpenAI]
Holotron-12B — High Throughput Computer Use Agent — HCompany 12B model optimized for high-throughput computer-use and agentic workflows; efficiency-focused architecture. [Hugging Face]

2026-W11

Can I run AI locally? — Hardware-compatibility checker for local LLM inference; 1,520 HN points, 353 comments — signal that on-device inference is now mainstream enough to need a dedicated lookup tool. [HN / canirun.ai]
BitNet: Inference framework for 1-bit LLMs — Microsoft’s inference runtime for 1-bit quantized models resurfaces with 370 HN points; production-facing implementation of the ternary-weight research. [GitHub / Microsoft]
Ulysses Sequence Parallelism: Training with Million-Token Contexts — Technical guide to sequence-parallel training that enables million-token context windows at scale. [Hugging Face]
Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries — Survey of async RL training throughput trade-offs across 16 open-source libraries. [Hugging Face]

2026-W10

Introducing GPT-5.4 — 1M-token context, state-of-the-art coding, computer use, and tool search; OpenAI’s most capable and efficient frontier model; 1,019 HN points. [OpenAI]
Gemini 3.1 Flash-Lite: Built for intelligence at scale — Google DeepMind’s fastest, most cost-efficient model in the Gemini 3 series; designed for throughput workloads. [DeepMind]
Show HN: I built a sub-500ms latency voice agent from scratch — Detailed architecture walkthrough achieving sub-500ms end-to-end latency for real-time voice inference; 570 HN points, 153 comments. [HN]
Bringing Robotics AI to Embedded Platforms: Dataset Recording, VLA Fine-Tuning, and On-Device Optimizations — NXP + Hugging Face guide for running vision-language-action models on constrained edge hardware. [Hugging Face / NXP]

2026-W09

Introducing Mercury 2: Fast Reasoning LLM powered by diffusion — Inception Labs’ diffusion-based reasoning LLM challenges autoregressive latency assumptions; competitive speed at a fraction of typical token-generation latency; 351 HN points. [HN / Inception Labs]
Qwen3.5 27B dense — 170k context at 100+ t/s on 2×3090 — 585 t/s throughput for 8 simultaneous requests at 170k context on commodity consumer hardware; 620 upvotes. [r/LocalLLaMA]

2026-W08

The path to ubiquitous AI (17k tokens/sec) — Taalas essay on chip-level LLM inference reaching 17k tokens/sec; argues dedicated silicon is approaching the cost-per-token crossover for always-on local models; 847 HN points, 455 comments. [HN / Taalas]
How Taalas “prints” LLM onto a chip — Technical explainer on Taalas’s architecture for imprinting inference directly onto dedicated silicon; 429 HN points, 256 comments. [HN]
GGML and llama.cpp join Hugging Face to ensure the long-term progress of Local AI — ggml.ai formalizes llama.cpp’s home on HF with infrastructure and maintainer support; 839 HN points — local inference secured as a first-class priority. [HN / Hugging Face]

2026-W07

Introducing GPT-5.3-Codex-Spark — OpenAI’s first real-time coding model: 15x faster generation than its predecessor with a 128k context window; 890 HN points. [OpenAI / HN]
Two different tricks for fast LLM inference — Sean Goedecke compares Anthropic’s fast mode (up to 2.5x token throughput) vs. GPT-5.3-Codex-Spark’s architectural speed gains; useful mental model for choosing inference strategy. [Sean Goedecke]

2026-W06

Nano-vLLM: How a vLLM-style inference engine works — Deep-dive on building a minimal vLLM-compatible inference engine; strong reference for understanding continuous batching, KV-cache, and scheduling internals; 271 HN points. [Neutree / HN]

2026-W04

GLM-4.7-Flash is even faster now — llama.cpp PR landing inference speed improvements for GLM-4.7-Flash; 264 upvotes, notable for showing inference-speed gains reaching community models via upstream optimizations. [r/LocalLLaMA]

2026-W03

Furiosa: 3.5x efficiency over H100s — Korean chip startup Furiosa claims its RNGD server delivers 3.5x better efficiency per watt than H100 for LLM inference at data-center scale; first credible alternative chip story for production inference in 2026; 160 HN comments examining methodology. [HN / Furiosa]
OpenAI partners with Cerebras — 750MW of Cerebras wafer-scale compute added to reduce inference latency for real-time ChatGPT workloads. [OpenAI]
4x AMD R9700 (128GB VRAM) + Threadripper 9955WX Build — Community build showcasing 128GB VRAM on AMD silicon as a viable local inference rig. [r/LocalLLaMA]
Raspberry Pi’s New AI Hat Adds 8GB of RAM for Local LLMs — Jeff Geerling benchmarks the Raspberry Pi AI Hat 2 with 8GB — enough to run small LLMs at the edge. [HN / jeffgeerling.com]

2026-W02

NVIDIA Jetson T4000 module — Blackwell-powered edge module with 4x greater energy efficiency. [NVIDIA]

2026-W01

Adaptive-P: a new sampler for creative text generation (llama.cpp PR) — Sampler that adapts top-p by token distribution shape; 118 upvotes. [r/LocalLLaMA]
Ratios of Active to Total Parameters on major MoE models — Community architecture-comparison post. [r/LocalLLaMA]

2025-W51

NanoGPT speedrun WR: 127.7s (was 8.2 min a year ago) — ~4x YoY training wall-clock improvement on fixed hardware. [r/LocalLLaMA]
Gemma Scope 2 — V2 sparse-autoencoder probes for Gemma. [DeepMind]

2025-W50

New in llama.cpp: Model Management — First-class downloads/caching for llama.cpp models. [Hugging Face]
2025 Open Models Year in Review — Nathan Lambert’s open-weights landscape recap. [Interconnects]

2025-W48

$900 for 192GB RAM on Oct 23 → $3k+ in late Nov — AI-driven memory price spike hits consumer builds; 1,012 upvotes. [r/LocalLLaMA]
Continuous batching from first principles — Inference-serving architecture explainer. [Hugging Face]

2025-W47

20x Faster TRL Fine-tuning with RapidFire AI — Speedup over stock TRL for RL fine-tuning. [Hugging Face]
Easily Build and Share ROCm Kernels with Hugging Face — AMD ROCm kernel-sharing on the HF Hub. [Hugging Face]

2025-W46

Faster Maya1 TTS (50s audio per 1s compute) — Community TTS speedup. [r/LocalLLaMA]
AMD Ryzen AI Max 395+ 256/512 GB Ram — Serious local-AI rigs on Strix Halo unified memory; 252 upvotes. [r/LocalLLaMA]

2025-W45

Faster Prompt Processing in llama.cpp: Smart Proxy + Slots + Restore — Community-built prompt-processing optimization; 77 upvotes. [r/LocalLLaMA]
GLM-4.5-Air Q4 at Full Context on Strix Halo vs Dual RTX 3090 — Cross-hardware benchmark for 106B MoE at full context. [r/LocalLLaMA]

2025-W44

Streaming datasets: 100x More Efficient — Datasets library efficiency win for streaming workloads. [Hugging Face]

2025-W43

Apple M5 Neural Accelerators on MLX — 3.3–4x faster TTFT vs M4; 14B dense under 10s, 30B MoE under 3s on MacBook Pro. [Apple]
M5 Neural Accelerator benchmarks in llama.cpp — Community benchmarks as llama.cpp support lands; 186 upvotes. [r/LocalLLaMA]
Intel Crescent Island (OCP Summit) — Xe3P GPU with 160GB LPDDR5X aimed at AI inference. [AI Business]

2025-W38

Jeff Geerling regrets $3000 Pi AI cluster — Pi hardware limitations for AI; 468 HN points. [HN]
Alibaba AI chip comparable to NVIDIA H20 — China’s compute self-sufficiency push; 297 HN points. [Futunn]
Qwen3-Coder-480B on M3 Ultra Mac Studio — 480B model running locally on Apple Silicon; 144 upvotes. [r/LocalLLaMA]

2025-W33

GPT-OSS-120B on 8GB VRAM + 64GB RAM — Consumer deployment of OpenAI’s 120B open model; 248 HN points. [r/LocalLLaMA]
GPT-OSS-20B at 10K tokens/sec on a 4090 — Blazing local inference speeds on consumer GPU. [r/LocalLLaMA]
M4 Max generation speed vs context size — Benchmarking Apple Silicon inference scaling; 232 upvotes. [r/LocalLLaMA]
Gemma 3 270M for edge AI — Tiny model targeting constrained device deployment. [Google]

2025-W32

GPT-OSS: MoE + MXFP4 quantization — gpt-oss-120b runs on a single 80GB GPU; gpt-oss-20b runs on 16GB edge devices. MXFP4 4-bit quantization enables consumer deployment. [OpenAI]
Claude Sonnet 4 gets 1M context window — Beta header enables 1M tokens for tier 4 API users, alongside GPT-5’s 400K context. [simonw]
How can ChatGPT serve 700M users when I can’t run GPT-4 locally? — Popular Ask HN on inference economics at scale; 574 HN points. [HN]
I want everything local — offline AI workspace — Comprehensive guide to fully local AI development; 1,143 HN points. [HN]

2025-W31

GLM-4.5-Air runs Space Invaders on a laptop — Simon Willison demonstrates GLM-4.5-Air (106B total, 12B active) running code generation locally on consumer hardware; 577 HN points. [simonwillison.net]
Tokens are getting more expensive — Analysis of rising AI subscription costs and the consumer “short squeeze”; 363 HN points. [HN]
Mac M3 + RooCode + Qwen3-Coder-30B local setup — Competitive local AI coding demonstrated with 4-bit quantized Qwen3-Coder on consumer Apple Silicon. [r/LocalLLaMA]

2025-W30

Cerebras launches Qwen3-235B at 1,500 tok/s with 131K context — Frontier-scale open weights running at wafer-scale speeds. [Cerebras]
Don’t bother parsing: use images for RAG — Morphik argues VLMs can skip document parsing entirely. [HN]
Price Per Token — LLM API pricing tracker — Live pricing comparison across LLM APIs. [HN]

2025-W28

K2-Mini: Kimi K2 compressed from 1.07T → 32.5B — 97% parameter reduction runs on a single H100. [r/LocalLLaMA]
BentoML LLM Inference Handbook — Complete handbook on serving LLMs at scale. [HN]
Cactus: Ollama for smartphones — Mobile-first local inference framework. [HN]
FP8 is ~100 tflops faster when the kernel name has “cutlass” in it — CUDA driver optimization quirk discovered and verified. [HN]

2025-W26

Life of an inference request (vLLM V1) — Detailed walkthrough of vLLM V1 internals. [Ubicloud]
LMCache: lossless LLM 3x throughput — Open-source KV-cache layer claiming 3x throughput over vLLM. [HN]
Transformer ASIC at 500k tokens/sec — r/LocalLLaMA discussion of custom transformer silicon. [r/LocalLLaMA]

2025-W25

Compiling LLMs into a MegaKernel — CMU research compiling entire LLMs into a single megakernel for low-latency serving. [HN]
OpenAI o3 price cut + reduced latency — OpenAI slashes o3 pricing to enable daily coding workloads. [OpenAI]

2025-W24

AMD’s AI Future Is Rack Scale ‘Helios’ — AMD’s rack-scale AI accelerator strategy to compete with NVIDIA NVL72. [HN]
Building an AI server on a budget — Practical build guide for home AI inference. [HN]

2025-W23

Cloud Run GPUs now GA — Google Cloud Run adds GPUs, simplifying serverless AI inference. [Google]
Tokasaurus: LLM inference engine for high-throughput workloads — Stanford’s new throughput-optimized inference engine. [Stanford]

2025-W20

KVSplit: 2-3x longer contexts on Apple Silicon — KV-cache splitting technique for M-series chips. [HN]
AutoThink: adaptive reasoning for local LLMs — Adaptive test-time reasoning harness for local models. [HN]

2025-W18

Tiny-LLM on Apple Silicon — Systems engineering course for serving LLMs on M-series chips; 297 HN points. [HN]
Anemll: LLMs on Apple Neural Engine — Running LLMs on Apple’s dedicated ML hardware; 286 HN points. New inference target beyond GPU/CPU. [HN]
GPT-2 on WebGL shaders — GPT-2 running via graphics shaders in browser; 228 HN points. Demonstrates LLMs on unconventional hardware. [HN]

2025-W17

DFloat11: lossless LLM compression via dynamic-length float — Achieves lossless compression of LLM weights with faster inference; 411 HN points. Significant because lossless compression typically doesn’t yield speed gains. [HN/arXiv]

2025-W16

Microsoft CPU-friendly model — BitNet-style hyper-efficient model running on CPUs; 146 HN points. [HN]
Fine-tuning LLMs to 1.58bit — Pushing quantization to near-binary weights; 75 upvotes. [r/LocalLLaMA]
China flash memory 10,000x faster — Hardware breakthrough with AI inference implications; 713 upvotes. [r/LocalLLaMA]

2025-W14

SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators — Apple Research approach to extreme LLM weight compression; 171 HN points. Novel for on-device deployment and model distribution. [Apple ML/HN]
ZClip: Adaptive Spike Mitigation for LLM Pre-Training — Adaptive gradient clipping to prevent training instability spikes; practical technique for large-scale training and fine-tuning. [arXiv via @huggingpapers]

2025-W13

Every Flop Counts: Scaling a 300B LLM Without Premium GPUs — Research on efficient 300B LLM training without top-tier GPU hardware; 117 HN points, relevant for orgs without H100 budget. [HN/arXiv]
Benchmark: RTX 3090/4090/4080 competitive for QwQ-32B inference — Community benchmarks show older consumer GPU generations remain viable for running 32B reasoning models locally. [r/LocalLLaMA]
MLX fork with speculative decoding in server — Community fork adds speculative decoding to the MLX inference server, improving throughput for Apple Silicon. [r/LocalLLaMA]
Accelerating LLM Inference with TGI on Intel Gaudi — HuggingFace guide to running TGI on Intel Gaudi accelerators; expands viable inference hardware options beyond NVIDIA. [Hugging Face Blog]

2025-W12

NVIDIA Dynamo — Datacenter-scale distributed inference: disaggregated prefill/decode, dynamic GPU scheduling, up to 30x throughput on Blackwell. [NVIDIA]
llama.cpp in pure Rust — Rust alternative achieving similar speeds with memory safety. [Reddit]

2025-W11

RTX 5090 llama.cpp benchmarks — 1.58x over RTX 4090, 213 tok/s on 8B models, 1.79TB/s GDDR7 bandwidth. [Phoronix]
Actual electricity consumption for local LLMs — Real-world power measurements from Gemma3 to QwQ. [Reddit]

2025-W10

LLM Inference on Edge via React Native — Hugging Face walkthrough of mobile on-device LLM inference using React Native; practical guide for edge deployment on iOS/Android. [Hugging Face]
DeepSeek claims 545% daily cost-profit ratio — DeepSeek’s reported unit economics highlight the extreme efficiency of their inference stack; raises competitive pressure on API pricing across the industry. [Reuters]

Baseline (through 2025-Q1)

Serving Frameworks

vLLM — Dominant open-source LLM serving engine. PagedAttention for efficient KV-cache management. Supports continuous batching, tensor parallelism, and most model architectures. Used in production by many companies.
- Source: https://github.com/vllm-project/vllm
Ollama — Local LLM runner that simplified running models on consumer hardware. One-command model downloads and inference. Huge adoption for local AI development. Supports GGUF format.
- Source: https://ollama.com/
llama.cpp — Georgi Gerganov’s C/C++ inference engine. Runs LLMs on CPUs and consumer GPUs. The backbone of local inference. GGUF format became the standard for quantized models.
- Source: https://github.com/ggerganov/llama.cpp
SGLang — Fast serving framework with RadixAttention for efficient prefix caching. Strong performance on multi-turn and agentic workloads.
- Source: https://github.com/sgl-project/sglang
TGI (Hugging Face) — Text Generation Inference server. Production-ready serving with Flash Attention, continuous batching, and Hugging Face model hub integration.
- Source: https://github.com/huggingface/text-generation-inference

Quantization

GGUF — Became the standard quantization format for local inference (via llama.cpp). Supports various bit-widths (Q2–Q8). Community regularly quantizes new models within hours of release.
AWQ (Activation-aware Weight Quantization) — 4-bit quantization with minimal quality loss. Widely adopted for GPU serving.
- Source: https://arxiv.org/abs/2306.00978
GPTQ — Early post-training quantization method. 4-bit with good quality. Still widely used.
- Source: https://arxiv.org/abs/2210.17323
FP8 training — DeepSeek-V3 demonstrated FP8 training at scale, reducing memory and compute requirements. NVIDIA H100 native FP8 support made this practical.
1-bit LLMs (BitNet) — Microsoft Research’s BitNet b1.58 showed ternary weight models ({-1, 0, 1}) can match full-precision on some tasks. Potential for massive efficiency gains if hardware adapts.
- Source: https://arxiv.org/abs/2402.17764

Hardware

NVIDIA H100/H200 — Dominated AI training and inference. H200 added HBM3e for larger memory bandwidth. Supply constraints eased through 2024 but still tight.
NVIDIA B200/GB200 (announced Mar 2024, shipping 2025) — Blackwell architecture. 2x training performance over H100. NVLink 5.0. GB200 “superchip” pairs two B200s with a Grace CPU.
- Source: https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing
Google TPU v5p/Trillium — Google’s latest TPUs for training and serving Gemini models. Available via GCP.
Groq LPU — Inference-specific chip (Language Processing Unit). Achieved ~500 tokens/sec on Llama 3 70B. Fastest inference for latency-sensitive applications. Limited model support.
- Source: https://groq.com/
Cerebras — Wafer-scale inference. Demonstrated fast inference speeds. Less flexible than GPU solutions.
- Source: https://cerebras.ai/
Apple Silicon — M-series chips became viable for local inference. MLX framework from Apple optimized for Apple Silicon. Community ran 70B+ models on Mac Studios.
- Source: https://github.com/ml-explore/mlx

Inference Optimization Techniques

Speculative decoding — Use a small draft model to generate candidate tokens, verified in parallel by the large model. 2-3x speedup with no quality loss. Adopted by most serving frameworks.
KV-cache optimization — PagedAttention (vLLM), prefix caching, and multi-query/grouped-query attention reduced memory usage for long contexts.
Mixture of Experts (MoE) — DeepSeek-V3 (671B total, 37B active) and Mixtral showed MoE dramatically reduces inference compute. Only a fraction of parameters are active per token.
Prompt caching — Anthropic (Oct 2024) and OpenAI introduced API-level prompt caching, reducing cost and latency for repeated prefixes by up to 90%.
- Source: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

Cost Trends

API pricing dropped ~10x through 2024-2025. GPT-4-class capabilities went from ~$30/M input tokens (GPT-4, early 2024) to ~$2.50/M (GPT-4o-mini, Claude 3.5 Haiku).
Open model hosting costs also fell. Running a 70B model went from requiring 4×A100s to 1-2× consumer GPUs with quantization.