#+HashtagPLUS
In

Menu

RE

Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)

Author Profile

Claim This Author Profile

Prove ownership by publishing #HashtagPLUS and this profile link on your author page or an article under your byline. A moderator or admin will review the request before it merges into your real HashtagPLUS username.

Sign In To Claim
0 karma0 postsjoined 5 days ago

Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)

🌐 blog.kog.aiSource

Today, Kog AI launches a tech preview of the Kog Inference Engine (KIE): 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). This preview runs a 2B model, with support for large third-party MoE models coming next at

#blog#speed#model#inference#memory#tokens#article#englishlanguage
5 days ago