LLM Inference Systems Engineer
I build the systems that serve large language models: GPU kernels, KV-cache stores, prefill/decode disaggregation, cache-aware routing, and the Kubernetes underneath.
I led serving for a frontier-scale MoE model (GLM-5.3) on multi-node B300 clusters, re-architecting it to pass a 200k-token SLA on 5 nodes instead of 11 with 65% more throughput, and I upstream the fixes to vLLM and LMCache.
Senior Software Engineer, MLOps · Multiverse Computing · San Sebastián, Spain · Remote, EU/UK hours
Same 200k-token SLA, 65% more throughput
Lower KV-cache p99 latency (84 s → 0.67 s)
KV-cache hit rate with cache-aware routing
Upstream to vLLM, LMCache, production-stack
How should KV cache move through a cluster? Disaggregated prefill/decode, tiered offload and cache-aware routing for long-context serving.
When does parallelism pay for itself? Context, expert and data parallelism for MoE models, measured against real SLAs rather than peak throughput.
How do we keep inference systems correct? Silent KV corruption, races and stalls that only appear at scale, and how to catch them before users do.
Re-architected GLM-5.3 FP8 serving on 8-GPU B300 nodes as P/D disaggregation with context-parallel prefill, KV transfer over EFA RDMA (vLLM NIXL), and a cache- and load-aware P/D router with failover and circuit breaking.
12.7M tokens/min on 5 nodes vs 7.7M on 11 · decode 25.8 → 125–139 tok/s per user
6 nodes freed. 4 prefill nodes (context + expert parallel) and 1 decode node; the KV cache moves between them over EFA RDMA with vLLM NIXL.
Per-user decode speed on a 200k-token prompt, in real time
Designed a KV store sharded across prefill context-parallel ranks, with a correctness proof for its prefix lookup and asynchronous per-rank loading. Profiled all 8 GPUs with Nsight Systems and applied a FlashInfer FP8 kernel fix for B300.
+34% prefill throughput · FP8 kernel 1.57 ms → 0.23 ms per call
Replayed a production pod's exact request stream on an isolated node. A cost model and queueing analysis showed a single-threaded Valkey L2 capped at 0.9 GB/s; scaled it to one instance per GPU rank and added asynchronous loading.
p99 84 s → 0.67 s (125×) · 6.3× cache bandwidth · no production impact
Built routing that sends requests to the replica holding their prefix, with safe fallbacks, and fixed burst herding and in-flight leaks in the router's load signal. Enabled cross-node KV sharing over EFA at 154–160 GB/s.
Hit rate 79–89% → 99% · warm P50 9.8 s → 1.3 s · 188k-token prompt in 1.35 s
Tuned single-model serving with CUDA graphs, speculative decoding, FlashAttention and FP8 quantization.
7.5× throughput (64 → 480 tok/s) · latency 5.49 s → 0.68 s
Production work for an enterprise customer at Multiverse Computing. Full detail in the CV.
A. Tomut, S. S. Jahromi, A. Sarkar, … M. Ibrahim, … R. Orús
arXiv:2401.14109, 2024
Led the 6-person engineering team building the 70%-compressed LLM, adopted by 3 enterprise clients for edge deployment.
Fixes found in production, upstreamed with tests to vLLM, LMCache and vLLM production-stack.
Agentic AI with compressed LLMs, hands-on workshop
AWS, Madrid · Compressed LLM driving a tool-calling agent; whole environment as code. Rated 4.6/5.
Subject Matter Expert, AWS Certified Generative AI Developer exam
AWS, Arlington VA · Jul 2025 · Designed, reviewed and validated exam content.
I'm interested in roles and collaborations on LLM inference, serving systems and efficient architectures. The fastest way to reach me is email.