GPU Detective Stories: PyTorch Recompiles & Hidden Bottlenecks

AI Performance Engineering Meetup (San Francisco, Global) Monday, November 16, 2026 · 9:00–10:00 AM PST · Online Zoom link: https://us02web.zoom.us/j/82308186562 Join us for two deep dives into hidden AI performance costs: surprise PyTorch recompilations during inference and compute, memory, and communication bottlenecks in Jev training. Talk #0: Introductions and Meetup Updates by Chris Fregly and Antje Barth Talk #1: The Recompile Ambush: Taming PyTorch Inference Spikes by Chaim Rand `torch.compile` can accelerate steady-state inference, yet a guard failure on a new input shape or layout can trigger an expensive recompilation in the request path. Drawing on his latest post, Chaim will compare six approaches using a dynamic graph-neural-network example: cache preloading, bounded recompilation with eager fallback, Dynamo and input-layout tuning, warmup seeding, background compilation, and ahead-of-time compilation with `torch.export` and AOTInductor. He’ll explain their tradeoffs and how the right choice depends on the model, input distribution, runtime environment, and latency requirements. Bio: Chaim Rand is a machine-learning algorithm developer who writes about PyTorch and practical AI/ML performance optimization. Based on: Overcoming the PyTorch Recompilation Dilemma in Real-Time Inference Workloads (https://chaimrand.medium.com/overcoming-the-pytorch-recompilation-dilemma-in-real-time-inference-workloads-069fac5485cd) Talk #2: Follow the Profile: Hunting Jev’s GPU Bottlenecks by Israel Ogbole, Co-founder and CEO of Zymtrace Israel will explore the latest Zymtrace/TypeSafe case study on training and serving Jev. He’ll discuss how continuous CPU/GPU profiling and MCP-accessible profile data supported an iterative optimization loop across compute, memory, communication, and inference synchronization. The published case study reports approximately 46% higher training throughput on the same hardware after three successive changes, plus approximately 10% better inference performance after removing a hidden device-to-host synchronization. The talk will focus on how the team found each bottleneck, measured each change, and identified the next constraint. Bio: Israel Ogbole is co-founder and CEO of Zymtrace, which develops continuous CPU/GPU profiling tools for performance optimization. Based on: How TypeSafe achieved 46% higher GPU throughput training Jev — with the help of Zymtrace (https://zymtrace.com/article/typesafe-zymtrace/) Zoom link: https://us02web.zoom.us/j/82308186562 Related Links Github Repo: http://github.com/cfregly/ai-performance-engineering/ O’Reilly Book: https://www.amazon.com/Systems-Performance-Engineering-Optimizing-Algorithms/dp/B0F47689K8/ YouTube: https://www.youtube.com/@AIPerformanceEngineering Generative AI Free Course on DeepLearning.ai: https://bit.ly/gllm