AI Infra Summit 2026 · Santa Clara
Where Is Your AI Training Time Actually Going?
NCCL collective profiling for distributed AI training, with no code changes and no recompilation. Thanks for visiting us at AI Infra Summit 2026.
+42%
More NCCL communication throughput on a multi-node 8-GPU cluster after tuning NCCL settings with Linaro MAP, with zero code changes.
The Problem: GPUs Idle While Communication Limits Throughput
As distributed training scales across larger GPU clusters, more wall-clock time is spent not in compute but in GPU-to-GPU communication. NCCL, the communication backbone of distributed training on NVIDIA GPUs, handles gradient synchronization and data redistribution through collectives such as AllReduce, AllGather and ReduceScatter across NVLink, NVSwitch and InfiniBand.
Scalability Challenges
Standard debuggers and profilers struggle with thousands of concurrent GPU threads and multi-node collective communication failures.
Hidden Choices
NCCL auto-selects algorithms and protocols that are not always optimal for your topology, leaving teams without a direct way to verify runtime settings or pinpoint bottlenecks.
Costly GPU Idle Time
Unoptimized GPU communication burns expensive compute cycles while ranks wait on collective barriers. On clusters costing six or seven figures a month, that invisibility is costly.
The Solution: Complete NCCL Visibility with Linaro MAP
Linaro Forge makes GPU-to-GPU communication observable across Python and C++/CUDA ML workloads. Linaro MAP collects dedicated NCCL metrics on every rank (time in NCCL operations, transfer rates in GB/s, and collective versus point-to-point rates) on one timeline alongside CPU, GPU kernel, power and I/O metrics.
Easy Setup
Prepend a single command to your job launch script. No recompilation required.
Full Cluster Context
View communication behavior across all ranks and nodes on one timeline to spot imbalances instantly.
Kernel-Level Verification
Confirm exact NCCL kernel variants to verify whether system overrides were honored.
Energy and Performance Alignment
Overlay GPU power draw against collective activity to optimize both throughput and energy expenditure.
One-Page Performance Summaries
Performance Reports generates clean before-and-after comparisons for quick stakeholder review.
Energy Savings
Monitor and optimize power draw against collective activity to reduce energy expenditure.
Case Study: Profiling NCCL AllReduce on a Multi-Node 8-GPU Cluster
Profiling NVIDIA’s nccl-tests AllReduce benchmark showed that NCCL’s default auto-configuration left substantial performance untapped. Tuning algorithm, protocol and transport settings, guided by Linaro MAP profiles, produced immediate gains.
+42%
Communication throughput
-33%
GPU kernel overhead
Zero
Code changes needed
-30%
Energy consumption
Performance metric
NCCL defaults
Tuned with Linaro MAP
Impact
AllReduce op duration
18.7 ms
10.9 ms
42% faster
NCCL send/receive rate
35.9 GB/s
50.8 GB/s
+42% throughput
Benchmark run time
18.1 s
12.8 s
41% shorter
Cumulative GPU kernel time
2.1 min
1.4 min
33% less GPU time
Energy consumption
7.09 Wh
4.95 Wh
30% less energy
For the full methodology and results, read Introducing Linaro Forge 26.0.
Enterprise AI Infrastructure Support
- Python and native profiling: profile Python-driven pipelines and C++/CUDA extensions side by side.
- Multi-vendor accelerators: observability across NVIDIA CUDA, AMD ROCm and Intel Xe-HPC platforms.
- Orchestration and schedulers: workflow support for the Slurm, Flux and PALS job managers.
- Enterprise environments: support for Amazon Linux 2023, RHEL 8+, SLES 15 and Ubuntu 22.04+.
- Scalability: the tools scale from a single process to thousands of nodes on large clusters.
Book a Live Demo on Your Workload
See NCCL collective profiling on your own cluster and training jobs. Email sales@forge.linaro.com to arrange a session.