Close Menu
    Facebook LinkedIn YouTube WhatsApp X (Twitter) Pinterest
    Trending
    • These Were My Favorite Things Samsung Unpacked During Its 2026 Galaxy Event
    • AI minister role boosted but tech department axed in Burnham shake-up
    • Loop Engineering for RAG Question Parsing: The Small Loop That Runs Before Retrieval
    • The risk of weather data sabotage is rising
    • Hand-E Now Reaches 100 mm Without Giving Up an Ounce of Precision
    • Weight loss drug effectiveness and long term maintenance
    • Here’s what Albo’s ‘Office of AI’ means for Australian tech
    • YouTube and X Have Become ‘Gateways’ to Nudify Apps
    Facebook LinkedIn WhatsApp
    Times FeaturedTimes Featured
    Thursday, July 23
    • Home
    • Founders
    • Startups
    • Technology
    • Profiles
    • Entrepreneurs
    • Leaders
    • Students
    • VC Funds
    • More
      • AI
      • Robotics
      • Industries
      • Global
    Times FeaturedTimes Featured
    Home»Artificial Intelligence»AI in Multiple GPUs: How GPUs Communicate
    Artificial Intelligence

    AI in Multiple GPUs: How GPUs Communicate

    Editor Times FeaturedBy Editor Times FeaturedFebruary 20, 2026No Comments5 Mins Read
    Facebook Twitter Pinterest Telegram LinkedIn Tumblr WhatsApp Email
    Share
    Facebook Twitter LinkedIn Pinterest Telegram Email WhatsApp Copy Link


    is a part of a sequence about distributed AI throughout a number of GPUs:

    Introduction

    Earlier than diving into superior parallelism strategies, we have to perceive the important thing applied sciences that allow GPUs to speak with one another.

    However why do GPUs want to speak within the first place? When coaching AI fashions throughout a number of GPUs, every GPU processes totally different knowledge batches however all of them want to remain synchronized by sharing gradients throughout backpropagation or exchanging mannequin weights. The specifics of what will get communicated and when relies on your parallelism technique, which we’ll discover in depth within the subsequent weblog posts. For now, simply know that trendy AI coaching is communication-intensive, making environment friendly GPU-to-GPU knowledge switch crucial for efficiency.

    The Communication Stack

    PCIe

    PCIe (Peripheral Component Interconnect Express) connects expansion cards like GPUs to the motherboard using independent point-to-point serial lanes. Here’s what different PCIe generations offer for a GPU using 16 lanes:

    • Gen4 x16: ~32 GB/s bidirectional
    • Gen5 x16: ~64 GB/s bidirectional
    • Gen6 x16: ~128 GB/s bidirectional (FYI 16 lanes × 8 GB/s/lane = 128 GB/s)

    High-end server CPUs typically offer 128 PCIe lanes, and modern GPUs need 16 lanes for optimal bandwidth. This is why you usually see 8 GPUs per server (128 = 16 x 8). Power consumption and physical space in server chassis also make it impractical to go beyond 8 GPUs in a single node.

    NVLink

    NVLink enables direct GPU-to-GPU communication within the same server (node), bypassing the CPU entirely. This NVIDIA-proprietary interconnect creates a direct memory-to-memory pathway between GPUs with huge bandwidth:

    • NVLink 3 (A100): ~600 GB/s per GPU
    • NVLink 4 (H100): ~900 GB/s per GPU
    • NVLink 5 (Blackwell): Up to 1.8 TB/s per GPU
    Source: GitHub (MIT license)

    Be aware: on NVLink for CPU-GPU communication

    Sure CPU architectures assist NVLink as a PCIe alternative, dramatically accelerating CPU-GPU communication by overcoming the PCIe bottleneck in knowledge transfers, comparable to shifting coaching batches from CPU to GPU. This CPU-GPU NVLink functionality makes CPU-offloading (a way that saves VRAM by storing knowledge in RAM as a substitute) sensible for real-world AI functions. Since scaling RAM is often more cost effective than scaling VRAM, this strategy presents important financial benefits.

    CPUs with NVLink assist embrace IBM POWER8, POWER9, and NVIDIA Grace.

    Nevertheless, there’s a catch. In a server with 8x H100s, every GPU wants to speak with 7 others, splitting that 900 GB/s into seven point-to-point connections of about 128 GB/s every. That’s the place NVSwitch is available in.

    NVSwitch

    NVSwitch acts as a central hub for GPU communication, dynamically routing (switching if you will) data between GPUs as needed. With NVSwitch, every Hopper GPU can communicate at 900 GB/s with all other Hopper GPUs simultaneously, i.e. peak bandwidth doesn’t depend on how many GPUs are communicating. This is what makes NVSwitch “non-blocking”. Each GPU connects to several NVSwitch chips via multiple NVLink connections, ensuring maximum bandwidth.

    While NVSwitch started as an intra-node solution, it’s been extended to interconnect multiple nodes, creating GPU clusters that support up to 256 GPUs with all-to-all communication at near-local NVLink speeds.

    The generations of NVSwitch are:

    • First-Generation: Supports up to 16 GPUs per server (compatible with Tesla V100)
    • Second-Generation: Also supports up to 16 GPUs with improved bandwidth and lower latency
    • Third-Generation: Designed for H100 GPUs, supports up to 256 GPUs

    InfiniBand

    InfiniBand handles inter-node communication. While much slower (and cheaper) than NVSwitch, it’s commonly used in datacenters to scale to thousands of GPUs. Modern InfiniBand supports NVIDIA GPUDirect® RDMA (Remote Direct Memory Access), letting network adapters access GPU memory directly without CPU involvement (no expensive copying to host RAM).

    Current InfiniBand speeds include:

    • HDR: ~25 GB/s per port
    • NDR: ~50 GB/s per port
    • NDR200: ~100 GB/s per port

    These speeds are significantly slower than intra-node NVLink due to network protocol overhead and the need for two PCIe traversals (one at the sender and one at the receiver).

    Key Design Principles

    Understanding Linear Scaling

    Linear scaling is the holy grail of distributed computing. In simple terms, it means doubling your GPUs should double your throughput and halve your training time. This happens when communication overhead is minimal compared to computation time, allowing each GPU to operate at full capacity. However, perfect linear scaling is rare in AI workloads because communication requirements grow with the number of devices, and it’s usually impossible to achieve perfect compute-communication overlap (explained next).

    The Importance of Compute-Communication Overlap

    When a GPU sits idle waiting for data to be transferred before it can be processed, you’re wasting resources. Communication operations should overlap with computation as much as possible. When that’s not possible, we call that communication an “exposed operation”.

    Intra-Node vs. Inter-Node: The Performance Cliff

    Modern server-grade motherboards support up to 8 GPUs. Within this range, you can often achieve near-linear scaling thanks to high-bandwidth, low-latency intra-node communication.

    Once you scale beyond 8 GPUs and start using multiple nodes connected via InfiniBand, you’ll see a large performance degradation. Inter-node communication is much slower than intra-node NVLink, introducing network protocol overhead, higher latency, and bandwidth limitations. As you add more GPUs, each GPU must coordinate with more peers, spending more time idle waiting for data transfers to complete.

    Conclusion

    Follow me on X for more free AI content @l_cesconetto

    Congratulations on making it to the top! On this put up you realized about:

    • The CPU-GPU and GPU-GPU communication fundamentals:
      • PCIe, NVLink, NVSwitch, and InfiniBand
    • Key design rules for distributed GPU computing
    • You’re now capable of make rather more knowledgeable selections when designing your AI workloads

    Within the subsequent weblog put up, we’ll dive into our first parallelism method, the Distributed Information Parallelism.

    1. NVIDIA Blog
    2. GPU Direct



    Source link

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Editor Times Featured
    • Website

    Related Posts

    Loop Engineering for RAG Question Parsing: The Small Loop That Runs Before Retrieval

    July 19, 2026

    How to Find the Optimal Coding Agent Interface

    July 9, 2026

    I Completed Five Years in Analytics Consulting: 5 Lessons That Changed How I Work

    June 29, 2026

    GPU-Resident Top-K for Agentic RAG: I Built a CUDA Kernel So My Retrieval Step Would Stop Bouncing Off the GPU

    June 19, 2026

    Can Machine Learning Predict the World Cup?

    June 9, 2026

    Automate Writing Your LLM Prompts

    June 5, 2026

    Comments are closed.

    Editors Picks

    These Were My Favorite Things Samsung Unpacked During Its 2026 Galaxy Event

    July 22, 2026

    AI minister role boosted but tech department axed in Burnham shake-up

    July 21, 2026

    Loop Engineering for RAG Question Parsing: The Small Loop That Runs Before Retrieval

    July 19, 2026

    The risk of weather data sabotage is rising

    July 18, 2026
    Categories
    • Founders
    • Startups
    • Technology
    • Profiles
    • Entrepreneurs
    • Leaders
    • Students
    • VC Funds
    About Us
    About Us

    Welcome to Times Featured, an AI-driven entrepreneurship growth engine that is transforming the future of work, bridging the digital divide and encouraging younger community inclusion in the 4th Industrial Revolution, and nurturing new market leaders.

    Empowering the growth of profiles, leaders, entrepreneurs businesses, and startups on international landscape.

    Asia-Middle East-Europe-North America-Australia-Africa

    Facebook LinkedIn WhatsApp
    Featured Picks

    New hope for drug-resistant epilepsy with RAP-219

    September 12, 2025

    21 Best Early Labor Day Sales on WIRED-Tested Gear (2025)

    August 21, 2025

    Bandcamp bans purely AI-generated music from its platform

    January 18, 2026
    Categories
    • Founders
    • Startups
    • Technology
    • Profiles
    • Entrepreneurs
    • Leaders
    • Students
    • VC Funds
    Copyright © 2024 Timesfeatured.com IP Limited. All Rights.
    • Privacy Policy
    • Disclaimer
    • Terms and Conditions
    • About us
    • Contact us

    Type above and press Enter to search. Press Esc to cancel.