Forgive me for beginning with a cliché, a chunk of finance jargon that has lately slipped into the tech lexicon, however I’m afraid I need to speak about “moats.” Popularized a long time in the past by Warren Buffett to confer with an organization’s aggressive benefit, the phrase discovered its method into Silicon Valley pitch decks when a memo purportedly leaked from Google, titled “We Have No Moat, and Neither Does OpenAI,” fretted that open-source AI would pillage Large Tech’s fort.
A couple of years on, the fort partitions stay protected. Other than a short bout of panic when DeepSeek first appeared, open-source AI fashions haven’t vastly outperformed proprietary fashions. Nonetheless, not one of the frontier labs—OpenAI, Anthropic, Google—has a moat to talk of.
The corporate that does have a moat is Nvidia. CEO Jensen Huang has known as it his most treasured “treasure.” It’s not, as you would possibly assume for a chip company, a chunk of {hardware}. It’s one thing known as CUDA. What appears like a chemical compound banned by the FDA could be the one true moat in AI.
CUDA technically stands for Compute Unified Gadget Structure, however very like laser or scuba, nobody bothers to increase the acronym; we simply say “KOO-duh.” So what is that this all-important treasure good for? If compelled to provide a one-word reply: parallelization.
Right here’s a easy instance. Let’s say we activity a machine with filling out a 9×9 multiplication desk. Utilizing a pc with a single core, all 81 operations are executed dutifully one after the other. However a GPU with 9 cores can assign duties so that every core takes a unique column—one from 1×1 to 1×9, one other from 2×1 to 2×9, and so forth—for a ninefold pace achieve. Fashionable GPUs may be even cleverer. For instance, if programmed to acknowledge commutativity—7×9 = 9×7—they’ll keep away from duplicate work, decreasing 81 operations to 45, practically halving the workload. When a single coaching run prices 100 million {dollars}, each optimization counts.
Nvidia’s GPUs have been initially constructed to render graphics for video video games. Within the early 2000s, a Stanford PhD pupil named Ian Buck, who first bought into GPUs as a gamer, realized their structure might be repurposed for common high-performance computing. He created a programming language known as Brook, was employed by Nvidia, and, with John Nickolls, led the event of CUDA. If AI ushers within the age of a everlasting white-collar underclass and autonomous weapons, simply know that it could all be as a result of somebody someplace taking part in Doom thought a demon’s scrotum ought to jiggle at 60 frames per second.
CUDA just isn’t a programming language in itself however a “platform.” I take advantage of that weasel phrase as a result of, not in contrast to how The New York Instances is a newspaper that’s additionally a gaming firm, CUDA has, through the years, develop into a nested bundle of software program libraries for AI. Every perform shaves nanoseconds off single mathematical operations—added up, they make GPUs, in trade parlance, go brrr.
A contemporary graphics card is not only a circuit board full of chips and reminiscence and followers. It’s an elaborate confection of cache hierarchies and specialised items known as “tensor cores” and “streaming multiprocessors.” In that sense, what chip corporations promote is sort of a skilled kitchen, and extra cores are akin to extra grilling stations. However even a kitchen with 30 grilling stations gained’t run any quicker and not using a succesful head chef deftly assigning duties—as CUDA does for GPU cores.
To increase the metaphor, hand-tuned CUDA libraries optimized for one matrix operation are the equal of kitchen instruments designed for a single job and nothing extra—a cherry pitter, a shrimp deveiner—that are indulgences for dwelling cooks however not when you’ve got 10,000 shrimp guts to yank out. Which brings us again to DeepSeek. Its engineers went beneath this already deep layer of abstraction to work instantly in PTX, a type of meeting language for Nvidia GPUs. Let’s say the duty is peeling garlic. An unoptimized GPU would go: “Peel the pores and skin along with your fingernails.” CUDA can instruct: “Smash the clove with the flat of a knife.” PTX permits you to dictate each sub-instruction: “Raise the blade 2.35 inches above the slicing board, make it parallel to the clove’s equator, and strike downward along with your palm at a pressure of 36.2 newtons.”
