Infrastructure · AI

How to Install CUDA: Drivers, Toolkit, cuDNN, and Avoiding Version Hell

CUDA is NVIDIA’s parallel computing platform, and it’s the foundation that everything GPU-accelerated sits on — PyTorch, TensorFlow, scientific simulation, and any code that runs on the GPU. Installing it isn’t conceptually hard, but it has a reputation for eating afternoons, and the reason is almost always version conflicts between the driver, the toolkit, and cuDNN. This guide walks a clean Linux installation that avoids those traps: checking compatibility first, using the right method, setting the environment correctly, and verifying that it actually works before you build on top of it.

What needs to be installed, and in what order?

The single most important thing to understand before touching a terminal is that CUDA is not one thing but a stack of components that must be installed in the right order, because installing them out of sequence is the root cause of most of the conflicts people hit. There are four layers: the NVIDIA driver (which lets the OS talk to the GPU), the CUDA Toolkit (the compiler and libraries you build against), cuDNN (a library of deep-learning primitives that frameworks need), and finally your framework like PyTorch or TensorFlow. The diagram shows the order.

The disciplined approach has five phases: plan (check your GPU’s compute capability, your OS and kernel, and what versions your framework needs, then pick a compatible set), clean (remove any old NVIDIA and CUDA packages so they don’t conflict), install in the order above, verify each layer works before moving up, and document the exact versions that ended up working so you can reproduce them. That planning step feels like overhead, but it’s what separates a ten-minute install from a two-day debugging session.

Why does nvidia-smi matter before you start?

Before installing the toolkit, you run nvidia-smi, and reading its output correctly prevents the most common compatibility mistake. The command shows your GPU and, in the top-right corner, a “CUDA Version” figure — and the crucial point is that this number is the highest CUDA Toolkit version your currently installed driver can support, not the version that’s actually installed. People routinely misread it as “CUDA is already installed,” when it’s really telling you the ceiling.

The practical consequence is a simple rule: if that reported version is lower than the toolkit you want to install, you update the NVIDIA driver first, because the driver must be new enough to support your target toolkit. Drivers are backward-compatible, so a newer driver happily runs older CUDA versions, which means erring toward a recent driver is safe. If nvidia-smi doesn’t run at all, the driver isn’t installed or loaded, and that’s your first job. Two pre-flight checks round this out: confirm the GPU is present with lspci, and check Secure Boot status — if Secure Boot is enabled, you’ll need to either sign the NVIDIA kernel module or disable it in UEFI, which is why most machines dedicated to GPU work have it disabled.

Cleaning out old installations

One step that quietly prevents a large share of installation failures is removing any old NVIDIA and CUDA packages before installing fresh ones, because leftover components from a previous attempt are a frequent source of mysterious conflicts. If you’ve ever installed a driver or toolkit on this machine, even a failed attempt, those packages can clash with the new versions in ways that produce confusing errors later. A clean slate is cheap insurance: purge the existing NVIDIA, CUDA, and cuDNN packages, run an autoremove to clear orphaned dependencies, remove any DKMS modules tied to old drivers, and reboot so nothing stale is still loaded in memory.

Two prerequisites belong in this same preparation. First, the CUDA compiler nvcc needs a compatible host C++ compiler installed — GCC on Linux, with a minimum version that’s risen over the toolkit’s history — so confirm GCC is present and reasonably current, since a missing or too-old compiler stops compilation cold. Second, revisit Secure Boot if you haven’t: with it enabled, the NVIDIA kernel module must be signed or the driver won’t load, which is why disabling Secure Boot in UEFI is standard on dedicated GPU machines. Handling these before you install means the actual installation runs without the surprises that come from a dirty or under-prepared system.

Choosing an installation method

NVIDIA offers several ways to install CUDA, and choosing the right one saves trouble later. The table compares the main options.

For most people on a server or workstation, the package-manager method is the right choice and NVIDIA’s own recommendation, because it integrates with your distribution’s native package system and makes future updates clean. The runfile is a distribution-independent standalone installer that gives more control and lets you target a specific older version, but it requires you to set up environment variables manually and doesn’t update your package manager — reach for it when the package method causes problems or you need a version it doesn’t offer. Conda and pip wheels suit Python-centric workflows where you want CUDA scoped to a particular environment rather than installed system-wide. The package-manager path is what the rest of this guide follows.

Installing the toolkit with apt

On a Debian-based system like Ubuntu, the installation itself is a short sequence once you’ve done the compatibility checks. You add NVIDIA’s repository by downloading and installing the cuda-keyring package, update your package lists, and then install the toolkit — and here is the single most important detail of the whole install: pin the minor version. The terminal shows the sequence.

# Pre-flight: GPU present? Secure Boot off? Old packages gone?
$ lspci | grep -i nvidia
$ mokutil —sb-state                 # want: SecureBoot disabled
$ sudo apt purge ‘^nvidia-.’ ‘^cuda-.’ ‘libcudnn*’ ; sudo apt autoremove
# Add NVIDIA’s apt repo (Ubuntu 24.04 example)
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
$ sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt-get update
# Install a PINNED toolkit — never bare ‘cuda’ (pulls newest major)
$ sudo apt-get install -y cuda-toolkit-12-6
# cuDNN, matched to the CUDA major version
$ sudo apt-get install -y cudnn9-cuda-12

Why pin to cuda-toolkit-12-6 rather than the bare cuda package? Because the unpinned package always pulls the newest major version available, which can silently break a workflow built against an older one — pinning the minor version gives you predictable, reproducible installs. The apt method installs to a versioned directory like /usr/local/cuda-12.6 with a /usr/local/cuda symlink pointing at it, which is what lets you keep multiple versions side by side later. Installing cuDNN, the deep-learning library that frameworks depend on, is a single matching package once the repository is configured — just make sure its version corresponds to your CUDA major version.

How do you set the environment variables?

After the packages install, CUDA still won’t work until you tell your shell where to find it, and missing this step is behind a huge fraction of “I installed CUDA but it doesn’t work” reports. Two environment variables matter: PATH must include CUDA’s bin directory so the system finds the nvcc compiler, and LD_LIBRARY_PATH must include its lib64 directory so programs find the CUDA shared libraries at runtime. You add both to your shell profile — typically ~/.bashrc — pointing at the /usr/local/cuda symlink so they survive version switches, then reload the profile.

There’s a subtlety worth knowing because it causes confusing failures: a ~/.bashrc only loads for interactive shells. If your IDE, a build system, or a service runs CUDA from a non-interactive shell, it won’t see those variables, and you’ll get nvcc-not-found errors even though it works fine in your terminal. The fix is to set the variables globally in /etc/environment as well, so every process inherits them. Many setups also define CUDA_HOME pointing at the same location, since some build tools look for it specifically. Getting these paths right is the difference between a working install and one that appears broken despite the packages being present.

Installing cuDNN for deep learning

If your goal is deep learning rather than general GPU computing, there’s one more essential layer: cuDNN, NVIDIA’s library of optimised primitives for deep neural networks. Frameworks like PyTorch and TensorFlow depend on it for the accelerated convolutions and other operations that make GPU training fast, so without it your framework may run but won’t perform the way it should. cuDNN sits on top of the CUDA Toolkit, which is exactly why the install order puts it after CUDA — it needs the toolkit already in place to build against.

The installation is straightforward once your CUDA repository is configured, since cuDNN is available as a single matching package — you install the cuDNN version that corresponds to your CUDA major version, and the apt repository you already added handles the rest. The one rule that bites people is version matching: a cuDNN built for a different CUDA major version will fail in confusing ways, so the version numbers must line up. Historically cuDNN required a free NVIDIA developer account to download manually, but the package-manager route through the configured repository avoids that friction entirely. If you want to confirm it works, cuDNN ships sample programs — building and running the bundled MNIST sample should end with a “Test passed!” message, which verifies the deep-learning layer end to end before you ever load a framework.

Verifying the installation

Never assume CUDA works just because the packages installed — you verify it explicitly in two or three steps. First, nvidia-smi should show your GPU and driver, confirming the driver loaded. Second, nvcc —version should print the compiler version, confirming the toolkit is installed and your PATH is correct. If nvidia-smi works but nvcc is “not found,” that’s a PATH problem, not a failed install — the toolkit is there, your shell just can’t find it.

The thorough final check is to compile and run a sample program. Since CUDA 11.6, NVIDIA no longer bundles the samples with the toolkit, so you clone them from GitHub on a branch matching your version, build the deviceQuery utility, and run it. A successful run prints detailed information about your GPU — compute capability, memory, core counts — and ends with “Result = PASS,” which means CUDA is genuinely working end to end, not just installed. This compile-and-run test catches problems that the version commands miss, because it actually exercises the compiler and runtime together. With that passing, you’re ready to install a framework like PyTorch, which is the foundation for work like fine-tuning a model, and to choose the right hardware to run it on as our GPU for AI guide describes — and for a controllable, properly provisioned GPU host, our dedicated servers in Toronto give CUDA a stable home, while careful version matching is what keeps the whole stack from fighting itself.

What goes wrong, and how do you fix it?

Even a careful install occasionally misbehaves, and the good news is that the failures cluster into a few recognisable patterns with reliable fixes. The most common is nvidia-smi working while nvcc is “not found” — as covered, this is a PATH issue, not a broken install; the toolkit is present and your shell simply can’t locate the compiler, so you fix your PATH or run a find to locate nvcc and point at it. The reassuring thing is that this looks alarming but is trivial to resolve.

The other patterns are equally tractable. If nvidia-smi itself fails, the driver didn’t load — you check whether the kernel modules are present with lsmod, look for errors in the kernel log with dmesg, and if you’re using DKMS, rebuild the module so it matches your current kernel, which commonly breaks after a kernel update. If you hit library-mismatch errors at runtime, your LD_LIBRARY_PATH is pointing at the wrong CUDA version or the library cache is stale — running ldconfig and confirming the paths usually clears it. The structured way to debug any of these is a quick checklist: does nvidia-smi show the GPU, does nvcc report a version, are the CUDA and cuDNN versions compatible, are the environment variables set, and is Secure Boot disabled. Walking that list in order isolates almost any problem to a single layer, which is exactly why the disciplined install order pays off — when something breaks, you know which layer to look at, and for isolating versions entirely across projects, containerising with the NVIDIA Container Toolkit (the subject of our Docker guide) sidesteps host-level conflicts altogether.