Top 3 Linux GPU Monitoring Command Line Tools

If you're a tech enthusiast or a Linux user who wants to keep a close eye on their GPU's performance, you've come to the right place. In this post, we'll explore the world of GPU monitoring software, specifically focusing on Ubuntu/Debian Linux. Whether you're a gamer, a data scientist, or a developer working with GPU-intensive tasks, having a reliable GPU monitor is essential for optimizing performance and ensuring your system stays cool under pressure. We'll dive into popular tools such as GPUStat, NVTOP, and NVITOP, discussing their features and how to install them to empower you with comprehensive CPU and GPU monitoring capabilities.

They are a fancy but very useful tool for GPU monitoring, an ncurses based GPU status viewer for NVIDIA GPUs, similar to the htop command or the top command. We can install it as follows using the pip/apt command on a Debian or an Ubuntu Linux. So, let's get started and unravel the power of GPU monitoring on Linux!

Install from PyPI. For more information, please visit GPUStat on Github: https://github.com/wookayin/gpustat

$ pip install gpustat

If you don't have root (sudo) privilege, please try installing gpustat on user namespace:

$ pip install --user gpustat

Commonly used gupstat commands are as follows:

# To periodically watch
$ gpustat --watch

# If something goes wrong, try
$ gpustat --debug

# Monitor gpu
$ gpustat -cp

For more usage information, please consult the usage help.

# See help details
$ gpustat --help
usage: gpustat [-h] [--force-color | --no-color] [--id ID] [-a] [-c] [-f] [-u] [-p] [-F] [-e [{,enc,dec,enc,dec}]] [-P [{,draw,limit,draw,limit,limit,draw}]] [--json] [-i [INTERVAL]] [--no-header] [--gpuname-width GPUNAME_WIDTH]
               [--debug] [--no-processes] [-v]

options:
  -h, --help            show this help message and exit
  --force-color, --color
                        Force to output with colors
  --no-color            Suppress colored output
  --id ID               Target a specific GPU (index).
  -a, --show-all        Display all gpu properties above
  -c, --show-cmd        Display cmd name of running process
  -f, --show-full-cmd   Display full command and cpu stats of running process
  -u, --show-user       Display username of running process
  -p, --show-pid        Display PID of running process
  -F, --show-fan-speed, --show-fan
                        Display GPU fan speed
  -e [{,enc,dec,enc,dec}], --show-codec [{,enc,dec,enc,dec}]
                        Show encoder/decoder utilization
  -P [{,draw,limit,draw,limit,limit,draw}], --show-power [{,draw,limit,draw,limit,limit,draw}]
                        Show GPU power usage or draw (and/or limit)
  --json                Print all the information in JSON format
  -i [INTERVAL], --interval [INTERVAL], --watch [INTERVAL]
                        Use watch mode if given; seconds to wait between updates
  --no-header           Suppress header message
  --gpuname-width GPUNAME_WIDTH
                        The width at which GPU names will be displayed.
  --debug               Allow to print additional informations for debugging.
  --no-processes        Do not display running process information (memory, user, etc.)
  -v, --version         show program's version number and exit

On Ubuntu Impish (21.10), Debian buster (stable) and more recent. For more information, please visit Nvtop on Github: https://github.com/Syllo/nvtop

# install nvtop
$ sudo apt install nvtop

On Ubuntu 20.04, 22.04 and newer

$ sudo add-apt-repository ppa:flexiondotorg/nvtop
$ sudo apt install nvtop

## RUN the tool ##
$ nvtop

For quick command line arguments help.

$ nvtop --help
nvtop version 1.0.0
Available options:
  -d --delay        : Select the refresh rate (1 == 0.1s)
  -v --version      : Print the version and exit
  -s --gpu-select   : Column separated list of GPU IDs to monitor
  -i --gpu-ignore   : Column separated list of GPU IDs to ignore
  -p --no-plot      : Disable bar plot
  -C --no-color     : No colors
  -N --no-cache     : Always query the system for user names and command line information
  -f --freedom-unit : Use fahrenheit
  -E --encode-hide  : Set encode/decode auto hide time in seconds (default 30s, negative = always on screen)
  -h --help         : Print help and exit

Install from PyPI. For more information, please visit Nvitop on Github: https://github.com/XuehaiPan/nvitop

# Install nvitop
$ pip3 install --upgrade nvitop

It is highly recommended to install nvitop in an isolated virtual environment. Simple installation and run via pipx

pipx run nvitop

Install from conda-forge

conda install -c conda-forge nvitop

# Query the status of all devices
$ nvitop -1  # or use `python3 -m nvitop -1`

# Specify query devices (by integer indices)
$ nvitop -1 -o 0 1  # only show <GPU 0> and <GPU 1>

# Run as a resource monitor
$ nvitop

Press h and the help screen will show, or type nvitop --help for more command options: