NCI’s peak supercomputer is Gadi, Australia's most powerful CPU-based research supercomputer.

A man holds open a door of the Gadi supercomputer while working on it.

A 4,962 node supercomputer comprising Intel Sapphire Rapids, Cascade Lake, Skylake and Broadwell CPUs and NVIDIA V100 and DGX A100 GPUs, Gadi supports diverse workloads with well over 10 petaflops of peak performance.

    Technical Specifications

    Gadi contains more than 250,000 CPU cores, 930 Terabytes of memory and 640 GPUs.

    • 3,074 nodes each containing two 24-core Intel Xeon Scalable ‘Cascade Lake’ processors and 192 Gigabytes of memory
      • including 50 nodes each offering 1.5 Terabytes of Intel Optane DC Persistent memory
    • 720 nodes each with two 52-core Intel Xeon Scalable 'Sapphire Rapids' processors and 512 Gigabytes of memory
    • 804 nodes each with two 14-core Intel 'Broadwell' processors
    • 192 nodes each with two 16-core Intel 'Skylake' processors
    • 160 nodes each containing four Nvidia V100 GPUs and two 24-core Intel Xeon Scalable 'Cascade Lake' processors.
    • 10 nodes each with two 14-core Intel 'Broadwell' processors and 512 Gigabytes of memory
    • 2 nodes of the NVIDIA DGX A100 system, with 8 A100 GPUs per node.
    • Linking the storage systems and Gadi is Mellanox Technologies' latest generation HDR InfiniBand technology in a Dragonfly+ topology, capable of transferring data at up to 200 Gb/s.
    • The storage sub-systems are NetApp enterprise class storage arrays, linked together in a DDN Lustre parallel file system.
    • Altair’s PBSPro software optimises job scheduling and workload management.
    • Gadi has more than 200 supported software packages.
    • Gadi uses the Rocky Linux 8 operating system.
    Tall panels decorated with blue Aboriginal artwork separated by sliding glass doors, seen from below.

    Gadi offers a 7 times increase in computational performance, and a significantly increased number of GPUs compared to its predecessor, the Raijin supercomputer. This provides significant benefits for the largest parallel codes in particular, and all users benefit from shorter queues and higher throughput. 

    Gadi first launched to NCI users in mid-November 2019. User documentation about the system can be found in our User Guides.

    The Gadi installation was documented on our dedicated blog. Read through to see each step of the progress.

    NCI’s previous supercomputer was a hybrid Fujitsu Primergy-Lenovo NeXtScale system called Raijin. Named after the Shinto god of thunder, lightning and storms, Raijin was a high-performance, distributed memory cluster with a peak performance of over 2 petaflops.

    Image of HPC system

    Raijin was originally installed in 2012 and entered full production in June 2013. Significant updates to the system entered production in November 2016 and again in mid-2017. 

    Raijin evolved from a relatively straightforward x86-based cluster to a heterogeneous one made up of many different systems. Over its operational life, Raijin was augmented with a wide variety of x86 and non-x86 based systems, including GPUs, many-core Xeon Phi processors and Power8 processors.

    The entire Raijin cluster ran the same operating system, an optimised version of a highly-customised, NCI-developed Linux kernel. 

    Raijin had:

    • 89,256 cores in 4,500 Intel Xeon Sandy Bridge, Broadwell and Skylake nodes
    • 128 GPUs in 32 NVIDIA Tesla K80 and P100 nodes
    • 32 64-core Intel Xeon Phi processors in 32 nodes
    • 64 cores in 4 IBM Power8 nodes 
    • 300 terabytes of memory
    • 8 petabytes of operational disk storage
    • Hybrid FDR/EDR Mellanox Infiniband full fat tree interconnect (up to 100 Gb/sec)