NCI is the home of Australia’s fastest supercomputerGadi. Supported by the Australian Government through its National Collaborative Research Infrastructure Strategy, Gadi represents a significant upgrade in computing power and performance over its predecessor, Raijin. Scroll down the page to see exactly how the construction of the Gadi supercomputer progressed, from floor work, to water pipes, electrical work and finally hardware delivery and installation.

Gadi appears in 24th on the Top500 ranking - 23 June 2020

A little over six months since users first logged in to Gadi, the new supercomputer has now appeared high up on the Top500 list released overnight at the ISC High Performance Digital Conference. Placing 24th with a peak performance of 9.26 Petaflops, Gadi is 8 times as powerful as Raijin, the supercomputer it replaces. We are thrilled at the great reception from our users, as well as the broader Australian science community. We are so proud to feature an artwork from Ngunnawal artist Lynnice Letty Church on our supercomputer – her art and the Gadi name perfectly encompass NCI’s mission of scientific research and high-performance computing: to search for knowledge that can make the world a better place through enabling innovative world-class research.

Tall panels decorated with blue Aboriginal artwork separated by sliding glass doors, seen from below.

Gadi is open for business - 18 November 2019 - Day 110

Racks of Gadi supercomputer.
When you think supercomputer, think Gadi.

The Gadi supercomputer has been opened to users for the very first time today as 'Phase 1' is completed. Eighteen racks of Gadi are now online and currently being battle-tested by Australian researchers, with the rest of the supercomputer to be made available in Q1 2020. 

Today marks a milestone in Australian HPC. Staff have been working round the clock to make sure that Gadi is the best possible environment for today's researchers to land that important scientific breakthrough, and make our world a better place.

A lot has changed in 110 days at NCI! 

An overhead view of some of the Gadi supercomputer racks.GPU nodes of the Gadi supercomputer.


A very cool delivery - 1 November 2019 - Day 92

All the pieces are coming together now. Some of the last critical components, the Cooling Distribution Units (CDUs), arrived on site last week and were rapidly connected up to the network of water pipes, tanks and cooling towers that maintain the temperature of our supercomputer. Manufactured to our specifications and specific cooling needs, these CDUs are a vital piece of the Gadi puzzle. The only way that a supercomputer like Gadi can operate is with a robust cooling infrastructure behind it to take away all of the heat its powerful processors produce.

A two-metre tall plywood box with one side removed revealing a tightly wrapped package inside.
A very important delivery!


A black metal box with sides removed showing a complex web of silver water pipes inside it.
All the connectors slot nicely into place.


Two tall silver pipes with blue handles go up the wall and along the ceiling.
Some of the new pipework transporting water to and from the CDUs.

Building a high-koalaty network - 30 October 2019 - Day 90

Nearing the final stages of the Gadi build, over the past few weeks a key component has been installed. Our Infiniband network is now in place, connecting the new Gadi compute nodes to their new /scratch filesystem, the /g/data filesystems, and MDSS. Hundreds of fibre-optic cables are now ready to deliver a 200Gb/s HDR Infiniband network to all our Gadi users. And all with a little help from our little NCI koala friend!

Dozens of thin blue-green cables in bundles stretch out into the distance across the length of the data hall.


A vertical stack of silver computer servers with very neat cables plugged in on left and right.


A koala toy is attached to a large bundle of thin blue cables stretching down from above.


Australia's most powerful supercomputer takes shape at NCI  – 16 October 2019 – Day 76

Equipment in NCI's data centre
Power, compute and water come together at NCI to make a supercomputer (left) | The new Gadi file system is up and running (right)

Compute nodes of GadiIt’s the home stretch. The core components of Gadi are now located in NCI’s data centre, and work is underway to ‘plumb it all in’ – literally and metaphorically. 

The plumbing that is used to keep Gadi cool has been connected to the in-row coolers. These coolers will – as the name suggests – keep Gadi running cool, and are placed at regular intervals beside compute hardware. 

These pipes will pump special water to prevent the build-up of minerals inside the plumbing, ensuring that Gadi will stay chill for years to come.

At the same time, connection of power and networking for each of the Gadi nodes is currently underway. Extra power cabinets have been installed along the western side of the data centre, and network cabling is being laid overhead.

Meanwhile, Gadi’s new file system has been configured and is ready for a supercomputer. The blinking lights are a dead giveaway.

Full to the brim  – 27 September 2019 – Day 57

With all our compute racks, cooling units, power distribution racks and more all filling up our data hall, it's getting pretty crowded in there! In the next few weeks, all the racks will be arranged into neat rows and connected up to power and water.

Long rows of black and silver racks standing next to each other.
On the left are Gadi's new CPU racks. On the right are the power distribution and in-row cooling racks.
A solid row of black server racks on the left facing new, unconnected server racks on the right.
Raijin's existing compute racks opposite the new Gadi racks waiting for installation.

It's all happening now! An update in THREE parts  – 18 September 2019 – Day 48

Part 1: Our new data arrays, installed during August, have now been joined by their management servers. These specialised nodes are the final major piece of the Gadi storage solution, so getting them in place is a big deal.

The front of a black rectangular computer server.The inside of a computer server showing cabling, green motherboards and cooling units.








Part 2: The electrical distribution boards that will send power through the Gadi system are currently being installed. Following the underfloor piping and electrical work, all those orange cables now need to be connected up to the big orange boxes that split the power up for use across the entire system.

An orange metal box several metres wide and tall sits on the back of a flat bed truck.An orange metal box with opened compartments showing where electrical cables will connect.








Part 3: Our all important compute racks have started arriving! Shipments of compute racks with pre-installed compute nodes are arriving this week, getting us ever closer to operations. Soon they will be mounted in place on the floor with their dedicated in-row coolers in position too. There are still some key components to come before we can switch them on, but the excitement is building!

A long row of black and grey compute racks standing next to each other on a white floor.
The first racks have arrived!

Fabulous flooring, fit for a supercomputer  – 13th September 2019 – Day 43

An empty space in a data centre with brand new flooring.
Hallowed ground - it's not much to look at, but the next major scientific breakthroughs may take place right here. This is where Gadi will live.

The super-extra-strong floor tiles are now in place. This void won't stay empty for long, as soon we'll have the first components of the Gadi supercomputing cluster being rolled into place.

The floor tiles conceal all of the complex plumbing and electrical infrastructure that was installed prior. When taking visitors on tours of NCI, we often say that the maze of pipes and cables is just as impressive as the computing cluster itself, perhaps even more so.

The floor tiles removed from the data centre reveal the plumbing underneath.
What it looked like before the new tiles were in place. All of this is new infrastructure.


Workers attending to piping in the ceiling.
Next door in the plant room, new plumbing is being installed to help keep Gadi nice and cool.

Meanwhile, new switches have been installed above the NetApp storage arrays, the last few pieces of a very complicated data storage puzzle. These file systems will be the fastest of their class in the Southern Hemisphere. 

A row of computer equipment in a data centre.
Looking much better with those gaps filled in...


Even supercomputers need to keep cool!  – 6th September 2019 – Day 36

A concrete floor covered with neatly arranged orange electrical cables, and some metal pipes lined up in rows.
From this...
A concrete floor with neatly arranged orange electrical cables, now with long lines of metal pipes in place along the floor.
... to this...
A concrete floor with orange electrical cables arranged in a neat grid, now with a full complement of silver metal pipes arranged in rows above the cables.
... and finally this.








We are almost finished installing the cooling infrastructure that will sit under the floor and provide Gadi with the circulating water it needs to keep cool. The water pipes are specifically designed for our needs, in particular the specific range of water temperatures that we will be working with to draw excess heat away from the supercomputer. Constructed off site, the pipes are trucked to our building and welded into place one by one.

We're almost ready for the floor to go back on!

Premium plumbing and a pint-sized supercomputer – 30th August 2019 – Day 30

Plumbing being installed and new compute equipment.
Plumbing fit for purpose (left) and some newfangled computing hardware (right).

You can't have a world-class supercomputer without a way to keep everything cool. Gadi will use evaporative cooling to beat the heat, just like Raijin. Instead of moving and removing heat using air (fans), heat from the supercomputer is transferred using water. The heated water is pumped away from Gadi, then evaporated via large cooling towers on the data centre roof, while cool water is then pumped back into the data centre to start the cycle all over again. 

These water 'loops' require purpose-built plumbing to stand up to the rigours of operation. The new plumbing for Gadi is especially treated to stop the build up of micro-organisms in the water supply, while also preventing the metal components themselves from degrading and contaminating the system. 

Computer experts huddle around computing equipment.
Some nice cable management here, right? Oh, and a bunch of extremely fast compute nodes, let's not forget about those.

While new plumbing is laid down for Gadi, a rack of mysterious looking equipment has since been plumbed in and switched on. These compute nodes are a miniature emulation of Gadi, and will be used to road-test the technology behind what is soon-to-be Australia's most powerful supercomputer. Our system administrators will be able to use this now, and in the future, to experiment with improvements to Gadi without interrupting the whole system.

With the last of the storage arrays being installed this week, we're really excited to see what's just around the corner...

Racks of storage arrays, unpowered.
Spooky looking. The storage arrays stand ready for Gadi. 


Data for days – 26th August 2019 – Day 26

NCI's new file servers are nearing completion
NCI's new file systems are nearly ready for Gadi.

The finishing touches are being readied for Gadi's new file systems. Nearly all of the storage arrays are now in place - now all that's left is for them to be plugged in and powered up (that might be a massive oversimplification). 

They won't become accessible until Gadi is brought online later in the year, but they'll be ready all the same. 

Lights on a power board.
There's definitely some power here! 
Electronics testing take place.
Some extra work taking place inside one of the in-row coolers.
Computer and electric cables installed.
Looking above the racks for the new file system. Check out those cables.


Data by the truckload – 21st August 2019 – Day 21

Hard drives stacked along a wall.
There are 1000 hard drives in this picture.

As soon as there is free space in the data centre, it's filled by fresh new hardware. Storage arrays and hard disk drives for Gadi's new file systems continues to be delivered by the truckload, and it won't be long before this new storage infrastructure is installed and ready to be fired up. 



Whoa we’re half way there… Whoa-oh! Look at all our hardware! – 14th August 2019 – Day 14

A long vertical stack filled with new grey and blue computer equipment extends to the back of the room.

Shiny, right?
We’re now past the half-way point for installing the new Gadi file systems. Drawers are being filled with fresh hard disk drives and cabling is being readied.
Every hard disk drive will be tested for performance and reliability before becoming a part of Gadi. These are all important steps before the compute hardware is delivered in the coming weeks and months. Work in the western side of the data centre is continuing in earnest.

A blue cable comes out from the back of a computer with some glowing green lights.
Signs of life.
A tray with 12 hard drives installed in a flat configuration.
Fully loaded and good to go.


Gadi grows in bits and bytes – 8th August 2019 – Day 8

Four empty black server racks are one-third filled from the bottom up with sets of four stacked orange and black draws.
New file systems are (almost) here!

The data centre has been a hive of activity today. While flooring continues to come up at one end the building, the other end is being populated with new storage for Gadi. Thousands of hard disk drives have been delivered over the past two days, and will make up the /home (‘slash home’) and scratch file systems for Gadi.

An open flat compartment with twelve rectangular slots, one of them filled with a hard drive.
The first hard disk drive is installed.
The back of a silver server box with visible fans and unused data ports.
Each of these holds sixty hard drives.

Floor tiling at the other end of the data centre has almost all been removed, and cable trays are currently being installed along the ceiling.

Neatly lined up orange cables in bundles run on the exposed concrete slab to connect underneath an orange electrical box.
There are several of these junction boxes in the data centre. That’s some serious power.


Floor the record, we’re making progress – 7th August 2019 – Day 7

It’s demo day! We have started pulling up floor tiles in the data centre to begin installation of essential plumbing and electrical services to support Gadi. Obviously, all of this work has to be done before anything else can happen.

A new set of in-row coolers also arrived today, giving us another behind-the-scenes look at our infrastructure. They look a lot like concert speakers, but these coolers are integral to NCI’s cooling requirements. Without them, everything would overheat very quickly.

Two in-row coolers with exposed sides revealing the blue radiator grills and fans inside.

Heat from the supercomputer is trapped in ‘hot aisles’, contained spaces between the rows and rows of computer equipment. This heat is then drawn over the cool-water radiators and transferred away from the equipment.

Meanwhile, cooled air is then drawn back into the data centre to begin the process all over again. The hot air makes its way up to the data centre roof, where it evaporates in Canberra’s dry air. This is commonly known as ‘free cooling’.

We have also taken delivery of hard disk drives and other storage hardware that will make up the /home (‘slash home’) file system of Gadi. These are being set aside for the moment while supporting infrastructure is assembled and put in place.

A cardboard box containing tightly packaged black plastic containers showing the outline of hard drives stored inside them.
Each of these hard drives stores 10 Terabytes of data. And we have a whole lot of them.
A silver box with a black front arranged in rows with orange highlights, this is a server that stores hard drives.
/home is now home (get it?)


It’s just a ‘phase’ we’re going through – 6th August 2019 – Day 6

I’ve got the power!
Each storage rack contains six of these super-sized power boards.

Neatly coiled thick black power cables with blue plugs lie on the floor ready for use.Coiled black power cables with three pronged ends visible.Close-up of the three prongs at the end of the 3 phase power cables ready for use.


New Storage Infrastructure Arrives – 2nd August 2019 – Day 2

Our very first deliveries are already here. These racks will be used to house new storage infrastructure that will expand NCI’s total storage capacity and support Gadi. Currently, NCI’s total data storage capacity stands around 50 Petabytes, or 50 million Gigabytes.

Two tall pallets wrapped in clear plastic sit on the loading bay of a delivery truck.
We are expecting dozens of deliveries just like this over the coming months.

They will be installed at the opposite end of the data centre, as Gadi itself will require all the space currently available on the western side of the building. Colocation of storage and compute at NCI has been an important part of our organisation for decades, and will continue to be once Gadi is up and running. Our HPC and data teams are expecting transfer speeds of up to 200Gb/s under the new system, more than twice as fast as what is available now with Raijin.

Six black server racks stand empty in rows surrounded by functioning supercomputer equipment.
It's almost time for a Gadi Party!

Small steps again today, but the best is yet to come.

Gadi is a GO – 1st August 2019 – Day 1

Following last week’s announcement, work has already commenced in NCI’s data centre to make way for the new hardware. From the very beginning, NCI’s data centre has kept an empty space available for Raijin’s eventual successor, and we are going to need every bit of it.

Men on ladders removing long fluorescent lights hanging from the ceiling a large room with a supercomputer in the background.
Our venerable supercomputer Raijin has a front row seat for all the action.

This forward thinking means that Raijin will stay online and available for our thousands of researchers while Gadi is assembled.

The overhead lights that spanned the empty space are starting to come down to make way for new lights and cable trays, while new lights will be installed to keep everything illuminated. This work is just the start of an incredibly busy period for NCI. The first truckloads of equipment are expected to arrive in the coming weeks.