NCI will be the site of Australia’s fastest supercomputerGadi, in 2020. Supported by the Australian Government through its National Collaborative Research Infrastructure Strategy, Gadi represents a ten-fold increase in computing power over its predecessor, Raijin. This page will keep you up-to-date with what’s happening behind the scenes with Gadi as the system comes online. Stay tuned!


It's all happening now! An update in THREE parts  – 18 September 2019 – Day 48

Part 1: Our new data arrays, installed during August, have now been joined by their management servers. These specialised nodes are the final major piece of the Gadi storage solution, so getting them in place is a big deal.

The front of a black rectangular computer server.The inside of a computer server showing cabling, green motherboards and cooling units.

 

 

 

 

 

 

 

Part 2: The electrical distribution boards that will send power through the Gadi system are currently being installed. Following the underfloor piping and electrical work, all those orange cables now need to be connected up to the big orange boxes that split the power up for use across the entire system.

An orange metal box several metres wide and tall sits on the back of a flat bed truck.An orange metal box with opened compartments showing where electrical cables will connect.

 

 

 

 

 

 

 

Part 3: Our all important compute racks have started arriving! Shipments of compute racks with pre-installed compute nodes are arriving this week, getting us ever closer to operations. Soon they will be mounted in place on the floor with their dedicated in-row coolers in position too. There are still some key components to come before we can switch them on, but the excitement is building!

A long row of black and grey compute racks standing next to each other on a white floor.
The first racks have arrived!

Fabulous flooring, fit for a supercomputer  – 13th September 2019 – Day 43

An empty space in a data centre with brand new flooring.
Hallowed ground - it's not much to look at, but the next major scientific breakthroughs may take place right here. This is where Gadi will live.

The super-extra-strong floor tiles are now in place. This void won't stay empty for long, as soon we'll have the first components of the Gadi supercomputing cluster being rolled into place.

The floor tiles conceal all of the complex plumbing and electrical infrastructure that was installed prior. When taking visitors on tours of NCI, we often say that the maze of pipes and cables is just as impressive as the computing cluster itself, perhaps even more so.

The floor tiles removed from the data centre reveal the plumbing underneath.
What it looked like before the new tiles were in place. All of this is new infrastructure.

 

Workers attending to piping in the ceiling.
Next door in the plant room, new plumbing is being installed to help keep Gadi nice and cool.
​​

Meanwhile, new switches have been installed above the NetApp storage arrays, the last few pieces of a very complicated data storage puzzle. These file systems will be the fastest of their class in the Southern Hemisphere. 

A row of computer equipment in a data centre.
Looking much better with those gaps filled in...

 

Even supercomputers need to keep cool!  – 6th September 2019 – Day 36

A concrete floor covered with neatly arranged orange electrical cables, and some metal pipes lined up in rows.
From this...
A concrete floor with neatly arranged orange electrical cables, now with long lines of metal pipes in place along the floor.
... to this...
A concrete floor with orange electrical cables arranged in a neat grid, now with a full complement of silver metal pipes arranged in rows above the cables.
... and finally this.

 

 

 

 

 

 

 

We are almost finished installing the cooling infrastructure that will sit under the floor and provide Gadi with the circulating water it needs to keep cool. The water pipes are specifically designed for our needs, in particular the specific range of water temperatures that we will be working with to draw excess heat away from the supercomputer. Constructed off site, the pipes are trucked to our building and welded into place one by one.

We're almost ready for the floor to go back on!

Premium plumbing and a pint-sized supercomputer – 30th August 2019 – Day 30

Plumbing being installed and new compute equipment.
Plumbing fit for purpose (left) and some newfangled computing hardware (right).

You can't have a world-class supercomputer without a way to keep everything cool. Gadi will use evaporative cooling to beat the heat, just like Raijin. Instead of moving and removing heat using air (fans), heat from the supercomputer is transferred using water. The heated water is pumped away from Gadi, then evaporated via large cooling towers on the data centre roof, while cool water is then pumped back into the data centre to start the cycle all over again. 

These water 'loops' require purpose-built plumbing to stand up to the rigours of operation. The new plumbing for Gadi is especially treated to stop the build up of micro-organisms in the water supply, while also preventing the metal components themselves from degrading and contaminating the system. 

Computer experts huddle around computing equipment.
Some nice cable management here, right? Oh, and a bunch of extremely fast compute nodes, let's not forget about those.

While new plumbing is laid down for Gadi, a rack of mysterious looking equipment has since been plumbed in and switched on. These compute nodes are a miniature emulation of Gadi, and will be used to road-test the technology behind what is soon-to-be Australia's most powerful supercomputer. Our system administrators will be able to use this now, and in the future, to experiment with improvements to Gadi without interrupting the whole system.

With the last of the storage arrays being installed this week, we're really excited to see what's just around the corner...

Racks of storage arrays, unpowered.
Spooky looking. The storage arrays stand ready for Gadi. 

 

Data for days – 26th August 2019 – Day 26

NCI's new file servers are nearing completion
NCI's new file systems are nearly ready for Gadi.

The finishing touches are being readied for Gadi's new file systems. Nearly all of the storage arrays are now in place - now all that's left is for them to be plugged in and powered up (that might be a massive oversimplification). 

They won't become accessible until Gadi is brought online later in the year, but they'll be ready all the same. 

Lights on a power board.
There's definitely some power here! 
Electronics testing take place.
Some extra work taking place inside one of the in-row coolers.
Computer and electric cables installed.
Looking above the racks for the new file system. Check out those cables.

 

Data by the truckload – 21st August 2019 – Day 21

Hard drives stacked along a wall.
There are 1000 hard drives in this picture.

As soon as there is free space in the data centre, it's filled by fresh new hardware. Storage arrays and hard disk drives for Gadi's new file systems continues to be delivered by the truckload, and it won't be long before this new storage infrastructure is installed and ready to be fired up. 

 

 

Whoa we’re half way there… Whoa-oh! Look at all our hardware! – 14th August 2019 – Day 14

A long vertical stack filled with new grey and blue computer equipment extends to the back of the room.

Shiny, right?
We’re now past the half-way point for installing the new Gadi file systems. Drawers are being filled with fresh hard disk drives and cabling is being readied.
Every hard disk drive will be tested for performance and reliability before becoming a part of Gadi. These are all important steps before the compute hardware is delivered in the coming weeks and months. Work in the western side of the data centre is continuing in earnest.

A blue cable comes out from the back of a computer with some glowing green lights.
Signs of life.
A tray with 12 hard drives installed in a flat configuration.
Fully loaded and good to go.

 

Gadi grows in bits and bytes – 8th August 2019 – Day 8

Four empty black server racks are one-third filled from the bottom up with sets of four stacked orange and black draws.
New file systems are (almost) here!

The data centre has been a hive of activity today. While flooring continues to come up at one end the building, the other end is being populated with new storage for Gadi. Thousands of hard disk drives have been delivered over the past two days, and will make up the /home (‘slash home’) and scratch file systems for Gadi.

An open flat compartment with twelve rectangular slots, one of them filled with a hard drive.
The first hard disk drive is installed.
The back of a silver server box with visible fans and unused data ports.
Each of these holds sixty hard drives.

Floor tiling at the other end of the data centre has almost all been removed, and cable trays are currently being installed along the ceiling.

Neatly lined up orange cables in bundles run on the exposed concrete slab to connect underneath an orange electrical box.
There are several of these junction boxes in the data centre. That’s some serious power.

 

Floor the record, we’re making progress – 7th August 2019 – Day 7

It’s demo day! We have started pulling up floor tiles in the data centre to begin installation of essential plumbing and electrical services to support Gadi. Obviously, all of this work has to be done before anything else can happen.

A new set of in-row coolers also arrived today, giving us another behind-the-scenes look at our infrastructure. They look a lot like concert speakers, but these coolers are integral to NCI’s cooling requirements. Without them, everything would overheat very quickly.

Two in-row coolers with exposed sides revealing the blue radiator grills and fans inside.

Heat from the supercomputer is trapped in ‘hot aisles’, contained spaces between the rows and rows of computer equipment. This heat is then drawn over the cool-water radiators and transferred away from the equipment.

Meanwhile, cooled air is then drawn back into the data centre to begin the process all over again. The hot air makes its way up to the data centre roof, where it evaporates in Canberra’s dry air. This is commonly known as ‘free cooling’.

We have also taken delivery of hard disk drives and other storage hardware that will make up the /home (‘slash home’) file system of Gadi. These are being set aside for the moment while supporting infrastructure is assembled and put in place.

A cardboard box containing tightly packaged black plastic containers showing the outline of hard drives stored inside them.
Each of these hard drives stores 10 Terabytes of data. And we have a whole lot of them.
A silver box with a black front arranged in rows with orange highlights, this is a server that stores hard drives.
/home is now home (get it?)

 

It’s just a ‘phase’ we’re going through – 6th August 2019 – Day 6

I’ve got the power!
Each storage rack contains six of these super-sized power boards.

Neatly coiled thick black power cables with blue plugs lie on the floor ready for use.Coiled black power cables with three pronged ends visible.Close-up of the three prongs at the end of the 3 phase power cables ready for use.

 

New Storage Infrastructure Arrives – 2nd August 2019 – Day 2

Our very first deliveries are already here. These racks will be used to house new storage infrastructure that will expand NCI’s total storage capacity and support Gadi. Currently, NCI’s total data storage capacity stands around 50 Petabytes, or 50 million Gigabytes.

Two tall pallets wrapped in clear plastic sit on the loading bay of a delivery truck.
We are expecting dozens of deliveries just like this over the coming months.

They will be installed at the opposite end of the data centre, as Gadi itself will require all the space currently available on the western side of the building. Colocation of storage and compute at NCI has been an important part of our organisation for decades, and will continue to be once Gadi is up and running. Our HPC and data teams are expecting transfer speeds of up to 200Gb/s under the new system, more than twice as fast as what is available now with Raijin.

Six black server racks stand empty in rows surrounded by functioning supercomputer equipment.
It's almost time for a Gadi Party!

Small steps again today, but the best is yet to come.

Gadi is a GO – 1st August 2019 – Day 1

Following last week’s announcement, work has already commenced in NCI’s data centre to make way for the new hardware. From the very beginning, NCI’s data centre has kept an empty space available for Raijin’s eventual successor, and we are going to need every bit of it.

Men on ladders removing long fluorescent lights hanging from the ceiling a large room with a supercomputer in the background.
Our venerable supercomputer Raijin has a front row seat for all the action.

This forward thinking means that Raijin will stay online and available for our thousands of researchers while Gadi is assembled.

The overhead lights that spanned the empty space are starting to come down to make way for new lights and cable trays, while new lights will be installed to keep everything illuminated. This work is just the start of an incredibly busy period for NCI. The first truckloads of equipment are expected to arrive in the coming weeks.