For the early development of supercomputing in Australia, and NCI's precessors, please read The Early Years and The APAC Years. This, the NCI Years, follows on from those narratives.
The NCI Years
The NCRIS investment framework for eResearch was progressively implemented from 2007 onwards, with the signing of funding agreements by the Commonwealth Government and the lead organisations for NCI (with ANU), ARCS (with VPAC), and ANDS (with Monash University). Indeed, it can be argued that APAC spawned the initial activities of each, with NCI conceptualised as solely a supercomputing facility, built from the base of the successful APAC National Facility program.
The First Phase of NCI
The Funding Agreement that established NCI was executed by ANU and the Commonwealth Government in June 2007 for an amount of $26M, with the objectives being to:
- Procure/sustain a capability computing system commensurate with international standing;
- Advise Government on further “specialised system” investments to provide for targeted application areas;
- Establish/operate a merit-based open access allocation scheme for researchers at publicly-funded research organisations;
- Provide support and expertise services; and
- Maintain a supported strategic plan for national computational needs.
The goal at the time was to procure a capability system at the scale of the “Track 2” systems being established in the USA at the time by the National Science Foundation. It became clear that this would be a difficult goal to realise without the injection of further investment/co-investment. This led to building the partnership and governance model that exists today, under the leadership of a Steering Committee chaired by Emeritus Professor Mark Wainwright, a former Vice-Chancellor of UNSW, and through the office of Professor Robin Stanton, a Pro Vice-Chancellor at ANU.
The Steering Committee (today’s Advisory Board) was established in 2007, and expanded in 2008 to include three of the national agencies—CSIRO, the national science agency, the Bureau of Meteorology, and Geoscience Australia—alongside ANU, as the host organisation. These were the first steps taken to establish the sustaining collaboration. ANU’s delivery on the goals of the contract was initiated with the appointment of the foundation Director of NCI, Professor Lindsay Botten in 2008. As in the APAC era, all services in this first phase of NCI were provided through the ANU Supercomputing Facility.
Early in 2008, at the time at which ANU was about to tender for a new peak system, the Bureau of Meteorology was planning to approach the market for a new operational weather forecasting system. With strong synergies apparent, it was agreed that BoM and ANU would issue a joint tender, with the goal of procuring interoperable systems that would facilitate research opportunities and collaboration in climate and weather science, and which also had the potential to enhance the service robustness for operational weather forecasting. This procurement took place under the governance of a Joint Steering Committee comprising BoM, ANU/NCI, and CSIRO representatives. The evaluations were completed by October 2008.
A contract was executed by ANU and Sun in March 2009, for a 16 rack, 12,000 core, 140 Teraflop Sun Constellation known as Vayu—a distributed memory cluster based on Intel Xeon Nehalem technology with a QDR Infiniband interconnect. The first phase of the Sun system at NCI was installed in September 2009, more than replacing the capability of previous SGI Altix 3700 system, with subsequent upgrades in February and April of 2010 bringing the system to its full capacity. Concurrently, BoM completed its negotiations with Sun at the same time, with its five rack system entering production during 2010.
The emerging NCI collaboration took a significant step forward late in 2008 with a decision by CSIRO to enter into partnership with ANU as the initial custodians of NCI—with CSIRO committing to its ongoing, strong level of co-investment in NCI by executing with ANU the Partner Service Agreement on 24 December 2008. QCIF and Pawsey (formerly iVEC) joined as minor partners shortly thereafter in 2009, followed in 2010 by Geoscience Australia, and Intersect (the NSW university consortium).
The Second Phase of NCI
The second phase of NCI’s development commenced relatively early, with announcements in the Commonwealth Government Budget of May 2009 of the new Super Science initiatives. Each of the Super Science investments, including the Climate HPC Centre Project through ANU/NCI, was funded from the Education Investment Fund, one of Australia’s Nation Building Funds.
The project objectives were:
- The establishment of a petascale HPC facility prioritised to the needs of research in climate change, earth system science, national water management, which simultaneously would support meritorious and high-impact research in all fields that required access to capability-class computing;
- The identification and support of data-intensive and flagship science application aligned with other government research and infrastructure investments;
- The development and implementation of an access model that would meet the priority use requirements, as well as providing open access on research merit to researchers at publicly-funded research organisations;
- The construction of a purpose-built data centre to house the HPC facility, capable which could be upgraded to handle future usage for at least five years;
The Collaboration Agreement establishes the objectives for the collaboration, puts in place its governance, lays out the access model, and underpins NCI’s business model and planning. This collaboration, which includes four national organisations, together leveraging of the Commonwealth’s infrastructure monies through substantial co-investment, are two distinctive features of NCI.
The Collaboration Agreement set foundations for the implementation of this current phase of NCI and formed the backdrop for an organisational change, from the beginning of 2012, by which the University brought together the NCI Project Office and the ANU Supercomputing Facility into a single operating unit. Under this governance model, ANU governs the operations of NCI on the advice of the NCI Board, to within the limits of its Statutes and policies.
With the Collaboration Agreement in place from July 2011, the data centre construction was initiated in August 2011, and was completed in September 2012, with the building formally handed over to ANU in November of that year. Concurrently, the tender for the HPC system also proceeded through a public tender from August 2011. The contract for the procurement was executed in mid-June 2012, leading to the debut of the system, now known as Raijin, on the Top500 list in November 2012 at rank 24, with a peak performance (Rmax) of 940 Teraflops (for the 93 per cent of the system that had been implemented at the time). Early user service began in late April 2013, with full user service from mid-June 2013. The Fujitsu contract also implemented a most significant collaboration framework to take NCI and its partners into the future, and, in particular, to deliver optimal value for the peak system procurement. This was done through a program of work to optimise the implementation and performance of Australia’s climate and earth system science modelling suite, ACCESS, on Raijin.
NCI’s Board took the strategic decision in 2011 that the NCI Collaboration, which had been established to support operations of a petascale supercomputer, should broaden its scope to provide the comprehensive and integrated services that had long been envisaged. The build-out of the integrated infrastructure solution was undertaken during 2013, with storage procurements from SGI and DDN, and with the establishment of a 3,200 core high-performance cloud from Dell.
The story since 2013 is told incrementally in NCI’s Annual Reports. These provide updates on Australian research outcomes and impacts that have been enabled by the NCI platforms and its expert support team, and the evolution of the infrastructure platform—with further incrementation of the storage from Netapp in 2015, a 40% upgrading of NCI’s supercomputer capacity in 2016 with a 22,000 core Lenovo NeXtScale (Broadwell) system co-funded by a substantial allocation from the NCRIS Agility Fund and the NCI Collaboration, together with major storage upgrades funded from the same source.
The next major stage in the NCI Years is marked by the procurement and installation of Raijin's successor supercomputer, a significantly increased machine named Gadi and comprising latest generation technologies from Intel, Nvidia, Mellanox and others. First appearing on the November 2019 Top500 list in its initial, half-built configuration at number 47, its first fully operational Top500 benchmark was released in June 2020 and placed it at number 24 with a peak performance of 9.26 Petaflops. Comprising more than 150,000 compute cores and 640 GPU accelerators, Gadi underpins the next generation of data and compute intensive research in Australia. Gadi is so-named because it means "to search for" in the language of the Ngunnawal people, the traditional owners of the land that NCI is built on.
Since its beginning, NCI has evolved as the national, high-end research computing service—one which is in the vanguard of international advanced computing, delivering solutions that encompass computational modelling and the needs of big data, that enable research of excellence and impact, which deliver in the national benefit, and which maintain Australian’s international competitiveness in research and innovation.
The collaboration that underpins NCI has grown in strength since its formation, anchored by the Australian National University as NCI’s host, and three of Australia’s national science agencies—the Australian Bureau of Meteorology, the national science agency, CSIRO, and the national geoscience agency, Geoscience Australia. Today, this partnership includes many of Australia’s research universities, or consortia thereof, medical research institutes, and the Australian Research Council, and sustains a large portion of the annual recurrent costs, with contributions from the Australian Government’s National Collaborative Research Infrastructure Strategy (NCRIS) providing the remainder.
Through its tightly-coupled, high-performance computing and data platforms, overlaid with internationally renowned expertise (~60 staff) in computational science, data science and data management, NCI provides essential services that underpin the requirements of research and industry, today and into the future.
Today, NCI plays a pivotal role in Australia’s research and innovation system, supporting the work of more than 5,000 researchers across more than 500 projects being undertaken in 35 universities, 5 national science agencies (including CSIRO, Bureau of Meteorology, and Geoscience Australia), 3 medical research institutes, and industry.
R&D which is reliant on NCI spans the full spectrum—from fundamental research, through the strategic and applied, and on to actual industrial applications. Today, NCI services are both necessary and influential in enhancing the competitiveness and impact of outcomes in every field of science and technology. An increasing number of fields are now highly dependent on the fusion of “big compute” and “big data” that NCI provides—in weather and climate science, the earth sciences, earth observation and remote sensing, medical research, and astronomy, amongst others.
Within the university sector, NCI provides the essential high-performance computing and data foundation for more than 200 research projects, ARC and NHMRC Centres of Excellence and Industry Hubs, and fellowships. Funding for these projects from the Australian Research Council and the National Health and Medical Research Council totals around $60 million per annum, or approximately $250 million over the lifetimes of these projects.
In the domain of the national science agencies, notably BoM, CSIRO and GA, NCI provides critical program-level support in earth system science, by serving as the development platform of the Australian national weather and climate modelling suite, ACCESS. NCI is also a national hub for major national and international satellite earth observation collections, used in the earth, marine and environmental sciences, agriculture for research, the informing of policy development, and the development of important information products for primary industry.