National Facility

Background

While the history of capability computing in Australia dates back twenty years into the late 1980s, it was not until the establishment of APAC, the antecedent of NCI, that there was either national policy for the development of advanced computing or a national high performance computing facility of an internationally significant scale. 

The first (APAC) National Facility was established at ANU in April 2001 and comprised a HP AlphaServer SC system, the performance of which was approximately one teraflop. 

The second (and current) generation National Facility comprising a SGI Altix 3700 system of approximately 10 teraflop performance was installed during the second phase of APAC operations and was commissioned in June 2005. 

The NCI business plan provides for a substantial upgrade to Australian capability computing during 2008-09 with a ten-fold increase in performance over the present facilities planned.  This is being pursued with a joint tender with the Bureau of Meteorology for a replacement system for the National Facility to be located at ANU, and with a system that can meet the Bureau’s operations requirements to be installed in Melbourne. 

In addition to the high performance systems, the National Facility resources include a comprehensive software library (see http://nf.nci.org.au/facilities/ac/hardware.php), mass storage systems, visualisation systems and data communications, and is staffed by a highly regarded support team.

Current Peak System

The current peak system (ac.nci.org.au) is a SGI Altix 3700 Bx2 cluster with 1928 Intel Itanium2 (1.6 GHz) processors (with 1920 processors dedicated to batch work), 5.6 TBytes of memory, approximately 100 Tbytes of disk space (30 TBytes of global filespace and 70 Tbytes of local scratch space), and a peak speed of 12 Teraflops (12 trillion calculations per second).  When it was commissioned in June 2005, it debuted on the Top500 list of the world’s most powerful computer systems (www.top500.org) at position 26. 

In more detail:

• 1928 1.6Ghz Itanium2 processors (1920 dedicated to batch jobs) with 6Mbyte L3 cache;
• processors are grouped into 30 partitions (single system image nodes) of 64 processors each, where 17 have 128 Gbytes of memory, 12 have 256 Gbytes of memory, and 1 has 384 Gbytes of memory;
• total memory is 5.6Tbytes;
• SGI's NUMAlink4 interconnect, both within and between partitions, provides 3.2 Gbytes/s of bidirectional bandwidth per link, with less than 2ms MPI latency;
• approximately 30 TBytes of global storage attached to five SGI InfiniteStorage TP9500 controllers.  The storage is available to all nodes under SGI’s CXFS filesystem with an aggregate bandwidth of more than 2Gbytes/s;
• a usable total of more than 27 TBytes (47+ TBytes of raw disk) of SGI TP9100 storage local to the partitions and connected to each by two dual port HBA’s (~ 800 MBytes/s).
• the operating system is based on SUSE SLES10 Linux with an enhanced Linux 2.6 kernel and SGI ProPack5;
• a total peak speed of over 11Tflops that was ranked 26 in the June 2005 Top500 list.
• an integrated SPECrate performance of 47820.

Applications are made using the forms for Merit Allocation Projects.

Other Compute Resources

The other computational system presently in the National Facility is a 152 processor Pentium Linux cluster (lc.nci.org.au) to support tasks which do not require the power, connectivity and I/O bandwidth of the SGI Altix supercomputer.

The detailed specification of the Linux cluster is:

• 152 Dell Precision 350 nodes, each containing 2.66 GHz Pentium 4 1 Gbyte (533 MHz dual channel) PC1066 RAMBUS memory, 100 GBytes IDE disks for job scratch, on-board gigabit;
• an aggregate SpecFPrate_base of (compute nodes only) ~1800; Peak theoretical performance of over 800 Gflops;
• a total of 152 Gbytes of RAM on compute nodes;
• 2 Dual Processor Dell PowerEdge 2600 servers for interactive use and fileserving;
• 1.4 TBytes global user storage (additional to node disks);
• 15 TBytes user scratch space for running jobs;
• 140 GBytes system storage;
• A total of over 16 TBytes of disk;
• Dell 5224 Gigabit switches for interconnecting the nodes.

This system is due for replacement and a tender for a substantial Linux cluster with an Infiniband interconnect and a Lustre filesystem has been let in June 2008.  The new facility, which should be available by the end of August 2008, will alleviate the current pressure on the SGI Altix and provide for new and existing projects to undertake computational research.

Mass Data Resources

The Data Cluster (dc.nci.org.au) is a system capable of storing both massive quantities of data and supporting complex data management projects and nationally significant data collections.  

• provides a resource for storing and retrieving data of magnitudes well beyond that of standard departmental or workstation storage capability;
• has high speed bandwidth and services to manage transfers between both national and international networks using multiple gigabit networks. The cluster and the peak system at the National Facility have the capability for appropriate connectivity to high data-generating instruments (in conjunction with AARNet);
• provides a management point for complex data needs, including fast (and/or large) relational database management;
• provides services for data grids, including capabilities for nationally and internationally federated datasets;
• has capability, services and software for allowing research groups to host significant data collections;
• provides computational capacity for data analysis.

Information about its usage including a user guide and available software are available on the National Facility website at http://nf.nci.org.au/facilities/mdss/.

Support Team

The National Facility is supported by a team of technical and research specialists within NCI/ANU Supercomputing Facility team (http://nf.nci.org.au/anusf_staff/index_photos.php ), with the NCI budget paying for around 15 effective full-time positions. 

 Past reviews of APAC have noted the high quality of the support service and the excellence of the support team, with the report of the final review of APAC in 2006 noting that users of the National Facility regularly commented that the National Facility was “world class” and a “resource we must be sure we are able to maintain”. 

Contact:

Dr Ben Evans

Manager, National Facility

Ben.Evans@anu.edu.au

Phone: 61 2 6125 4967