National Computational Infrastructure

NCI

Tenjin User Guide

      1. Login to the Tenjin Dashboard at https://tenjin.nci.org.au/ with your NCI credentials. The name of your OpenStack project (also known in OpenStack as a ‘tenant’) is the same as the name of your NCI project. If you are a member of multiple projects, then please choose the desired project from the drop-down list at top left of the Dashboard.
      2. If you are using Tenjin for the first time, Click “Access & Security” and create/add your KeyPair. On Linux or Mac, you can “cat” and copy your public key and paste it here. You may give it any name. This is one time only operation. However depending on your workflow/security model you may chose to have a number of key pairs.
      3. Click on the “Instances Tab” and press “Launch Instance” to start a virtual machine.
      4. Click “Details” and select appropriate Image Name and Flavor.

        Flavors Explained
        NCI offers a number of virtual machine flavors to suit the needs of a research group. The name of the flavor gives you details of the number of cpus, memory and local disk space.

        E.g. 8c16m80d
        – CPUS: 8
        – Memory: 16GB
        – Local Disk: 80 GB*

        Overview of Storage types

        Storage Type Device Intended use Pros Cons
        Root disk /dev/sda, automatically mounted on / Operating system and applications
        • High IOPS and throughput (Solid State Disks in RAID 0)
        • Included in instance snapshots
        • Limited size
        • No redundancy
        • Deleted when virtual machine is deleted
        • Not automatically backed up
        Ephemeral disk /dev/sdb, automatically mounted on /mnt Temporary storage (scratch space)
        • High IOPS and throughput (Solid State Disks in RAID 0)
        • Limited size
        • No redundancy
        • Deleted when virtual machine is deleted
        • Not included in instance snapshots
        • Not automatically backed up
        Attached volumes aka Block Storage aka Cinder (Backed by Ceph) /dev/sdc and subsequent, not automatically mounted Persistent storage of large volumes of data
        • Large size
        • High throughput
        • Built-in redundancy
        • Exists until deleted by owner
        • Can be detached from one VM, then re-attached to another.
        • Can create point-in-time volume snapshots
        • Lower IOPS
        • Not automatically backed up


        Local Disk and Cinder Volume

        No storage of any type is automatically backed up. Volume storage has built-in redundancy. Local disk and ephemeral are only for operating system and scratch; they are both local to the compute blade and have NO redundancy. We strongly suggest that you use an automated virtual machine creation process, so that your virtual machines are reproducible. For this purpose, we recommend use of configuration management tools such as Puppet, Chef, Ansible, Salt, cfengine or custom-made scripts.

        For persistent storage, NCI provides cinder volume and projects should use cinder volume to store critical data. While cinder volume (based on Ceph) is replicated, we strongly suggest projects to ask /pay for long term storage on NCI’s tape drives. The data on NCI’s tape drives is backed up across two remote sites. For more information please send an email to help@nci.org.au.

      5. Click “Access & Security” and select the Key pair you want to use for logging into the virtual machine once it is provisioned.
      6. Click “Networking” and select the IP address. Your project may have multiple IP address associated depending upon the requirements.
      7. Click Launch.
      8. Use “ssh –i /path/to/keypair root@IP.ADDRESS” to access the virtual machine.
      9. We do not recommend putting in useful data on the VDA (root) and (VDB) ephemeral storage. At the time of creation of the project, NCI gives 10GB (minimum) quota for block storage (we use Ceph).
      10. Click “Volumes” tab and create a volume.
      11. Attach the volume it to the virtual machine. It will most probably get attached as /dev/vdc but it is always a good idea to check. On your virtual machine fdisk –l command will give you a clear idea.
        [root@awesome]# fdisk -l
        
        Disk /dev/vda: 10.7 GB, 10737418240 bytes
        255 heads, 63 sectors/track, 1305 cylinders
        Units = cylinders of 16065 * 512 = 8225280 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x000c62bf
        
           Device Boot      Start         End      Blocks   Id  System
        /dev/vda1   *           1        1306    10484736   83  Linux
        
        Disk /dev/vdb: 32.2 GB, 32212254720 bytes
        16 heads, 63 sectors/track, 62415 cylinders
        Units = cylinders of 1008 * 512 = 516096 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x00000000
        
        
        Disk /dev/vdc: 10.7 GB, 10737418240 bytes
        16 heads, 63 sectors/track, 20805 cylinders
        Units = cylinders of 1008 * 512 = 516096 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disk identifier: 0x00000000
        
        
      12. Create a filesystem on /dev/vdc
        [root@awesome /]# mkfs.ext4 /dev/vdc
        mke2fs 1.41.12 (17-May-2010)
        
      13. Mount the volume on your virtual machine.
        [root@awesome /]# mkdir /data; mount /dev/vdc /data
        

Limitations:
Tenjin is different from Amazon EC2 or the NeCTAR Cloud in terms of features and specifications. The specifications of Tenjin are similar to Raijin- our supercomputer. It uses 56G Ethernet and SRIOV for low latency and high bandwidth network. Introduction of these features have resulted in a few limitations that are due to inherent nature of hardware and operating system design.

1. No Snapshots on Running Virtual Machines: 
We use SRIOV (Single Root IO Virtualization) for fast 56G Ethernet (with RDMA support) and it does not support snapshot feature on a running virtual machine. If you want to snapshot, you will have to shutdown the virtual machine and then snapshot. Snapshot on a live virtual machine will appear to have hung. We plan to patch the dashboard to prevent this bug but it is quite low on priority list. 


        • EthN interface increments on RHEL/CentOS-6 when using snapshot as an image: To stop ethernet interfaces from incrementing by one after each snapshot, please remove the following file before taking the snapshot. This is not a limitation or a bug but just the way udev rules work.
          [root@awesome /]# rm /etc/udev/rules.d/70-persistent-net.rules
            

2. No Live Migration: 
Due to inherent nature of SRIOV design, we cannot perform live migration of virtual machines between the hypervisors. Cold migrations are fully supported. 


NCI uses the IP address range supplied by the Australian National University. These IP addresses are regularly scanned for security vulnerabilities and monitored for suspicious network traffic and behavior. NCI reserves the right to shutdown and lock your virtual machine in the case your virtual machine is not secure, has been hacked and/or is involved in a suspicious behavior. NCI staff will inform the virtual machine owner and the project CI with the reasons for shutting down the virtual machine.

In Collaboration With