Spartan HPC

Spartan is a High Performance Computing (HPC) system operated by Research Computing Services at The University of Melbourne. To process your data on Spartan, you will need to transfer it from Mediaflux to Spartan, generally either into a project directory or scratch space (see: Where to Store Your Data on Spartan).  

Spartan Modules System

There is some Mediaflux functionality available on Spartan via the modules system.

Unimelb Mediaflux command-line clients

The university has implemented command line clients that speak the native Mediaflux protocol. This protocol provides fast, reliable, restartable data transfers. The main clients are:

• unimelb-mf-download for downloads from Mediaflux

• unimelb-mf-upload for uploads to Mediaflux

• unimelb-mf-check for comparison/verification between the local file system and Mediaflux (confirming the source and destination match)

For example, to get data from Mediaflux to Spartan:

• log in to the Spartan login node

• load the Mediaflux clients module

• download the data from Mediaflux to Spartan storage using unimelb-mf-download.

This will transfer the data directly from one server to another bypassing your client machine which is faster and doesn't require local storage on your client machine.

To load the Mediaflux clients module on the Spartan login node:

This may be run directly in your terminal to load the Mediaflux clients module for this session only, or added to your ~/.bashrc to load the module each time you log in.

• Clients available on Spartan

  • Mediaflux Unimelb Command-Line Clients.  These have the ability to run multiple parallel threads which can speed up data movement (provided there is sufficient bandwidth).  Remember that you are working in a shared environment, and the resources are not infinite. Therefore we ask that you don't use more than 4 threads. You may need to do some testing and see if you get any performance benefit from additional threads.  The command line tools require that you set up a configuration file specifying the details of the Mediaflux server and your account.

  • aterm this is the basic command-line Mediaflux client (generally used by administrators).  It has a GUI and a shell version.  First set up a Configuration File specifying the details of the Mediaflux server and your account. To execute the shell version, at the Unix prompt just type aterm and you will be prompted for your password.

Mediaflux Data Mover

Data Mover is used to transfer data between Mediaflux and a file system using pre-created shareable links (for upload and download). The shareable (or token) includes information that allows you to access data for download or upload to a specific location in Mediaflux. Data Mover has a Graphical User Interface (GUI) which you can use with an Open OnDemand session or using the Command-Line Interface (CLI), which may be easier to manage. To load the Data Mover module:

To see a list of command-line options, you can run the following command:

See the Data Mover Reference documentation for further details.

Mediaflux Explorer

Mediaflux Explorer (HTTPS Protocol) is a Mediaflux GUI client for users to manage their data in Mediaflux. It is similar to a GUI FTP client.  You can access a GUI on Spartan through an Open OnDemand session. Note: University VPN connection is required if you are off campus.

Within your Spartan Open OnDemand session, open a terminal and type the following command to load the Mediaflux Explorer module:

Once the module is loaded, you can type the command below to start Mediaflux Explorer:

Mediaflux explorer send to sink

Using Mediaflux Explorer you can transfer data from the Mediaflux server directly to Spartan through a data sink.  See the Sinks In Explorer section of the Mediaflux Explorer page.  This method is best for smaller datasets; if you have a large dataset, consider using the unimelb-mf-clients on the command line or Mediaflux Data Mover on an Open OnDemand session.

Other protocols

Mediaflux supports many other protocols which may be used on Spartan if they fit your workflow. Examples are:

SFTP

Mediaflux supports the SFTP protocol which can be used with an sftp client of your choice. Examples available on spartan are:

• BSD sftp, installed and available by default on Spartan, no module needed

• lftp, available in the lftp module

SMB

Mediaflux supports the SMB protocol. There are many SMB client implementations, including the ability to mount the share as a network drive on your client machine. There are many ways to interact with SMB shares on Spartan, for example:

• smbclient

• smbget

• smbtar

• smbtree

For more information, use the man command on a Spartan login node, e.g. man smbclient

rsync

Mediaflux supports the rsync protocol. The rsync client is installed on Spartan by default, no module needed. For more information see man rsync on a Spartan login node.