Mediaflux on Spartan HPC

Owned by Andy Tseng
Mar 21, 2024
3 min read

Spartan is a High Performance Computing (HPC) system operated by Research Computing Services at The University of Melbourne.  There is some Mediaflux functionality available on Spartan via the modules system.

1. Module unimelb-mf-clients 

To process your data on Spartan, you will need to transfer it from Mediaflux to Spartan, generally either into a project directory or scratch space (see: Where to Store Your Data on Spartan).  If you have a Mediaflux account, the recommended way to get data from Mediaflux to Spartan is to log in to the Spartan login node, load the Mediaflux clients module, and download the data from Mediaflux to Spartan storage using one of the clients.  This will transfer the data directly from one server to another bypassing your client machine which is faster and doesn't require local storage on your client machine.

The Mediaflux clients are available via the module system on Spartan.  To load the Mediaflux clients module on the Spartan login node:

module load unimelb-mf-clients

This may be run directly in your terminal to load the Mediaflux clients module for this session only, or added to your ~/.bashrc to load the module each time you log in.

  • Clients available on Spartan
    • Mediaflux Unimelb Command-Line Clients.  These have the ability to run multiple parallel threads which can speed up data movement (provided there is sufficient bandwidth).  Remember that you are working in a shared environment, and the resources are not infinite. Therefore we ask that you don't use more than 4 threads. You may need to do some testing and see if you get any performance benefit from additional threads.  The command line tools require that you set up a Configuration File specifying the details of the Mediaflux server and your account.
    • aterm -  this is the basic command-line Mediaflux client (generally used by administrators).  It has a GUI and a shell version.  First set up a Configuration File specifying the details of the Mediaflux server and your account.
    • To execute the shell version, at the Unix prompt just type aterm and you will be prompted for your password.


2. Module mediaflux-data-mover

The Data Mover Reference is used to transfer data between Mediaflux and a file system using pre-created shareable links (for upload and download). The shareable (or token) includes information that allows you to access data for download or upload to a specific location in Mediaflux. The Data Mover Reference has a Graphical User Interface (GUI) which you can use with Spartan Virtual Desktop (best with Chrome as copy/paste can be problematic with other browsers) or a Command-Line Interface (CLI), which may be easier to manage. To load the Data Mover module:

module load mediaflux-data-mover

See the Data Mover Reference documentation for further details.


3. Module mediaflux-explorer

Mediaflux Explorer (HTTPS Protocol) A Mediaflux GUI client for users to manage their data in Mediaflux.  This does not use a configuration file as all server details are entered into the GUI.  You can access a GUI on Spartan through a Virtual Desktop session. (Note: University VPN connection is required to access Spartan Virtual Desktop if you are off campus.)

Within your Spartan Virtual Desktop, open a terminal and type the following command to load the Mediaflux Explorer module:

module load mediaflux-explorer

Once the module is loaded, you can type the command below to start Mediaflux Explorer:

mediaflux-explorer

4. Mediaflux explorer send to sink

Using Mediaflux Explorer you can transfer data from the Mediaflux server directly to Spartan through a data sink.  See the Sinks In Explorer section of the Mediaflux Explorer page.  This method is best for smaller datasets; if you have a large dataset, consider using the unimelb-mf-clients on the command line or Mediaflux Data Mover on the FastX GUI (see above).