Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Mediaflux supports the sFTP (secure File Transfer Protocol) and so you can use standard sFTP clients to upload and download data to and from Mediaflux.  sFTP encrypts data during transmission and some clients (usually the graphical ones) can transfer data in parallel.

Some clients we recommend:

  • Filezilla is recommended for all desktop platforms (Windows, macOS, Linux)
  • WinSCP also works well on Windows (you must use it with the sFTP protocol - it does support other protocols like SCP but Mediaflux does not).

It is also common for Unix-based systems to provide a command-line sFTP tool.

It is also possible to mount mediaflux as a network share using an SSHFS client over the SFTP protocol.  Be aware that there are certain performance penalties to doing this, so if you can mount using one of the other protocols available such as SMB then you should probably do so.  But this can be a convenient way of accessing your files under certain circumstances.

An example of an SSHFS client is GNOME Files on Linux (see the example below).  There are also ports of this functionality to both Windows and macOS though they don't ship with it out of the box.

Finally, we have also made available the ability to download from an sFTP server directly into Mediaflux via the web-based Desktop interface. Please see the Desktop Plugin section.  This complements being able to upload into Mediaflux via an sFTP client.

Uploading and downloading data

The /Volumes directory on the Mediaflux server will list all projects, but you will only be able to access your own projects, and you may see an error if trying to list a project that you don't have access to.

The workflow for uploading and downloading data will depend on the sFTP client that you are using

  • Command-line clients generally use get/put commands (get -r and put -r to work recursively).
  • GUI clients often support drag and drop or have a button to transfer selected items from local to remove and remote to local.

If your sFTP client supports parallel transfers, it has some advantage for big downloads.  Use carefully, starting with perhaps 2 parallel transfers and generally not more than 4. If you use large numbers of threads you will compromise your own computer as well as perhaps the server.

Usually, graphical clients like FileZilla also support restarts in that they allow you to configure what to do with files that already exist on the client (your) side.

Make sure that client timeouts are set to something large (or disabled) if you are dealing with large files.

Connection details

Any sFTP tool needs to be configured with the details of the sFTP server that you are interacting with. The details for our Mediaflux sFTP service are

  • protocol : sftp 
  • host/server  : mediaflux.researchsoftware.unimelb.edu.au
  • port : 22 or 9003

The service runs on port 9003 and port 22 is forwarded to 9003.

User name format

Most sFTP tools allow you to pre-configure your user name.  The sFTP user name is a colon-separated combination of your Mediaflux authentication domain and username as follows.  You can leave the password field blank, and when you connect you will be prompted to enter your password.

University of Melbourne Account

Your username is of the form domain:username where domain is:

  • unimelb for staff e.g. unimelb:sclaus

  • student for students e.g. student:jfrost

Local account

If your account is a local account (i.e. you are not using your institutional account), you will have been provided with the details for domain, user and password.  Your username is of the form domain:username, e.g. local:dwho

Example - FileZilla

This example uses the FileZilla desktop tool executing on a Mac OS X computer and a University of Melbourne account.   Windows will be very similar (for the same tool, FileZilla).

Configure connection and connect

  • Start Filezilla
  • From the File menu, click Site Manager
  • Click the New Site button.  Give your site a name (in the tree in left-hand pane)
  • Enter the connection details (see above) on the General tab in the right hand pane
    • Set the Protocol to SFTP
    • By selecting a login type of Ask for password, FileZilla will prompt for the password each time we connect.  You can choose the Normal login type if you wish to save your password.
  • Click Connect and a new window will ask for your password.  Enter it and click OK.
  • As you have never connected to this host before, you will be prompted about an Unknown host key.  If you tick Always trust this host you will not be prompted next time.

Navigate

Click on the Volumes folder in the upper-right pane.  All Mediaflux projects are listed here.

Click on the project you want to work with.  If you attempt to do this for a project you do not have access to, Mediaflux will deny you access and FileZilla will present an error message.  You can drag and drop to upload/download data from your local machine (left side) to/from the Mediaflux project (right side).

Example - command line sftp client

Most Unix-based operating systems (macOS, Linux, etc.) offer  a command-line sFTP tool. The following examples were executed on macOS, but should work for most Unix variants.  If not, refer to the local help for the tool you are using.

Log in with University of Melbourne

For UoM central accounts or local accounts can use the domain:username or domain/username syntax for the sFTP username. For University of Melbourne Staff:

sftp -o User=unimelb:sclaus mediaflux.researchsoftware.unimelb.edu.au

or

sftp unimelb/sclaus@mediaflux.researchsoftware.unimelb.edu.au


For University of Melbourne students:

sftp -o User=student:astudent mediaflux.researchsoftware.unimelb.edu.au

or

sftp student/astudent@mediaflux.researchsoftware.unimelb.edu.au


Log in with a Local Account

To log in with a local account, on the alternate port of 9003:

sftp -P 9003 -o User=local:nkilleen mediaflux.researchsoftware.unimelb.edu.au

or

sftp -P 9003 local/nkilleen@mediaflux.researchsoftware.unimelb.edu.au

Navigate and download

sftp> cd /Volumes/proj-demonstration-1128.4.15
sftp> get -r Photos
Fetching /Volumes/proj-demonstration-1128.4.15/Photos/ to Photos

Example - using cURL

cURL can be used to transfer files over sftp.  It cannot use the domain:username syntax; only the domain/username syntax is supported.  To download a file:

curl -u 'unimelb/sclaus' -O sftp://mediaflux.researchsoftware.unimelb.edu.au/Volumes/proj-demonstration-1128.4.15/hello.txt

To upload a file:

curl -u 'unimelb/sclaus' --upload-file hello2.txt sftp://mediaflux.researchsoftware.unimelb.edu.au/Volumes/proj-demonstration-1128.4.15/hello2.txt


Example - using sshfs

GNOME Files on Ubuntu 18.04

Open GNOME Files and click Other Locations on the left panel.  In the address field at the bottom, type in the connection string using either port 22 or 9003:

  • sftp://mediaflux.researchsoftware.unimelb.edu.au:22
  • sftp://mediaflux.researchsoftware.unimelb.edu.au:9003

And click Connect.


Put in your credentials, based on the Username Format section and click Connect.


You will see the network drive mounted in the left panel, and you can navigate to your project, under /Volumes.

Example - rclone for transfers to or from cloud storage

Rclone is a tool for transferring or syncing files to cloud storage.  Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.  Rclone has cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat.

You can transfer data to or from Mediaflux over both the SFTP and SMB protocols.  Some endpoints that might be of interest to University users are SharePoint, OneDrive, Google Drive, S3 and CloudStor.

We have a detailed page on rclone with examples transferring data from many cloud storage endpoints.


  • No labels