Download and upload with aterm

Owned by Andy Tseng
Mar 21, 2024
2 min read

You can use aterm commands to upload and download data over the HTTPS protocol.  This allows for secure, multi-threaded upload with good data integrity guarantees.  See Mediaflux Terminal (aterm) for information on acquiring and logging in to aterm.

Basic commands

When you first log in to aterm, you will be presented with the command prompt.  You can use the help command to get a list of available commands, and help commandname to get specific help on a command.  Some unix-like commands are available.  Some examples that are worth running to get started:

help ls
help cd
help pwd

In aterm, namespaces present just like directories on a regular filesystem.  On the University of Melbourne server, generally projects appear in the /projects namespace.

cd /projects
ls
cd proj-demonstration-1128.4.15

While the ls command allows you to work with relative paths, most commands in Mediaflux require absolute paths (which specify a location starting at the root, e.g. /projects/proj-demonstration-1128.4.15).  When your project is created, you will be advised what its path is.  You will only see namespaces that you have access to.

Ping

A really useful capability is the ability to ping the Mediaflux server to measure the bandwidth available from your client to the Mediaflux server, similar to the unix perf tool. Here is an example:

ping 100MB

Download

The download command allows you to download namespaces (directories) or assets (files) to your local filesystem.  It only downloads assets that have content (not zero-size).

help download

Upload (import)

The import command allows you to upload directories or files from your local machine to Mediaflux as new namespaces/assets.

help import

Examples

In the following examples:

  • -ncsr 2 means use two simultaneous threads (don't use too many you will exhaust your bandwidth)
  • -filename-collisions skip directs the command to skip files you already downloaded (so you can restart).
  • The last argument is the path on the local file system where the data will be saved.

Downloads

Recursively download an entire namespace (and its children).

download -ncsr 2 -namespace "/projects/proj-demo-1128.1.3/baskets" -filename-collisions skip "/Users/nebk/data/baskets"

Download assets from a particular namespace which were created more recently than 7 days ago (uses the where query clause)

download -ncsr 2 -where "namespace>=/projects/proj-demo-1128.1.3/baskets and ctime>='TODAY-7DAY'" -filename-collisions skip "/Users/nebk/data/baskets"

Uploads

Create a new namespace (directory) and recursively upload the contents of a local directory into it.

mkns "/projects/proj-demo-1128.1.3/panniers"
import -ncsr 2 -namespace "/projects/proj-demo-1128.1.3/panniers" "/Users/nebk/data/panniers"

Upload the contents of a local directory into an existing  namespace and compress the files into a zip archive (compression level 5) on-the-fly. So the result is a single asset holding all the recursive contents of the local directory.  You can download and unpack that asset with, e.g. unzip

import -archive zip:5 -ncsr 2 -namespace "/projects/proj-demo-1128.1.3/panniers" "/Users/nebk/data/panniers"


Downloading previous versions of an asset

You can download previous versions of assets using aterm commands.  For example, this command will download two versions of hello.txt.  It uses a TCL for loop.  This will work in the Java version of aterm (aterm GUI or aterm CLI) but not web aterm.

for {set a 1} {$a < 3} {incr a} { asset.content.get :id path=/projects/proj-demonstration-1128.4.15/hello.txt -version $a :out "hello.$a.txt" }