Introduction

The Data Mover tool has both a Graphical User Interface (GUI) and a Command Line Interface (CLI).  The GUI is more fully featured and easy to use, but the CLI is important for environments where the user working with the Data Mover does not have a windowing environment.  This is most common in server and high-performance computing Unix systems.

The GUI version of the Data Mover offers a light-weight process for the user to update it when new releases are made available (from Arcitecta, the vendor).  However, this is not the case with the CLI version.  The only way to update the CLI version is to update the installation, and so, if possible, it should be centrally maintained.

Script wrappers are provided with the distribution to aid execution of the CLI version.  These wrappers set up the environment and use the self-contained version of Java that is installed with the Data Mover.

The wrapper script that you are going to execute is called mediaflux-data-mover-cli. 

Installation

The CLI is a part of the standard installation. 

The wrappers are found in the following locations (relative to the root of the installation) and should be included in your PATH environment variable.

• Linux
  • bin/mediaflux-data-mover-cli
• Mac OS

  • Contents/MacOS/mediaflux-data-mover-cli

• Windows
  • mediaflux-data-mover-cli.cmd

Command Line Arguments

The examples below are drawn from a Unix environment where the folder containing the mediaflux-data-mover-cli wrapper is assumed to be in your PATH (see above).  The  Unix prompt is represented by the % sign.

Help

Executing the CLI with the switch -help will print out all of the switches, options  and arguments.

% mediaflux-data-mover-cli -help

Basic Syntax

The basic syntax is

% mediaflux-data-mover-cli [Switches] [Options] [Token] [Argument]

where the Switches, Options  and Argument control the behaviour.  

Arguments

• Generic Switches
  • -download : run the CLI Data Mover to execute a download
  • -upload : run the CLI Data Mover to execute an upload 
  • -follow-symlinks : if uploading, will attempt to resolve symbolic links
  • -help : print out all help
  • -info : print out information about the shareable you are working with
  • -mkdir  : if doing a download, and the parent directory does not exist, create it
  • -v : display additional information (verbose) as the transfer proceeds
  • -version : display the version and exit
  • -watch : allow the upload shareable to keep watching the upload source directory for any changes
• History Related Switches.
  • The GUI version helps you manage (inspect, download csv,  delete) your upload/download tasks (every execution is a stored task). So there are some CLI switches to help with this.  However, please note that execution of the CLI form of the Data Mover does not generate new history entries (if completed).  It will generate new entries if the task is incomplete (hence the switches history.resume and history.retry).  Otherwise, the other history switches pertain to pre-existing history entries generated by the GUI version.

  • The switches are

    • -history.completed.all.delete : delete all history entries

    • -history.delete : delete a specified history entry

    • -history.describe  : describe all previous transfers.
    • -history.list  : list all previous transfers.
    • -history.resume  : resume execution of an incomplete history entry.
    • -history.retry  : retry failed execution of an incomplete history entry.
• Options
  • -exists [Value]: what to do if the target directory on download pre-exists. Choose from one of [rename, update, overwrite, skip, fail]
  • -csv [Path]: If specified, the full path to a CSV file report of the upload/download
  • -settings [Path] : Path to Data Mover settings file (defaults to $HOME/.Arcitecta/DataMover)
  • -upload-args : A semi-colon separated set of arguments required for upload.  
    • These are deployment specific, however, for University of Melbourne instrument operators with an  instrument upload shareable, the standard upload arguments are share-with (the email address of the person who receives an email with URLs to download their data), keywords (set on the manifest upload asset), name-suffix (appended to the transactional namespace name), email-subject-suffix (this is only available if your shareable has explicitly been enabled with email-subject-suffix) and data-note (note about the data being uploaded included in the email to the recipient user. This is only available if your shareable has explicitly been enabled with data-note). Examples are as follows (note the double quotes):
      • -upload-args "share-with=santa.claus@northpole.com;keywords=fish,carp;name-suffix=aquatic"
      • -upload-args "share-with=santa.claus@northpole.com;keywords=fish,carp;name-suffix=aquatic;email-subject-suffix=GOOD"
      • -upload-args "share-with=santa.claus@northpole.com;keywords=fish,carp;name-suffix=aquatic;email-subject-suffix=GOOD;data-note=this is a test"
• Token
  • The  Token (or shareable) is supplied to you by somebody (it's a long string) and is required for uploads and downloads, but not for any of the history.* switches.
• Argument depends on the switches as follows
  • If switch -upload is given the value of Argument is the source path of the data to upload and the Token is required
  • If switch -download is given the value of Argument is the target  path for the download and the Token is required
  • If switch -history.completed.all.delete is given the value of Argument is 'yes' (to confirm) and Token is omitted.
  • If switch -history.delete is given the value of Argument is the ID of the entry you want to delete and Token is omitted
  • If switch -history.list is given the values of Argument and Token are omitted 
  • If switch -history.describe is given the value of Argument is the ID of the entry you want to describe and Token is omitted.

Examples

In these examples, we will pretend the upload token (shareable) value is 2iqkvmijugyydjrqr2jq8g and the download token (shareable) value is jrt1nw0rg3dq2k70l23 (in reality they would be longer).  The upload token (shareable) specifies where in the Mediaflux the data will go - the uploader has no visibility on that.  Similarly, the download token (shareable) specifies what data you are going to download.

Note that you do have to be able to paste in the token (shareable) which is a long string - you will have it in your email generally as an end user. If you cannot read your email via a command-line client on the platform you are going to operate the DM CLI then you need some other way of accessing the shareable. The easiest approach is to put it in a text file, transfer that to the platform and then copy and paste out of that file with command-line tools (e.g. on Unix, 'cat' or 'more').

% mediaflux-data-mover-cli -upload -csv ./upload.csv 2iqkvmijugyydjrqr2jq8g /data/uploads/data1

% mediaflux-data-mover-cli -download -csv ./download.csv -exists rename jrt1nw0rg3dq2k70l23 /data/downloads

% mediaflux-data-mover-cli -history.list

% mediaflux-data-mover-cli -history.describe 3

% mediaflux-data-mover-cli -history.completed.all.delete

% mediaflux-data-mover-cli -history.delete 3