/
Data Mover Reference

Data Mover Reference

Apr 09, 2024

Introduction

Please see the Terminology section first.

 

The Mediaflux Data Mover tool is used to move data to and from Mediaflux.   It has two primary capabilities: 

  1. download from mediaflux - via a  download link which can also be emailed to someone to use.  

  2. upload to mediaflux - via an upload link which can also be emailed to someone to use.

These two basic capabilities are further combined at the University of Melbourne into two additional specialised capabilities:

 

The Data Mover also has the following attributes:

  • Secure - it utilises the HTTPS protocol so that data are encrypted in transport.  

  • Efficient - it moves data in parallel to maximise throughput

  • Robust - it has built in integrity checking ensuring source and destination data are identical including the number of files uploaded. 

  • Restartable - it can restart where it left off if a transfer fails for some reason (e.g. a network error).

  • Automatic - it will self-install and self update as needed - the Data Mover tool is actually a fully self-contained Java application.

The Data Mover is formally supported for

  • macOS

  • Windows 10

    • It will likely work on other older Windows platforms

    • Windows 7 has been demonstrated to work

  • Linux 

    • It is provided as a Java bundle (Java included)  - no particular installer, so it is very portable to most flavours of Linux so long as OpenJDK 17 is supported (see below)

The Data Mover tool (client application) is  free to use for all users. The user does not need a Mediaflux system to use it - it's just like the Zoom video conferencing tool (you don't need a Zoom system to use the Zoom client tool).

 

 

Installing and Configuring the Data Mover

You can fetch the Data Mover manually and install it, or you can wait until you receive a Shareable (for upload or download). Currently for Linux it is only possible to install manually. Clicking on the Shareable link will initiate the download and installation.

The Data Mover is formally supported for macOS, Windows 10 and Linux.  Data Mover is built using Java, so you can see the system requirements for OpenJDK 17.0.2 for a list of supported platforms.  In addition to those listed, we know that Windows 7 does currently work.

 

  1. Download the Data Mover manually and Install 

    1. Preparation

      1. When doing a new install, remove any .jar files that you may have downloaded (part of the auto-update process see section 3) in your .Arcitecta/DataMover/updates folder (under your home directory).

      2. If you don't remove these, they will conflict with the newly installed version.

      3. Uninstall the old version of the app (or just install over the top of it).

      4. Do not rename the old application to something else as this will cause conflicts. If you want to keep an earlier version, then zip up the old application first.

    2. Download 

      1. The URL is https://mediaflux.researchsoftware.unimelb.edu.au/mflux/data/mover/index.html

      2. From there you can download the version for your operating system (OSX, Windows 10 and Linux)

      3. If you'd like to download a specific version with a tool like curl or wget the URLs are:

        1. macOS : https://mediaflux.researchsoftware.unimelb.edu.au/mflux/data/mover/installers/mac/Mediaflux%20Data%20Mover.dmg

        2. Windows : https://mediaflux.researchsoftware.unimelb.edu.au/mflux/data/mover/installers/windows/Mediaflux%20Data%20Mover.msi

        3. Linux : https://mediaflux.researchsoftware.unimelb.edu.au/mflux/data/mover/installers/linux/mediaflux-data-mover.zip

    3. Install

      1. macOS

        1. Double click on the file Mediaflux Data Mover.dmg file to install

          1. If you encounter an issue where macOS complains that the .dmg file is damaged, you can resolve that by removing the extended attributes.  Start the Terminal (Command line) Application, change directory to the directory where you downloaded the .dmg (in the example it's the Downloads directory) and issue a command like this

            cd ~/Downloads xattr -cr Mediaflux\ Explorer-1.5.0.dmg

        2. Using the GUI that is presented, drag the Data Mover to /Applications

      2. Windows 

        1. Double click on the file Mediaflux Data Mover.msi to install.

      3. Linux

        1. Unpack the mediaflux-data-mover.zip file.  Note that the resulting directory mediaflux-data-mover must not be stored in a directory that contains bin as one of its elements due to a Linux Java bug (avoid /usr/local/bin or ~/bin for example).

        2. Set the binary to be executable with a command like:

          chmod +x mediaflux-data-mover/bin/mediaflux-data-mover

        3. Optionally, add  <path>/mediaflux-data-mover/bin to your PATH variable

        4. Optionally, read the README.txt for instructions on setting up your web browser to automatically open arcio links with the Data Mover\

      4. Check

        1. After you install, start up the Data Mover and make sure the version running (see bottom left of GUI) is the version you expect.

  2. Use a Shareable link to download and install

    1. Paste a Shareable (upload or download) into a browser

    2. It will be detected whether you have the Data Mover already or not. If not, it will download the correct operating system installer (Windows 7/10, MacOS, Linux)

    3. Execute the installer (for whatever platform you are on as in Section 1 above)

  3. An optional XML configuration file called settings.xml can be created in the  .Arcitecta/DataMover (beneath the home directory) folder. This file controls varies Data Mover behaviour. Details are found on this page.

 

 

Updating the Data Mover via the GUI

You will download the initial install of the DataMover from our Mediaflux server (see section 2 above).  However, thereafter, the Data Mover tool GUI will offer you updates when they are available (from Arcitecta, the vendor).  When you start it, there will be a prompt in the bottom right of the main screen where you can update and relaunch the Data Mover.  

Please note that Data Mover configuration file (settings.xml)is not affected by update processes.

In general, the update process is  light-weight - this means that all the update does is download a new Java .jar file and locate it in the .Arcitecta/DataMover/updates folder (under your home directory).  However, occasionally, the full application needs to be updated in a heavy-weight process.  This accommodates the occasional need for the application to be repackaged (e.g. a new version of Java). This multi-step heavy-weight process is really targeted at external end-users who may have installed the Data Mover to receive data from a University of Melbourne instrument (we don't know who they are so we cannot write to them).  

Otherwise, it is easier to

  • uninstall the Data Mover and remove any files in the .Arcitecta/DataMover/updates folder

  • download and install the current release from our server (see section 2).

If you do follow the heavy-weight update process, for whatever reason, and this will happen between versions 1.0.11 (the initial deployment) and 1.1.15 (the next release), then carefully  follow the on-screen instructions.  This is the outline of the steps to go from v 1.0.11 to v 1.1.15

  • Light-weight update to v 1.0.13

  • Manually quit and restart the Data Mover (the relaunch button does not work in this release) v 1.0.13

  • Heavy-weight update to v 1.1.10

    • download the full package

    • quits Data Mover

    • prompts you to install it 

  • Manually start the newly installed v 1.1.10

  • Light-weight update to v 1.1.13

  • Relaunch (now running v 1.1.13)

  • Light-weight update to v 1.1.15

  • Relaunch v 1.1.15

  • Check the right version is running