Data Mover Shareables and Tokens

Owned by Andy Tseng
Mar 21, 2024
3 min read

1 Introduction

A shareable is a Mediaflux entity that is used by the Data Mover tool to upload (an upload shareable) data to Mediaflux or download (a download shareable) data from Mediaflux. Inside the shareable is what is called the token.  The shareable token is used to construct URLs (links) that are emailed to people.  When clicked on, those links will start the Data Mover tool (or install it), and present GUIs that enable the upload or download to take place.


2 Types of Shareable

2.1 General

General shareables can be created (and emailed) by all Mediaflux users via the Mediaflux Explorer tool. In this way, a user can share their data in Mediaflux with other colleagues (download) or other colleagues can upload data to the Mediaflux user's Mediaflux workspace to share it with them.

2.2 Instrument

A special kind of shareable is one that is created as part of our specialised Instrument Upload workflow.  This type of shareable is configured so that

  • It always uploads to a specific workspace (project) in Mediaflux (usually each instrument has its own workspace). This location is only directly accessible by the instrument staff.
  • The upload GUI has a particular set of entry boxes, including keywords and the email address of the user for whom the data were acquired.
  • After upload, a handler is executed which creates a download shareable and emails it to the end user for whom the data were acquired.  That user can then download the data with the Data Mover tool (the shareable provides the indirect access).

The token from the instrument shareable is located in a settings.xml file  in the instrument acquisition computer.  On startup, the  Data Mover tool reads the token from that file. An instrument shareable is persistent, and it can be deployed in multiple computers.  For example, it may be that the acquisition workflow is

  • Acquire data
  • Send to pre-processing workstation
  • Pre-process
  • Upload and dispatch

In this case, the settings.xml file and token can be deployed on the pre-processing computers (rather than the acquisition computer).  The token is in no way specific to a particular computer, just to a specific process and end-point in Mediaflux.

Instrument shareables are created only by the Data Solutions Team of Research Computing Services.  Generally this is the result of a consultation process which establishes if the Data Mover Instrument upload capability is appropriate for your needs.



3 Configuration File

An optional XML configuration file called settings.xml can be created in the following folder .Arcitecta/DataMover (beneath your home directory).  This file contains various Data Mover behaviours.

  1.  The configuration file (plain text XML) can have the following elements:
    1. token : a shareable token that the Data Mover will utilise rather than you having to paste it in to the GUI.  This is primarily used for instrument uploads.
    2. software-updates : allowed values are none, published, and all which controls whether the Data Mover will present software updates.  The default, if not set (and the recommended setting) is published which will only present software updates marked as production ready. If you specify all then pre-release updates would also be presented (we do not recommend this).
    3. upload-start-time : specifies a time (HH:MM or HH) before which uploads cannot commence (unless manually started).  At the specified time, all queued upload tasks will commence.
    4. download-directory : specify the default path for downloads. 
    5. log-rotate-size : specifies the size (in MiB) of log files (in the .Arcitecta/DataMover/logs directory) in which errors (only) are logged.  Defaults to 10MiB.  This parameter is generally not of end-user interest.

  2. Example of XML configuration file contents showing possible parameters (with pretend values) :

    <settings>
<token>dhshsdghahgaghaklasouidsdetyhkdqeljcadfnc</token>
<software-updates>stable</software-updates>
<download-directory>/Users/sclaus/Downloads</download-directory>
<upload-start-time>18:00</upload-start-time>
<log-rotate-size>100</log-rotate-size>
</settings>