unimelb-mf-download

Owned by Andy Tseng
Mar 21, 2024
1 min read

This is a command-line Java application that you can use to efficiently download your data (folders recursively or individual files) from Mediaflux. Installation instructions are available in the Mediaflux Unimelb Command-Line Clients page.

You will need to know where your data are in Mediaflux (the last argument of the command) and where you want to locate (--out) the data on the destination computer.

  • This client can download files in parallel (--nb-workers). There is no magic in this, it will only go faster if there is sufficient network capacity. Therefore, please don't use more than 4 download threads. You may even find that if the network is heavily congested, 4 threads is no faster than 1. You may have to experiment a little to find the optimum.
  • The client can run in daemon (background) mode wherby it will keep on downloading data as it arrives in Mediaflux.
  • The client can synchronise from Mediaflux to the local file system and delete files that no longer exist in Mediaflux from the local file system.
  • Please see all command-line arguments with the --help switch

Examples

Example 1

Download data with one worker thread and skip pre-existing files, checking for files that pre-exist by their name and size only.

unimelb-mf-download --mf.config ~/.Arcitecta/mflux.cfg --out /data/projects/punim0058 /projects/proj-myproject-1128.1.59/12Jan2018

Example 2

Download data with four worker threads and we overwrite pre-existing files, and we check files pre-exist by their name and size and checksum (slower but safer). We don't need to specify the path to the config file as the client will look for it in the standard places.

unimelb-mf-download --overwite --csum-check --nb-workers 4 --out /data/projects/punim0058 /projects/proj-myproject-1128.1.59/12Jan2018