Object Storage (S3-compatible)

Owned by Andy Tseng
Last updated: Jul 31, 2024 by Scott LewisVersion comment
3 min read


Overview

Our S3-Compatible Object Storage service provides S3-style access to files - typically through either a Rest API, or a dedicated S3 client program.

When is S3-Compatibile Object Storage right for you?

S3-Compatible Object Storage is significantly different to more traditional forms of file storage (eg. shared network drives). Therefore, its use is only recommended when you're using a service (or piece of software) that's been specifically written for the S3 API.

If you are unfamiliar with working with an S3 storage environment, then we recommend considering an alternative storage platform.

What is meant by 'S3-Compatible'?

Unlike many S3 environments (which rely on Amazon Web Services {AWS} ), our system is powered by Ceph's RADOS Gateway (RGW).
We are therefore able to provide a great many features that are offered by Amazon, yet a handful of Amazon's RESTful S3 API calls are not supported. For a full list of supported S3 features, please refer to the following: https://docs.ceph.com/en/latest/radosgw/s3/.

Applying for new storage

Any UoM Researcher is eligible to request new S3-Compatible object storage, via the links below:

Accessing your S3-Compatible Object Storage

Method 1: via an API library

The object storage gateway is designed to be accessed programmatically. Please refer to the RGW API Documentation for instruction on using most popular S3 API libraries. For specific API examples that have been tested by the object storage team, please refer to the following:

  • Boto 3 - Using Python Boto 3 library

Method 2: via an S3-compatible client

Most S3-compatible clients can be used to access the object storage gateway. The object storage team only official support the following clients:

  • AWS CLI (AWS Command Line Interface) - AWS's command-line client for Windows, Mac and Linux

  • S3cmd - command-line interface to S3 for Linux and Mac

  • S3 Browser - a graphical S3 interface for Windows computers only.

  • RClone - a utility offering both Graphical (GUI) and command-line (CLI) options available on Windows, Linux and Mac.
    (Tip: On Windows computers, S3 buckets may be 'mounted' {akin to a normal network drive} after rclone + winFsp are installed. See the rclone mount documentation for more information).  

Multiple Buckets

A single S3 bucket (and single corresponding access keypair) will be administered per storage request.
If additional buckets are required: then please lodge an additional Research Data Storage request for each required bucket. (Users will not be able to manually create additional buckets of their own accord).

Creating a new folder (aka. directory structure) within a bucket

  • Via a Graphical interface (eg. S3 Browser, RClone WebUI): simply locate + click on the 'new folder' button.
  • Via a command-line client: Most Command-Line S3 clients operated via a series of put (upload) or get (download) commands. In order to create a new 'folder' when uploading a file, simply insert a forward-slash character ( / ) , followed by your new directory name , into your upload path.
       
      For example (using S3cmd): 
                                 s3cmd -c mybucket.cfg put results1.txt s3://mybucket/allresults/results1.txt
           (this uploads the results1.txt file into a new folder, called 'allresults'). 

Sharing Data

S3 data is less secure than our other storage platforms. For example: the University VPN is NOT REQUIRED in order to access these buckets.
  It is therefore essential that researchers:
    (a) are 100% certain that this S3-Compatible Object storage is right for them - and
    (b) it is paramount that any S3 bucket credentials are stored securely.

The s3 platform is also not designed for regular sharing of objects (ie. files), and we strongly recommend utilising an alternative platform if file sharing is desired. There are some very limited ways that S3 objects might be shared, but these are unsupported. For information on making your data available to others, please refer to the following article: Sharing Ceph-S3 data.

Backup of S3-Compatible data

Our S3-Compatible Object Storage is operated by the University of Melbourne, in the University data centre. Please be aware that the data is only housed in one geographical region.

Although we employ hardware redundancy (to protect against hardware failure), there is no official data backup system for S3. We can attempt to backup S3 data on request, however the S3 platform has specifically been labelled as 'unsupported' by the University's primary recoverability vendors. Therefore, we can not guarantee recovery of deleted S3 data.

API Specification