Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »


Overview

Our S3-Compatible Object Storage service provides S3-style access to files - typically through either a Rest API, or a dedicated S3 client program.

When is S3-Compatibile Object Storage right for you?

S3-Compatible Object Storage is significantly different to more traditional forms of file storage (eg. shared network drives). Therefore, its use is only recommended when you're using a service (or piece of software) that's been specifically written for the S3 API.

If you are unfamiliar with working with an S3 storage environment, then we recommend considering an alternative storage platform.

What is meant by 'S3-Compatible'?

Unlike many S3 environments (which rely on Amazon Web Services {AWS} ), our system is powered by Ceph's RADOS Gateway (RGW).
We are therefore able to provide a great many features that are offered by Amazon, yet a handful of Amazon's RESTful S3 API calls are not supported. For a full list of supported S3 features, please refer to the following: https://docs.ceph.com/en/latest/radosgw/s3/.

Applying for new storage

Any UoM Researcher is eligible to request new S3-Compatible object storage, via the links below:

Accessing your S3-Compatible Object Storage

Method 1: via an API library

The object storage gateway is designed to be accessed programmatically. Please refer to the RGW API Documentation for instruction on using most popular S3 API libraries. For specific API examples that have been tested by the object storage team, please refer to the following:

  • Boto 3 - Using Python Boto 3 library

Method 2: via an S3-compatible client

Most S3-compatible clients can be used to access the object storage gateway. The object storage team only official support the following clients:

  • AWS CLI (AWS Command Line Interface) - AWS's command-line client for Windows, Mac and Linux

  • S3cmd - command-line interface to S3 for Linux and Mac

  • S3 Browser - a graphical S3 interface for Windows computers only.

  • RClone - a utility offering both Graphical (GUI) and command-line (CLI) options available on Windows, Linux and Mac.
    (Tip: On Windows computers, S3 buckets may be 'mounted' {akin to a normal network drive} after rclone + winFsp are installed. See the rclone mount documentation for more information).  

Multiple Buckets

A single S3 bucket (and single corresponding access keypair) will be administered per storage request.
If additional buckets are required: then please lodge an additional Research Data Storage request for each required bucket. (Users will not be able to manually create additional buckets of their own accord).

Creating a new folder (aka. directory structure) within a bucket

  • Via a Graphical interface (eg. S3 Browser, RClone WebUI): simply locate + click on the 'new folder' button.
  • Via a command-line client: Most Command-Line S3 clients operated via a series of put (upload) or get (download) commands. In order to create a new 'folder' when uploading a file, simply insert a forward-slash character ( / ) , followed by your new directory name , into your upload path.
       
      For example (using S3cmd): 
                                 s3cmd -c mybucket.cfg put results1.txt s3://mybucket/allresults/results1.txt
           (this uploads the results1.txt file into a new folder, called 'allresults'). 

Sharing Data

By default, S3 bucket data is private – that is, a bucket's access is restricted to the holder of its credentials (keypair). Therefore, it's essential that your bucket credentials be stored securely.
For information on making your data available to others, please refer to the following article: Sharing Ceph-S3 data.

Backup of S3-Compatible data

Our S3-Compatible Object Storage is operated by the University of Melbourne, in the University data centre. Please be aware that the data is only housed in one geographical region.

Although we employ hardware redundancy (to protect against hardware failure), there is no official data backup system for S3. We can attempt to backup S3 data on request, however the S3 platform has specifically been labelled as 'unsupported' by the University's primary recoverability vendors. Therefore, we can not guarantee recovery of deleted S3 data.

API Specification

  • No labels