Boto 3

Owned by Andy Tseng
Mar 21, 2024
2 min read

Boto is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access. Boto can also be used for accessing UniMelb object storage gateway.

For installation and quickstart, please refer to: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html.

Authenticate

To authenticate against to Unimelb object storage gateway:

boto3quickstart.py
import boto3
 
 
def authenticate():
    """Return the authenticated client"""
    return boto3.client(
            's3',
            aws_access_key_id=ACCESS_KEY,
            aws_secret_access_key=SECRET_KEY,
            endpoint_url='https://objects.storage.unimelb.edu.au',
            use_ssl=True,
            )
 
 
client = authenticate()

Use S3Transfer

S3Tranfer module can automate the multipart chunking and multithreaded uploading, making it easy to upload files to the object storage gateway.

boto3transfer.py
from boto3.s3.transfer import S3Transfer, TransferConfig
 
KB = 1024
MB = KB * KB
GB = MB * KB
 
# CONCURRENCY TUNING
MULTIPART_THRESHOLD = 15 * MB
MAX_CONCURRENCY = 30
 
 
def configure():
    """Return configuration for S3Transfer"""
    return TransferConfig(
        multipart_threshold=MULTIPART_THRESHOLD,
        max_concurrency=MAX_CONCURRENCY
        )
 
client = authenticate()
config = configure()
transfer = S3Transfer(client, config)
transfer.upload_file(FILENAME, BUCKET_NAME, FILENAME)

Adding A Progress Percentage

To add an upload percentage, use the following code snippet:

class ProgressPercentage(object):
    """Display upload percentage"""
    def __init__(self, filename):
        self._filename = filename
        self._size = float(os.path.getsize(filename))
        self._seen_so_far = 0
        self._lock = threading.Lock()
 
    def __call__(self, bytes_amount):
        # To simplify we'll assume this is hooked up
        # to a single filename.
        with self._lock:
            self._seen_so_far += bytes_amount
            percentage = (self._seen_so_far / self._size) * 100
            sys.stdout.write(
                "\r%s  %s / %s  (%.2f%%)" % (
                    self._filename, self._seen_so_far, self._size,
                    percentage))
            sys.stdout.flush()
 
transfer.upload_file(FILENAME, BUCKET_NAME, FILENAME,
                     callback=ProgressPercentage(FILENAME))