How to Upload large files to OCI Object Storage using the AWS CLI

In this short blog post I am going to share how to upload a large file to Oracle Cloud Infrastructure (OCI) Object Storage using the AWS CLI to perform a concurrent multipart upload.

For this post I will use a large 4GB file to demonstrate an upload.

Let’s start by creating a file called bigfile using the Linux dd command, for example.

dd if=/dev/random of=bigfile bs=1024 count=4096000

AWS Configuration Values

The AWS CLI documentation details a number of configuration options which can be set for the aws s3 and aws s3api commands.

Update AWS S3 multipart_threshold, multipart_chunksize and max_concurrent_requests values using the AWS CLI utility, the AWS CLI maintains the ~/.aws/config file without the need edit it directly.

• multipart_threshold – The size threshold the CLI uses for multipart transfers of individual files.
• multipart_chunksize – When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.
• max_concurrent_requests – The maximum number of concurrent requests.

By specifying ––profile oci I can update my dedicated Oracle Cloud AWS configuration profile, ensuring other profiles are not impacted.

aws configure set s3.multipart_threshold 64MB --profile oci
aws configure set s3.multipart_chunksize 8MB --profile oci
aws configure set s3.max_concurrent_requests 300 --profile oci

If we now inspect the ~/.aws/config file we can see it’s been updated.

[profile oci]
s3 =
   addressing_style = path
   multipart_threshold = 64MB
   multipart_chunksize = 8MB
   max_concurrent_requests = 300

Object Storage Namespace

Use the OCI command line interface to determine your OCI Object Storage Namespace using oci os ns get, for example.

% oci os ns get
{
  "data": "<Object Storage Namespace>"
}

Alternatively, logon to your Oracle Cloud account, and navigate to Profile -> Tenancy and look for the Object storage namespace in the Object storage settings.

Object Storage Upload

Now we have captured the OCI Object Storage namespace we can create an OCI URI, using the following format:

https://<Object Store Namespace>.compat.objectstorage.<region>.oraclecloud.com

For example,

https://lrqsxxxxxx.compat.objectstorage.uk-london-1.oraclecloud.com

Now let’s upload the bigfile to an OCI Object Storage bucket called testbucket1.

% aws --endpoint-url https://${ENDPOINT} s3 cp /Users/rekins/objectdata/bigfile s3://testbucket1/bigfile --profile oci
upload: ./bigfile to s3://testbucket1/bigfile

We can see the bigfile has been successfully uploaded by navigating to Object Storage -> Buckets -> testbucket1.

The WebUI shows the Name, Version, Size and Storage Tier.

Using the OCI CLI object list-object-versions we can also the object version details within a bucket.

oci os object list-object-versions --namespace <object_storage_namespace> --bucket-name <bucket_name>

For example, providing the OCI Object Storage namespace and bucket name.

$ oci os object list-object-versions --namespace  <Object Storage Namespace>  --bucket-name testbucket1
{
  "data": [
    {
      "archival-state": null,
      "etag": "fb2dff24-de03-40de-8350-39e8d585545e",
      "is-delete-marker": false,
      "md5": "33kTjYeVLmAj8x3vKNrXPg==-500",
      "name": "bigfile",
      "size": 4194304000,
      "storage-tier": "Standard",
      "time-created": "2023-07-25T11:13:41.144000+00:00",
      "time-modified": "2023-07-25T11:13:41.144000+00:00",
      "version-id": "7ea3acc2-908a-4efe-baca-bdce1ef96d5b"
    }
  ],
  "prefixes": []
}

We can also confirm the size of the bigfile uploaded using the AWS CLI aws s3api head-object syntax, for example:

$ aws --endpoint-url https://${ENDPOINT} s3api head-object --bucket testbucket1 --key bigfile --profile oci
{
    "AcceptRanges": "bytes",
    "LastModified": "2023-07-25T11:13:41+00:00",
    "ContentLength": 4194304000,
    "ETag": "\"df79138d87952e6023f31def28dad73e-500\"",
    "VersionId": "7ea3acc2-908a-4efe-baca-bdce1ef96d5b",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

Summary

In this post I have shared how to upload a large file to an Oracle Cloud Infrastructure (OCI) Object Storage bucket using the AWS CLI aws s3api.