How to Replicate FlashBlade S3 Objects to Oracle Cloud Infrastructure (OCI) Object Storage

Introduction

The popularity and adoption of Object Storage continues to grow, with more and more applications and databases using or having an option to utilise Object Storage.

I have previously shared a number of posts discussing S3 including:

• Performing Oracle RMAN backups to S3 to Amazon S3 Object Storage
• Performing Oracle RMAN backups to SS to on-premises FlashBlade S3 Object Storage
• How to access OCI Object Storage from your on-premises Oracle 19c Database

In this post I will show how we can replicate object data from an on-premises Pure Storage FlashBlade Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket.

Potential Use Cases

There are a number of reasons why you may want to replicate on-premises data to a Cloud Object store, including:

• Data Protection
  • Replicating Oracle RMAN backups to Cloud Object Storage
• Data Archiving
  • Replicating on-premises Oracle RMAN backups to Cloud Archive Object Storage
• Database Cloud Migration
  • Transferring on-premises data to Cloud Object Storage for migration using for example:
    • SQL Developer Data Import Wizard
    • SQL Developer Migration Workbench
    • Oracle Data Pump
    • Database Links
    • DBMS_CLOUD package
    • External Tables
• Data Warehouse and Data Lake workloads
  • Transferring on-premises data for use with data warehouse and data lake solutions
• Data Mobility
  • Replicating data for the creation of Cloud development and test environments.

Benefits of Object Storage

Using Oracle Object Store as a staging area for data loading or data access has a number of benefits, these include:

• Simplifying architectures; simplifying ELT/E-TL on Object Storage
• Options to Query-in-Place from ADM
• Object Storage is independent from the data processing engines
• Less expensive storage costs

Oracle Cloud Infrastructure (OCI)

Before we start, if you don’t already have an OCI account, you may want to consider creating an account. The OCI Always Free Service provides a number of free services including 10 GB Standard Object Storage and 10 GB Infrequent Access Object Storage.

Ok, let’s go!

Create OCI Bucket

Logon to Oracle Cloud Infrastructure (OCI) and from the Dashboard select Object Storage or via the hamburger menu navigate to Storage -> Object Storage -> Buckets.

Select your required Compartment, and click Create Bucket

Provide a Bucket Name, and confirm storage options, for this post I will use the Standard Default Storage Tier. OCI also provides an archive storage tier for long-term storage which may be of interest as it is one-tenth of the cost of Standard object storage

Please Note: you need to enable Object Versioning as this is required for FlashBlade replication.

Note: Bucket names must be unique, if you have used the name even in another Compartment expect to see an error message.

By default the bucket is Private, however if required you can change the visibility by clicking on the 3 vertical dots on the right hand side and updating it to Public.

Customer Secret Key

Before we can replicate to our OCI private bucket from the on-premises Pure Storage FlashBlade we need to provide authentication.

To do this from the OCI dashboard, Click on your Profile icon (top right), navigate to: 

User Settings -> Customer Secret Keys and click Generate Secret Key providing a descriptive name.

Enter secret key Name

Once generated you will need to select Copy, this will be the aws_secret_access_key, which we will be using later.

Now, click on the newly created Secret Name to obtain the aws_access_key, again save this for later.

Determine Object Storage Namespace Name

From the OCI dashboard, Click on your Profile icon (top right), navigate to Tenancy

The Object Storage Namespace and default S3 compartments should be visible.

Alternatively, you can use the OCI command line interface, for example.

% oci os ns get
{
  "data": "<Object Storage Namespace>"
}

Test OCI Object Storage URL

To test access to the OCI Object Storage Bucket we can use the AWS s3api command line interface.

Update your local ~/.aws/credentials file with a profile and query the OCI Object Storage URL, for example.

% aws s3api list-buckets --query "Buckets[].Name" --endpoint-url https://<Object Storage Namespace>.compat.objectstorage.uk-london-1.oraclecloud.com  --profile oci
[
    "FlashBlade"
]

The OCI Bucket URL format is https://<Object Storage Namespace>.compat.objectstorage.<region>.oraclecloud.com

A full list of OCI Object Storage endpoints is available in the OCI Documentation.

Pure Storage FlashBlade

Now the FlashBlade side.

Create S3 Target Connection

Logon to the Pure Storage FlashBlade and navigate to Storage -> S3 Target Connections

Click the “+” to the right of Connect S3 Target

Enter Name and Address

Add Remote S3 Credentials

Navigate to Protection -> Object Replication -> Remote Credentials

Click the “+” to the right of Create Remote Credentials

Select the OCI Target from the pick list for Remote Array or Target

Enter a Name for the connection, for example: OCI-UK-London1

Paste the OCI Access Key ID and Secret Key we created earlier.

Click “Create”

Setup Bucket Replication to OCI Object Storage

Navigate to Protection -> Object Replication -> Bucket Replica Link

Select Local Bucket Name, Remote Array or Target, Remote Credential and provide the OCI Remote Bucket name.

From the FlashBlade we can now see we have established a Bucket Replica Link.

Test Replication

To test the Replication Link upload a test file to the FlashBlade bucket, for example.

$ aws s3api put-object --bucket oracle-bucket --key VagrantAPEX.mp4 --body /home/oracle/osbws/VagrantAPEX.mp4 --endpoint-url ${ENDPOINT} --profile default

OCI Object Storage

And returning to OCI, navigate to OCI Object Storage -> Bucket Details and confirm replication.

---

Summary

In this post I have shared some of the use cases and reasons why we may want to replicate on-premises data to the Cloud. I have also demonstrated how we can replicate an on-premises Pure Storage FlashBlade Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket.

Hope you found this post useful.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]