Performing Oracle RMAN backups to S3 – Part 1: Amazon S3

Background

Historically most of the conversations I have regarding Oracle Recovery Manager (RMAN) are around the use of NFS storage such as a Pure Storage FlashBlade, and Oracle Direct NFS (dNFS).

However, with more Oracle customers deploying databases in Oracle Cloud Infrastructure (OCI) and / or using Object Storage for backups I thought it was about time I had a look using RMAN with S3.

Preparation

Before we can start using the Oracle OSB (Oracle Secure Backup) Module for Oracle RMAN (Recovery Manager) backups we need to have the following to hand:

  • OTN account details
  • AWS Access Key ID and Secret Access Key.
  • The OBS installer (osbws_installer.jar)
    • Oracle 19c ships the osbws_installer.zip within the $ORACLE_HOME/lib directory.
    • Alternatively you can find the Oracle OSB Web Services Module for Amazon S3 here.
Oracle Secure Backup Cloud Module for Amazon S3

Create AWS S3 User Account

Logon to the AWS console and create a dedicated AWS user for your RMAN backups using the ‘Add User’ wizard, you can find this in the Identity and Access Management (IAM) area.

In this example I have specified Programmatic access, as this account will not be connecting via the console.

Create User

At step 2, create a group and attach the AmazonS3FullAccess policy.

Create Group

Complete the wizard to create the user account.

Check AWS S3 Bucket

From the AWS Console, select S3 and create a new bucket.

Create s3 Bucket

Test S3 Access

Install and configure AWS CLI to test access if not already available.

The AWS CLI provides a rich set of options which you read-up on here, we can list use the AWS to show our newly created bucket with the aws s3api list-buckets command.

$ aws s3api list-buckets --query "Buckets[].Name" --endpoint-url http://s3.amazonaws.com --profile aws
-------------------
|   ListBuckets   |
+-----------------+
|  rontestbucket  |    
+-----------------+

Prepare Oracle Environment

You will need to create an Oracle wallet in your Oracle Home is you don’t already have one. e.g.

$ cd $ORACLE_HOME/dbs
$ mkdir osbws_wallet

The Oracle Secure Cloud Backup Module requires a Java version of 1.7 or higher, you can check this with java -version e.g.

$ java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)

Unzip the Oracle 19c osbws_installer.zip or the file previously downloaded.

Please note the installer needs internet access.

The installer will download the library appropriate to the platform it is running on. It will also create the library configuration file and the Oracle Wallet where the S3 credentials are stored.

Let’s run the installer without any options to find a list of command line parameters.

$ java -jar osbws_install.jar
Oracle Secure Backup Web Service Install Tool, build 19.3.0.0.0DBRU_2019-04-17
No arguments supplied
Usage: java -jar osbws_install.jar
-AWSID:          AWS Access Key ID
-AWSKey:         AWS Secret Access Key
-IAMRole:        AWS IAM role name
-IAMRoleMetaUri: Metadata URI for the specified IAM role
-awsEndPoint:    non default host name
-awsPort:        non default HTTP/HTTPS port
-location:       Location to store backups
-useHttps:       setup HTTPS
-useSigV2:       setup authentication scheme
-walletDir:      Directory to store wallet
-trustedCerts:   SSL certificates to be imported
-import-all-trustcerts:
                 Import all certificates from Java truststore
-configFile:     File name of config file
-libDir:         Directory to store library
-libPlatform:    Platform of library to download
-lib-download-only:
                 Download library only
-proxyHost:      HTTP proxy host
-proxyPort:      HTTP proxy port
-proxyID:        HTTP proxy userid, if needed
-proxyPass:      HTTP proxy password, if needed
-argFile:        File name of arguments file
-help:           Print this usage information and exit

Oracle OSB Installer

The osbws_install Java utility will try to perform the following:

  • Install an osbws library file into ‘-libDir’ location e.g. ‘$ORACLE_HOME/lib’
  • Create a configuration file in ‘$ORACLE_HOME/dbs’
  • Create a wallet file in the ‘-walletDir’ location e.g. ‘$ORACLE_HOME/dbs/osbs_wallet’

Below is my Oracle Secure Backup script for AWS.

#!/bin/bash
export AWSID=<AWS ID>
export AWSKey=<AWS Secret Key> 

java -jar osbws_install.jar \ 
-AWSID ${AWSID} \ 
-AWSKey ${AWSKey} \ 
-walletDir ${ORACLE_HOME}/dbs/osbws_wallet \ 
-location ${location} \ 
-libDir $ORACLE_HOME/lib 

Ok, now let’s now install OBS

$ ./osbws_install_AWS.sh 
Install OSB Library
 Oracle Secure Backup Web Service Install Tool, build 19.3.0.0.0DBRU_2019-04-17
 Debug: os.name        = Linux
 Debug: os.arch        = amd64
 Debug: os.version     = 4.14.35-1902.3.1.el7uek.x86_64
 Debug: file.separator = /
 Debug: Platform = PLATFORM_LINUX64
 Debug: Verifying AWS account using endpoint s3.amazonaws.com
 Debug: Canoical Request:
...
 Oracle Secure Backup Web Service wallet created in directory /u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbws_wallet.
 Oracle Secure Backup Web Service initialization file /u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbwsDEMO1.ora created.
 Downloading Oracle Secure Backup Web Service Software Library from file osbws_linux64.zip.
 Debug: Temp zip file = /tmp/osbws_linux644593000752882300726.zip
 Debug: Downloaded 27721116 bytes in 19 seconds.
 Debug: Transfer rate was 1459006 bytes/second.
 Download complete.

OSB Database file

From the above we can see a osbw<ORACLE_SID>.ora has been created in ${ORACLE_HOME}/dbs.

Before we perform a backup let’s have a look at the file created.

$ cat $ORACLE_HOME/dbs/osbwsDEMO1.ora 
#OSB_WS_HOST=http://s3.amazonaws.com
OSB_WS_WALLET='location=file:/u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbws_wallet CREDENTIAL_ALIAS=rekins_aws'
OSB_WS_BUCKET=rontestbucket
OSB_WS_LOCATION=eu-west-2

I have customised the OSB parameter file to use the S3 bucket previously created, if you do not specify OSB_WS_BUCKET Oracle will automatically create an s3 bucket on your behalf.

OSB supports a number of other parameters and a few undocumented, non-supported underscore parameters which may be useful for debugging purposes e.g. _OBS_WS_TRACE_LEVEL

Set this to 100 to enable tracing and 0 to disable e.g. _OSB_WS_TRACE_LEVEL=100

If used the trace information can be found in sbtio.log file in $ORACLE_BASE/diag/rdbms/../../trace/


RMAN Backup to AWS s3

Now we have every thing in place, let’s try a performing a test backup of the user tablespace.

$ rman target=/
RMAN> RUN
2> {
3> allocate channel c1_s3 device type sbt
4> parms='SBT_LIBRARY=/u01/app/oracle/product/19.0.0/dbhome_1/lib/libosbws.so,SBT_PARMS=(OSB_WS_PFILE=/u01/app/oracle/product/19.0.0/dbhome_1/dbs/osbwsDEMO1.ora)';
5> backup tablespace users;
6> }
 
using target database control file instead of recovery catalog
allocated channel: c1_s3
channel c1_s3: SID=5691 instance=DEMO1 device type=SBT_TAPE
channel c1_s3: Oracle Secure Backup Web Services Library VER=19.0.0.1 

Starting backup at 25-JAN-2021 150728
channel c1_s3: starting full datafile backup set
channel c1_s3: specifying datafile(s) in backup set
input datafile file number=00007 name=+DATA/DEMO/DATAFILE/users.379.1040572917
channel c1_s3: starting piece 1 at 25-JAN-2021 150728
channel c1_s3: finished piece 1 at 25-JAN-2021 150731
piece handle=c6vlh9lg_1_1 tag=TAG20210125T150728 comment=API Version 2.0,MMS Version 19.0.0.1
channel c1_s3: backup set complete, elapsed time: 00:00:03
Finished backup at 25-JAN-2021 150731
 
Starting Control File and SPFILE Autobackup at 25-JAN-2021 150731
piece handle=c-3784643325-20210125-01 comment=API Version 2.0,MMS Version 19.0.0.1
Finished Control File and SPFILE Autobackup at 25-JAN-2021 150735
released channel: c1_s3 
RMAN> list backup of tablespace users;
 
List of Backup Sets
===================
BS Key  Type LV Size       Device Type Elapsed Time Completion Time   
------- ---- -- ---------- ----------- ------------ ------------------
1892    Full    2.50M      SBT_TAPE    00:00:01     25-JAN-2021 150729
        BP Key: 3489   Status: AVAILABLE  Compressed: NO  Tag: TAG20210125T150728
        Handle: c6vlh9lg_1_1   Media: s3-eu-west-2.amazonaws.com/rontestbucket
  List of Datafiles in backup set 1892
  File LV Type Ckp SCN    Ckp Time           Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ------------------ ----------- ------ ----
  7       Full 69186257   25-JAN-2021 150728              NO    +DATA/DEMO/DATAFILE/users.379.1040572917

Finally let’s log back into the AWS Console to confirm our RMAN backup has arrived in our specified AWS S3 bucket.

AWS s3 Bucket

From the above we can see our RMAN backup piece handle c6vlh9lg_1_1 has been created in the rontestbucket S3 bucket.


OSB Parameters

Below I have listed the supported and undocumented Oracle 19c OSB parameters for troubleshooting / educational purposes.

$ strings $ORACLE_HOME/lib/libosbws.so  | grep '^_OSB_' 
$ strings $ORACLE_HOME/lib/libosbws.so  | grep '^OSB_'
Documented / SupportedUndocumented / Unsupported
OSB_WS_PFILE_OSB_WS_ALLOCBUF_DISABLE
OSB_WS_HOST_OSB_WS_AUTH_SCHEME
OSB_WS_PROXY_OSB_WS_SEND_BUF_SIZE
OSB_WS_BUCKET_OSB_WS_RECV_BUF_SIZE
OSB_WS_LOCATION_OSB_WS_SDU_SIZE
OSB_WS_CHUNK_SIZE_OSB_WS_TDU_SIZE
OSB_WS_LICENSE_ID_OSB_WS_REUSE_CONNECTION
OSB_WS_LICENSE_MAX_SESSIONS_OSB_WS_SESSION_RETENTION_TIME
OSB_WS_WALLET_OSB_WS_CHECKER_RETENTION_TIME
OSB_WS_VIRTUAL_HOST_OSB_WS_PURGE_LIMIT
OSB_WS_IAM_ROLE_OSB_WS_CLEANER
OSB_WS_IAM_ROLE_META_URI_OSB_WS_CLEANER_RETENTION_TIME
OSB_WS_CREDENTIAL_OBJECT_OSB_WS_RUN_CLEANER
_OSB_WS_CONNECT_TIMEOUT
_OSB_WS_RESPONSE_TIMEOUT
_OSB_WS_SEND_TIMEOUT
_OSB_WS_RETRY_WAIT_TIME
_OSB_WS_UPLOAD_DELAY
_OSB_WS_EVENT
_OSB_WS_TRACE_LEVEL
_OSB_WS_100_CONTINUE
_OSB_WS_DEFERRED_DELETE
_OSB_WS_FOLLOW_REDIRECT
_OSB_WS_BUFFER_WRITE
_OSB_WS_BUFFER_READ
_OSB_WS_USE_IPV6
_OSB_WS_NO_SSL
_OSB_WS_VALIDATE_CERT
_OSB_WS_UNLOAD_DLL
_OSB_WS_NO_PROXY
_OSB_WS_USE_BULK_DELETE
_OSB_WS_CHUNK_PARTS
Please Note underscore parameters should only be used for trouble shooting or under the direction of Oracle Support

Summary

In Part 1 I have shown how we can use RMAN and Oracle Secure Backup (OSB) to backup an on-premises database to an AWS S3 Object Store.

In Part 2 I will show how we can do the same but this time to Oracle Cloud Infrastructure Object Storage via the Swift API Integration.

And in the final part I plan to repeat again, but this time using an on-premises S3 Object Storage.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: