IBM i - Having fun with IBM Cloud Object Storage (COS)

In previous articles I've been talking about IBM Cloud Object Storage and how to connect from IBM i.

In this small article I'll try to make a short cheatsheat and give you some tips on how to ease your experience.

Installing AWS CLI on IBM i

• Add '/QOpenSys/pkgs/bin' to your PATH
• Install Python using YUM or ACS. In my case Python 3.9
• Connect with SSH and run the following:

pip3 install awscli
        

• To setup credentials

aws configure
        

Commands

1. With the prefix "s3://" you can run commands on IBM COS:

• ls = List buckets or bucket content
• cp = Copy to or from your buckets
• rm = Delete an object in your bucket
• mv = Move or rename file

2. For more commands type: aws s3 help?

How to use IBM COS on IBM i

Get the URL Endpoint for your buckets on IBM Cloud portal and use the option "--endpoint-url"

aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 l


#This command lists your buckets.
        

aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 cp file.txt s3://ibmitxts/file.tx


This copies the local file to the bucket "ibmitxts"
        

TIPS

1. If you want to use shorter commands try creating an alias (can be saved in your .profile , .bash_profile or .bashrc)

alias ibmcos=aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3
        

Now your commands will look similar to this:

ibmcos ls
ibmcos cp file.txt s3:/ibmitxts/file.txt
        

2. You can use piping (a way to chain commands used in Unix-Like environments) to make some tasks more efficient

Using piping to upload files?

system "WRKACTJOB" | aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 cp - s3://ibmitxts/wrkactjob_20220523.txt
        

This example list the content of WRKACTJOB and send the results to the bucket

A more complex example:

cat /QSYS.LIB/BACKUPSAV.LIB/MYSAVF.FILE | gzip -9 | aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 cp - s3://ibmitxts/MYSAVF.gz
        

This example sends the content from MYSAVF save file , compress and save the content in your bucket

3. You can use piping to read the content of a file in your bucket

aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 cp s3://ibmitxts/wrkactjob_20220523.txt -
        

This will show the content of this file and you can process this text stream with grep, cut, sed, awk or save to a file using ">" .?

You can also extract information from a compressed file while performing a download operation.

aws --endpoint-url=https://s3.us-south.cloud-object-storage.appdomain.cloud s3 cp s3://ibmitxts/wrkactjob_20220523.txt - |grep RUN > waj_20220523_RUN.txt

        

This will filter the content from the cloud and save only matching records to your file.

WARNING: This will read all the file content from the cloud. With large files will take some time and you will consume cloud credits.

4. Improving performance

You can make some adjustments to improve performance when copying files from/to your bucket.

Edit the file "config" in your /home/USER/.aws/ directory and add this to the [default] section

s3 =


?max_concurrent_requests = 20


?max_queue_size = 10000


?multipart_threshold = 256MB


?multipart_chunksize = 128MB
        

Now try again your uploads or downloads

As always, if any doubt arises just send me a message

Good luck !