Amazon S3 cloud storage access from command line in Linux

How to access Amazon S3 BY command line

Amazon S3 is stand for Simple storage service that is storage for the Internet. It as a, “simple storage service that offers software developers a highly-scalable, reliable, and low-latency data storage infrastructure at very low costs”.

Amazon S3 provides a simple web service interface which you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Using this web service, developers can easily build applications that make use of Internet storage.

Here below i will describe how to access Amazon S3 cloud storage from the command line in Linux Step by step you can learn and execute.

One of the most popular Amazon S3 command line clients is s3cmd, which is written in python. As a simple AWS S3 command line tool, s3cmd is ideal to use when you want to run scripted cron jobs such as daily backups.

Step 1: Install s3 cmd on your system (Linux-Ubuntu, RedHat etc)

To install s3cmd on CentOS or RHEL, download .rpm package from
its official site, and install it manually.
For 64-bit CentOS or RHEL 6

$ sudo rpm -ivh s3cmd-1.0.0-4.1.x86_64.rpm

$ sudo apt-get install s3cmd       To install on Ubuntu

$ sudo yum install s3cmd           To install s3cmd on Fedora

Step 2: Configure s3cmd

Now you have to configure it by the running following command, when you run s3 cmd first time

$ s3cmd --configure

It will pop-up prompt  you for few questions like this given in below:

• Access key and secret key for AWS S3
• Encryption password for encrypted data transfer to and from AWS S3.
• Path to GPG program used to encrypt data (e.g., /usr/bin/gpg)
• Whether to use HTTPS protocol
• Name and port of HTTP proxy if used

Configuration will then be saved as a plain text in ~/.s3cfg.

Basic s3cmd commands

To list all existing buckets in your AWS S3 account:

$ s3cmd  ls

2011-05-28 22:30  s3://Gaurav
2011-05-29 00:14  s3://GauravN

To view the content of an existing bucket:

$ s3cmd ls s3://GauravN

To create a new bucket:

$ s3cmd mb s3://prince16

Bucket 's3://prince16/' created

To upload files to an existing bucket:

$ s3cmd put gaurav.png gaurav.txt shani.mp3 s3://prince16

1.png -> s3://prince16/gaurav.png  [1 of 3]
 26261 of 26261   100% in    5s     4.33 kB/s  done
2.png -> s3://prince16/gaurav.txt  [2 of 3]
 201430 of 201430   100% in    2s    98.05 kB/s  done
3.png -> s3://prince16/shani.mp3  [3 of 3]
 46630 of 46630   100% in    0s    56.62 kB/s  done

The default access permission for the uploaded files is “private”, i.e only owner, with correct access and secret keys, will be able to access the files.

To upload files to an existing bucket with public access permission:

$ s3cmd put --acl-public Linux_Point.png s3://prince16

4.png -> s3://prince16/Linux_Point.png  [1 of 1]
 30778 of 30778   100% in    8s     3.34 kB/s  done
Public URL of the object is: http://prince16.s3.amazonaws.com/Linux_Point.png

With public access permission granted, the uploaded file can be accessed by anyone by going to the HTTP URL.

To download files Linux_Point contained in an existing bucket prince16:

$ s3cmd  get  s3://prince16/Linux_Point.png

s3://prince16/Linux_Point.png -> ./Linux_Point.png  [1 of 4]
 26261 of 26261   100% in    0s    39.39 kB/s  done

To remove files in an existing bucket:

$ s3cmd  del   s3://prince16/Linux_Point.png

File s3://prince16/Linux_Point.png deleted

To encrypt a file before uploading it to an existing S3 bucket:

$ s3cmd  -e  put encrypt.png s3://Prince16

/tmp/tmpfile-pzT1zV3kLZlxDwqA0kwy -> s3://prince16/encrypt.png  [1 of 1]
 196890 of 196890   100% in    1s    99.51 kB/s  done

When downloading an encrypted file with s3cmd, it will automatically detect the encryption, and decrypt the file on the fly upon downloading it.

Thus, to download and access an encrypted file, simply run as usual:

$ s3cmd get s3://Prince16/encrypt.png

s3://de```v99/encrypt.png -> ./encrypt.png  [1 of 1]
 196890 of 196890   100% in    1s   131.29 kB/s  done

To remove an existing bucket:

$ s3cmd rb s3://prince16

Bucket 's3://prince16/' removed

Note, that you cannot remove a bucket if it is not empty.

To get information about an existing bucket, including its storage location and access control lists:

$ s3cmd info s3://GauravN

s3://GauravN/ (bucket):
   Location:  us-east-1
   ACL:       dan.nanni: READ
   ACL:       dan.nanni: WRITE
   ACL:       dan.nanni: READ_ACP
   ACL:       dan.nanni: WRITE_ACP

So these are basic commands and more in deep i will give you an intro and examples continuously in coming  posts related to AWS_POINT.

 I hope you all liked it and enjoyed.