Use Amazon Simple Storage Service(S3) as an object store to manage Python data structures.<\/p>\n
Amazon S3 is extensively used as a file storage system to store and share files across the internet.\u00a0 Amazon S3 can be used to store any type of objects, it is a simple key value store.\u00a0 It can be used to store objects created in any programming languages, such as Java, JavaScript, Python etc.\u00a0 AWS DynamoDB recommends to use S3 to store large items of size more than 400KB.\u00a0 This article focuses on using S3 as an object store using Python.<\/p>\n
The Boto3 is the official AWS SDK to access AWS services using Python code.\u00a0 Please ensure Boto3 and awscli are installed in the system.<\/p>\n
$pip install boto3<\/pre>\n$pip install awscli<\/pre>\nAlso configure the AWS credentials using \u201caws configure\u201d command or set up environmental variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY store your keys in the environment.\u00a0 Please DO NOT hard code your AWS Keys inside your Python program.<\/p>\n
To configure aws credentials, first install awscli and then use \u201caws configure\u201d command to setup.\u00a0 For more details refer AWS CLI Setup<\/a> and Boto3 Credentials<\/a>.<\/p>\n
Configure the AWS credentials using command:<\/p>\n
$aws configure<\/pre>\nDo a quick check to ensure you can reach AWS.<\/p>\n
$aws s3 ls<\/pre>\nThe above CLI must show the S3 buckets created in your AWS account.\u00a0 The AWS account will be selected based on the credentials configured.\u00a0 In case, multiple AWS accounts are configured, use the \u201c\u2013profile \u201d option in the AWS CLI.\u00a0 If you don\u2019t mention \u201c\u2013profile \u201d option the CLI takes the profile \u201cdefault\u201d.<\/p>\n
Use the below commands to configure development profile named \u201cdev\u201d and validate the settings.<\/p>\n
$aws configure -profile dev\n$aws s3 ls --profile dev<\/pre>\nThe above command show s3 buckets present in the account which belongs to \u201cdev\u201d profile.<\/p>\n
3. Connecting to S3<\/h2>\n
3.1 Connecting to Default Account (Profile)<\/h3>\n
The client() API connects to the specified service in AWS.\u00a0 The below code snippet connects to S3 using the default profile credentials and lists all the S3 buckets.<\/p>\n
import boto3\n\ns3 = boto3.client('s3')\nbuckets = s3.list_buckets()\nfor bucket in buckets['Buckets']:\n print bucket['CreationDate'].ctime(), bucket['Name']<\/pre>\n3.2 Connecting to Specific Account (Profile)<\/h3>\n
To connect to a specific account, first create session using Session() API.\u00a0 The Session() API allows to mention the profile name and region.\u00a0 It also allows to specify the AWS credentials.<\/p>\n
The below code snippet connects to an AWS account configured using \u201cdev\u201d profile and lists all the S3 buckets.<\/p>\n
import boto3\n\nsession = boto3.Session(profile_name=\"dev\", region_name=\"us-west-2\")\ns3 = session.client('s3')buckets = s3.list_buckets()\nfor bucket in buckets['Buckets']:\n print bucket['CreationDate'].ctime(), bucket['Name']<\/pre>\n4. Storing and Retrieving a Python LIST<\/h2>\n
Boto3 supports put_object()<\/strong> <\/em>and get_object()<\/strong><\/em> APIs to store and retrieve objects in S3.\u00a0 But the objects must be serialized before storing.\u00a0 \u00a0The python pickle library supports serialization and deserialization of objects.\u00a0 Pickle is available by default in Python installation.<\/p>\n
The APIs pickle.dumps() and pickle.loads() is used to serialize and deserialize Python objects.<\/p>\n
4.1 Storing a List in S3 Bucket<\/h3>\n
Ensure serializing the Python object before writing into the S3 bucket.\u00a0 The list object must be stored using an unique \u201ckey\u201d.\u00a0 If the key is already present, the list object will be overwritten.<\/p>\n
import boto3\nimport pickle\n\ns3 = boto3.client('s3')\nmyList=[1,2,3,4,5]\n\n#Serialize the object \nserializedListObject = pickle.dumps(myList)\n\n#Write to Bucket named 'mytestbucket' and \n#Store the list using key myList001\n\ns3.put_object(Bucket='mytestbucket',Key='myList001',Body=serializedListObject)<\/pre>\nThe put_object()<\/strong> <\/em>API may return a \u201cNoSuchBucket<\/em>\u201d exception, if bucket does not exists in your account.<\/p>\n
NOTE:\u00a0 Please modify bucket name to your S3 bucket name.\u00a0 I don\u2019t won this bucket.<\/strong><\/p>\n
4.2 Retrieving a List from S3 Bucket<\/h3>\n
The list is stored as a stream object inside Body.\u00a0 It can be read using read() API of the get_object()<\/strong><\/em> returned value.\u00a0 It can throw an \u201cNoSuchKey\u201d exception, if the key is not present.<\/p>\n
import boto3\nimport pickle\n\n#Connect to S3\ns3 = boto3.client('s3')\n\n#Read the object stored in key 'myList001'\nobject = s3.get_object(Bucket='mytestbucket',Key='myList001')\nserializedObject = object['Body'].read()\n\n#Deserialize the retrieved object\nmyList = pickle.loads(serializedObject)\n\nprint myList<\/pre>\n5 Storing and Retrieving a Python Dictionary<\/h2>\n
Python dictionary objects can be stored and retrieved in the same way using put_object()<\/strong> <\/em>and get_object()<\/strong> <\/em>APIs.<\/p>\n
5.1 Storing a Python Dictionary Object in S3<\/h3>\n
import boto3\nimport pickle\n\n\n#Connect to S3 default profile\ns3 = boto3.client('s3')\n\nmyData = {'firstName':'Saravanan','lastName':'Subramanian','title':'Manager', 'empId':'007'}\n#Serialize the object\nserializedMyData = pickle.dumps(myData)\n\n#Write to S3 using unique key - EmpId007\ns3.put_object(Bucket='mytestbucket',Key='EmpId007')<\/pre>\n5.2 Retrieving Python Dictionary Object from S3 Bucket<\/h3>\n
Use the get_object() API to read the object.\u00a0 The data is stored as a stream inside the Body object.\u00a0 This can be read using read() API.<\/p>\n
import boto3\n\ns3 = boto3.client('s3')\n\nobject = s3.get_object(Bucket='mytestbucket',Key='EmpId007')\nserializedObject = object['Body'].read()\n\nmyData = pickle.loads(serializedObject)\n\nprint myData<\/pre>\n6 Working with JSON<\/h2>\n
When working with Python dictionary, it is recommended to store it as JSON, if the consumer applications are not written in Python or do not have support for Pickle library.<\/p>\n
The api json.dumps() converts the Python Dictionary into JSON and json.loads() converts a JSON to a Python dictionary.<\/p>\n
6.1 Storing a Python Dictionary Object As JSON in S3 bucket<\/h3>\n
import boto3\nimport json\n\ns3 = boto3.client('s3')\n\nmyData = {'firstName':'Saravanan','lastName':'Subramanian','title':'Manager', 'empId':'007'}\nserializedMyData = json.dumps(myData)\n\ns3.put_object(Bucket='mytestbucket',Key='EmpId007')<\/pre>\n6.2 Retrieving a JSON from S3 bucket<\/h3>\n
import boto3\nimport json\n\ns3 = boto3.client('s3')\nobject = s3.get_object(Bucket='mytestbucket',Key='EmpId007')\nserializedObject = object['Body'].read()\n\nmyData = json.loads(serializedObject)\n\nprint myData<\/pre>\n7 Upload and Download a Text File<\/h2>\n
Boto3 supports upload_file() and download_file() APIs to store and retrieve files to and from your local file system to S3.\u00a0 As per S3 standards, if the Key contains strings with \u201c\/\u201d (forward slash) will be considered as sub folders.<\/p>\n
7.1 Uploading a File<\/h3>\n
import boto3\n\ns3 = boto3.client('s3')\ns3.upload_file(Bucket='mytestbucket', Key='subdir\/abc.txt', Filename='.\/abc.txt')<\/pre>\n7.2 download a File from S3 bucket<\/h3>\n
import boto3\n\ns3 = boto3.clinet('s3')\ns3.download_file(Bucket='mytestbucket',Key='subdir\/abc.txt',Filename='.\/abc.txt')<\/pre>\n8 Error Handling<\/h2>\n
The Boto3 APIs can raise various exceptions depends on the condition.\u00a0 For example, \u201cDataNotFoundError\u201d<\/em>,\u201dNoSuchKey\u201d,\u201dHttpClientError<\/em>\u201c, \u201cConnectionError<\/em>\u201c,\u201dSSLError<\/em>\u201d are few of them.\u00a0 The Boto3 exceptions inherit Python \u201cException\u201d class.\u00a0 So handle the exceptions by looking for Exceptions<\/em> class in error and exception handling in the code.<\/p>\n
import boto3\n\ntry:\ns3 = s3.client('s3')\nexcept Exceptions as e:\n print \"Exception \",e<\/pre>\n9.Summary<\/h2>\n
Storing python objects to an external store has many use cases.\u00a0 For example,\u00a0 a game developer can store intermediate state of objects and fetch them when the gamer resumes from where left, API developer can use S3 object store as a simple key value store are few to mention.\u00a0 Please refer the URLs in the Reference sections to learn more.\u00a0 Thanks.<\/p>\n
References<\/h2>\n
[i] Boto3 \u2013 https:\/\/boto3.amazonaws.com\/v1\/documentation\/api\/latest\/index.html<\/a><\/p>\n
[ii] Boto3 S3 API<\/a> \u2013 https:\/\/boto3.amazonaws.com\/v1\/documentation\/api\/latest\/reference\/services\/s3.html<\/a><\/p>\n
[iii] AWS CLI \u2013 https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/cli-chap-install.html<\/a><\/p>\n
[iv] AWS Boto3 Credentials https:\/\/boto3.amazonaws.com\/v1\/documentation\/api\/latest\/guide\/configuration.html<\/a><\/p>\n
[v]Python 2.7 Pickle Library \u2013 https:\/\/docs.python.org\/3\/library\/pickle.html<\/a><\/p>\n
[vi] Boto3 Exceptions\u00a0\u00a0https:\/\/github.com\/boto\/botocore\/blob\/develop\/botocore\/exceptions.py<\/a><\/p>\n
\n