get started on cloud now!
by Mohdqaz
AWS S3
What is AWS S3
What does AWS S3 stands for?
AWS S3 stands for Simple Storage Service. As the name tells us, S3 provides storage to store files in the Cloud (Amazon Cloud).
And it is not only to store normal files, it can also be used as archival and even to host a static website.
It have unlimited storage and file size can be from 0 byte to 5TB.
Files are stored in bucket and we always hear people mention S3 bucket. But what is actually an Amazon S3 bucket? It is just AWS term for folders. Meaning, we are storing our files or S3 objects in folders.
S3 Objects
S3 is object based where we can imagine that each of the file that we can upload is an object
S3 Objects consists of the following:
- Key (name of the object)
- Value (data is made up of a sequence of bytes)
- Version ID (for versioning)
- Metadata (data about the data that you are storing)
- Subresources
S3 Features
S3 has following features (you can click on each of the features to know more details) :
- Tiered Storage (Storage Class)
- Lifecycle Management
- For example, when a file is 30 days old, move to another tier and after 90 days move to the Glacier for example for archiving purpose
- Versioning (version control)
- Encryption
- MFA Delete (this feature can be enabled so that MFA is required to delete any object in S3)
- Secure data using Access Control List & Bucket Policy (related to S3 permission)
S3 Data Consistency
How does data consistency work for S3?
- Read after write Consistency for PUTS of new Objects
- If the user write a new file & read it immediately, user will be able to read the data
- Eventual Consistency for overwrite PUTS and DELETES (will take a bit of time to propagate)
- If the user update EXISTING file or DELETE a file and read it immediately, user may get either
- Older version
- New version
- And will get the new version after few seconds
- If the user update EXISTING file or DELETE a file and read it immediately, user may get either
S3 Guarantees (based on S3 standard)
AWS S3 is built for 99.99% availability for AWS S3 platform
Amazon guarantee 99.9% availability (SLA) for their S3 service
- What S3 availability means is probability that you will be able to get the content back the moment you try to access it
- So, 0.1% of the time user cannot retrieve the file is due to few possible reason such as file corruption
Amazon Guarantee durability of 99.999999999% (11 9s if we want to remember this)
- As for S3 Durability, it means that probability the user is going to get the file back
- How does S3 durability being achieved? By storing those objects redundantly across multiple facilities within an S3 region
S3 Storage Classes (S3 Storage Types)
What is S3 Storage classes? It’s type of S3 storage that user can go for according to their requirements. To understand more about the options, below are S3 storage classes explanation:
- S3 Standard
- 99.99% availability
- 99.999999999% durability
- Stored redundantly across multiple devices in multiple facilities (multiple host in multiple datacenter)
- Thus can sustain loss of 2 facilities (datacenter) concurrently
- S3 Standard – IA (infrequently access)
- Data accesses less frequently (eg: access only once a month)
- but requires rapid access when needed
- Lower fee than S3 standard
- But will be charged a retrieval fee
- S3 One zone – IA (One Zone Infrequently Access)
- Lower-cost option for infrequently access data
- But do not require multiple AZ (Availability zone)
- S3 – Intelligent tiering
- Optimized cost by automatically moving data to the most cost efficient access tier
- Without performance impact or operational overhead
- Using machine learning
- Eg: S3 standard can be moved to S3-IA
- S3 Glacier
- Secure, durable & low cost storage class for data archiving
- Reliably store any amount of data at costs that are competitive or cheaper than on-premise solution
- Retrieval time configurable from minutes to Hours
- S3 Glacier deep archive
- Lowest cost storage class
- REtrieval time of 12 hours is accepted
- Need to put a request
S3 Storage Classes Comparison
| S3 Standard | S3 Intelligent Tiering | S3 Standard-IA | S3 One Zone-IA | S3 Glacier | S3 Glacier Deep Archive | |
|---|---|---|---|---|---|---|
| Designed for Durability | 99.999999999% (11 9's) | 99.999999999% (11 9's) | 99.999999999% (11 9's) | 99.999999999% (11 9's) | 99.999999999% (11 9's) | 99.999999999% (11 9's) |
| Designed for Availability | 99.99% | 99.9% | 99.9% | 99.5% | N/A | N/A |
| Availability SLA | 99.9% | 99% | 99% | 99% | N/A | N/A |
| Availability Zones | >= 3 | >= 3 | >= 3 | 1 | >= 3 | >= 3 |
| Minimum capacity charge per object | N/A | N/A | 128KB | 128KB | 40KB | 40KB |
| Minimum storage duration charge | N/A | 30 days | 30 days | 30 days | 90 days | 180 days |
| Retrieval Fee | N/A | N/A | per GB retrieved | per GB retrieved | per GB retrieved | per GB retrieved |
| First byte latency | Milliseconds | Milliseconds | Milliseconds | Milliseconds | Select minutes or hours | Select hours |
AWS S3 Object Lifecycle (Lifecycle Management)
As we have gone through type of storages that AWS is offering, it is now easier to understand what is S3 Object Lifecycle or Lifecycle management means to us.
S3 Object Lifecycle is set of rules that automate the migration of the Object Storage Class to a different storage class based on specified time intervals.
For example on the scenario:
- I have a work file that I’m going to access daily for 30 days
- Usage: Frequent
- Best storage: S3 standard
- After 30 days, I may only need to access the file once a week for the next 60 days
- Usage: Infrequent access
- Best storage: S3-IA (Infrequent access)
- After 90 days, I probably will never access that file but still want to keep the file just in case somebody ask for it.
- Usage: Most likely will never need the file
- Best storage: Glacier
Although there is a cost being charged if we use lifecycle management, there is also a saving that we can enjoy when implementing it. This is due to the more efficient usage of the storage based on what we required. Keep in mind that S3 standard is the most expensive and it will be cheaper towards Glacier Deep archive.
So, why not move the files to the cheaper storage if we can save some money from there.
Difference between S3 Object Lifecycle and S3 Intelligent tiering
After understanding what is S3 Object Lifecycle and storage class particularly S3 Intelligent tiering, one may ask “what is the difference between those two feature then?”
Actually what makes those two differs from each other are:
- S3 Intelligent tiering only support moving object from S3 Standard to S3 Standard – IA
- Whereas Lifecycle Management support all S3 storage class
This might change in the future, so stay tune for future updates from AWS.
S3 Version Control
S3 versioning with S3
- Stores all versions of an object (writes / delete)
- Can be used as Backup tool as we can easily revert back to the previous version of the object
- Once enabled, Versioning cannot be disabled!! >> only suspended
- You can only delete the bucket and create a new one if you don’t want versioning anymore
- Integrates with Life cycle rules
- Versioning have MFA Delete capability
- Use MFA to prevent accidentally deletion. This will become another layer of security
- When we enable versioning, please remember that it will incur some cost as each of the version will add additional size to the object
S3 Encryption
It is really important to secure data that we transfer to and from AWS S3.
This is to prevent any data that we stored to fall into wrong hands such as hackers.
There are two scenarios that we need to take into consideration when protecting our data:
- Data while in-transit (Encryption in transit)
- Data as it travels to and from S3 (AWS Datacenter)
- Data at rest (Encryption at Rest)
- While data is stored on disks in AWS datacenter
- Encryption at rest can then be split into another two types:
- Server-Side Encryption
- S3 Managed Keys (SSE-S3)
- AWS Key Management Service (SSE KMS)
- Server-side encryption with customer-provided key (SSE-C)
- Client-side encryption
- Server-Side Encryption
Below is an example of how encryption is being done for server-side encryption
Amazon S3 Pricing
How much does Amazon S3 cost (S3 Pricing)?
Amazon S3 charges will be a bit different for each of the region. But we’ll get S3 pricing example based on North Virginia this time
Below are examples of S3 charges or S3 pricing:
| S3 Standard | S3 Standard Infrequent Access | S3 One Zone - IA | S3 Glacier | |
|---|---|---|---|---|
|
STORAGE |
|
|
|
|
|
First 50 TB / mo |
$0.023 / GB |
$0.0125 / GB |
$0.01 / GB |
$0.004 / GB |
|
Next 450 TB / mo |
$0.022 / GB |
$0.0125 / GB |
$0.01 / GB |
$0.004 / GB |
|
Over 500 TB / mo |
$0.021 / GB |
$0.0125 / GB |
$0.01 / GB |
$0.004 / GB |
|
REQUEST & RETRIEVAL |
|
|
|
|
|
Put, Copy, Post, List |
$0.005 / 1000 request |
$0.01 / 1000 request |
$0.01 / 1000 request |
$0.05 / 1000 request |
|
Data retrievals |
$0.000 / GB |
$0.01 / GB |
$0.01 / GB |
* Refer below |
*For S3 Glacier:
- Object archived have a minimum of 90 days storage. Object deleted before 90 days will be charged pro-rated charge which is equal to the remaining storage charge.
- Object deleted, overwritten and transferred to a different storage before minimum period (90 days), will be charged normal storage usage + pro-rated request charge for the remaining of minimum storage duration
- Object stored longer than 90 days WILL NOT be charged minimum request charge
Above is just portion of the S3 charges as an example on how S3 pricing works. To get more information for different S3 storage classes cost and to check out on S3 pricing based on your own region, head on to https://aws.amazon.com/s3/pricing/
S3 Permission
When we touch on S3 permission, we cannot run from talking about Bucket policies and access control list.
Anyway, all buckets and objects are PRIVATE by default. Only resource owner have access at first. Owner will then grant access to other users through:
- S3 resource based policy
- IAM User policy
Let’s dive deep into what is S3 resource based policy and IAM User Policy?
S3 Bucket Policy
- S3 Bucket Policy is a policy attached to S3 bucket.
- Permission in the policy applied to all objects in the bucket.
- Policies determine what actions are allowed or denied for particular user
S3 Access Control List
- Grant access to users in other Account or to Public
- Both bucket or object has ACL (Access Control List)
- Object Access Control list allow us to share S3 object via URL with the Public
AWS S3 FAQ
- A: No, we explained a little bit on S3 pricing at S3 pricing section above. But if you register for AWS Free tier account, you will get 5GB of S3 standard storage for 12 months. Not really free I would say, it’s more to the customer to try it out
- A: There is an S3 plugin for Jenkins available to integrate AWS S3 with Jenkins available here: https://plugins.jenkins.io/s3/
- A: Yes, Amazon S3 support FTP more precisely SFTP. There is a good article explaning about this here: https://aws.amazon.com/blogs/aws/new-aws-transfer-for-sftp-fully-managed-sftp-service-for-amazon-s3/
- A: The most common way to perform this is to go to AwS Management Console specifically to S3 portion of it at: https://console.aws.amazon.com/s3/
- A: S3 is part of AWS Free Tier offering and during the 12 months free tier period, you will able to enjoy 5GB of S3 standard storage for free. For more information, go to: https://aws.amazon.com/free
Tips for AWS Solution Architect Associate Exam (related to S3 topic)
For those who is planning to sit for AWS Solution Architect Associate exam or currently doing your revision, below are some tips to prepare for the exam especially if you are currently preparing on AWS S3 topic. Take note that there a lot of questions on S3 in this exam so, S3 topic is definitely one of the topic that you need to master on.
- Go through all the S3 F.A.Q in AWS website: https://aws.amazon.com/s3/faqs/
- Find a good AWS Online training. One of the best I went through so far is from Ryan Kroonenburg on Udemy
- Make sure you do a lot of lab exercise (practical knowledge) and practice questions. I find Whizlabs having a good resources for this (practice labs and practice questions) with an affordable price too. (you can try the free test first before purchasing the practice question). What I like about the practice question is they have around 6 set of full questions (all topics) and also question on specific topic for us to focus on the topic that we are weak on.
Good luck for your exam
Whizlabs
Hands-on Lab, Exam Practice Questions, E-Learning
AWS Document
Official information from AWS
Table Of Content