Amazon launched their Simple Storage Service (S3) service about 10 years ago followed shortly by Elastic Compute Cloud (EC2). In the past 10 years, Amazon has learned a few things about running these services. In his keynote at LinuxCon Europe, Chris Schlaeger, Director Kernel and Operating Systems at the Amazon Development Center in Germany, shared 10 lessons from Amazon.
1. Build evolvable systems
The cloud is all about scale and being able to get compute power only when you need it and getting rid of it when you don’t need it anymore. Schlaeger says that “the lesson that we learned isn’t to design for a certain scale, you always get it wrong. What you want to do instead is design your system so you can evolve it … over time without the customers or users knowing it.”
2. Expect the unexpected
Hardware has a finite lifespan, so things will fail, but you can design your systems to check for failure, deal with it, isolate failures, and then react to them. “Control the blast radius and raise failure as a natural occurrence of your software and hardware, all the time,” Schlaeger suggests.
3. Primitives, not frameworks
Amazon doesn’t know what every customer wants to do, and they don’t want try to tell customers how to do their work. However, they do want to evolve quickly to follow the needs of their customers, and this agility is something that is much easier to accomplish with primitives rather than frameworks.
4. Automation is key
Schlaeger points out that “if you want to scale up, you need to have some form of automation in place.” If someone can log into your servers and make changes on the fly, then you can’t track what changes have been made over time.
5. APIs are forever
APIs can be tricky because if you want to keep your customers happy, you can’t keep changing your APIs. “You need to be very, very cautious and conscious about the APIs you have and make sure you don’t change them,” Schlaeger says.
6. Know your resource usage
When Amazon first launched S3, they charged for storage space and transactions, so people quickly learned that storing and retrieving tiny thumbnail images for items on eBay was quite cheap. However, the large numbers of API calls generated a big enough load on Amazon’s servers that they had to start including call rates in the pricing model. Understanding all of your costs and building them into your prices is important.
7. Build security in from the ground up
It is important that you get the security involved in the design of a system in addition to the implementation. You should also do regular check-ins as your service evolves over time to make sure that it stays secure.
8. Encryption is a first class citizen
Schlaeger points out that “the best way you can prove to your customers that the data is safe from access from other parties … is to have them encrypted.” Within AWS, customers can encrypt all of their data and only the customer has access to the keys used to encrypt and decrypt the data.
9. Importance of the network
This is probably the hardest part to get right, because the network is a shared resource for everybody across all use cases. Various customers have unique and often contradictory requirements for using the network.
10. No gatekeepers
“The more open you are with your platform, … the more success you will have,” Schlaeger says. Amazon doesn’t try to limit what their customers can do beyond what they need to protect the instances or services of other customers.
For more details about each of these 10 lessons, watch the full video below.
Interested in speaking at Open Source Summit North America on September 11 – 13? Submit your proposal by May 6, 2017. Submit now>>
Not interested in speaking but want to attend? Linux.com readers can register now with the discount code, LINUXRD5, for 5% off the all-access attendee registration price. Register now to save over $300!
