Moving to S308 09 2009, Michiel van Vlaardingen on PicturePush
For some time we have used Amazon S3 as back up for our pro users on PicturePush. However, with the growing amount of users we decided that it was time to switch to S3 for all storage. The reason being that we got tired of managing our own storage: disks got full at unexpected times, etc.
The approach we followed was to first get all files into S3 using the backup scripts we already had for the pro-accounts. Then we wrapped al our code that touched the image files. So we ended up with a filesystem based storage api with methods like:
- getLocator($image, $format)
- putFromFile($locator, $file)
Furthermore, we needed to account for metadata storage such as filesize, resolution, etc. because we could no longer get these from the files directly.
Implementing all these abstraction was the most elaborate part of the transition. (If you ever do anything with files: wrap it! ) After this, the actual switch to using S3 was trivial. After a few days of bug fixing everything seems to be stable and quick again.
As we needed to preserve our existing URL structure, lazy image generation and session based authentication ( for private images ) we could not serve from S3 directly. To keep things quick, we decided on using a large local cache for the most common files. This enables us to serve files fast and using our existing infrastructure without relying on it for permanent storage.
For example: To further speed up the serving of images, we are now experimenting with custom CDN nodes. Hopefully we will start some nodes in the US soon, so our US customers will get a much better experience.
17 03 2011: Brand New Observu Coming
14 12 2010: Print Details
13 07 2010: Save Your Queries