If you don't find an answer, please click here to post your question.
Engineering Blog

Safe Deletion in Amazon S3

by Community Manager ‎12-06-2012 09:40 AM - edited ‎12-07-2012 03:01 PM (2,512 Views)

Written by Patrick Fairbank
 
 

At Ooyala, we store a lot of large files in Amazon S3. When a customer wants to host a video with us, they upload it to our servers and we store a copy in S3. Our distributed transcoding system converts the source video into the various formats, resolutions, and bitrates required for streaming, and all the resulting files are also sent to S3. In total, we have on the order of several hundred terabytes of video data stored.

 

Not all of this data is useful, however. Until recently, when a customer re-transcoded a video they had previously uploaded (to change the quality settings, for example), or deleted a video they didn’t need anymore, we would delete all references to the files, but not the files themselves. The reason for this is that deletions on Amazon S3 are irreversible, and we didn’t want to risk accidentally losing customer data if the wrong files were deleted. S3 does offer a “versioning” feature that you can turn on for an entire bucket to retrieve files after they have been deleted or overwritten, but it didn’t meet our use case as we often overwrite files without wanting to keep the previous versions.

 

As our platform gained more usage and our storage volume increased exponentially, it became clear that we could no longer justify the cost of storing files that were not being used. We talked with some folks on the S3 team at Amazon, and while they couldn’t offer us the safe-deletion functionality we wanted right away, they shared a few tips that helped us implement it ourselves. Another engineer, Noam Szpiro, and I decided to tackle this problem as a Newyala project, and so we took a two-week break from our regular teams to develop a safe-deletion mechanism as well as a system for identifying which set of the files in our S3 can safely be deleted.

 

We wanted the ability to delete individual S3 objects but be able to restore them within a 30-day window. After this period, the files would be deleted permanently and the space they occupied reclaimed. Since S3 doesn’t provide that out of the box, we reduced the problem to these equivalent requirements:

  • Individual objects within an S3 bucket should be made totally inaccessible, so that failed legitimate access attempts are detectable.
  • Deleted files should expire automatically after a certain amount of time and no longer be recoverable.

Based on this simplification, we came up with the following mechanism:

  • Move the files to the “trash” bucket (actually a server-side copy followed by a delete, since S3 doesn’t have an atomic move operation).
  • Set a lifecycle configuration on the whole trash bucket to auto-expire files created more than 30 days ago.
  • If a file restore is needed, simply move the file from the trash bucket back to its original location.

 

To make this mechanism easy to use across the company, we threw together a Sinatra server exposing a small API, dubbed WALL-E (after Pixar’s trash-collecting robot). Here’s the form it took:

 

require "aws-sdk"
require "sinatra/base"

class WallEServer < Sinatra::Base
  TRASH_BUCKET = "ooyala-trash"

  configure do
    @@s3 = AWS::S3.new(:access_key_id => ENV["AWS_ACCESS_KEY_ID"],
                       :secret_access_key => ENV["AWS_SECRET_ACCESS_KEY"])
  end

  post "/delete" do
    move_object(params[:key], params[:bucket], "#{params[:bucket]}/#{params[:key]}", TRASH_BUCKET, true)
  end

  post "/restore" do
    move_object("#{params[:bucket]}/#{params[:key]}", TRASH_BUCKET, params[:key], params[:bucket], false)
  end

  def move_object(src_key, src_bucket, dest_key, dest_bucket, use_reduced_redundancy)
    src_object = @@s3.buckets[src_bucket].objects[src_key]
    dest_object = @@s3.buckets[dest_bucket].objects[dest_key]
    src_object.move_to(dest_object, :reduced_redundancy => use_reduced_redundancy)
  end
end

Once we started deleting our old files, it quickly became apparent that scale was an issue. We had identified somewhere on the order of 200 million files that could be safely cleaned up, and our Sinatra server wasn’t able to hit up the S3 APIs fast enough to make any dent in the backlog. So instead of doing the deletion synchronously right in the API server, we used Resque, a Redis-backed library for creating and running background jobs. The Sinatra server shovels the deletion requests into Resque as jobs, and a scalable fleet of c1.xlarge Spot Instances (rented for just $0.07/hr) do the actual work of calling the Amazon APIs.

We ended up deleting more than a third of the volume of movie files we had stored in S3, for cost savings of hundreds of thousands of dollars per year. The savings keep increasing in proportion with the amount of media our customers continue hosting with us, and the delete/restore functionality has helped us avoid losing customer data on more than one occasion.