Now using S4CMD

S3CMD’s distinct lack of multi-threading led me to hunt for alternatives. While I tried many alternatives, such as s3-multipart (great when I did use it), s3funnel and s3cp among others, none quite fit the bill of supporting the key features I found important. 1) Listing/Downloading/Uploading/etc of files and “folders” 2) Multi-threaded 3) Synchronization handled so… Continue reading

Use regex to select only certain files via s3cmd

Recently I wanted to download only a small subset of thousands of files from an S3 “folder.” The s3cmd docs were lacking with regard to how to select only a subset. This is how I downloaded only log files from 11PM: s3cmd get –recursive –rinclude “.*2014-10-28-23.*” –rexclude “.*” s3://my-bucket-of-logs/folder/2014-10-28/ Note that I used both rinclude… Continue reading