Recently I wanted to download only a small subset of thousands of files from an S3 “folder.” <\/p>\n

The s3cmd docs were lacking with regard to how to select only a subset. <\/p>\n

This is how I downloaded only log files from 11PM:
\ns3cmd get --recursive --rinclude \".*2014-10-28-23.*\" --rexclude \".*\" s3:\/\/my-bucket-of-logs\/folder\/2014-10-28\/<\/code><\/p>\n

Note that I used both rinclude and rexclude. If you only use rinclude (or include), it won’t restrict only to what you specified, but it will ensure what you specified is taken. So you have to exclude everything first (--rexclude \".*\"<\/code>), THEN include only what you want (--rinclude \".*2014-10-28-23.*\"<\/code>)<\/p>\n","protected":false},"excerpt":{"rendered":"

Recently I wanted to download only a small subset of thousands of files from an S3 “folder.” The s3cmd docs were lacking with regard to how to select only a subset. This is how I downloaded only log files from 11PM: s3cmd get –recursive –rinclude “.*2014-10-28-23.*” –rexclude “.*” s3:\/\/my-bucket-of-logs\/folder\/2014-10-28\/ Note that I used both rinclude… Continue reading→<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/31"}],"collection":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/comments?post=31"}],"version-history":[{"count":2,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/31\/revisions"}],"predecessor-version":[{"id":97,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/posts\/31\/revisions\/97"}],"wp:attachment":[{"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/media?parent=31"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/categories?post=31"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/garrens.com\/blog\/wp-json\/wp\/v2\/tags?post=31"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}