S3CMD’s distinct lack of multi-threading led me to hunt for alternatives. While I tried many alternatives, such as s3-multipart (great when I did use it), s3funnel and s3cp among others, none quite fit the bill of supporting the key features I found important. 1) Listing/Downloading/Uploading/etc of files and “folders” 2) Multi-threaded 3) Synchronization handled so… Continue reading
Recently I wanted to download only a small subset of thousands of files from an S3 “folder.” The s3cmd docs were lacking with regard to how to select only a subset. This is how I downloaded only log files from 11PM: s3cmd get –recursive –rinclude “.*2014-10-28-23.*” –rexclude “.*” s3://my-bucket-of-logs/folder/2014-10-28/ Note that I used both rinclude… Continue reading
Wired access is better than wireless generally, so when I wanted to have two machines in opposite areas with wired internet, something had to give. My initial setup before using two routers was to have the modem and main router in one area and just use wireless everywhere else. But that was cramping my speed… Continue reading
Sometimes we need to store integers, but we have too many to store for a traditional relational database like MySQL, and Hadoop wouldn’t bat an eye at the amount. However we want to use SQL, because it poses a new challenge (and may be necessary for other reasons we won’t go into). A typical table… Continue reading