Oh do you mean to simulate a stream? That’s a pretty cool idea. My approach, which worked, is a little different. I used python’s zipfile module. Since I had the files stored locally I was able to use python to append json files together to make bigger files of at least 128MB and at most 1GB. So far it seems to have worked. I went from having ~2M mini json files to 28 appropriately-sized json files (~300MB each). Your blog was super helpful, thanks!

Mohammad, if you already have an army of tiny files, you can do a batch conversion to parquet by reading in chunks of the data set (e.g. 1 hour partition at a time of a 24 hour day)

While 1,000 2MB files on HDFS is not optimal due to being smaller than HDFS block size (64 or 128MB default), it shouldn’t be a big issue. The number of partitions should be between 8 and 1,000 depending on compression (if any). If using DataFrames or Datasets, you can do df.rdd.getNumPartitions to get the total number of partitions.