Moving files into S3 as fast as possible has been a side-quest of mine for, well, two decades (yep, S3 is more than 20 years old ). My first encounter with moving large numbers of files to S3 was from an on-premise system, working over a backlog of many thousands of files, with thousands of new ones appearing each day. For that project I tried the obvious path (a foreach loop doing uploads one by one) and failed. I discovered success with concurrency -- running multiple uploads simultaneously from multiple workers. It worked, and a similar technique has been built into the AWS CLI (CRT and concurrent connections config setting). Concurrency was helpful because bandwidth wasn't the limit. The files were small; at most a 1-2 MB each, and many just a few KB. The bandwidth to S3 was relatively low by today's standards, but not the bottleneck. It was the round-trip latency, connection negotiation and retransmission getting in the way.…