Planet London Python

July 03, 2018

Menno Smits

Introducing influx-spout

As well as my main gig, I do some work with the excellent folks at Jump Trading. My main focus there so far has been finalising and open sourcing a project - implemented in Go - called influx-spout.

Read more… (1 min remaining to read)

by Menno Finlay-Smits at July 03, 2018 01:41 AM

June 26, 2018

Menno Smits

Listing S3 objects with NodeJS

I recently had to write some NodeJS code which uses the AWS SDK to list all the objects in a S3 bucket which potentially contains many objects (currently over 80,000 in production). The S3 listObjects API will only return up to 1,000 keys at a time so you have to make multiple calls, setting the Marker field to page through all the keys.

It turns out there's a lot of sub-optimal examples out there for how to do this which often involve global state and complicated recursive callbacks. I'm also a fan of the clarity of JavaScript's newer async/await feature for handling asynchronous code so I was keen on a solution which uses that style.

Read more… (1 min remaining to read)

by Menno Finlay-Smits at June 26, 2018 10:21 AM