Each year for April Fools’, rather than a prank, we like to create a project that explores the way that humans interact at large scales. This year we came up with Place, a collaborative canvas on which a single user could only place a single tile every five minutes. This limitation de-emphasized the importance of the individual and necessitated the collaboration of many users in order to achieve complex creations. Each tile placed was relayed to observers in real-time.
This post details how we approached building Place from a technical perspective.
Next to detailing how the data is stored, synced, and drawn on the canvas; the article also covers some realtime problem solving after the RabbitMQ CPU load avg shot up from 6 to 60 (!) …
The 72h timelapse – condensed to 4’30 – is embedded above. The source code of r/Place is also available.
At Instagram, they’re using Redis (comparable to Memcached, but with more options) to map photoIds to userIds.
After tweaking their setup — by using Redis hashes (dictionaries that are can be encoded in memory very efficiently) — in Redis, they’ve brought memory usage down from 70MB to 16MB to store 1,000,000 records.
While prototyping this solution, we found that Redis needed about 70 MB to store 1,000,000 keys this way. Extrapolating to the 300,000,000 we would eventually need, it was looking to be around 21GB worth of data.
With our 1,000,000 key prototype (encoded into 1,000 hashes of 1,000 sub-keys each), Redis only needs 16MB to store the information. Expanding to 300 million keys, the total is just under 5GB
By comparison: Memcached needed 52MB of memory to hold the 1,000,000 records.