abelanger 13 hours ago

While we didn't look too closely into this, our buffers used to scale capacity based on a Fibonacci sequence if we hit capacity within the flush interval, up to a max capacity. I'm sure there's much better prior work on the ideal scaling laws here but this worked really well for us, and I'm sure any exponential would do fine.

When we rewrote our inserts system, we opted to just ensure that the buffers would flush immediately on first write, which makes writes fast on anything that's on the order of < 1k writes/s with constant load over the second (10 buffers, 10ms flush interval), which is a pretty good baseline for our system. This is simpler and has been working really well for us, so there wasn't a need at this point to reintroduce any sort of exponential scaling, but I'll be experimenting more with this.

1
atombender 12 hours ago

So you measure the throughput and slowly increase the batch size until the throughput stabilizes?

That's better than nothing, of course, but I would very much like to also be able to ramp down if the throughput suddenly drops (e.g. due to competing activity).