b5cc68a321
This adds batching support to crdt-consensus per #1008 . The crdt component can now take advantage of the BatchingState, which uses the batching-crdt datastore. In batching mode, the crdt datastore groups any Add and Delete operations in a single delta (instead of just 1, as it does by default). Batching is enabled in the crdt configuration section by setting MaxBatchSize **and** MaxBatchAge. These two settings control when a batch is committed, either by reaching a maximum number of pin/unpin operations, or by reaching a maximum age. Batching unlocks large pin-ingestion scalability for clusters, but should be set according to expected work loads. An additional, hidden MaxQueueSize parameter provides the ability to perform backpressure on Pin/Unpin requests. When more than MaxQueueSize pin/unpins are waiting to be included in a batch, the LogPin/LogUnpin operations will fail. If this happens, it is means cluster cannot commit batches as fast as pins are arriving. Thus, MaxQueueSize should be increase (to accommodate bursts), or the batch size increased (to perform less commits and hopefully handle the requests faster). Note that the underlying CRDT library will auto-commit when batch deltas reach 1MB of size. |
||
---|---|---|
.. | ||
crdt | ||
raft |