We’re excited to announce that Sweet Security has officially released CandyStore – an extremely fast open source key-value store, with a very low memory footprint.
Why We Created CandyStore
It’s not every day that a cyber security company develops its own database (or its little cousin, a key-value store), but Sweet is not your average security company. We believe that the best product requires the best-of-breed technology, even if it means we have to build it ourselves.
Before getting there, we tested several well-established databases and key-value stores (sqlite, RocksDB, to name a few), but none of them met our performance, memory or concurrency requirements. After a few weeks of rigorous testing, we set off writing our own key-value store and thus CandyStore was born.
Key Benefits
CandyStore is unique in that it’s not based on LSM or B-Trees, as most existing key-value stores, but rather on a novel algorithm. This algorithm can be seen as a “zero overhead” extension of a hash table that’s stored over files. Because it operates in O(1) like a hash table, rather than log(N) like a tree, we are able to achieve sub-microsecond latency, normally requiring a single IO for lookup / insert / removal, and 2 IOs for an update..
Another huge benefit of CandyStore is that it does not require a journal or a write-ahead log (WAL), thus reducing the number of disk operations required while still being crash-consistent. This is especially important in cloud instances, where disk IOs are capped. It also does not require costly merges or maintenance, and its compaction operations are bounded in size.
CandyStore allows us to reduce our sensor’s memory footprint even further, and is already deployed to thousands of sensors in the field.
Being a Rust shop since day one, and relying a lot on open source technologies ourselves, we felt obliged to share this project with the open source community. The code is now available on Github, under the Apache 2.0 license. Feel free to check it out. For a technical deep-dive of how the DB works, see this blog.
If you’d like to learn more about our technology and offering, schedule a demo today!
