Post

With default params (bucket sketch with 10k buckets), it can sketch 900MB/s of input data per thread, and then can do 1 million pairwise distance comparisons per second per thread. So a full triangle of 1000 2Mbp bacterial genomes is 2.3s + 0.5s.

or with 6 threads, only 0.7s. That's with bucket size b=8. Sketching is >50x faster than BinDash[-rs], which takes at least 110s. For b=8 as above, computing distances is ~2x faster than BinDash, but for b=1, SimdSketch is again 10x faster. curiouscoding.nl/posts/simd-s...

Also note: for comparison speed, 1-bit hashes beat everything else (top-right red line here). Because each bit is an independent sample, the entropy is very high. (Whereas comparing 8-bit sketches loses some information, because if two 8-bit values differ in the first bit, the other 7 are unused.)

Compared to `mash triangle` for 208 bgzipp'd cattle genomes on 4 threads, it only took 65 min vs 230 min. Used slightly more RAM (8.2 GB vs 3.3 GB), but easily worth the speedup! Mash defaults are also k=21 and 1k sketches, so not a fair comparison to k=31 and 10k (not sure about sketches v buckets)

Simd-sketch perf should be independent of k. 10k 8-bit buckets is mostly equivalent to a 10k bottom-sketch, so you're getting some more accuracy here. But since you only have 208 genomes, the 200^2 comparisons will take less than a second anyway.

I wonder how much time is spent on decompresion here; that might well be the bottleneck. I get nearly 1GB/s sketching locally, so that's 3s/genome or 200*3s=600s=10min total on a single thread, or only 2.5min on 4 threads.