Post

Ragnar {Groot Koerkamp}
@curiouscoding.nl

Looking for people to test the latest version of simd-sketch. It's now 2x as fast at sketching, and supports skipping over kmers containing N and other ambiguous bases (which is only ~35% slower). 'cargo install simd-sketch' is right there under your fingertips ;) github.com/RagnarGrootK...

github.comGitHub - RagnarGrootKoerkamp/simd-sketch: Compute bottom-s sketches and s-buckets sketches, using simd-minimizers crate.Compute bottom-s sketches and s-buckets sketches, using simd-minimizers crate. - RagnarGrootKoerkamp/simd-sketch
Ragnar {Groot Koerkamp}
@curiouscoding.nl

With default params (bucket sketch with 10k buckets), it can sketch 900MB/s of input data per thread, and then can do 1 million pairwise distance comparisons per second per thread. So a full triangle of 1000 2Mbp bacterial genomes is 2.3s + 0.5s.

Ragnar {Groot Koerkamp}
@curiouscoding.nl

or with 6 threads, only 0.7s. That's with bucket size b=8. Sketching is >50x faster than BinDash[-rs], which takes at least 110s. For b=8 as above, computing distances is ~2x faster than BinDash, but for b=1, SimdSketch is again 10x faster. curiouscoding.nl/posts/simd-s...

curiouscoding.nlSimdSketch: a fast bucket sketchTable of Contents 1 Jaccard similarity 2 Hash schemes 2.1 MinHash 2.2 $s$-mins sketch 2.3 Bottom-\(s\) sketch 2.4 FracMinHash 2.5 Bucket sketch 2.6 Mod-bucket hash (new?) 2.7 Variants 3 Compressing sk...
Ragnar {Groot Koerkamp}
@curiouscoding.nl

Also note: for comparison speed, 1-bit hashes beat everything else (top-right red line here). Because each bit is an independent sample, the entropy is very high. (Whereas comparing 8-bit sketches loses some information, because if two 8-bit values differ in the first bit, the other 7 are unused.)

1 repost0 likes