← back
Vector Search Benchmark[eting] - Philipp Krenn, Elastic
Takeaway
Treat vendor vector-search benchmarks as marketing — only trust automated, reproducible, recall-anchored benchmarks across realistic read-write workloads.
Summary
- Philipp Krenn (Elastic) coins 'benchmarketing' — every vendor publishes benchmarks showing they're faster than every competitor, which is suspicious.
- Common tricks: read-only benchmarks when real workloads are read-write; HNSW filtering counterintuitively makes vector search slower (more candidates needed), so vendors pick filter-friendly scenarios.
- Defaults are weaponized — shard size, memory allocation, instance type, compression settings tuned for your own stack, not the competitor's.
- Skipping precision/recall is a classic dodge — you can produce crap results fast with low recall and headline as 'faster'.
- Better practice: nightly automated reproducible benchmarks across versions; ANN-benchmarks/open data sets; publish recall numbers alongside latency.
vector-searchbenchmarkselastic
Original description
Every vector database out there is both faster and slower than any other competitor — if you believe all the benchmarketing out there. Let's turn the marketing into useful benchmarks that actually help you: 1. How not to benchmark (spoiler: don’t trust the glossy charts). 2. What’s uniquely tricky about benchmarking vector search. 3. How to build meaningful benchmarks tailored to your use case. PS: Yes, you will have to get your hands dirty. Never believe a benchmark that you haven't tweaked yourself. About Philipp Krenn Philipp leads Developer Relations at Elastic — the company behind the Elasticsearch, Kibana, Beats, and Logstash. Based in San Francisco, he lives to demo interesting technology and solve challenging problems — all with a smile and a terminal window. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter