.jpg)
Zilliz, the creators of the open-source vector database Milvus, has launched VDBBench 1.0, an innovative benchmarking platform designed to evaluate vector databases under realistic production conditions. This open-source solution addresses the gap between traditional benchmarks and actual enterprise performance, empowering organizations to make informed infrastructure decisions.
VDBBench 1.0 tests vector databases with real-world production workloads.
Features streaming data ingestion and metadata filtering for accuracy.
Uses modern datasets from OpenAI and Cohere, with 768–1,536 dimensions.
Supports custom datasets for industry-specific performance testing.
Prioritizes P95/P99 latency, throughput, and recall for capacity planning.
Available on GitHub with interactive dashboard and community support.
The vector database market has surged since 2023, but traditional benchmarks often rely on static data, failing to reflect real-world complexities. “The vector database market has experienced explosive growth since 2023, but our benchmarking methodologies haven’t kept pace,” said James Luan, VP of Engineering at Zilliz. VDBBench 1.0 introduces a new standard by simulating production environments, including streaming data ingestion and concurrent workloads, to deliver authentic performance insights.
VDBBench 1.0 offers advanced features like metadata filtering analysis across selectivity levels (50% to 99.9%), revealing performance bottlenecks that impact query speeds and recall accuracy. Its streaming read/write testing mimics continuous data ingestion while serving queries, conditions where many databases underperform despite strong static benchmarks. These capabilities ensure organizations can assess vector databases under realistic AI-driven workloads.
Unlike legacy benchmarks using outdated datasets like SIFT or GloVe, VDBBench employs vectors from state-of-the-art embedding models like OpenAI and Cohere, with dimensions ranging from 768 to 1,536. It also supports custom datasets, allowing organizations to test with their own production data and embedding models, ensuring relevance for specific use cases like Retrieval-Augmented Generation (RAG).
VDBBench prioritizes metrics critical for production environments, such as P95/P99 tail latency, sustainable throughput, and recall accuracy. Its redesigned dashboard offers interactive visualizations, enabling engineers to quickly identify performance gaps. Results for platforms like Milvus, Zilliz Cloud, Pinecone, and Elasticsearch are accessible on the VDBBench Leaderboard, fostering transparency and comparison.
VDBBench 1.0, freely available on GitHub with comprehensive documentation, empowers organizations to move beyond misleading benchmarks. By providing a robust, open-source tool for realistic vector database testing, Zilliz is setting a new industry standard, ensuring AI applications perform reliably in production environments.
Zilliz builds next-generation database technologies that help organizations unlock the value of unstructured data and rapidly develop AI applications. Headquartered in Redwood Shores, California, Zilliz is backed by leading investors, including Aramco's Prosperity7 Ventures and Temasek's Pavilion Capital.
TechIntelPro empowers B2B technology decision-makers with the latest industry trends, deep insights, leadership perspectives and actionable intelligence to drive both organizational and professional milestones—and thrive on the tech pulse.