AutoDBBench: An Automated Benchmarking Workbench for Relational, Graph, Vector, and Lakehouse Databases
Main article
Abstract
The modern data stack now routinely combines relational engines, property graph databases, vector indexes, and lakehouse storage in the same analytical workflow, yet there is no widely accepted benchmarking infrastructure that treats these four families as comparable members of a single evaluation universe. Existing benchmarks such as TPC-C, TPC-H, LDBC SNB, and ANN-Benchmarks are excellent within their respective domains but use incompatible workload generators, metric reporters, and reproducibility conventions, so a practitioner comparing alternatives must stitch together heterogeneous tools and accept that the resulting numbers will not be directly comparable. This article presents AutoDBBench, an automated benchmarking workbench that unifies workload generation, query execution, resource monitoring, and visualization across the four database families through a common internal data model. AutoDBBench centers its design on the database itself: a documented schema, a typed field dictionary, indexed metric storage, a quality control pipeline, and a reusable application programming interface together turn the workbench into a research database rather than a one-off scripting harness. We describe the architecture, the internal data model, the workload-template grammar, and the fault-injection facility, and we report a runnable experiment over eight target databases on a five-node test cluster. Across the chosen workloads AutoDBBench surfaces a 5.6 percent throughput regression in a graph engine that none of the upstream vendor benchmarks detected, attributes a 41.7 percent latency tail to a buffer pool misconfiguration, demonstrates linear scalability up to 16 nodes for three of the four families, and recovers from injected network partitions in 35 to 88 seconds. The full workbench, including configuration files, dictionaries, and reproducible containers, is released under an open license.
