Strengthening LargeRDFBench for Interoperable Federated SPARQL Evaluation

LargeRDFBench is one of the most comprehensive benchmarks for evaluating federated SPARQL query engines, combining a large collection of real, interlinked datasets with a rich query suite that has made it a reference point for the community. As the benchmark continues to be widely adopted and the range of conformant engines grows, the interoperability of its artifacts, and thus the reproducibility of comparisons across engines, becomes ever more important. Yet several of its data dumps violate the RDF specifications, and its expected query results are distributed in an ad hoc format and contain discrepancies with respect to the source datasets. We systematically identify and categorize these data-quality issues and repair them, producing standards-conformant serializations of every affected dataset; we re-encode the benchmark’s expected results in the W3C SPARQL 1.1 Query Results JSON Format and correct their discrepancies. We contribute a standards-compliant edition of LargeRDFBench, produced by a reproducible cleaning pipeline, together with its expected query results in a standard, machine-verifiable format. Every dataset now parses under strict, specification-compliant RDF parsers, and the expected results are machine-verifiable through a standard format, extending the benchmark’s reach to the full range of conformant engines while staying faithful to the original data. Running the modernized benchmark end-to-end, we validate and correct its expected results, and find that the hosting engine can change the answers the benchmark returns, so that results obtained on it are not always reproducible across engines. We further perform a preliminary comparison, not previously explored, of ASK- and COUNT-based source selection in the FedX algorithm. This work strengthens an already valuable community resource by aligning its artifacts with the RDF standards. In doing so, we broaden the set of engines that can be fairly and reproducibly compared, and open the question of how the server-side SPARQL engines that host the data influence algorithmic choices and, through them, federated query performance.

@inproceedings{tam_dmkg_largerdfbench_2026,
  author = {Tam, Bryan-Elliott and Muhammad, Saleem and Taelman, Ruben},
  title = {Strengthening LargeRDFBench for Interoperable Federated SPARQL Evaluation},
  month = oct,
  booktitle = {Proceedings of the 2nd International Workshop on Data Management for Knowledge Graphs},
  year = {2026}
}