Query-driven Knowledge Integration for a Decentralized Web

Data is highly decentralized

• Produced by governments, companies, individuals, …
• Stored at different locations across the world
• Consumed through interactive applications, processes, intelligent agents, …

Decentralized data integration is challenging for application developers

• Discovering data

  How can I find data sources? How can I find data within a source?

• Combining data

  How to combine data across different data sources?

• Preserving privacy

  How to not leak sensitive data?

Query engines abstract access to decentralized data

Hide the complexities of reading and writing for app developers

↔ ↔

Image credit

Focus on Knowledge Graphs

• Graph-based data model

•   When data does not fit into a fixed relation model

• Based on Semantic Web / Linked Data technologies

•   RDF, SPARQL, …

•   Interlinking data across multiple data sources

SPARQL processing over centralized data

• Dataset is collocated with query engine

  All data is known beforehand

• Single dataset

  With precomputed indexes

• Execute queries of any complexity

  SELECT ?name ?deathDate WHERE {
    ?person a dbpedia-owl:Artist;
            rdfs:label ?name;
            dbpedia-owl:birthPlace [ rdfs:label "York"@en ].
    FILTER LANGMATCHES(LANG(?name),  "EN")
    OPTIONAL { ?person dbpprop:dateOfDeath ?deathDate. }
  }
                  

Centralization not always possible

• Private data

  Technical and legal reasons

• Evolving data

  Requires continuous re-indexing

• Web scale data

  Indexing the whole Web is infeasible (for non-tech-giants)

How to query over decentralized data?

• Data and query engine are not collocated

  Query engine runs on a separate machine

• Not just one datasets

  Data is spread over the Web into multiple data sources

Approaches for querying over decentralized data

• Federated Query Processing

  Distributing query execution across known sources

• Link Traversal Query Processing

  Local query execution over sources that are discovered by following links

Client distributes query over query APIs

• Clients do limited effort

  Split up the query, distribute it (source assignment), and combine results

• Servers perform most of the effort

  They actually execute the queries, over potentially huge datasets

Federation over SPARQL endpoints

• Servers are SPARQL endpoints (most common)

  They accept any valid SPARQL query

• Manual vs. automatic source selection

  Using SERVICE clauses

SELECT ?drug ?title WHERE {
  ?drug db:drugCategory dbc:micronutrient.
  ?drug db:casRegistryNumber ?id.
  ?keggDrug rdf:type kegg :Drug.
  ?keggDrug bio2rdf:xRef ?id.
  ?keggDrug purl:title ?title.
}

SELECT ?drug ?title WHERE {
  SERVICE <http://example.com/drb> {
    ?drug db:drugCategory dbc:micronutrient.
    ?drug db:casRegistryNumber ?id.
  }
  SERVICE <http://example.com/kegg> {
    ?keggDrug rdf:type kegg :Drug.
    ?keggDrug bio2rdf:xRef ?id.
    ?keggDrug purl:title ?title.
  }
}

Federation over heterogeneous sources

• Public SPARQL endpoints have low availability (95%)

  ~1.5 days downtime per month!
  Due to high query expressivity is high and public nature

• Alternative interfaces besides SPARQL endpoints

  Linked Data Fragments: TPF, WiseKG, brTPF, Passage, ...
  Different levels of server expressivity

• Trade-off between server and client effort

  Low-cost publishing prevents server availability issues
  Executing parts of queries client-side

Limitations of federated querying

• All federation members must be known before execution starts

  Source selection distributes query across list of sources
  No discovery of new sources

• Limited scalability in terms of number of endpoints

  Current federation techniques scale to the order of 10 sources

• Public endpoints have restrictions

  Timeouts, rate limits, …
  Current federation techniques can not cope with them

Federated querying over public endpoints in practise does not work due to restrictions

• Crum, Elias, et al. "Does SPARQL federation work in the real world? A case study over large biological SPARQL endpoints." ISWC 2026.

Exploit interlinking of documents

• Linked Data documents are linked to each other

  Following the Linked Data principles

• Query engine can follow links

  Start from one document, and discover new documents on the fly

Example: decentralized address book

Example: Find Alice's contact names

SELECT ?name WHERE {
    <https://alice.pods.org/profile#me>
        foaf:knows ?person.
    ?person foaf:name ?name.
}

Query process:

1. Start from Alice's address book
2. Follow links to profiles of Bob and Carol
3. Query over union of all profiles
4. Find query results: [ { "name": "Bob" }, { "name": "Carol" } ]

Link Traversal: too slow for querying over Linked Open Data

• Massive number of possible links on the open Web

  New content is produced faster than you can follow links

• Inefficient query plans

  Traversal engine can not optimize sufficiently due to lack of statistics

Link Traversal becomes feasible with structural assumptions

• Environments with structural properties

  Subsets of the Web that follow specific structures

• Traversal engines can make additional assumptions

  Assumptions about how data is structured and interlinked
  → Guided link traversal (Taelman 2023)

Solid pods follow structural properties

• Pod contents listed through as REST API

  Linked Data Platform

• WebID profile

  User name and link to storage

• Type index

  Type-based resource discovery

Query results arrive incrementally
within human attention limits

• SPARQL query with 6 triple patterns
• Across 1.531 simulated Solid pods (158.233 files, 3.556.159 triples)
• Taelman, Ruben, and Ruben Verborgh. "Link traversal query processing over decentralized environments with structural assumptions." International Semantic Web Conference 2023.

If pods expose more information, complex querying can become faster

• When following links, engines can discover optimizations

  If pods expose them

• A pod can guide engines more efficiently towards relevant data

  Type indexes, shape trees, shape indexes, …

• Cross-pod indexes can prune out irrelevant pods

  Aggregations or summarization of data across collections of pods
  Dangers: staleness, trust, censorship, …

Pods exposing shape information allows query engines to skip many links

• Tam, Bryan-Elliott, et al. "Traveling with a Map: Reducing the Search Space of Link Traversal Queries Using RDF Shapes." Semantic Web Journal (2026).

Pods exposing cardinality information allows engines to make better query plans

• Hanski, Jonni, et al. "Link Traversal over Decentralised Environments using Restart-Based Query Planning." ICWE 2025.

Client-side caching across improves performance within user sessions

• Eschauzier, Ruben, et al. "SolidSessionBench: Realistic Query Sequences for User-Oriented Decentralized Environments." ISWC 2026.

Conclusion: Query engines are a good match for integrating decentralized data

• This requires intelligent query processing algorithms

  Abstract away complexities of decentralized data
  

• Trade-offs between federated querying and link traversal

  Federation faster, but lacks source discovery and limited in #sources
  Link traversal currently only works over documents
  Need for a hybrid between federation and link traversal?
• 

  Active research

  https://comunica.dev/: Engine for federation + link traversal

We develop the Comunica framework

• Abstraction layer to query decentralized data

  Federation, link traversal, Solid, …
  SPARQL, GraphQL-LD, …

• Modular query engine framework

  Flexible experimentation ground for research (link traversal, federation, …)
  

• Open-source (MIT)

  Used for research and beyond
  Used by 1.500+ open-source projects, 380.0000+ monthly downloads

The future of RDF and SPARQL

• Contributing to RDF and SPARQL W3C working group

  Towards RDF and SPARQL 1.2

• And towards SPARQL 1.3

  Better list support, lateral joins, rate limit announcement, …