NL2SHACL-Bench
A Benchmark Suite for Natural Language to SHACL Translation
Paper
NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
Yuchen Zhou, Niels Bobet, Maribel Acosta
Technical University of Munich, Munich, Germany
Resources
| Framework | Source code for the NL2SHACL-Framework |
| Dataset (GitHub) | NL2SHACL-Dataset repository |
| Dataset (Zenodo) | Dataset archive with DOI |
| LLM Evaluation | Prompts, outputs, and evaluation results |
Overview
The Shapes Constraint Language (SHACL) is a W3C recommendation for validating RDF knowledge graphs. It is widely used to ensure data quality and consistency in knowledge graph-based systems. However, authoring SHACL shapes requires expertise in both the application domain and semantic web technologies, which most domain experts lack.
Translating natural language requirements into SHACL (NL2SHACL) offers a promising way to lower this barrier. Despite recent advances in large language models (LLMs), the NL2SHACL problem has not been systematically studied. No benchmark currently supports systematic evaluation of NL2SHACL generation.
NL2SHACL-Bench addresses these challenges. It is the first benchmark suite dedicated to natural language to SHACL translation.
What is NL2SHACL-Bench?
NL2SHACL-Bench consists of three main components:
-
:material-pipe: NL2SHACL-Framework
An extensible pipeline for dataset construction and evaluation, covering the full end-to-end benchmarking workflow.
-
:material-database: NL2SHACL-Dataset
240 manually verified NL-SHACL pairs across six subsets covering five domain-specific settings and one general domain (DBpedia), annotated with ontology information.
-
:material-chart-bar: NL2SHACL-Metrics
Eight metrics across syntactic, structural, and semantic dimensions, tailored to assess the quality of translated SHACL shapes.
Dataset at a Glance
| Subset | # Records | # Node Shapes | # Prop. Shapes | Avg. NL Length | Avg. Shapes per Record |
|---|---|---|---|---|---|
| CHEMROF | 74 | 74 | 441 | 154.5 | 6.96 |
| DCAT | 20 | 35 | 103 | 116.2 | 6.90 |
| ePO | 50 | 50 | 139 | 116.5 | 3.78 |
| Invoice | 78 | 78 | 113 | 75.1 | 2.45 |
| SNIK | 8 | 11 | 49 | 52.0 | 7.50 |
| DBpedia | 10 | 10 | 11 | 9.5 | 2.10 |
| Overall | 240 | 258 | 856 | 108.1 | 4.64 |
Key Findings
Our evaluation of four state-of-the-art LLMs shows that current models can reliably generate syntactically valid SHACL shapes, but struggle to preserve semantic correctness for complex logical and structural patterns. This indicates that NL2SHACL-Bench is not yet saturated and provides a meaningful basis for measuring advances in the state of the art.
Citation
@dataset{zhou2025nl2shacl,
author = {Zhou, Yuchen and Bobet, Niels and Acosta, Maribel},
title = {NL2SHACL-Dataset},
year = {2025},
publisher = {Zenodo},
doi = {10.5281/zenodo.20082565},
url = {https://doi.org/10.5281/zenodo.20082565}
}