Skip to content

NL2SHACL-Framework

The NL2SHACL-Framework is an extensible pipeline that supports end-to-end benchmarking of NL2SHACL systems. It covers dataset construction and evaluation while remaining agnostic to the underlying translation model.

Framework Overview

Overview

The framework consists of three components:

  1. Dataset Construction Module - builds paired NL-SHACL datasets from raw shape files and ontologies
  2. NL2SHACL Translator Module - performs the translation task (model-agnostic)
  3. Shapes Evaluation Module - evaluates generated shapes against reference shapes

The framework is implemented in Python and is publicly available on GitHub.

Design Principles

The framework is designed around three key properties:

Modularized Each component can be used independently. Users can focus on dataset construction, model evaluation, or the full pipeline as needed.

Extensible Users can incorporate custom input files to generate new datasets, replace the translation module with arbitrary models, and extend the evaluation component with additional metrics.

Reproducible The framework defines a consistent pipeline for dataset construction and evaluation, ensuring that results can be reliably reproduced.

Component 1: Dataset Construction Module

This module is a semi-automated pipeline for constructing paired NL-SHACL datasets. Its input consists of raw RDF shape graph files and domain ontology files.

Data Pre-processor

The pre-processor performs three steps:

  1. Shape decomposition - each input SHACL file is split into individual records. For each record, a shape is extracted together with all shapes it references via shape-based constraint components (e.g., sh:node, sh:property, sh:qualifiedValueShape). These referenced shapes are included to form a self-contained shapes graph. Shapes containing SPARQL-based constraints are discarded.

  2. Syntactic validity check - each SHACL file is parsed into an RDF graph. Shapes that fail parsing are removed.

  3. Ontology enrichment - each record is enriched with ontology information. Domain terms are extracted and linked to ontology metadata. Records with unresolved domain terms are discarded.

Description Constructor

This component generates a natural language description for each SHACL record.

  • If a record already contains a sh:description, it is manually reviewed and retained if usable.
  • If the description is missing or insufficient, a description draft is generated using an LLM. The model is prompted with the reference SHACL shape and instructed to express the underlying constraints in natural language as a domain expert would, without using RDF or SHACL terminology.

Description Reviewer

A two-round human validation process is used to ensure quality and mitigate hallucinations.

  • Round 1: Two annotators independently review each description alongside its SHACL record, identifying incorrectly described constraints, missing or redundant information, and the use of ontology-specific terms.
  • Round 2: Annotators discuss flagged cases and agree on revisions. The final descriptions are refined to faithfully reflect the constraints encoded in the corresponding SHACL shapes.

A graphical user interface (GUI) was developed to support this review process.

Component 2: NL2SHACL Translator Module

This module performs the NL2SHACL translation task. Given a natural language description and the corresponding ontology snippets, it produces a translated shapes graph.

The module is model-agnostic and does not impose any constraints on the underlying implementation. Users can plug in arbitrary translation systems, including LLM-based or rule-based approaches.

Component 3: Shapes Evaluation Module

This module evaluates the quality of translated shapes by comparing them against the reference shapes. Given a translated shapes graph and its corresponding reference shapes graph, the module assesses their equivalence across syntactic, structural, and semantic dimensions.

The module is extensible and allows additional metrics to be incorporated. See the Metrics page for details.