NL2SHACL-Metrics
Evaluating NL2SHACL requires going beyond surface-level comparison, as syntactically different shapes may still be semantically equivalent. NL2SHACL-Bench defines six metrics across three complementary dimensions.
Overview
| Dimension | Metric | Symbol |
|---|---|---|
| Validity | RDF Parsing Validity | RDF-VR |
| Validity | SHACL Specification Validity | Spec-VR |
| Validity | SHACL Vocabulary Validity | Vocab-VR |
| Structural | Exact Matching | EMR |
| Structural | Partial Matching | PMS |
| Semantic | Semantic Equivalence Rate | SER |
Validity Metrics
Validity metrics ensure that generated outputs are well-formed and conform to SHACL standards. For each data point, a binary result is assigned. At the dataset level, results are reported as the proportion of outputs satisfying the corresponding condition.
Let $N$ be the total number of data points, and let $P_i, S_i, V_i \in {0,1}$ denote whether the $i$-th output satisfies RDF parsing, SHACL specification, and vocabulary validity, respectively.
RDF Parsing Validity (RDF-VR)
Checks whether the output can be parsed as a valid RDF graph.
$$\text{RDF-VR} = \frac{1}{N}\sum_{i=1}^{N} P_i$$
SHACL Specification Validity (Spec-VR)
Verifies whether the parsed graph conforms to the SHACL specification.
$$\text{Spec-VR} = \frac{1}{N}\sum_{i=1}^{N} P_i S_i$$
SHACL Vocabulary Validity (Vocab-VR)
Ensures that only standard SHACL constraint components are used, under a closed-world assumption.
$$\text{Vocab-VR} = \frac{1}{N}\sum_{i=1}^{N} P_i S_i V_i$$
Structural Metrics
Structural metrics measure how closely a generated shape matches the reference at the graph level. Both graphs are canonicalized to handle blank nodes, and their triple sets are compared.
Exact Matching (EMR)
A binary metric indicating whether the generated and reference shapes are structurally identical after canonicalization. This provides a strict criterion for structural correctness.
Partial Matching (PMS)
A continuous metric measuring the degree of overlap between the two graphs based on triple-level similarity. It is computed as the macro-averaged F1 score across all data points, providing a finer-grained similarity signal than exact matching.
Semantic Metrics
While structural metrics measure similarity at the graph level, they do not capture whether two shapes enforce the same constraints. Semantic metrics assess equivalence in terms of actual validation behavior on RDF data graphs.
Semantic Equivalence Rate (SER)
For each data point, a synthetic RDF data graph is generated using rdf-graph-gen, with the reference shape serving as the schema. The same data graph is then validated against both the reference shape $\mathcal{S}^{*}_i$ and the generated shape $\hat{\mathcal{S}}_i$, producing two validation reports.
From each report, the set of violating focus nodes is extracted and compared. Two shapes are considered equivalent if and only if the sets of violating nodes are identical.
$$\text{SER} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\left[\text{Equiv}(\hat{\mathcal{S}}_i, \mathcal{S}^{*}_i)\right]$$
This metric can also be applied using real-world knowledge graphs, allowing evaluation under application-specific data distributions.