Syntactic similarity

Syntactic similarity metrics compare the intersection or difference of the atoms in the action preconditions and effects between an evaluated domain model \(M\) and reference model \(M^{*}\). Let \(pre_M(a)\) be the set of preconditions of an action \(a\) according to \(M\).

  • True Positives: \(TP_{pre}(a)=|(pre_M(a)\cap pre_M^{*}(a)|\)

  • False Positives: \(FP_{pre}(a)=|(pre_M(a)\setminus pre_M^{*}(a)|\)

  • False Negatives: \(FN_{pre}(a)=|(pre_M^{*}(a)\setminus pre_M(a))|\)

The preconditions precision \(P_{pre}\) and recall \(R_{pre}\) are defined as:

  • Syntactic precision of \(pre_a\) : \(P_{pre}(a) = \frac{TP_{pre}(a)}{TP_{pre}(a)+FP_{pre}(a)}\)

  • Syntactic recall of \(pre_a\): \(R_{pre}(a) = \frac{TP_{pre}(a)}{TP_{pre}(a)+FN_{pre}(a)}\)

The precision of an action \(a\) is defined by summing up \(TP(a)\) over the preconditions \(pre\), positive effects \(eff^+\) and negative effects \(eff^-\):

  • Syntactic precision of \(a\) : \(P(a) = \frac{TP(a)}{TP(a)+FP(a)}\)

  • Syntactic recall of \(a\) : \(R(a) = \frac{TP(a)}{TP(a)+FN(a)}\)

Finally, the overall precision and recall of a domain model averages over the actions precision and recall:

  • Syntactic precision of \(M\): \(P = \frac{1}{|A|}\sum_{a\in A} P(a)\)

  • Syntactic recall of \(M\): \(R = \frac{1}{|A|}\sum_{a\in A} R(a)\)

amlgym.metrics._syntactic.syntactic_precision(evaluated_path, reference_path)[source]

Evaluate the syntactic precision metric of a domain model \(M\) with respect to a reference model \(M^{*}\).

Parameters:

  • evaluated_path (str) – path of the PDDL model to evaluate.

  • reference_path (str) – path of the PDDL reference model.

Return type:

Dict[str, float]

Returns:

the syntactic precision grouped by preconditions and effects

amlgym.metrics._syntactic.syntactic_recall(evaluated_path, reference_path)[source]

Evaluate the syntactic recall metric of a domain model \(M\) with respect to a reference model \(M^{*}\).

Parameters:

  • evaluated_path (str) – path of the PDDL model to evaluate.

  • reference_path (str) – path of the PDDL reference model.

Return type:

Dict[str, float]

Returns:

the syntactic recall grouped by preconditions and effects