Problem solving

Problem solving metrics assess a domain model ability of solving problems. The solving ratio evaluates the model ability of producing plans that are applicable in the environment and achieve the given goal. The false plans ratio measures the ratio of produced plans that are either not applicable in the environment or do not achieve the given goal. Problem solving metrics are defined with respect to a set of problems and a planner.

amlgym.metrics._solving.problem_solving(model_learn_path, model_ref_path, problem_paths, timeout=60, show_progress=True)[source]
Solve the given problems using the model to be evaluated and return:

  1. the solving plans ratio in the environment defined by the reference model

  2. the false plans ratio in the environment defined by the reference model

  3. the ratio of problems deemed unsolvable

  4. the ratio of problems where the planning process timed out

  5. the ratio of problems that raised syntax errors during parsing in unified planning

Parameters:

  • model_learn_path (str) – learned model path

  • model_ref_path (str) – reference model path

  • problem_paths (List[str]) – list of problem paths

  • timeout – planner timeout

Return type:

Dict[str, float]

Returns:

solving and false plans ratios, deemed unsolvable problems, timed out planning processes

amlgym.metrics._solving.validate_plan(model_path, problem_path, plan_path)[source]

Validate the solution plan to the given problem in the environment defined by the given model. The definition of valid plan follows the one provided in “PDDL2. 1: An extension to PDDL for expressing temporal planning domains”, M. Fox and D. Long, JAIR 2003.

Parameters:

  • model_path (str) – path to the PDDL model defining the environment.

  • problem_path (str) – path to the PDDL problem to be solved.

  • plan_path (str) – path to the solution plan for the PDDL problem.

Return type:

float

Returns:

true if the plan is valid, otherwise false.