Evaluation schema - Satisficing tracks
The following rules apply for satisficing tracks:
- Each planner is given 30 mins to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
The quality of the resulting plans is important! Suppose a problem with an optimal solution having quality Q* and a planner finding a plan with quality Q<Q* (assume higher qualities are considered best). This planner would aggregate the quality ratio Q/Q* to its total score (use Q*/Q in case lower qualities are considered best). The organizers will try to find optimal solutions for all problems, through the development of specialized domain-specific solvers. In cases where this is not feasible, Q* will be considered the best plan quality found by any competing planner or by the organizers.
- The planner with the highest sum of quality ratios is declared the winner in each category.
Some additional notes:
- Problem instances for the satisficing tracks will be drawn from the same domains as for the optimal tracks. However, it is expected that problem instances for the satisficing tracks will be larger that those for the optimal tracks.