Günther Charwat, Giovambattista Ianni, Thomas Krennwallner, Martin Kronegger, Andreas Pfandler, Christoph Redl, Martin Schwengerer, Lara Spendier, Johannes Peter Wallner and Guohui Xiao
VCWC: A Versioning Competition Workflow Compiler
System competitions evaluate solvers and compare state-of-the-art implementations on benchmark sets in a dedicated and controlled computing environment comprising of multiple hosts. An important task for running a competition is the benchmark execution platform that schedules the workload on available benchmark machines, keeps track of failed and finished jobs, and calculates the competition statistics and solver ranking.
In this paper we present VCWC, the Versioning Competition Workflow Compiler. This tool takes as input the participating solvers and dedicated benchmark sets and generates a workflow description for executing all necessary (sub-)tasks for generating the final solver rankings and statistics. As jobs may fail during the execution, VCWC supports a gradual refinement of the competition workflow and allows to add or update solvers, instances, benchmarks, or further runs after the machinery has been brought up. We introduce an abstract model for a competition and present the implementation and system architecture for VCWC. Based on this we report how VCWC is used for the Answer Set Programming Competition 2013.