Online Proceedings of the IJCAI-99 Workshop on Empirical Artificial Intelligence

Since the beginning of AI, experiments have been used to demonstrate the value and potential of new techniques. While in earlier years, the demonstration of a new technique in a toy-domain was sufficient, the standards for acceptable experiments have significantly changed. In later years, more emphasis was put on "real-world problems" as well as on more stringent evaluations of experi ments. At least in part stimulated by Paul Cohen's book "Empirical Methods for AI", recent years have seen a massive increase in the use of statistical techniques to assess the validity of empirical results. However, what issues can and should be addressed by such quantitative approaches is not clear.

This one day workshop aims at clarifying the role of empirical methods in AI. Attention will be given to the traditional factors found in other empirical sciences (such as the problem of representativity of case studies, the proper use of statistics, etc.) as well as to the more traditional objectives in AI research: instead of quantifying the relative performance of certain techniques, we are often interested in the development of new concepts allowing a new and hopefully more adequate understanding of certain issues, such as our own cognitive processes, the nature of the tasks we want to solve, etc. What kind of empirical studies can help us in the development of new concepts and frameworks? To stimulate the discussion in the workshop, it will begin with an invited talk by Paul Cohen, author of "Empirical Methods for AI".

This workshop continues a sequence of workshops on Empirical AI held at previous the IJCAI-97, ECAI-96 and ECAI-98 conferences. These workshops were preceded by more specialised workshops at AAAI-94 on Experimental Evaluation of Reasoning and Search Methods and the CADE-12 Workshop on Evaluation of Automated Theorem Provers.

While previous workshops had an emphasis on performance comparisons of heuristics for combinatorial problems, the IJCAI-99 workshop promises to reflect a broader spectrum of research areas in AI.

On behalf of the organising committee I wish to thank Paul Cohen for agreeing to present the invited lecture. A big thank you goes also to all authors for their efforts and to all other participants for contributing to the success to the workshop. Finally, I wish to thank the members of the organising committee for the smooth cooperation during the preparation of this workshop.

Achim Hoffmann

(Chair of the Organising Committee)

Table of Contents

Hoffmann	Introduction
Anbulagan	Meticulous Empirical Study on Phase Transition
Bhattacharyya and Laird	Empirical Methods for Plan Execution Architectures
Endres-Niggemeyer	Grounded Theory Methodology for Knowledge Engineering
Bonet and Geffner	Planning as Heuristic Search: New Results
Gini and Benfenati	What experiments are suitable to access the prediction of toxic effects of chemicals
Hoos and Stuetzle	On the Empirical Evaluation of Las Vegas Algorithms - Position Paper
Kaindl and Kainz	Guidelines for the Experimental Comparison of Search Algorithms
Korb	Calibration and the Evaluation of Predictive Learners
Menzies	The TIM-BO model and the Fairmont Effect
Mitchell	A Remark on Benchmarks and Analysis
Thakar	Industrial experiences in empirical evaluation of domain modelling approaches