In what follows you will be introduced to the next of the so-called Sisyphus Experiments.
These experiments began in 1990 as an attempt by the knowledge acquisition community to compare and contrast different methods and techniques used in the construction of knowledge-based systems.
The original domain was the now infamous Room Allocation Problem proposed by Marc Linster. A large number of different sites contributed solutions to the problem. This experiment significantly enhanced our ability to liken and contrast approaches that had hitherto seemed difficult to compare. Many important insights and publications were forthcoming.
A criticism of Sisyphus I was that the application was somewhat simple. In particular, only a small amount of KA material was provided and this was supplied in its entirity at the outset of the problem. There was also a concern that the various approaches might not be able to scale up to larger problem domains.
Sisyphus II tackled a much larger problem - namely, the design of complex mechanical devices. Originally Sandra Marcus had built a KBS called VT to design elevators. Elevator design involves many components and complex constraints. Sisyphus II used the VT domain for its second experiment. On this occassion Greg Yost provided a much more substantial document detailing the knowledge required to build such a system. Yost's document was drawn from a number of sources but again only a limited range of acquisition methods were employed and the material was provided in a single "one-off" document.
Once again important lessons and insights were forthcoming. Many of the solutions are accessible on the web together with: the original problem description, and the Yost document. During Sisyphus II Ontolingua was used as a means of encoding and documenting a standard ontology that the various approaches could contribute to and take advantage of.
At the Banff 95 KA workshop consideration was given as to whether a third Sisyphus should be undertaken. There was a general consensus that it ought and that a number of specific objectives should be set out to complement the progress made in earlier Sisyphus work.
In both Sisyphus I and II the majority of effort was devoted to the knowledge modelling and implementation aspects of KBS construction. In particular, a dominant theme was the use of methodologies that use Problem Solving Models of various sorts to mediate the analysis and modelling of a domain. Whilst important the actual process of knowledge acquistion and the process of transforming the acquistion results into knowledge models has gone largely undocumented. Indeed the process of knowledge engineering employed in the earlier experiments has not been documented in any quantitative fashion.
This provided a context for the principal objectives of Sisyphus III. These are
How are they to be realised? The objectives together impose a number of requirements on any potential domain. It should not be too large and needs to be capable of having achievement metrics associated with it. There neeeds to be an existing corpus of varied but systematic KA material that can be released in a controlled fashion. It should be sufficiently complex to exercise the interpretive skills of knowledge engineers but not so complex as to necessitate the formulation of novel problem solving architectures.
We hope that the domain described will enable such measure to be taken - not least because we have at Nottingham used this domain extensively in past projects for quantitative assessment of the efficacy and utility of varous KA techniques. This research has resulted in us amassing a large amount of raw and processed KA material. It also required us to formulate a so-called "gold standard" against which to assess aspects of the knowledge engineering process. The gold standard comprises a set of necessary and sufficient knowledge to tackle a core subset of the problem domain. We have also recently begun large scale experiments to understand the effectiveness models for model based knowledge acquistion. This experience we hope places us in a good position to facilitate a Sisyphus III experiment.
The Rules of the Game detail the terms and conditions by which we undertake Sisyphus III. We welcome any constructive suggestions about how the procedures could be improved.
The original timetable envisaged presentation of initial results at EKAW'96. Further discussion at a working group meeting, led to a refined definition of the experiment and a revised, but final set of dates.
There will be staged release of material. The first release will include a detailed problem specification together with a first set of acquisition material that will be common to all participants. In future releases participants will be free to select some or all of the KA material offered.
A second release of material will be available together with an announced expansion of the problem set to be solved.
Finally, a third stage will anounce a significant extension to the functionality of the system that should be incorporated if possible.
All KA material will have an associated price according to an economic model. Participants will keep a record of KA spent as well as project logs to record the amount of time spent on various activities in building their models and implementing their system. This will also be recorded in terms of a standard Sisyphus knowledge engineer day rate.
To move from stage to stage we will request that current versions of the system and associated models are submitted to us together with metaprotocols of your treatment of selected KA material and your project logs showing tasks undertaken, and time spent.
It is not our intention to impose either onerous nor bureaucratic constraints - simply to acquire sufficient information to deepen our undertstanding of the qualitative and quantitative aspects of various KA methodologies.
If you are interested in taking part in this particular Sisyphus experiment, have a look at the problem statement and and mail sisyphus@psychology.nottingham.ac.uk with an indication of the time period over which you intend to carry out the work. People who have signed up will have the initial data set released to them on November 22nd 1996.