Hand, Sexton and Mullan [HSM94] believe that perhaps a grammar-based approach to understanding gestures might be feasible. While they did not directly apply their techniques to sign language, as a personal opinion, I believe further investigation in this area might prove fruitful.
To do this, they define several postures that were the terminals of the language. These included things like HO -- hand open, HF --hand fist, IF index finger outstretched. To these, they added several direction movements, like: MU, MD -- move up and down; ML, MR -- move left and right; and MT, MA -- move towards and away.
They then defined the production rules, a few of which are shown below:
<Gesture> :== <Pick> | <Point> |
<Translate> | <Undo> | <Redo> | <Stop>
<Pick> :== <ChooseLocation> <Grab> <Drag> <Drop>
<ChooseLocation> :== TIF-Motion <ChooseLocation>
<Grab> :== TIFC
<Drag> :== TIFC-Motion <Drag>
<Drop> :== TIF
<Point> :== <MovePoint> <SelectPoint>
<MovePoint> :== IF-Motion <MovePoint>
<SelectPoint> :== TIF
<Translate> :== FTFO <Direction>
<Direction> :== ML | MR | MU | MD | MT | MA
<Undo> :== IMFO MA
<Redo> :== IMFO MT
<Stop> :== HF HO
This indicates for example that to stop, you would make a fist and then release it -- or to drag an object, you would hold your thumb and index finger closed and continue to move it.
It is easy to see how such an approach could be adopted for sign language use, with the terminal symbols being handshapes, with additional terminal symbols for changes in orientation and position. Of course, there would be some complexity in the symbol generator, which would take the input from the glove and produce symbols. The second major drawback is that currently, automatic grammar construction is not sufficiently advanced to support this sort of approach, although several techniques exist (as discussed in [Cha93]).
Thus all new signs would have to be added and tested manually. Since the size of the grammar is not known in advance, it is possible that the grammar of Auslan might be ``ambiguous'' -- that is it may not be possible to distinguish all signs, on the basis of structure alone.