In order to assess the feasibility of extending the existing GRASP to a larger lexicon, it would be useful to know how the system ``scales'' with increasing numbers of signs. In section 6.2.2, it was shown that with 95 signs the error rate was 20 per cent. If we increase the number of signs to 200, will the error rate double to 40 per cent? If we went to 1000 signs, would we then get an error rate that was 100 per cent -- would GRASP always be wrong?
It is clear that the error rate is likely to increase as the the number of signs increases. If we use the ``random guess'' comparison, then if you have 1000 signs, then you only have a 0.1 per cent chance of guessing the correct sign. A similar situation applies for learning algorithms. If there are more classes to distinguish between, then the algorithms have to be more capable of discriminating between signs. Essentially the feature space becomes filled with more instances, and thus developing concept boundaries that are tight enough to be accurate is more difficult.
We are still interested in the extent in the change of error rate, however. For example, one of the best possible results would be that we find that the error increases relative to the logarithm of the number of signs. This means, for example, that if we go from 100 signs to 200 signs, then the increase in the error is the same as going from 50 signs to 100 signs. In effect, this would mean large lexicon systems are feasible.
Thus tests were run on each of the three large datasets to observe their behaviour as the number of signs increased. 5-fold cross-validation was used, and we used the most effective attributes as discussed in section 6.2.2. The subsets of signs were selected at random, but were identical for each of the three datasets, to allow a fair comparison (since it is possible to choose some sets of signs that are easier to tell apart than others).
The results are shown in figure 6.3.
Figure 6.3: The effect of an increasing number of
signs on the error rate.
As we would expect, the error increases as the number of signs learnt does. However, there is a noticeable tapering off as the number of signs increases. The data is too noisy for IBL1 to really make any more substantial claims about its behaviour, although it appears that the error rate is increasing less than linearly.
Out of curiosity, the x-axis was logarithmically plotted, in the hope we we would see a more interesting pattern in the data.
This is shown in figure 6.4.
Figure 6.4: The effect of an increasing number of
signs on the error rate using IBL1, this time with a logarithmic x-axis.
It appears that there is a strong logarithmic relationship for C4.5, but the results are more ambiguous for IBL. In either case, however, it is clear from this graph that the rate of growth of the error rate is less than linear.
This bodes well for extension to larger systems. Because of this plateau effect, it is more likely that larger lexicons will not lead to proportionately larger error rates.
More significantly however, is that the C4.5 error rate seems to be increasing faster than that of the IBL datasets.
In conclusion, it appears that both systems behave better than linearly in the number of signs. The results for IBL1 are a little ambiguous and noisy, but for C4.5 it is plausible to suggest that the error rate increases as the logarithm of the number of signs. As mentioned above, although this does not guarantee performance with larger collections of signs, it does indicate that there is potential for the expansion of GRASP to a larger lexicon. Furthermore, it appears that IBL has an advantage over C4.5 in that the rate at which the error increases with increasing number of signs is lower. This suggests that IBL is more suitable for development of large lexicon systems.