Experimental results

In his thesis, de Chazal took each of the ECGs with its heartbeats, and selected what is termed the ``dominant'' heartbeat. The dominant heartbeat is the one that is the most free of noise. Each ECG consisted of between 4 and 16 heartbeats, with an average of approximately 8. Typically, the dominant heartbeat was the 6th heartbeat; however, domain experts analysed each example to see if they were ``typical'' examples of the beat. This of course, requires domain knowledge.

For comparison with de Chazal's, we used the same dominant beats as he did for our first group of experiments.

It is interesting to note that these data files are much larger than a typical instance in a propositional problem: each training instances was between 20 kilobytes and 84 kilobytes, with the average being 47 kilobytes. Even considering just dominant beats, then that is still approximately 24 megabytes of raw data.

Each dominant beat was extracted and labelled by the class. Using 10-fold cross validation (as de Chazal did), we used TClass to learn all 7 classes. The results of the experiments are shown in Table 6.17. For TClass we used both relative time and height because this makes more sense in this domain: The amplitudes depend greatly on the peculiarities of the ECG electrodes; and the beats of the heart vary greatly in duration.

Table 6.17: Error rates for the ECG data (dominant beats only)

Approach	Error
TClass with J48	$51.4 \pm 2.0$
TClass with PART	$45.2 \pm 3.0$
TClass with IB1	$52.4 \pm 3.1$
TClass with NB	$36.6 \pm 2.2$
TClass with Bag	$39.0 \pm 2.6$
TClass with AB	$35.4 \pm 3.8$
Naive Segmentation	$34.2 \pm 2.7$
Hidden Markov Model	$34.4 \pm 2.0$
de Chazal	$28.6 \pm 2.4$
Human expert	$29.7$
Human panel	$25.2$

The results in Table 6.17 seems to be very close between TClass and the baseline learners. However, it does not compare well with de Chazal's work, or even human experts. The results for human cardiologists are shown from de Chazal's thesis. The cardiologists were asked to classify the same database. The ``Human expert'' result is the median value for a single expert and the ``Human panel'' for a voted panel of experts. Note that we are learning based on a small training set of only 500 examples (or in fact, 450 examples once we take into account cross validation); and if we were given more data we may very well perform better.

As before, we can improve the results of TClass by using multiple runs and then voting them. Figure 6.35 shows the result of such a voting scheme on accuracy. Although this shows some improvement, it is still not competitive with de Chazal's work. Perhaps this is not unexpected; given the amount of work de Chazal put in to feature extraction.

Figure 6.35: Voting TClass learners improves accuracy.

However, there is another way to improve accuracy: As mentioned before, each ECG recording consists of multiple beats, with an average of 8 beats per recording. Unfortunately, these other beats do not represent ``good'' data for the following reasons:

• They are likely to very strongly correlated with other beats from the same recording - so it is not really new data in some sense. What is more, we'll be multiplying the data by a factor of 8, which will consequently have an impact on performance and time consumed.
• Some instances have more examples than others, so we may be ``biasing'' the learning to people with faster heartbeats (since they will have more heartbeats recorded).
• The data has not been ``vetted'' by domain experts as typical heartbeats.

Hence our algorithm will have to be robust to noise and able to cope with biases in the input data, as well as able to cope with large amounts of data. In fact, using this approach will use approximately 200 megabytes of input data.

TClass was able to cope with this amount of data, even on a computer with only 512 megabytes of RAM. It necessitated setting some of the parameters for the feature extractors to not produce superfluous events - e.g. tiny maxima and minima of no importance to classification. Tests were run to compare the effect on accuracy and TClass's function was not affected by having stricter constraints that only picked up minima that were a certain width. The results are shown in Table 6.18 for the dominant beats only.

Table 6.18: Effect of only allowing ``wide'' maxima and minima on dominant beats. There is not a significant difference in terms of accuracy.

Approach	Error	Reduced events
TClass with J48	$51.4 \pm 2.0$	$50.2 \pm 2.5$
TClass with PART	$45.2 \pm 3.0$	$46.6 \pm 2.2$
TClass with IB1	$52.4 \pm 3.1$	$52.6 \pm 2.0$
TClass with Bag	$39.0 \pm 2.6$	$35.4 \pm 3.3$
TClass with AB	$35.4 \pm 3.8$	$37.4 \pm 2.4$

One other observation is that the results can easily be voted across all the heartbeats in the test set. Also note that this is applicable not just to the TClass methods but to naive segmentation and hidden Markov models as well.

Table 6.19 shows how this worked in practice.

Table 6.19: Error rates for the ECG data with all beats used

Approach	Error
TClass with J48	$45.5 \pm 1.7$
TClass with PART	$41.9 \pm 2.1$
TClass with IB1	$45.3 \pm 1.3$
TClass with Bag	$35.1 \pm 2.6$
TClass with AB	$32.9 \pm 2.4$
Naive Segmentation	$\mathbf{28.5 \pm 2.6}$
Hidden Markov Model	$33.5 \pm 1.7$

These results can again be improved on by applying voting across different runs of TClass. Figure 6.36 demonstrates the effect of repeated voted of TClass using AdaBoost as a base learner.

Figure 6.36: Voting TClass learners asymptotes to error the hand-selected feature extraction technique.

These results show that even after two runs of TClass with AdaBoost, our performance is indistinguishable from either a human expert or the performance that de Chazal achieved. With minimal background knowledge about the domain, we are able to achieve equivalent accuracy results to someone who used years of domain knowledge to prepare an appropriate set of features.