Assessing Accuracy of Weak TC

On a single element $S$, one way to measure success is to say that if $\ensuremath{\mathit{class}_P}(S) = \ensuremath{\mathit{class}}(S)$ then it is accurate, and inaccurate otherwise.

This works for most cases. However, in some domains, the above is too simplistic - not all inaccuracies are equally bad. Some errors may be worse than others. For example, consider working on a medical TC application involving a diagnosis, where $\mathit{CL} = \{yes, no\}$, with ``yes'' indicating they have some condition and ``no'' indicating they do not. A ``false positive'' classification (i.e. misclassifying a negative as a positive) may not be as bad as a ``false negative'' (i.e. misclassifying a positive as a negative). This is an old problem in machine learning, and a field termed cost-sensitive learning has studied this problem.

Typically, it is solved by introducing a function $\ensuremath{\mathit{cost}}(i, j): CL \times CL \rightarrow [0..1]$ which tells us what the cost of misclassifying an $i$ as a $j$. The function need not be i-j symmetric, i.e. $\ensuremath{\mathit{cost}}(i,j) \neq \ensuremath{\mathit{cost}}(j,i)$. Universally, $cost(i,i) = 0$.

We can represent the above simple case (where all errors are equally bad) as:

$$cost(i,j) = 0, \mathrm{if}\ i = j$$

$$= 1, \mathrm{otherwise}$$

However, we can always have a more complex cost function.

Secondly, it is better to get higher accuracy on frequently occurring elements than on rare ones. So to give a more accurate measure of accuracy, this too must be included. We use the function $P_\ensuremath{\mathit{SS}}(S)$ to indicate the probability that a stream $S$ has of occurring in the stream set $\mathit{SS}$.

Our goal can therefore be defined as minimising:

$$\sum_{S \in \ensuremath{\mathit{SS}}} \ensuremath{\mathit{cost}}(... ...{class}}(S),\ensuremath{\mathit{class}_P}(S))\ P_{\ensuremath{\mathit{SS}}}(S)$$