1. |
- Johansson, Ulf, et al.
(författare)
-
Accurate and Interpretable Regression Trees using Oracle Coaching
- 2014
-
Konferensbidrag (refereegranskat)abstract
- In many real-world scenarios, predictive models need to be interpretable, thus ruling out many machine learning techniques known to produce very accurate models, e.g., neural networks, support vector machines and all ensemble schemes. Most often, tree models or rule sets are used instead, typically resulting in significantly lower predictive performance. The over- all purpose of oracle coaching is to reduce this accuracy vs. comprehensibility trade-off by producing interpretable models optimized for the specific production set at hand. The method requires production set inputs to be present when generating the predictive model, a demand fulfilled in most, but not all, predic- tive modeling scenarios. In oracle coaching, a highly accurate, but opaque, model is first induced from the training data. This model (“the oracle”) is then used to label both the training instances and the production instances. Finally, interpretable models are trained using different combinations of the resulting data sets. In this paper, the oracle coaching produces regression trees, using neural networks and random forests as oracles. The experiments, using 32 publicly available data sets, show that the oracle coaching leads to significantly improved predictive performance, compared to standard induction. In addition, it is also shown that a highly accurate opaque model can be successfully used as a pre- processing step to reduce the noise typically present in data, even in situations where production inputs are not available. In fact, just augmenting or replacing training data with another copy of the training set, but with the predictions from the opaque model as targets, produced significantly more accurate and/or more compact regression trees.
|
|
2. |
- Johansson, Ulf, et al.
(författare)
-
Conformal Prediction for Accuracy Guarantees in Classification with Reject Option
- 2023
-
Ingår i: Modeling Decisions for Artificial Intelligence. - : Springer. - 9783031334979 ; , s. 133-145, s. 133-145
-
Konferensbidrag (refereegranskat)abstract
- A standard classifier is forced to predict the label of every test instance, even when confidence in the predictions is very low. In many scenarios, it would, however, be better to avoid making these predictions, maybe leaving them to a human expert. A classifier with that alternative is referred to as a classifier with reject option. In this paper, we propose an algorithm that, for a particular data set, automatically suggests a number of accuracy levels, which it will be able to meet perfectly, using a classifier with reject option. Since the basis of the suggested algorithm is conformal prediction, it comes with strong validity guarantees. The experimentation, using 25 publicly available two-class data sets, confirms that the algorithm obtains empirical accuracies very close to the requested levels. In addition, in an outright comparison with probabilistic predictors, including models calibrated with Platt scaling, the suggested algorithm clearly outperforms the alternatives.
|
|
3. |
- Johansson, Ulf, et al.
(författare)
-
Fish or Shark : Data Mining Online Poker
- 2009
-
Konferensbidrag (refereegranskat)abstract
- In this paper, data mining techniques are used to analyze data gathered from online poker. The study focuses on short-handed Texas Hold’em, and the data sets used contain thousands of human players, each having played more than 1000 hands. The study has two, complementary, goals. First, building predictive models capable of categorizing players into good and bad players, i.e., winners and losers. Second, producing clear and accurate descriptions of what constitutes the difference between winning and losing in poker. In the experimentation, neural network ensembles are shown to be very accurate when categorizing player profiles into winners and losers. Furthermore, decision trees and decision lists used to acquire concept descriptions are shown to be quite comprehensible, and still fairly accurate. Finally, an analysis of obtained concept descriptions discovered several rather unexpected rules, indicating that the suggested approach is potentially valuable for the poker domain.
|
|
4. |
- Johansson, Ulf, et al.
(författare)
-
Locally Induced Predictive Models
- 2011
-
Konferensbidrag (refereegranskat)abstract
- Most predictive modeling techniques utilize all available data to build global models. This is despite the wellknown fact that for many problems, the targeted relationship varies greatly over the input space, thus suggesting that localized models may improve predictive performance. In this paper, we suggest and evaluate a technique inducing one predictive model for each test instance, using only neighboring instances. In the experimentation, several different variations of the suggested algorithm producing localized decision trees and neural network models are evaluated on 30 UCI data sets. The main result is that the suggested approach generally yields better predictive performance than global models built using all available training data. As a matter of fact, all techniques producing J48 trees obtained significantly higher accuracy and AUC, compared to the global J48 model. For RBF network models, with their inherent ability to use localized information, the suggested approach was only successful with regard to accuracy, while global RBF models had a better ranking ability, as seen by their generally higher AUCs.
|
|
5. |
- Johansson, Ulf, et al.
(författare)
-
One Tree to Explain Them All
- 2011
-
Konferensbidrag (refereegranskat)abstract
- Random forest is an often used ensemble technique, renowned for its high predictive performance. Random forests models are, however, due to their sheer complexity inherently opaque, making human interpretation and analysis impossible. This paper presents a method of approximating the random forest with just one decision tree. The approach uses oracle coaching, a recently suggested technique where a weaker but transparent model is generated using combinations of regular training data and test data initially labeled by a strong classifier, called the oracle. In this study, the random forest plays the part of the oracle, while the transparent models are decision trees generated by either the standard tree inducer J48, or by evolving genetic programs. Evaluation on 30 data sets from the UCI repository shows that oracle coaching significantly improves both accuracy and area under ROC curve, compared to using training data only. As a matter of fact, resulting single tree models are as accurate as the random forest, on the specific test instances. Most importantly, this is not achieved by inducing or evolving huge trees having perfect fidelity; a large majority of all trees are instead rather compact and clearly comprehensible. The experiments also show that the evolution outperformed J48, with regard to accuracy, but that this came at the expense of slightly larger trees.
|
|
6. |
- Johansson, Ulf, et al.
(författare)
-
Oracle Coached Decision Trees and Lists
- 2010
-
Konferensbidrag (refereegranskat)abstract
- This paper introduces a novel method for obtaining increased predictive performance from transparent models in situations where production input vectors are available when building the model. First, labeled training data is used to build a powerful opaque model, called an oracle. Second, the oracle is applied to production instances, generating predicted target values, which are used as labels. Finally, these newly labeled instances are utilized, in different combinations with normal training data, when inducing a transparent model. Experimental results, on 26 UCI data sets, show that the use of oracle coaches significantly improves predictive performance, compared to standard model induction. Most importantly, both accuracy and AUC results are robust over all combinations of opaque and transparent models evaluated. This study thus implies that the straightforward procedure of using a coaching oracle, which can be used with arbitrary classifiers, yields significantly better predictive performance at a low computational cost.
|
|
7. |
- Johansson, Ulf, et al.
(författare)
-
Using Genetic Programming to Obtain Implicit Diversity
- 2009
-
Konferensbidrag (refereegranskat)abstract
- When performing predictive data mining, the use of ensembles is known to increase prediction accuracy, compared to single models. To obtain this higher accuracy, ensembles should be built from base classifiers that are both accurate and diverse. The question of how to balance these two properties in order to maximize ensemble accuracy is, however, far from solved and many different techniques for obtaining ensemble diversity exist. One such technique is bagging, where implicit diversity is introduced by training base classifiers on different subsets of available data instances, thus resulting in less accurate, but diverse base classifiers. In this paper, genetic programming is used as an alternative method to obtain implicit diversity in ensembles by evolving accurate, but different base classifiers in the form of decision trees, thus exploiting the inherent inconsistency of genetic programming. The experiments show that the GP approach outperforms standard bagging of decision trees, obtaining significantly higher ensemble accuracy over 25 UCI datasets. This superior performance stems from base classifiers having both higher average accuracy and more diversity. Implicitly introducing diversity using GP thus works very well, since evolved base classifiers tend to be highly accurate and diverse.
|
|
8. |
- Sönströd, Cecilia, et al.
(författare)
-
Evaluating Algorithms for Concept Description
- 2009
-
Konferensbidrag (refereegranskat)abstract
- When performing concept description, models need to be evaluated both on accuracy and comprehensibility. A comprehensible concept description model should present the most important relationships in the data in an accurate and understandable way. Two natural representations for this are decision trees and decision lists. In this study, the two decision list algorithms RIPPER and Chipper, and the decision tree algorithm C4.5, are evaluated for concept description, using publicly available datasets. The experiments show that C4.5 performs very well regarding accuracy and brevity, i.e. the ability to classify instances with few tests, but also produces large models that are hard to survey and contain many extremely specific rules, thus not being good concept descriptions. The decision list algorithms perform reasonably well on accuracy, and are mostly able to produce small models with relatively good predictive performance. Regarding brevity, Chipper is better than RIPPER, using on average fewer conditions to classify an instance. RIPPER, on the other hand, excels in relevance, i.e. the ability to capture a large number of instances with every rule.
|
|
9. |
- Sönströd, Cecilia, et al.
(författare)
-
Evolving Accurate and Comprehensible Classification Rules
- 2011
-
Konferensbidrag (refereegranskat)abstract
- In this paper, Genetic Programming is used to evolve ordered rule sets (also called decision lists) for a number of benchmark classification problems, with evaluation of both predictive performance and comprehensibility. The main purpose is to compare this approach to the standard decision list algorithm JRip and also to evaluate the use of different length penalties and fitness functions for evolving this type of model. The results, using 25 data sets from the UCI repository, show that genetic decision lists with accuracy-based fitness functions outperform JRip regarding accuracy. Indeed, the best setup was significantly better than JRip. JRip, however, held a slight advantage over these models when evaluating AUC. Furthermore, all genetic decision list setups produced models that were more compact than JRip models, and thus more readily comprehensible. The effect of using different fitness functions was very clear; in essence, models performed best on the evaluation criterion that was used in the fitness function, with a worsening of the performance for other criteria. Brier score fitness provided a middle ground, with acceptable performance on both accuracy and AUC. The main conclusion is that genetic programming solves the task of evolving decision lists very well, but that different length penalties and fitness functions have immediate effects on the results. Thus, these parameters can be used to control the trade-off between different aspects of predictive performance and comprehensibility.
|
|
10. |
- Sönströd, Cecilia, et al.
(författare)
-
Generating Comprehensible QSAR Models
- 2009
-
Konferensbidrag (refereegranskat)abstract
- This paper presents work in progress from the INFUSIS project and contains initial experimentation, using publicly available medicinal chemistry datasets, on obtaining comprehensible QSAR models. Three techniques are evaluated on both predictive performance, measured as accuracy, and comprehensibility, measured as model size. The chosen techniques are J48 decision trees and JRip and Chipper decision lists. The results show that J48 obtains superior accuracy and that Chipper performs best of the two decision list algorithms on accuracy. Furthermore, it is seen that, regarding accuracy, all techniques benefit from feature reduction, which almost always results in increased accuracy. Regarding comprehensibility, JRip obtains the smallest models, followed by Chipper, with J48 producing the largest models. For model size, feature reduction is not seen to be universally beneficial; only J48 produces smaller models for the reduced datasets, while both decision list algorithms actually produce larger models on average. The overall conclusion is that, for these datasets, there exists a definite tradeoff between accuracy and comprehensibility that needs to be investigated further.
|
|