1. Introduction; 2. Machine learning and statistics overview; 3. Performance measures I; 4. Performance measures II; 5. Error estimation; 6. Statistical significance testing; 7. Data sets and experimental framework; 8. Recent developments; 9. Conclusion; Appendix A: statistical tables; Appendix B: additional information on the data; Appendix C: two case studies.
Gives a solid basis for conducting performance evaluations of learning algorithms in practical settings with an emphasis on classification algorithms.
Nathalie Japkowicz is an Associate Professor at the School of Information Technology and Engineering of the University of Ottawa. She is a former assistant professor at Dalhousie University and lecturer at Ohio State University. Japkowicz co-organized numerous workshops on classifier evaluation and the class imbalance problem at AAAI and ICML. She has published many articles in peer-reviewed journals and conference proceedings. Mohak Shah is a Postdoctoral Fellow at the Centre for Intelligent Machines at McGill University. He is a former CIHR Postdoctoral Fellow at the CHUL Genomics research centre and Laval University in Quebec. He has been named the Arnold Smith Commonwealth Scholar in 2002 and a National Scholar in India in 1995. Shah has served on program committees of various conferences and symposiums in addition to reviewing for major journals and conferences in the field.
"This treasure-trove of a book covers the important topic of performance evaluation of machine learning algorithms in a very comprehensive and lucid fashion. As Japkowicz and Shah point out, performance evaluation is too often a formulaic affair in machine learning, with scant appreciation of the appropriateness of the evaluation methods used or the interpretation of the results obtained. This book makes significant steps in rectifying this situation by providing a reasoned catalogue of evaluation measures and methods, written specifically for a machine learning audience and accompanied by concrete machine learning examples and implementations in R. This is truly a book to be savoured by machine learning professionals, and required reading for Ph.D students." Peter A. Flach, University of Bristol
"This book has the merit of organizing most of the material about the evaluation of learning algorithms into a homogeneous description, covering both theoretical aspects and pragmatic issues. It is a useful resource for researchers in machine learning, and provides adequate material for graduate courses in machine learning and related fields." Corrado Mencar, Computing Reviews