Multiple Classifier Systems

Morethanadecadeago,combiningmultipleclassi?erswasproposedasap- siblesolutiontotheproblemsposedbythetraditionalpatternclassi?cation approachwhichinvolvedselectingthebestclassi?erfromasetofcandidates basedontheirexperimentalevaluation. Asnoclassi?erisknowntobethebest forallcasesandtheselectionofthebestclassi?erforagivenpracticaltaskis verydi?cult,diverseresearchcommunities,includingMachineLearning,N- ralNetworks,PatternRecognition,andStatistics,addressedtheengineering problemofhowtoexploitthestrengthswhileavoidingtheweaknessesofd- ferentdesigns. Thisambitiousresearchtrendwasalsomotivatedbyempirical observationsaboutthecomplementarityofdi?erentclassi?erdesigns,natural requirementsofinformationfusionapplications,andintrinsicdi?cultiesasso- atedwiththeoptimalchoiceofsomeclassi?erdesignparameters,suchasthe architectureandtheinitialweightsforaneuralnetwork. Afteryearsofresearch, thecombinationofmultipleclassi?ershasbecomeawellestablishedandexciting researcharea,whichprovidese?ectivesolutionstodi?cultpatternrecognition problems. Aconsiderablebodyofempiricalevidencesupportsthemeritof- signingcombinedsystemswhoseaccuracyishigherthanthatofeachindividual classi?er,andvariousmethodsforthegenerationandthecombinationofm- tipleclassi? ershavebecomeavailable. However,despitetheprovedutilityof multipleclassi?ersystems,nogeneralanswertotheoriginalquestionaboutthe possibilityofexploitingthestrengthswhileavoidingtheweaknessesofdi?erent classi?erdesignshasyetemerged. Otherfundamentalissuesarealsoamatterof on-goingresearchindi?erentresearchcommunities. Theresultsachievedd- ingthepastyearsarealsospreadoverdi?erentresearchcommunities,andthis makesitdi?culttoexchangesuchresultsandpromotetheircross-fertilization. Theacknowledgmentofthefundamentalrolethatthecreationofacommon internationalforumforresearchersofthediversecommunitiescouldplayfor theadvancementofthisresearch?eldmotivatedthepresentseriesofwo- shopsonmultipleclassi?ersystems. Followingitspredecessors,MultipleCl- si?erSystems2000(SpringerISBN3-540-67704-6)and2001(SpringerISBN 3-540-42284-6),thisvolumecontainstheproceedingsoftheThirdInternational WorkshoponMultipleClassi?erSystems(MCS2002),heldattheGrandHotel ChiaLaguna,Cagliari,Italy,onJune24-26,2002. The29papersselectedby thescienti?ccommitteehavebeenorganizedinsessionsdealingwithbagging andboosting,ensemblelearningandneuralnetworks,combinationstrategies, designmethodologies,analysisandperformanceevaluation,andapplications. Theworkshopprogramandthisvolumeareenrichedwiththreeinvitedtalks givenbyJoydeepGhosh(UniversityofTexas,USA),TrevorHastie(Stanford University,USA),andSarunasRaudys(VilniusGediminasTechnicalUniversity, Lithuania). Papersweresubmittedfromresearchersofthefourdiversecom- nities,socon?rmingthatthisseriesofworkshopscanbecomeacommonforum VI Foreword forexchangingviewsandreportinglatestresearchresults. Asfortheprevious editions,thesigni?cantnumberofpapersdealingwithrealpatternrecognition applicationsareproofofthepracticalutilityofmultipleclassi?ersystems. This workshopwassupportedbytheUniversityofCagliari,Italy,theUniversityof Surrey,Guildford,UnitedKingdom,andtheDepartmentofElectricalandEl- tronicEngineeringoftheUniversityofCagliari. Allthesesupportsaregratefully acknowledged. WealsothanktheInternationalAssociationforPatternRecog- tionanditsTechnicalCommitteeTC1onStatisticalPatternRecognitionTe- niquesforsponsoringMCS2002. Wewishtoexpressourappreciationtoallthose whohelpedtoorganizeMCS2002. Firstofall,wewouldliketothankallthe membersoftheScienti?cCommitteewhoseprofessionalismwasinstrumental increatingaveryinterestingtechnicalprogram. Specialthanksareduetothe membersoftheOrganizingCommittee,GiorgioFumera,GiorgioGiacinto,and GianLucaMarcialisfortheirindispensablecontributionstotheMCS2002web sitemanagement,localorganization,andproceedingspreparation. April2002 FabioRoliandJosefKittler WorkshopChairs F. Roli(Univ. ofCagliari,Italy) J. Kittler(Univ. ofSurrey,UnitedKingdom) Scienti?cCommittee J. A. Benediktsson(Iceland) M. Kamel(Canada) H. Bunke(Switzerland) L. I. Kuncheva(UK) L. P. Cordella(Italy) L. Lam(HongKong) B. V. Dasarathy(USA) D. Landgrebe(USA) R. P. W. Duin(TheNetherlands) Dar-ShyangLee(USA) C. Furlanello(Italy) D. Partridge(UK) J. Ghosh(USA) A. J. C. Sharkey(UK) T. K. Ho(USA) K. Tumer(USA) S. Impedovo(Italy) G. Vernazza(Italy) N. Intrator(Israel) T. Windeatt(UK) A. K. Jain(USA) LocalCommittee G. Fumera(Univ. ofCagliari,Italy) G. Giacinto(Univ. ofCagliari,Italy) G. L. Marcialis(Univ. ofCagliari,Italy) Organizedby Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari UniversityofSurrey Sponsoredby UniversityofCagliari UniversityofSurrey Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari TheInternationalAssociationforPatternRecognition Supportedby UniversityofCagliari Dept. ofElectricalandElectronicEngineeringoftheUniversityofCagliari UniversityofSurrey TableofContents InvitedPapers Multiclassi? erSystems:BacktotheFuture...1 J. Ghosh SupportVectorMachines,KernelLogisticRegressionandBoosting...16 J. Zhu,T. Hastie MultipleClassi?cationSystemsintheContextofFeatureExtractionand Selection...27 ? S. Raudys BaggingandBoosting BoostedTreeEnsemblesforSolvingMulticlassProblems...42 T. Windeatt,G. Ardeshir DistributedPastingofSmallVotes...52 N. V. Chawla,L. O. Hall,K. W. Bowyer,T. E. Moore,Jr. , W. P. Kegelmeyer BaggingandBoostingfortheNearestMeanClassi?er:E?ectsofSample SizeonDiversityandAccuracy...62 M. Skurichina,L. I. Kuncheva,R. P. W. Duin HighlightingHardPatternsviaAdaboostWeightsEvolution ...72 B. Caprile,C. Furlanello,S. Merler UsingDiversitywithThreeVariantsofBoosting:Aggressive,Conservative, andInverse ...81 L. I. Kuncheva,C. J. Whitaker EnsembleLearningandNeuralNetworks MultistageNeuralNetworkEnsembles...91 S. Yang,A. Browne,P. D. Picton ForwardandBackwardSelectioninRegressionHybridNetwork...98 S. Cohen,N. Intrator TypesofMultinetSystem...108 A. J. C.