weka.attributeSelection
Class RaceSearch

java.lang.Object
  extended by weka.attributeSelection.ASSearch
      extended by weka.attributeSelection.RaceSearch
All Implemented Interfaces:
java.io.Serializable, RankedOutputSearch, OptionHandler, TechnicalInformationHandler

public class RaceSearch
extends ASSearch
implements RankedOutputSearch, OptionHandler, TechnicalInformationHandler

Races the cross validation error of competing attribute subsets. Use in conjuction with a ClassifierSubsetEval. RaceSearch has four modes:

forward selection races all single attribute additions to a base set (initially no attributes), selects the winner to become the new base set and then iterates until there is no improvement over the base set.

Backward elimination is similar but the initial base set has all attributes included and races all single attribute deletions.

Schemata search is a bit different. Each iteration a series of races are run in parallel. Each race in a set determines whether a particular attribute should be included or not---ie the race is between the attribute being "in" or "out". The other attributes for this race are included or excluded randomly at each point in the evaluation. As soon as one race has a clear winner (ie it has been decided whether a particular attribute should be inor not) then the next set of races begins, using the result of the winning race from the previous iteration as new base set.

Rank race first ranks the attributes using an attribute evaluator and then races the ranking. The race includes no attributes, the top ranked attribute, the top two attributes, the top three attributes, etc.

It is also possible to generate a raked list of attributes through the forward racing process. If generateRanking is set to true then a complete forward race will be run---that is, racing continues until all attributes have been selected. The order that they are added in determines a complete ranking of all the attributes.

Racing uses paired and unpaired t-tests on cross-validation errors of competing subsets. When there is a significant difference between the means of the errors of two competing subsets then the poorer of the two can be eliminated from the race. Similarly, if there is no significant difference between the mean errors of two competing subsets and they are within some threshold of each other, then one can be eliminated from the race.

For more information see:

Andrew W. Moore, Mary S. Lee: Efficient Algorithms for Minimizing Cross Validation Error. In: Eleventh International Conference on Machine Learning, 190-198, 1994.

BibTeX:

 @inproceedings{Moore1994,
    author = {Andrew W. Moore and Mary S. Lee},
    booktitle = {Eleventh International Conference on Machine Learning},
    pages = {190-198},
    publisher = {Morgan Kaufmann},
    title = {Efficient Algorithms for Minimizing Cross Validation Error},
    year = {1994}
 }
 

Valid options are:

 -R <0 = forward | 1 = backward race | 2 = schemata | 3 = rank>
  Type of race to perform.
  (default = 0).
 -L <significance>
  Significance level for comaparisons
  (default = 0.001(forward/backward/rank)/0.01(schemata)).
 -T <threshold>
  Threshold for error comparison.
  (default = 0.001).
 -A <attribute evaluator>
  Attribute ranker to use if doing a 
  rank search. Place any
  evaluator options LAST on 
  the command line following a "--".
  eg. -A weka.attributeSelection.GainRatioAttributeEval ... -- -M.
  (default = GainRatioAttributeEval)
 -F <0 = 10 fold | 1 = leave-one-out>
  Folds for cross validation
  (default = 0 (1 if schemata race)
 -Q
  Generate a ranked list of attributes.
  Forces the search to be forward
  and races until all attributes have
  selected, thus producing a ranking.
 -N <num to select>
  Specify number of attributes to retain from 
  the ranking. Overides -T. Use in conjunction with -Q
 -J <threshold>
  Specify a theshold by which attributes
  may be discarded from the ranking.
  Use in conjuction with -Q
 -Z
  Verbose output for monitoring the search.
 
 Options specific to evaluator weka.attributeSelection.GainRatioAttributeEval:
 
 -M
  treat missing values as a seperate value.

Version:
$Revision: 1.24 $
Author:
Mark Hall (mhall@cs.waikato.ac.nz)
See Also:
Serialized Form

Field Summary
static Tag[] TAGS_SELECTION
           
static Tag[] XVALTAGS_SELECTION
           
 
Constructor Summary
RaceSearch()
           
 
Method Summary
 java.lang.String attributeEvaluatorTipText()
          Returns the tip text for this property
 java.lang.String debugTipText()
          Returns the tip text for this property
 java.lang.String foldsTypeTipText()
          Returns the tip text for this property
 java.lang.String generateRankingTipText()
          Returns the tip text for this property
 ASEvaluation getAttributeEvaluator()
          Get the attribute evaluator used to generate the ranking.
 int getCalculatedNumToSelect()
          Gets the calculated number of attributes to retain.
 boolean getDebug()
          Get whether output is to be verbose
 SelectedTag getFoldsType()
          Get the xfold type
 boolean getGenerateRanking()
          Gets whether ranking has been requested.
 int getNumToSelect()
          Gets the number of attributes to be retained.
 java.lang.String[] getOptions()
          Gets the current settings of BestFirst.
 SelectedTag getRaceType()
          Get the race type
 double getSelectionThreshold()
          Returns the threshold so that the AttributeSelection module can discard attributes from the ranking.
 double getSignificanceLevel()
          Get the significance level
 TechnicalInformation getTechnicalInformation()
          Returns an instance of a TechnicalInformation object, containing detailed information about the technical background of this class, e.g., paper reference or book this class is based on.
 double getThreshold()
          Get the threshold
 java.lang.String globalInfo()
          Returns a string describing this search method
 java.util.Enumeration listOptions()
          Returns an enumeration describing the available options.
 java.lang.String numToSelectTipText()
          Returns the tip text for this property
 java.lang.String raceTypeTipText()
          Returns the tip text for this property
 double[][] rankedAttributes()
          Returns a X by 2 list of attribute indexes and corresponding evaluations from best (highest) to worst.
 int[] search(ASEvaluation ASEval, Instances data)
          Searches the attribute subset space by racing cross validation errors of competing subsets
 java.lang.String selectionThresholdTipText()
          Returns the tip text for this property
 void setAttributeEvaluator(ASEvaluation newEvaluator)
          Set the attribute evaluator to use for generating the ranking.
 void setDebug(boolean d)
          Set whether verbose output should be generated.
 void setFoldsType(SelectedTag d)
          Set the xfold type
 void setGenerateRanking(boolean doRank)
          Records whether the user has requested a ranked list of attributes.
 void setNumToSelect(int n)
          Specify the number of attributes to select from the ranked list (if generating a ranking).
 void setOptions(java.lang.String[] options)
          Parses a given list of options.
 void setRaceType(SelectedTag d)
          Set the race type
 void setSelectionThreshold(double threshold)
          Set the threshold by which the AttributeSelection module can discard attributes.
 void setSignificanceLevel(double sig)
          Sets the significance level to use
 void setThreshold(double t)
          Sets the threshold for comparisons
 java.lang.String significanceLevelTipText()
          Returns the tip text for this property
 java.lang.String thresholdTipText()
          Returns the tip text for this property
 java.lang.String toString()
          Returns a string represenation
 
Methods inherited from class weka.attributeSelection.ASSearch
forName, makeCopies
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

TAGS_SELECTION

public static final Tag[] TAGS_SELECTION

XVALTAGS_SELECTION

public static final Tag[] XVALTAGS_SELECTION
Constructor Detail

RaceSearch

public RaceSearch()
Method Detail

globalInfo

public java.lang.String globalInfo()
Returns a string describing this search method

Returns:
a description of the search method suitable for displaying in the explorer/experimenter gui

getTechnicalInformation

public TechnicalInformation getTechnicalInformation()
Returns an instance of a TechnicalInformation object, containing detailed information about the technical background of this class, e.g., paper reference or book this class is based on.

Specified by:
getTechnicalInformation in interface TechnicalInformationHandler
Returns:
the technical information about this class

raceTypeTipText

public java.lang.String raceTypeTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setRaceType

public void setRaceType(SelectedTag d)
Set the race type

Parameters:
d - the type of race

getRaceType

public SelectedTag getRaceType()
Get the race type

Returns:
the type of race

significanceLevelTipText

public java.lang.String significanceLevelTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setSignificanceLevel

public void setSignificanceLevel(double sig)
Sets the significance level to use

Parameters:
sig - the significance level

getSignificanceLevel

public double getSignificanceLevel()
Get the significance level

Returns:
the current significance level

thresholdTipText

public java.lang.String thresholdTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setThreshold

public void setThreshold(double t)
Sets the threshold for comparisons

Specified by:
setThreshold in interface RankedOutputSearch
Parameters:
t - the threshold to use

getThreshold

public double getThreshold()
Get the threshold

Specified by:
getThreshold in interface RankedOutputSearch
Returns:
the current threshold

foldsTypeTipText

public java.lang.String foldsTypeTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setFoldsType

public void setFoldsType(SelectedTag d)
Set the xfold type

Parameters:
d - the type of xval

getFoldsType

public SelectedTag getFoldsType()
Get the xfold type

Returns:
the type of xval

debugTipText

public java.lang.String debugTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setDebug

public void setDebug(boolean d)
Set whether verbose output should be generated.

Parameters:
d - true if output is to be verbose.

getDebug

public boolean getDebug()
Get whether output is to be verbose

Returns:
true if output will be verbose

attributeEvaluatorTipText

public java.lang.String attributeEvaluatorTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setAttributeEvaluator

public void setAttributeEvaluator(ASEvaluation newEvaluator)
Set the attribute evaluator to use for generating the ranking.

Parameters:
newEvaluator - the attribute evaluator to use.

getAttributeEvaluator

public ASEvaluation getAttributeEvaluator()
Get the attribute evaluator used to generate the ranking.

Returns:
the evaluator used to generate the ranking.

generateRankingTipText

public java.lang.String generateRankingTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setGenerateRanking

public void setGenerateRanking(boolean doRank)
Records whether the user has requested a ranked list of attributes.

Specified by:
setGenerateRanking in interface RankedOutputSearch
Parameters:
doRank - true if ranking is requested

getGenerateRanking

public boolean getGenerateRanking()
Gets whether ranking has been requested. This is used by the AttributeSelection module to determine if rankedAttributes() should be called.

Specified by:
getGenerateRanking in interface RankedOutputSearch
Returns:
true if ranking has been requested.

numToSelectTipText

public java.lang.String numToSelectTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setNumToSelect

public void setNumToSelect(int n)
Specify the number of attributes to select from the ranked list (if generating a ranking). -1 indicates that all attributes are to be retained.

Specified by:
setNumToSelect in interface RankedOutputSearch
Parameters:
n - the number of attributes to retain

getNumToSelect

public int getNumToSelect()
Gets the number of attributes to be retained.

Specified by:
getNumToSelect in interface RankedOutputSearch
Returns:
the number of attributes to retain

getCalculatedNumToSelect

public int getCalculatedNumToSelect()
Gets the calculated number of attributes to retain. This is the actual number of attributes to retain. This is the same as getNumToSelect if the user specifies a number which is not less than zero. Otherwise it should be the number of attributes in the (potentially transformed) data.

Specified by:
getCalculatedNumToSelect in interface RankedOutputSearch

selectionThresholdTipText

public java.lang.String selectionThresholdTipText()
Returns the tip text for this property

Returns:
tip text for this property suitable for displaying in the explorer/experimenter gui

setSelectionThreshold

public void setSelectionThreshold(double threshold)
Set the threshold by which the AttributeSelection module can discard attributes.

Parameters:
threshold - the threshold.

getSelectionThreshold

public double getSelectionThreshold()
Returns the threshold so that the AttributeSelection module can discard attributes from the ranking.


listOptions

public java.util.Enumeration listOptions()
Returns an enumeration describing the available options.

Specified by:
listOptions in interface OptionHandler
Returns:
an enumeration of all the available options.

setOptions

public void setOptions(java.lang.String[] options)
                throws java.lang.Exception
Parses a given list of options.

Valid options are:

 -R <0 = forward | 1 = backward race | 2 = schemata | 3 = rank>
  Type of race to perform.
  (default = 0).
 -L <significance>
  Significance level for comaparisons
  (default = 0.001(forward/backward/rank)/0.01(schemata)).
 -T <threshold>
  Threshold for error comparison.
  (default = 0.001).
 -A <attribute evaluator>
  Attribute ranker to use if doing a 
  rank search. Place any
  evaluator options LAST on 
  the command line following a "--".
  eg. -A weka.attributeSelection.GainRatioAttributeEval ... -- -M.
  (default = GainRatioAttributeEval)
 -F <0 = 10 fold | 1 = leave-one-out>
  Folds for cross validation
  (default = 0 (1 if schemata race)
 -Q
  Generate a ranked list of attributes.
  Forces the search to be forward
  and races until all attributes have
  selected, thus producing a ranking.
 -N <num to select>
  Specify number of attributes to retain from 
  the ranking. Overides -T. Use in conjunction with -Q
 -J <threshold>
  Specify a theshold by which attributes
  may be discarded from the ranking.
  Use in conjuction with -Q
 -Z
  Verbose output for monitoring the search.
 
 Options specific to evaluator weka.attributeSelection.GainRatioAttributeEval:
 
 -M
  treat missing values as a seperate value.

Specified by:
setOptions in interface OptionHandler
Parameters:
options - the list of options as an array of strings
Throws:
java.lang.Exception - if an option is not supported

getOptions

public java.lang.String[] getOptions()
Gets the current settings of BestFirst.

Specified by:
getOptions in interface OptionHandler
Returns:
an array of strings suitable for passing to setOptions()

search

public int[] search(ASEvaluation ASEval,
                    Instances data)
             throws java.lang.Exception
Searches the attribute subset space by racing cross validation errors of competing subsets

Specified by:
search in class ASSearch
Parameters:
ASEval - the attribute evaluator to guide the search
data - the training instances.
Returns:
an array (not necessarily ordered) of selected attribute indexes
Throws:
java.lang.Exception - if the search can't be completed

rankedAttributes

public double[][] rankedAttributes()
                            throws java.lang.Exception
Description copied from interface: RankedOutputSearch
Returns a X by 2 list of attribute indexes and corresponding evaluations from best (highest) to worst.

Specified by:
rankedAttributes in interface RankedOutputSearch
Returns:
the ranked list of attribute indexes in an array of ints
Throws:
java.lang.Exception - if the ranking can't be produced

toString

public java.lang.String toString()
Returns a string represenation

Overrides:
toString in class java.lang.Object
Returns:
a string representation