Outcome measures based on classification performance fail to predict the intelligibility of binary-masked speech (bibtex)
by , and
Abstract:
To date, the most commonly used outcome measure for assessing ideal binary mask estimation algorithms is based on the difference between the hit rate and the false alarm rate (H-FA). Recently, the error distribution has been shown to substantially affect intelligibility. However, H-FA treats each mask unit independently and does not take into account how errors are distributed. Alternatively, algorithms can be evaluated with the short-time objective intelligibility (STOI) metric using the reconstructed speech. This study investigates the ability of H-FA and STOI to predict intelligibility for binary-masked speech using masks with different error distributions. The results demonstrate the inability of H-FA to predict the behavioral intelligibility and also illustrate the limitations of STOI. Since every estimation algorithm will make errors that are distributed in different ways, performance evaluations should not be made solely on the basis of these metrics.
Reference:
Outcome measures based on classification performance fail to predict the intelligibility of binary-masked speechA.A. Kressner, T. May and C.J. Rozell. Journal of the Acoustical Society of America, 139(6), pp. 3033–3036, June 2016.
Bibtex Entry:
@Article{kressner.16,
  author = 	 {Kressner, A.A. and May, T. and Rozell, C.J.},
  title = 	 {Outcome measures based on classification performance fail to predict the intelligibility of binary-masked speech},
  abstract =     {To date, the most commonly used outcome measure for assessing ideal binary mask estimation algorithms is based on the difference between the hit rate and the false alarm rate ({H-FA}). Recently, the error distribution has been shown to substantially affect intelligibility. However, {H-FA} treats each mask unit independently and does not take into account how errors are distributed. Alternatively, algorithms can be evaluated with the short-time objective intelligibility ({STOI}) metric using the reconstructed speech. This study investigates the ability of {H-FA} and {STOI} to predict intelligibility for binary-masked speech using masks with different error distributions. The results demonstrate the inability of {H-FA} to predict the behavioral intelligibility and also illustrate the limitations of {STOI}. Since every estimation algorithm will make errors that are distributed in different ways, performance evaluations should not be made solely on the basis of these metrics.},
year = 2016,
journal = {Journal of the Acoustical Society of America},
month = jun,
volume = {139},
number = {6}, 
pages = {3033--3036},
url = {http://scitation.aip.org/content/asa/journal/jasa/139/6/10.1121/1.4952439}
}
Powered by bibtexbrowser