Centre for Digital Music

 
 

IEEE AASP Challenge:
Detection and Classification of Acoustic Scenes and Events

Event Detection - Office Synthetic Results

These are the results for the running of the Event Detection - Office Synthetic subtask. For background information about this task set please refer to the subtask specifications and the subtask datasets. See also the IEEE TMM paper summarising the D-CASE challenge results and the EUSIPCO 2013 paper describing the baseline systems.

General Legend

Code Abstract Participants
DHV PDF Aleksandr Diment, Toni Heittola and Tuomas Virtanen
GVV PDF Jort F. Gemmeke, Lode Vuegen, Bart Vanrumste, Hugo Van hamme
VVK PDF Lode Vuegen, Bert Van Den Broeck, Peter Karsmakers, Jort F. Gemmeke, Bart Vanrumste, Hugo Van hamme

Frame-based Results

F (%) Pre (%) Rec (%) AEER
DHV 18.68 13.64 45.26 7.980
GVV 21.28 38.51 15.23 1.318
VVK 13.51 22.55 12.30 1.888
Baseline 12.76 14.39 14.90 2.804

Event-based Results

F (%) Pre (%) Rec (%) F_off (%) Pre_off (%) Rec_off (%) AEER AEER_off
DHV 16.06 12.37 26.00 11.12 8.63 17.71 3.5157 3.7643
GVV 17.00 22.29 14.22 13.60 18.00 11.40 1.681 1.766
VVK 13.78 20.93 11.44 9.16 14.04 7.61 1.683 1.798
Baseline 7.75 4.92 21.65 0.57 0.37 1.36 6.263 6.871

Class-wise Event-based Results

F (%) Pre (%) Rec (%) F_off (%) Pre_off (%) Rec_off (%) AEER AEER_off
DHV 18.69 17.18 26.08 12.18 11.08 17.68 2.903 3.155
GVV 14.16 17.14 14.26 11.40 13.55 11.52 1.216 1.298
VVK 10.51 12.31 11.49 7.48 9.13 7.69 1.377 1.491
Baseline 9.47 7.09 21.79 0.33 0.19 1.34 5.332 5.945

Note: The original OS results published at the time of the challenge differ from the results published here due to a systematic fault affecting a subset of the labels in the original OS development and test datasets. This was found and fixed, and the three teams who submitted systems to the OS task were contacted and invited to revise their systems. The DHV system was re-trained on the corrected OS development data; the configurations of the other systems (GVV, VVK, and baseline) were not affected, and were left unchanged. All three systems were re-evaluated on the corrected test datasets to obtain the results here. The corrections to the data generally improved system performance, which is to be expected since they improved the correspondence between training and test.

Back to main challenge page