CochlScene: Acquisition of acoustic scene data using crowdsourcing

We first selected the target acoustic scene classes and added/removed some during the collection process. We mainly considered the following:

• •

  Participants population We focused on the acoustic scenes that many participants could conveniently visit. Therefore, we chose the target classes in urban areas and excluded some nature scenes (e.g. Moutain, Sea, Waterfall).

• •

  Cultural bias Since a company in Korea performed the crowdsourcing process, we expected that the participants were mostly located on the same specific cultural basis. We tried to avoid this bias in the dataset and excluded scenes which are uncommon in Korea (e.g. Tram) or rare outside of Korea (e.g. Traditional Korean Palace).

• •

  Ambiguity During the planning and actual collection process, we found out that some acoustic scenes are difficult to be defined (e.g. Parking lot) or hard to be expected to have meaningful acoustic cues (e.g. Bakery or Florist). We excluded such scenes to eliminate ambiguity.

• •

  Accessibility The places that require permission to enter and record sounds (e.g. Factory, Spa) are not suitable for the crowdsourcing-based collection since it is not available for unspecified participants. Inducing the participation of people who can access this place by increasing the reward is another option, while we just excluded those classes in this work.

• •

  Submission rate Although it passed all the above considerations, some classes (e.g. Bus station) had a relatively low submission rate than others. We stopped collecting those classes if it seemed they could not reach the target quantity.

• •

  Acoustic similarity Some scenes might be semantically similar (e.g. Cafe and Restaurant), or have similar acoustic characteristics (e.g. Street and Residential Area), while those were not excluded for this reason.

A smartphone application developed by SelectStar was used for the participants to upload the recordings. Fig. 1 shows its basic interface. Most of the participants were existing users of the application who had experience participating in other collection tasks, including image and speech data. To submit the recordings, users could record the sound only through the application right before the submission, while uploading the existing files was not allowed to prevent manipulation. Each participant could upload a limited number of recordings for each class, which was slightly increased for the classes with a low submission rate.

We also used crowdsourcing to check the quality of the submitted data. 3 to 5 participants listened to each submission and made the binary decision on its quality, and the submission was accepted or rejected based on the majority voting. The following were considered, though these criteria would be subjective and limited as the validation was based solely on the audio recording and no additional information.

• •

  Recording gain The submissions whose gain was extremely low (not perceivable in normal volume) or high (causes clipping) were rejected.

• •

  Privacy The submissions containing speech that could be easily identified or contained private topics were rejected. Some submissions, however, may contain speech from the participant who recorded under their agreement.

• •

  Acoustic cue The submissions were rejected if no cue about its target scene was found from the recording. However, we did not restrict that these acoustic cues have to be only related to the target scene; for example, if the submitted data was estimated to be recorded either in Cafe or Restaurant when the target class was Cafe, it was accepted.

All the accepted submissions were normalized to have the same format. First, We divided each recording into files of 10-second segments and discarded the remainders. It is noted that the acoustic cue in section II.C was considered for the entire recording in data validation, so each segment may not have a clear cue for the acoustic scene. Each file was saved with a file name of the form {Class}_{User ID}_{Submission ID}_{Segment index} e.g. Bus_user0017_14896147_000.wav.

CochlScene consists of 76,115 single-channel audio files among 13 acoustic scene classes, and each file has 10 seconds length in 44.1kHz sample rate. The detailed numbers regarding the CochlScene are listed in Table 1. Each class has a similar but not identical number of files, from 5,744 to 5,969. These files were obtained from the original 12,349 submissions with varied lengths, from 50.68 to 301.45 seconds. 90.70% of the submissions have a length between 60 and 70 seconds.

In total, 831 participants submitted their recordings. The number of submissions per participant varies, and 85 participants submitted one recording while the others submitted up to 103 each.

To reduce the effect of skewed submission distributions on participants, we provide manual data partitioning for training, validation, and testing. The main aims are 1) to avoid the submissions from the same participants being included in more than one partition and 2) to maximize the number of participants in the test set to improve the reliability of the evaluation results.

We first counted the number of files (10-second segments) for each user and sorted them. The user with the most files was assigned to training sets if any class did not exceed the target quantity after assigning. It continues for each user in order until no more users can be assigned to the training set. After that, the same process was repeated for the validation set using the unassigned data, and the rest was assigned to the test set. The target quantity for training and validation was 80% and 10% of the total files for each class, respectively.

The baseline model architecture is shown in Table  2. The proposed baseline system is designed to have the same structure as DCASE 2022 task 1 baseline⁴⁴4https://github.com/marmoi/dcase2022_task1_baseline [7]. The differences can be found in the input and output shapes; we assume the input audio has a length of 10 seconds with 44.1kHz in sample rate, whereas the DCASE baseline model receives 1-second audio with the same sample rate. To recognize the 13 scenes in the dataset, we also modified the output shape of the Dense_2 layer from 10 to 13. We also apply batch normalization, ReLU activation, and dropout like the DCASE model, even though they are omitted in the table. The model has a total of 116,821 trainable parameters.

To convert the audio input to a Mel-scale spectrogram, the Kapre library was used [15]. The Window size and hop size were set to 40ms and 20ms, respectively, as in the DCASE model. In addition, 40-dimensional log Mel filterbanks were used with Kapre’s basic Mel-scale conversion library. Note that the time and frequency dimensions are swapped using the Permute layer after the melspectrogram conversion.

The Adam optimizer with a learning rate of 1e-3 was used for training. The training procedure was stopped when the validation accuracy was not improved for ten successive epochs or when it reached the 100th epoch. The implementation of the baseline system is publicly available in Cochl’s GitHub repository⁵⁵5https://github.com/cochlearai/cochlscene.

The overall performance of the baseline model, including precision, recall, and F-score, is summarized in Table 3. We found that the baseline model shows a worse F-score in Cafe, CrowdedIndoor, Park, and ResidentialArea classes while it performs comparably better for Car, Kitchen, and Restroom. We expect that the classes with higher F-score are easier to be classified because they may contain unique sound events, such as engine sounds for Car, chopping sound for Kitchen, and flushing sound for Restroom. For the classes with lower F-score, we assume several classes may have semantic or acoustic similarities, so each can be easily confused.

Fig. 2 shows a confusion matrix of baseline model for a more in-depth examination. We found that Cafe, CrowdedIndoor, and Restaurant are likely to be wrongly recognized as each other. Besides, we can find asymmetries in the confusion matrix; for example, only 34 Cafe data are classified as CrowdedIndoor, whereas 152 CrowdedIndoor data are classified as Cafe. We can assume that Cafe data often contain certain clues that CrowdedIndoor data do not. For example, we can expect that sounds from a coffee grinder or espresso machine may rarely occur in the CrowdedIndoor data. Similar relations can be found between CrowdedIndoor and Restaurant because dish clanking and silverware rattling sounds could only be found in the Restaurant data. We can also find the asymmetric confusion among Park, ResidentialArea, and Street, where the confusion mostly occurs in the direction of Park$\rightarrow$ResidentialArea and ResidentialArea$\rightarrow$Street. We leave a detailed investigation of the acoustic characteristics of each class as future work.