MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection

A critical requirement for the MEVA dataset is broad releasablility to the research community. Possible restrictions include prevention of commercial use, which discourages commercial companies from using the dataset for research or collaborating on teams which do use the dataset. Another restriction is when data containing human activity is collected without oversight of an Institutional Review Board (IRB) or collected outside of its initially IRB-approved conditions. While the data may address other needs of the research community, lack of proper IRB oversight or participant consent can render the data unusable, wasting significant resources in the collection and curation process.

The MEVA dataset was carefully crafted to avoid such restrictions to maximize its value to the research community. The data collection effort was overseen by an IRB to ensure that the data collected would be usable. We wrote a human subjects research protocol and informed consent documents that detailed the collection process, what types of data would be collected, how the data will be released and may be used, and the risks and benefits to anyone consenting to participate in the study. The informed consent document was provided to prospective participants in advance and reviewed in an informed consent briefing session prior to beginning the data collection. Included in the protocol and consent briefing was a sample license that may be used for the data, a CC-BY 4.0, to make it fully clear the intent and potential future use of the data. The consent presentation was followed by small group discussions and question/answer periods to thoroughly address and answer any participant concerns prior to them consenting to participate the research. Once the individuals who voluntarily chose to participate signed the consent forms, we were able to begin the data collection process.

To further ensure broad releasability of the data, the collection occurred at an access-controlled facility to minimize non-participating individuals entering the fields of view of the cameras and appearing in the data.

To guarantee collection of minimum activity instance counts, ensure diversity in the data, and produce the most realistic video data, MEVA data was collected for both scripted and unscripted activities. The scripting challenges for the data collection effort can broadly be broken down into two categories: (1) scripting for satisfying program requirements such as quantity and diversity, and (2) scripting for realism in activity behavior.

It was critical to design the data collection to represent a variety of realistic scenarios in ground-based security video collection. We determined seven overarching scenarios of interest to the public safety community. For example a footrace with multiple phases including registration, participant arrival, race and cleanup. The scenarios were collected over a period of three weeks on different days and at different times of days to capture video variations due to weather and lighting. At the extreme, two sub-collects in March and May produced dramatic contrast in weather from snow to extreme heat, and thus diversity in natural human behavior (e.g., clustering together in the cold) and wardrobe (i.e., jackets in the winter and t-shirts in the summer).

Scripted into these scenarios were 37 primitive, complex and threat-based human activity types to ensure minimum instance counts useful to the program; these are listed in Figure 2. These activities were scripted to be performed by actors with different age, sex and ethnicity as allowed by the actor pool. Activities involving vehicles or props were specifically scripted to rotate the vehicle pool and theatrical property. Scripted activities occurred across various scenarios and at multiple locations in the facility. Activities were scripted in overlapping and singular fields of view, indoor and outdoor cameras, and various camera types (e.g., stationary EO and IR, roaming EO). Activities were scripted to be performed by the same individual in different, singular cameras with the goal of associating the same individual across different cameras. This is relevant when a central activity is detected involving a subject, and then a derived complex activity is to detect other individuals that meet with the original subject of interest. Scenarios also included confusers for the scripted activity types to add value for performer evaluation in the final dataset. Figure 5 shows two approximately synchronized views from the gym.

Though scenarios had an overarching theme, multiple iterations of the same scenario (called takes) were designed to produce visibly distinct but comparable results by varying the parameters described above. Additionally, actor density at various location in the facility and traffic flow were modified to produce differences in subsequent takes of the same scenario. Scripting the scenarios and activities in this level of detail guaranteed the diversity and quantity of data satisfied program requirements.

To address the second broad goal, we hired professional actors to act scenarios in a natural manner to create an equivalent challenge for computer vision event detectors, as if the events occurred in real-life scenarios without acting. The actors were divided into squads which each had a squad lead in charge of managing a group of 5-8 individuals. The squads were designed, grouped and shuffled to produce variations in demographics, behaviors and inter-actor behaviors. Each squad was given a direction card which provided information on timing and locations for activities to be performed, and group behavior such as social groupings within the squad. By giving the squad lead the ability to designate roles within the group, natural social dynamics, such as those observed in married couples or families, were preserved and increased the realism of the collected data. The squad lead was responsible for assigning roles to the actors, managing vehicle and prop assignment, providing feedback to individual actors for corrections on activity behavior, and reporting descriptions of complex or threat-based activities for use in expediting later annotation.

We found that providing scripting at the squad rather than individual level produced the most realistic actor behavior. When actor behavior was scripted on the individual level at high temporal and spatial resolution, the actors, overwhelmed by the details, focused on achieving all activities at the specified time and location rather than performing activities that were natural in appearance. Using a small pilot collect of data, we were able to determine which activities occurred naturally in the data collection without direction (e.g., person_opens_door). Taking this into account when scripting, lower instances of these activities were scripted and a mix of scripted and unscripted occurrences were collected. On the other hand, rare and threat-based activities needed to be specifically scripted to ensure minimum counts of these activities were achieved and later located for annotation. However, allowing the actors to have artistic licence with how to accomplish activities, especially complex activities, produced more a natural and diverse dataset. Actor familiarity with the scripted scenario also increased realism. Most scenarios varied in time from 15 minutes to 1 hour. Multiple takes could be run consecutively with minor modifications to scripted behaviors and activity instances, and minimal reset time (5 to 10 minutes) to efficiently produced varied instances of a scenario and communicate modifications to the squad leads.

In addition to activity-focused scripting, scene-level behaviors also needed to be scripted to ensure realism. For example, train and bus arrival schedules were designed to reflect public transit schedules scaled for the population depicted by our actors. Vehicle traffic was scripted to have ebbs and flows associated with similar patterns to real-world versions scripted scenarios. For example, an increase in vehicles dropping off people at the bus station just prior to a bus’s scheduled arrival. Driving routes, vehicles and drivers were shuffled to provide variations between different scenarios and takes. Again, these tasks were assigned at the squad level and managed by squad leads.

In addition to scripted scenarios, periods of completely unscripted data were collected with the aim of collecting naturally occurring common activities with no actor direction. We also took advantage of unscripted data to interject threat-based and rare activities to capture undirected response by casual observers. These unscripted times were naturally occurring and added value to reset times between scenario takes.

The MEVA video dataset was collected over a total of three weeks at the Muscatatuck Urban Training Center (MUTC) with a team of over 100 actors and 10 researchers. MUTC is an access-controlled facility run by the Indiana National Guard that offers a globally unique, urban operating environment. The real and operational physical infrastructure, including curbed roads and used buildings, set it apart from other access-controlled facilities for collecting realistic video data.

The camera infrastructure included commercial-off-the-shelf EO cameras; thermal IR camerasas part of several EO-IR pairs, two DJI Inspire 1 v2 drones, and a range of video and still images from handheld cameras used by the actors. The fields of view, both overlapping and non-overlapping, capture person and vehicle activities in indoor and outdoor settings.

Onsite staging and actor instruction also improved the realism in scenes for the MEVA scenario collection. Areas were staged with props, such as signs, tables, chairs, and trashcans, to provide scenario-specific context necessary to increase the natural appearance and usefulness of the area to both actors and recording cameras. Additionally, areas were staged to look similar but distinct between different scenarios to add diversity to the dataset collected. For example, the foyer of one building was converted into an operational cafe where the actors were able to grab drinks and snacks during scripted or unscripted camera time.

Actors were selected to provide a diverse pool of individuals in background, ethnicity and gender. As part of our iterative process to receive and incorporate feedback, all actors attended briefings in which camera fields of view and common issues (e.g., erratic driving or aimless walking) were discussed in a group setting, raising awareness of key factors required to collaboratively produce realistic behaviors in video. Squad leads also held smaller briefings, providing feedback to the scripting team and receiving instructions to pass to their squad members. During morning briefings, all actors and MEVA team members were registered with a unique identifying number matched to their GPS logger unit and wardrobe enrollment photograph(s) as shown in Figure 6. The photos, paired with the GPS logs, enable reidentification of individual actors across fields of view in a single scenario, as seen in Figure 7. Each day before filming, squad leads were given direction cards to assign roles to their actors and inform prop distribution, then data collection was performed. Shorter scenario takes of less than 1 hour were performed in rapid succession with only minor modifications relayed during a brief reset. One day short-scenario data collection would contain between 4 and 8 takes; for longer scenarios of approximately 8 hours, the scenario would be repeated across multiple sequential days. Direction and course correction was provided by both directly interjecting a MEVA team member into the scenario for a brief interval or incorporating actor briefings naturally into the scenario schedule.

The final dataset contains 9,303 hours of ground-camera video (both EO and thermal IR), 42 hours of UAV footage, and 46 hours from hand-held and body-worn cameras. Additionally we collected over 2.7 million GPS trackpoints from 108 unique loggers.

We annotated MEVA through a sequential process of activity identification by a trained, dedicated team of third-party annotators, followed by a quality control audit from an internal team of MEVA experts intimately involved in defining the activities. The initial activity identification step detects all MEVA activities in a 5 minute video clip, specifying the start and end times for each activity, and identifying the initial and final location for each actor or object involved. The subsequent audit step catches any missed or incorrectly labeled activity instances, and confirms strict adherence to MEVA definitions.

Ambiguity in activity definitions is a fundamental annotation challenge. To minimize this, we define activities very explicitly and require that events are always annotated on pixel evidence and not human interpretation. We developed detailed annotation guidelines for each activity, including definition of the start and end times for an activity, and the actors and objects involved. Definitions also include extensive discussion of corner cases.

We explored several methods to increase annotation efficiency and quality, including completely crowdsourced methods on Amazon Mechanical Turk (AMT) and a solely in-house team of experts. The optimized annotation process used a team of third-party annotators dedicated to MEVA annotation to lower the overall cost and supply surge capacity via dynamic team scaling while still guaranteeing quality results. After comparing the results of multiple annotators performing activity annotation in parallel and combined results in a post-processing step with annotating in series with a quality assurance step, serial annotation was selected to enable efficiently annotating a larger dataset.

Annotation consistency is essential to providing high-quality ground truth annotations. When using a diverse team of annotators, it was essential to have a clear set of guidelines and iterative definition modifications to include corner cases to ensure annotator agreement across the team. Additionally, we found that project-specific training, including effective use of the annotation tools, in-depth discussion of definitions including examples, and iterative feedback on annotation quality produced improvements in annotation quality and speed. Finally, a quality audit from an internal team of MEVA annotators was used to guarantee no missed or incorrect instances. We observed a 3-fold increase in audit efficiency over a two month period of working in a tight feedback loop with these procedures and a project-dedicated team of annotators.

Once activities and participating objects were identified by MEVA experts, objects were tracked with bounding boxes for the duration of their involvement in the activity. The annotation of bounding boxes was primarily conducted through crowdsourced annotation on AMT, via an iterative process of bounding box creation and quality review.

In the first step of bounding box refinement, videos were broken up into segments that corresponded to a single activity annotation. An AMT task was created for each activity, displaying a set of start and end boxes with linearly interpolated boxes on intervening frames for all objects involved in the activity. Workers were instructed to complete the tracks by annotating bounding boxes for the interpolated frames which were as tight as possible around the object; for example, all the visible limbs of a person should be in the bounding box while minimizing ”buffer” pixels. These two characteristics are required in the MEVA dataset to ensure that high-quality activity examples with minimal irrelevant pixels are provided for testing and evaluation. These traits were enforced by using other AMT annotators to assess the quality of the resulting tracks.

In the quality review step, the bounding box annotations were shown to several AMT workers who were asked to evaluate them as acceptable according our guidelines. If agreement was achieved between the AMT workers, then the results were considered acceptable and complete; otherwise, a new AMT task would be created for bounding box refinement. The results of the next round of refinement would be provided for quality review, and the process would repeat until acceptable annotations were produced or a threshold number of iterations were performed. If the threshold number of iterations were performed, the clips would be vetted and edited manually as needed by an MEVA auditor. The percentage of activities requiring expert intervention was less than 5%. Specialized web interfaces were developed for each of these tasks to allow workers (AMT or in-house) to easily provide the necessary results. In order to eliminate most systematic low quality jobs, annotations were sampled and workers were allowed to continue based on quality.

Figure 9 illustrates bounding box annotations for tracks from a variety of activities and fields of view that demonstrate the quality produced by our track annotation procedure. Box size varied dramatically due to the scale variation in the video, with a mean area of 13559$\pm$23799. Annotations span 5 track types (person, vehicle, other, bag, and bicycle) with a distribution of 90.71%, 4.5%, 4.51%, 0.15%, and 0.05%, respectively. As part of quality control, we compared MEVA annotations against high-confidence performer false alarms from the ActEV evaluation; our false negative rate (i.e., confirmed missed instances to total activity count in reviewed clips) is less than 0.6% .

Additional Data In addition to the annotations, we have provided supplemental data such as camera models for the camera models which register into a common geo-referenced coordinate system. We have also provided a 3D model of the outdoor component of the collection site, provided as a PLY file and visualized in Figure 10.