Everyday Activity Science and Engineering Table Setting Dataset – Scientific Data
Understanding how people plan and execute everyday tasks is pivotal for building cognition-enabled robots and intelligent systems. The Everyday Activity Science and Engineering Table Setting Dataset (EASE-TSD) offers a rare, richly detailed window into human behavior by capturing synchronized, high-dimensional biosignals as volunteers set a table in a controlled lab environment.
Why this dataset matters
From assistive robots that can anticipate a user’s next move to AI systems that interpret human intent, models need more than images or isolated sensor traces—they need temporally aligned, multimodal evidence of what people do and why they do it. EASE-TSD delivers just that: comprehensive recordings that pair physical motion with physiology, vision, language, and context, enabling research across robotics, human–computer interaction, neuroscience, and machine learning.
What EASE-TSD captures
EASE-TSD comprises 78 recorded sessions. In each session, a participant completes six table-setting trials, allowing researchers to observe both planning and execution over repeated, comparable tasks. The dataset aligns eight biosignal and contextual streams to the same timeline, including:
- Marker-based motion capture for precise body and limb kinematics
- Environmental and first-person video cameras for scene and egocentric perspectives
- Eye-tracking to reveal visual attention and gaze dynamics
- Electromyography (EMG) to measure muscle activity
- Electrodermal activity (EDA) as an indicator of arousal
- Acceleration for movement and inertial cues
- Microphones to capture speech and ambient audio
- Electroencephalography (EEG) for brain activity
This multimodal design connects intention, perception, and action: from where the eyes look, to how muscles activate, to how hands and objects move through space.
Inside the lab: protocol and think-aloud methods
Participants set a table in a standardized setting, ensuring consistent object layouts and environmental conditions. Crucially, they are instructed to think aloud in two modes:
- Concurrent think-aloud during the task, verbalizing moment-to-moment decisions and observations
- Retrospective think-aloud after the task, reflecting on choices and strategies
This pairing grounds the sensor data in explicit accounts of cognitive processes, turning opaque behavior patterns into interpretable sequences of goals, plans, and adjustments.
Annotations that bridge intention and action
EASE-TSD is richly annotated using a hierarchical schema that spans multiple layers of behavior, including:
- Phases that segment overarching task structure
- Activities that describe goal-directed steps
- Motions that mark fine-grained physical actions
- Interacted objects to capture human–object relations
The think-aloud protocols are further coded using dedicated TA labels, linking verbalized cognition to specific time-aligned events in the sensor streams. Together, these annotations support analyses that tie “what happened” to “what the participant was thinking.”
From raw signals to research-ready data
After recording, the data undergo semi-automatic labeling, post-processing, and analysis using modern biosignal processing and machine learning techniques. This pipeline accelerates the transition from raw streams to structured datasets, enabling tasks such as:
- Action and activity segmentation across modalities
- Human intention and plan recognition
- Sensor fusion for robust behavior modeling
- Benchmarking of multimodal learning algorithms
Who can use EASE-TSD—and how
Because it integrates synchronized kinematics, physiology, gaze, audio, and video with cognitive overlays, EASE-TSD is valuable for:
- Robotics and HRI: Learning policies for assistive manipulation, anticipation, and collaboration
- AI and multimodal ML: Training and evaluating sequence models, transformers, and fusion methods
- Cognitive science: Testing hypotheses about planning, attention, and sensorimotor control
- Ergonomics and design: Understanding task efficiency, workload, and interaction with everyday objects
At a glance
- Task: Human table-setting in a controlled lab
- Sessions: 78 total, each with six repeated trials
- Modalities: Eight synchronized streams, including motion capture, video (environmental and first-person), eye-tracking, EMG, EDA, acceleration, microphones, and EEG
- Verbal data: Concurrent and retrospective think-aloud protocols
- Annotations: Hierarchical labels for phases, activities, motions, interacted objects; TA codes for verbal reports
- Processing: Semi-automatic labeling, post-processing, and analysis with state-of-the-art methods
Why EASE-TSD sets a new bar
Many datasets capture what people do; far fewer capture why. By uniting synchronized biosignals, multiple camera perspectives, and think-aloud reasoning with detailed annotations, EASE-TSD empowers research that connects intention to action with scientific rigor. It is a foundation for next-generation systems that don’t just see and react—but understand and collaborate.
