Restricted access

Research article

First published online October 30, 2023

Re-use permissions

The effects of selected object features on a pick-and-place task: A human multimodal dataset

Linda Lastrico, Valerio Belcamino, […], Alessandro Carfì https://orcid.org/0000-0001-9208-6910 [email protected], Alessia Vignolo, Alessandra Sciutti, Fulvio Mastrogiovanni, and Francesco Rea+4View all authors and affiliations

Volume 43, Issue 1

https://doi.org/10.1177/02783649231210965

Abstract
References

Abstract

We propose a dataset to study the influence of object-specific characteristics on human pick-and-place movements and compare the quality of the motion kinematics extracted by various sensors. This dataset is also suitable for promoting a broader discussion on general learning problems in the hand-object interaction domain, such as intention recognition or motion generation with applications in the Robotics field. The dataset consists of the recordings of 15 subjects performing 80 repetitions of a pick-and-place action under various experimental conditions, for a total of 1200 pick-and-places. The data has been collected thanks to a multimodal setup composed of multiple cameras, observing the actions from different perspectives, a motion capture system, and a wrist-worn inertial measurement unit. All the objects manipulated in the experiments are identical in shape, size, and appearance but differ in weight and liquid filling, which influences the carefulness required for their handling.

Get full access to this article

View all access and purchase options for this article.

References

Antonsson EK, Mann RW (1985) The frequency content of gait. Journal of Biomechanics 18(1): 39–47.

Apicella T, Slavic G, Ragusa E, et al. (2022) Container localisation and mass estimation with an RGB-D camera. In: The 32nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Singapore, 9152–9155.

Bingham G (1987) Kinematic form and scaling: further investigations on the visual perception of lifted weight. Journal of Experimental Psychology: Human Perception and Performance 13(2): 155–177.

Breazeal C (2003) Toward sociable robots. Robotics and Autonomous Systems 42(3): 167–175. Socially Interactive Robots.

Carfì A, Foglino F, Bruno B, et al. (2019) A multi-sensor dataset of human-human handover. Data in Brief 22: 109–117.

Chaminade T, Cheng G (2009) Social cognitive neuroscience and humanoid robotics. Journal of Physiology Paris 103(3): 286–295. Neurorobotics.

Dragan AD, Lee KCT, Srinivasa SS (2013) Legibility and predictability of robot motion. In: Proceedings of the 8th ACM/IEEE International Conference on Human-Robot Interaction. Tokyo, Japan, 301–308.

Duarte NF, Chatzilygeroudis K, Santos-Victor J, et al. (2020) From human action understanding to robot action execution: how the physical properties of handled objects modulate non-verbal cues. In: 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 1–6.

Dwarampudi M, Reddy NVS (2019) Effects of Padding on LSTMS and CNNS, 07288. ArXiv abs/1903.

Fan Z, Taheri O, Tzionas D, et al. (2023) ARCTIC: a dataset for dexterous bimanual hand-object manipulation. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Garello L, Lastrico L, Rea F, et al. (2021) Property-aware robot object manipulation: a generative approach. In: 2021 IEEE International Conference on Development and Learning (ICDL), 1–7.

Hamilton A, Joyce D, Flanagan J, et al. (2007) Kinematic cues in perceptual weight judgment and their origins in box lifting. Psychological Research 71: 13–21.

Huang Y, Sun Y (2019) A dataset of daily interactive manipulation. The International Journal of Robotics Research 38(8): 879–886.

Huang Y, Bianchi M, Liarokapis M, et al. (2016) Recent data sets on object manipulation: a survey. Big Data 4(4): 197–216.

Khusainov R, Azzi D, Achumba IE, et al. (2013) Real-time human ambulation, activity, and physiological monitoring: taxonomy of issues, techniques, applications, challenges and limitations. Sensors 13(10): 12852–12902.

Kratzer P, Bihlmaier S, Midlagajni NB, et al. (2021) Mogaze: a dataset of full-body motions that includes workspace geometry and eye-gaze. IEEE Robotics and Automation Letters 6(2): 367–373.

Lastrico L, Carfì A, Vignolo A, et al. (2021) Careful with that! observation of human movements to estimate objects properties. In: Proceedings of the 13th International Workshop of Human-Friendly Robotics (HFR). Innsbruck, Austria: Springer International Publishing, 127–141.

Lastrico L, Garello L, Rea F, et al. (2022) Robots with different embodiments can express and influence carefulness in object manipulation. In: 2022 IEEE International Conference on Development and Learning (ICDL), 280–286.

Lastrico L, Ferreira Duarte N, Carfì A, et al. (2023) Expressing and inferring action carefulness in human-to-robot handovers. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Accepted for publication.

Metta G, Fitzpatrick P, Natale L (2006) Yarp: yet another robot platform. International Journal of Advanced Robotic Systems 3: 43–48.

Metta G, Sandini G, Vernon D, et al. (2008) The icub humanoid robot: an open platform for research in embodied cognition. In: Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems (PerMIS). Gaithersburg, Maryland, USA, 50–56.

Mottaghi R, Schenck C, Fox D, et al. (2017) See the glass half full: reasoning about liquid containers, their volume and content. In: 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 1889–1898.

Nicora E, Goyal G, Noceti N, et al. (2020) The MoCA dataset, kinematic and multi-view visual streams of fine-grained cooking actions. Scientific Data 7(1): 432.

Pang YL, Xompero A, Oh C, et al. (2021) Towards safe human-to-robot handovers of unknown containers. In: 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN). Virtual, 51–58.

Pezzulo G, Donnarumma F, Dindo H (2013) Human sensorimotor communication: a theory of signaling in online social interactions. PLoS One 8: 1–11.

Sanchez-Matilla R, Chatzilygeroudis K, Modas A, et al. (2020) Benchmark for human-to-robot handovers of unseen containers with unknown filling. IEEE Robotics and Automation Letters 5(2): 1642–1649.

Sandini G, Sciutti A, Rea F (2019) Movement-based communication for humanoid-human interaction. In: Humanoid Robotics: A Reference. Dordrecht: Springer Netherlands, 2169–2197.

Stein S, Mckenna S (2013) Combining embedded accelerometers with computer vision for recognizing food preparation activities. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 729–738.

Tenorth M, Bandouch J, Beetz M (2009) The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: 2009 IEEE 12th International Conference on Computer Vision Workshops. ICCV Workshops, 1089–1096.

Vignolo A, Noceti N, Rea F, et al. (2017) Detecting biological motion for human–robot interaction: a link between perception and action. Frontiers in Robotics and AI 4.

Wang H, Zhu C, Ma Z, et al. (2022) Improving generalization of deep networks for estimating physical properties of containers and fillings. In: The 32nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Singapore, 9147–9151.

Xiong Y, Quek FKH (2006) Hand motion gesture frequency properties and multimodal discourse analysis. International Journal of Computer Vision 69: 353–371.

Xompero A, Donaher S, Iashin V, et al. (2022) The Corsmal Benchmark for the Prediction of the Properties of Containers. IEEE Access.

Yu LF, Duncan N, Yeung SK (2015) Fill and transfer: a simple physics-based approach for containability reasoning. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 711–719.

|

You currently have no access to this content. Visit the access options page to authenticate.