RIKEN Information R&D and Strategy Headquarters Multimodal Data Recognition Research Team

Team Director: Yasutomo Kawanishi (Ph.D.)

Japanese Page

Research Summary

Our goal is to recognize surrounding environments and humans around a robot. We are focusing on signal processing and pattern recognition using sensory data in various modality.

We mainly tackle these topics.

Human activity recognition, Facial expression recognition, and Intention estimation.
Multimodal sensor fusion for more various and much detailed object recognition.
Detection and description of unknown events/objects.

Main Research Fields

Informatics

Related Research Fields

Engineering
Complex Systems
Perceptual information processing
Intelligent informatics
Intelligent robotics

Keywords

Computer Vision
Pattern Recognition
Multimedia
Robotics

Selected Publications

1. Trung Thanh Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
"Hierarchical Global-Local Fusion for One-stage Open-vocabulary Temporal Action Detection"
ACM Transactions on Multimedia Computing, Communications, and Applications, (2025).
2. Hao-Yu Hou, Chun-Yi Lee, Motoharu Sonogashira, Yasutomo Kawanishi
"FROSS: Faster-Than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images"
The 20th International Conference on Computer Vision, (2025).
3. Trung Thanh Nguyen, Yasutomo Kawanishi, Vijay John, Takahiro Komamizu, Ichiro Ide
"Action Selection Learning for Weakly Labeled Multi-modal Multi-view Action Recognition"
"ACM Transactions on Multimedia Computing, Communications, and Applications, (2025).
4. Jia-Yi Chen, Vijay John, Yasutomo Kawanishi
"Cross-modal Emotion-specific Attention Model for Multimodal Emotion Recognition"
The 19th IEEE International Conference on Automatic Face and Gesture Recognition, (2025).
5. Yasutomo Kawanishi, Yutaka Nakamura, Taiken Shintani, Carlos T. Ishi, Seiya Kawano, Koichiro Yoshino, Takashi Minato, Michihiko Minoh
"RoboDJ: Live Commentary Robots System Driven by Physical- and Cyber-world Observations"
The 31st International Conference on Multimedia Modeling, (2025). (Best Demo Honorable Mention)
6. Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi
"Relationship-Aware Unknown Object Detection for Open-Set Scene Graph Generation"
IEEE Access, vol.12, pp.122513 - 122523, (2024) (open access).
7. Masaya Mizuno, Tomohiro Fujita, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi
"MuraseSubjective Baggage-Weight Estimation based on Human Walking Behavior"
IEEE Access, vol. 12, pp. 39390 - 39398, (2024).
8. Daiju Kanaoka, Hakaru Tamukoh, Motoharu Sonogashira, Yasutomo Kawanishi
"ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields"
In Proceedings of the 34th British Machine Vision Conference, .
9. Shu Nakamura, Yasutomo Kawanishi, Shohei Nobuhara, Ko Nishino
"DeePoint: Visual Pointing Recognition and Direction Estimation"
In Proceedings of the 19th International Conference on Computer Vision, (2023)./li>
10. Yasutomo Kawanishi
"Label-Based Multiple Object Ensemble Tracking with Randomized Frame Dropping"
In Proceedings of the 26th International Conference on Pattern Recognition (2022).

Lab Members

Principal investigator

Yasutomo Kawanishi: Team Director

Core members

Motoharu Sonogashira: Research Scientist
Itthisak Phueaksri: Postdoctoral Researcher
Daiju Kanaoka: Postdoctoral Researcher
Tingwei Liu: Junior Research Associate

Contact Information

3rd Floor, Advanced Telecommunications Research Institute International
2-2-2 Hikaridai Seika-cho, Sorakugun, Kyoto
619-0288 Japan (Kansai Science City)
Email: yasutomo.kawanishi@riken.jp