VisionMembersProjectsPublicationsNews

Prof. Dr. Shoya Ishimaru

Shoya Ishimaru is a Junior Professor in Computer Science at the University of Kaiserslautern and the director of Psybernetics Lab. Shoya received his PhD in Engineering with an award summa cum laude from the University of Kaiserslautern in 2019. He has also received the title of MITOU Super Creator, which is given to outstanding young developers (around 10 people per year) by the Ministry of Economy, Trade, and Industry in Japan.

He defines human intelligence as "the ability of learning, reasoning, and communicating," and has been developing systems to augment or amplify them (e.g., Wordometer, HyperMind, CoALA, Thermometer for the Mind). He has co-founded Immersive Quantified Learning Lab (a living lab for assistive learning technologies) and Alphaben (a company for assistive reading technologies) in order to develop practical applications using his research results.

Personal website: https://shoya.io

Projects

template

Quantifying Discussion by Using Camera as a Smart Sensor

template

CoALA: Confidence-Aware Learning Assistant

template

HyperMind: Intelligent Digital Textbook

template

Wordometer: Reading Activity Tracker

template

Sensor-Based Mental States Estimation

Publications

  1. David Dembinsky, Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Eye Movement in a Controlled Dialogue Setting”. In Proceedings of the 2024 Symposium on Eye Tracking Research and Applications (ETRA '24), 2024.
  2. Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Metacognition-EnGauge: Real-time Augmentation of Self-and-Group Engagement Levels Understanding by Gauge Interface in Online Meetings”. In Proceedings of the Augmented Humans International Conference 2024 (AHs '24), p. 301-303, 2024.
  3. Jayasankar Santhosh, Akshay Palimar Pai and Shoya Ishimaru. “Toward an Interactive Reading Experience: Deep Learning Insights and Visual Narratives of Engagement and Emotion”. IEEE Access, p. 6001-6016, 2024.
  4. Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Accelerating Knowledge Transfer by Sensing and Actuating Social-Cognitive States”. In Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing (UbiComp/ISWC '23 Adjunct), p. 258–262, 2023.
  5. Jayasankar Santhosh, David Dzsotjan and Shoya Ishimaru. “Multimodal Assessment of Interest Levels in Reading: Integrating Eye-Tracking and Physiological Sensing”. IEEE Access, p. 93994-94008, 2023.
  6. Ankur Bhatt, Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Appearance-Based Gaze Estimation with Deep Neural Networks: From Data Collection to Evaluation”. International Journal of Activity and Behavior Computing 2024 (1), p. 1-15, 2024.
  7. Ko Watanabe, Tanuja Sathyanarayana, Andreas Dengel and Shoya Ishimaru. “EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network”. IEEE Access, pp. 52886–52898, 2023.
  8. Ko Watanabe, Hacer Kübra Kücük, Sarah Gonzales, Andrew Vargo, Koichi Kise and Shoya Ishimaru. “Combining the Knowledge of Experienced Programmers to Extract Useful Web Resources for Solving Programming Tasks”. In Proceedings of the Asian HCI Symposium 2023 (Asian CHI '23), p. 22–27, 2023.
  9. Kanta Yamaoka, Ko Watanabe, Koichi Kise, Andreas Dengel and Shoya Ishimaru. “Experience is the Best Teacher: Personalized Vocabulary Building Within the Context of Instagram Posts and Sentences from GPT-3”. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '22 Adjunct), pp. 313–316, 2022.
  10. Haruki Suzawa, Ko Watanabe, Masakazu Iwamura, Koichi Kise, Andreas Dengel and Shoya Ishimaru. “Supporting Smooth Interruption in a Video Conference by Dynamically Changing Background Music Depending on the Amount of Utterance”. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '22 Adjunct), pp. 299–302, 2022.
  11. Andrew Vargo, Motoi Iwata, Mathilde Hutin, Sofiya Kobylyanskaya, Ioana Vasilescu, Olivier Augereau, Ko Watanabe, Shoya Ishimaru, Benjamin Tag, Tilman Dingler, Koichi Kise, Laurence Devillers and Andreas Dengel. “Learning Cyclotron: An Ecosystem of Knowledge Circulation”. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '22 Adjunct), pp. 308–312, 2022.
  12. Akshay Palimar Pai, Jayasankar Santhosh and Shoya Ishimaru. “Real-Time Feedback on Reader's Engagement and Emotion Estimated by Eye-Tracking and Physiological Sensing”. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '22 Adjunct), pp. 97–98, 2022.
  13. Chenhao Chen, Yutaka Arakawa, Ko Watanabe and Shoya Ishimaru. “Quantitative Evaluation System for Online Meetings Based on Multimodal Microbehavior Analysis”. Sensors and Materials 34 (8), pp. 3017–3027, 2022.
  14. Ko Watanabe, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa and Shoya Ishimaru. “How do Programmers Use the Internet? Discovering Domain Knowledge from Browsing and Coding Behaviors”. In 2022 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics), pp. 605–610, 2022.
  15. Andrew Vargo, Shoya Ishimaru, Md. Rabiul Islam, Benjamin Tag and Koichi Kise. “Obtaining Labels for In-the-Wild Studies: Using Visual Cues and Recall”. IEEE Pervasive Computing 21 (1), pp. 9–17, 2022.
  16. Ko Watanabe, Yusuke Soneda, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa, Andreas Dengel and Shoya Ishimaru. “DisCaaS: Micro Behavior Analysis on Discussion by Camera as a Sensor”. Sensors 21 (17), p. 5719, 2021.
  17. David Dzsotjan, Kim Ludwig-Petsch, Sergey Mukhametov, Shoya Ishimaru, Stefan Küchemann and Jochen Kuhn. “The Predictive Power of Eye-Tracking Data in an Interactive AR Learning Environment”. In Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '21 Adjunct), pp. 467–471, 2021.
  18. Pramod Vadiraja, Jayasankar Santhosh, Hanane Moulay, Andreas Dengel and Shoya Ishimaru. “Effects of Counting Seconds in the Mind while Reading”. In Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '21 Adjunct), pp. 486–490, 2021.
  19. Prerna Garg, Jayasankar Santhosh, Andreas Dengel and Shoya Ishimaru. “Stress Detection by Machine Learning and Wearable Sensors”. In Proceedings of the 26th International Conference on Intelligent User Interfaces Companion (IUI '21), pp. 237–240, 2021.
  20. Pramod Vadiraja, Andreas Dengel and Shoya Ishimaru. “Text Summary Augmentation for Intelligent Reading Assistant”. In Proceedings of the 2nd Augmented Humans International Conference (AHs '21), pp. 319–321, 2021.
  21. Shoya Ishimaru, Takanori Maruichi, Andreas Dengel and Koichi Kise. “Confidence-Aware Learning Assistant”. In arXiv preprint arXiv:2102.07312, 2021.
  22. Iuliia Brishtel, Anam Ahmad Khan, Thomas Schmidt, Tilman Dingler, Shoya Ishimaru and Andreas Dengel. “Mind Wandering in Multimodal Reading Setting: Behavior Analysis & Automatic Detection using Eye-Tracking and EDA Sensor”. Sensors 20 (9), p. 2546, 2020.
  23. Shoya Ishimaru, Takanori Maruichi, Koichi Kise and Andreas Dengel. “Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback”. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20 Asian CHI Symposium 2020), 2020.
  24. Jayasankar Santhosh, Shoya Ishimaru and Andreas Dengel. “Generating Heatmap for Unknown Documents Towards Readability Measurement”. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (IUI '20), pp. 47–48, 2020.
  25. Takanori Maruichi, Shoya Ishimaru and Koichi Kise. “Self-confidence Estimation on Vocabulary Tests with Stroke-level Handwriting Logs”. In Proceedings of the 15th IAPR International Conference on Document Analysis and Recognition (ICDAR HDI'19), pp. 18–22, 2019.
  26. Shoya Ishimaru, Takanori Maruichi, Manuel Landsmann, Koichi Kise and Andreas Dengel. “Electrooculography Dataset for Reading Detection in the Wild”. In Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '19 Adjunct), pp. 85–88, 2019.
  27. Shoya Ishimaru, Ko Watanabe, Nicolas Großmann, Carina Heisel, Pascal Klein, Yutaka Arakawa, Jochen Kuhn and Andreas Dengel. “HyperMind Builder: Pervasive User Interface to Create Intelligent Interactive Documents”. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '18 Adjunct), pp. 357–360, 2018.
  28. Soumy Jacob, Shoya Ishimaru and Andreas Dengel. “Interest Detection While Reading Newspaper Articles by Utilizing a Physiological Sensing Wristband”. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '18 Adjunct), pp. 78–81, 2018.
  29. Dayananda Herurkar, Shoya Ishimaru and Andreas Dengel. “Combining Software-Based Eye Tracking and a Wide-Angle Lens for Sneaking Detection”. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '18 Adjunct), pp. 54–57, 2018.
  30. Yuya Ohbayashi, Shoya Ishimaru, Andreas Dengel and Koichi Kise. “Investigating Gaze and Physiological Features to Estimate Comprehension on E-learning Video Lectures”. In Proceedings of the 1st international interdisciplinary Symposium on Reading Experience and Analysis of Documents (READ '18), 2018.
  31. Jayasankar Santhosh, Shoya Ishimaru and Andreas Dengel. “Estimating Fixation Durations for Each Word in Documents towards Readability Measurement”. In Proceedings of the 1st international interdisciplinary Symposium on Reading Experience and Analysis of Documents (READ '18), 2018.
  32. Nicolas Großmann, Iuliia Brishtel, Shoya Ishimaru, Andreas Dengel, Carina Heisel, Pascal Klein and Jochen Kuhn. “iQL - Immersive Quantified Learning Lab”. In Proceedings of the 1st international interdisciplinary Symposium on Reading Experience and Analysis of Documents (READ '18), 2018.
  33. Soumy Jacob, Shoya Ishimaru, Syed Saqib Bukhari and Andreas Dengel. “Gaze-Based Interest Detection on Newspaper Articles”. In Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction (ETRA '18), p. 4, 2018.
  34. Iuliia Brishtel, Shoya Ishimaru, Olivier Augereau, Koichi Kise and Andreas Dengel. “Assessing Cognitive Workload on Printed and Electronic Media using Eye-Tracker and EDA Wristband”. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion (IUI '18), p. 45, 2018.
  35. Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, Pascal Klein, Jochen Kuhn and Andreas Dengel. “Augmented Learning on Anticipating Textbooks with Eye Tracking”. In Positive Learning in the Age of Information (PLATO), pp. 387–398, 2018.
  36. Shoya Ishimaru, Soumy Jacob, Apurba Roy, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, Michael Thees, Jochen Kuhn and Andreas Dengel. “Cognitive State Measurement on Learning Materials by Utilizing Eye Tracker and Thermal Camera”. In Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition (ICDAR HDI '17), pp. 32–36, 2017.
  37. Shoya Ishimaru, Kensuke Hoshika, Kai Kunze, Koichi Kise and Andreas Dengel. “Towards Reading Trackers in the Wild: Detecting Reading Activities by EOG Glasses and Deep Neural Networks”. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '17 Adjunct), pp. 704–711, 2017.
  38. Koichi Kise, Olivier Augereau, Yuzuko Utsumi, Masakazu Iwamura, Kai Kunze, Shoya Ishimaru and Andreas Dengel. “Quantified Reading and Learning for Sharing Experiences”. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '17 Adjunct), pp. 724–731, 2017.
  39. Shoya Ishimaru and Andreas Dengel. “ARFLED: Ability Recognition Framework for Learning and Education”. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '17 Adjunct), pp. 339–343, 2017.
  40. Sabine Hoffmann, Helga Tauscher, Andreas Dengel, Shoya Ishimaru, Sheraz Ahmed, Jochen Kuhn, Carina Heisel and Yutaka Arakawa. “Sensing Thermal Stress at Office Workplaces”. In Proceedings of the 5th International Conference on Human-Environment Systems (ICHES '16), 2016.
  41. Shoya Ishimaru. “Cognitive State Recognition for Developing Anticipating Textbook”. In Proceedings of the 9th International Conference on Mobile Computing and Ubiquitous Networking (ICMU '16), pp. 1–2, 2016.
  42. Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Jochen Kuhn and Andreas Dengel. “Towards an Intelligent Textbook: Eye Gaze Based Attention Extraction on Materials for Learning and Instruction in Physics”. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '16 Adjunct), pp. 1041–1045, 2016.
  43. Shoya Ishimaru, Kai Kunze, Koichi Kise and Andreas Dengel. “The Wordometer 2.0: Estimating the Number of Words You Read in Real Life using Commercial EOG Glasses”. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '16 Adjunct), pp. 293–296, 2016.
  44. Shoya Ishimaru, Tilman Dingler, Kai Kunze, Koichi Kise and Andreas Dengel. “Reading Interventions: Tracking Reading State and Designing Interventions”. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '16 Adjunct), pp. 1759–1764, 2016.
  45. Christophe Rigaud, Thanh-Nam Le, J-C Burie, Jean-Marc Ogier, Shoya Ishimaru, Motoi Iwata and Koichi Kise. “Semi-Automatic Text and Graphics Extraction of Manga using Eye Tracking Information”. In Proceedings of the 2016 12th IAPR Workshop on Document Analysis Systems (DAS '16), pp. 120–125, 2016.
  46. Shoya Ishimaru and Koichi Kise. “Quantifying the Mental State on the Basis of Physical and Social Activities”. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '15 Adjunct), pp. 1217–1220, 2015.
  47. Kai Kunze, Katsuma Tanaka, Shoya Ishimaru, Yuji Uema, Koichi Kise and Masahiko Inami. “MEME: Eye Wear Computing to Explore Human Behavior”. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '15 Adjunct), pp. 361–363, 2015.
  48. Kai Kunze, Katsutoshi Masai, Masahiko Inami, Ömer Sacakli, Marcus Liwicki, Andreas Dengel, Shoya Ishimaru and Koichi Kise. “Quantifying Reading Habits: Counting How Many Words You Read”. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15), pp. 87–96, 2015.
  49. Kai Kunze, Kazutaka Inoue, Katsutoshi Masai, Yuji Uema, Sean Shao-An Tsai, Shoya Ishimaru, Katsuma Tanaka, Koichi Kise and Masahiko Inami. “MEME: Smart Glasses to Promote Healthy Habits for Knowledge Workers”. In ACM SIGGRAPH 2015 Emerging Technologies (SIGGRAPH '15), p. 17, 2015.
  50. Katsuma Tanaka, Shoya Ishimaru, Koichi Kise, Kai Kunze and Masahiko Inami. “Nekoze!: Monitoring and Detecting Head Posture while Working with Laptop and Mobile Phone”. In Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health '15), pp. 237–240, 2015.
  51. Shoya Ishimaru, Kai Kunze, Katsuma Tanaka, Yuji Uema, Koichi Kise and Masahiko Inami. “Smart Eyewear for Interaction and Activity Recognition”. In Extended Abstracts of the 2015 CHI Conference on Human Factors in Computing Systems (CHI '15 EA), pp. 307–310, 2015.
  52. Oliver Amft, Florian Wahl, Shoya Ishimaru and Kai Kunze. “Making Regular Eyeglasses Smart”. IEEE Pervasive Computing 14 (3), pp. 32–43, 2015.
  53. Shoya Ishimaru, Kai Kunze, Koichi Kise and Masahiko Inami. “Position Paper: Brain Teasers - Toward Wearable Computing That Engages Our Mind”. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct), pp. 1405–1408, 2014.
  54. Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise, Masahiko Inami and Katsuma Tanaka. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct), pp. 239–242, 2014.
  55. Shoya Ishimaru, Jens Weppner, Andreas Poxrucker, Paul Lukowicz, Kai Kunze and Koichi Kise. “Shiny: An Activity Logging Platform for Google Glass”. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct), pp. 283–286, 2014.
  56. Shoya Ishimaru, Jens Weppner, Kai Kunze, Andreas Bulling, Koichi Kise, Andreas Dengel and Paul Lukowicz. “In the Blink of an Eye: Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass”. In Proceedings of the 5th Augmented Human International Conference (AH '14), pp. 150–153, 2014.
  57. Shoya Ishimaru, Kai Kunze, Yuzuko Utsumi, Masakazu Iwamura and Koichi Kise. “Where Are You Looking At? - Feature-Based Eye Tracking on Unmodified Tablets”. In Proceedings of the 2013 2nd Asian Conference on Pattern Recognition (ACPR '13), pp. 738–739, 2013.
  58. Kai Kunze, Shoya Ishimaru, Yuzuko Utsumi and Koichi Kise. “My Reading Life: Towards Utilizing Eyetracking on Unmodified Tablets and Phones”. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '13 Adjunct), pp. 283–286, 2013.
  59. Kai Kunze, Yuki Shiga, Shoya Ishimaru and Koichi Kise. “Reading Activity Recognition Using an Off-the-Shelf EEG – Detecting Reading Activities and Distinguishing Genres of Documents”. In Proceedings of the 2013 12th International Conference on Document Analysis and Recognition (ICDAR '13), pp. 96–100, 2013.

© 2021 PsyberLab | Privacy Policy