Shoya Ishimaru

• This thesis investigates how smart sensors can quantify the process of learning. Traditionally, human beings have obtained various skills by inventing technologies. Those who integrate technologies into daily life and enhance their capabilities are called augmented humans. While most existing augmenting human technologies focus on directly assisting specific skills, the objective of this thesis is to assist learning -- the meta-skill to master new skills -- with the aim of long-term augmentations. Learning consists of cognitive activities such as reading, writing, and watching. It has been considered that tracking them by motion sensors (in the same way as the recognition of physical activities) is a challenging task because dynamic body movements could not be observed during cognitive activities. I have solved this problem with smart sensors monitoring eye movements and physiological signals. I propose activity recognition methods using sensors built into eyewear computers. Head movements and eye blinks measured by an infrared proximity sensor on Google Glass could classify five activities including reading with 82% accuracy. Head and eye movements measured by electrooculography on JINS MEME could classify four activities with 70% accuracy. In a wild experiment involving seven participants who wore JINS MEME more than two weeks, deep neural networks could detect natural reading activities with 74% accuracy. I demonstrate Wordometer 2.0, an application to estimate the number of rear words on JINS MEME, which was evaluated in a dataset involving five readers with 11% error rate. Smart sensors can recognize not only activities but also internal states during the activities. I present an expertise recognition method using an eye tracker which performs 70% classification accuracy into three classes using one minute data of reading a textbook, a positive correlation between interest and pupil diameter (p < 0.01), a negative correlation between mental workload and nose temperature measured by an infrared thermal camera (p < 0.05), an interest detection on newspaper articles, and effective gaze and physiological features to estimate self-confidence while solving multiple choice questions and spelling tests of English vocabulary. The quantified learning process can be utilized for feedback to each learner on the basis of the context. I present HyperMind, an interactive intelligent digital textbook. It can be developed on HyperMind Builder which may be employed to augment any electronic text by multimedia aspects activated via gaze. Applications mentioned above have already been deployed at several laboratories including Immersive Quantified Learning Lab (iQL-Lab) at the German Research Center for Artificial Intelligence (DFKI).