Multimodal learning with audio description: an eye tracking study of children's gaze during a visual recognition task

Krzysztof Krejtz , Izabela Krejtz , Andrew T. Duchowski , Agnieszka Szarkowska , Agnieszka Walczak

Abstract

The paper explores the effects of adding audio description to an educational film on children's learning behaviour, manifested by a visual recognition task. We hypothesize that the multimodal educational setting, consisting of both verbal (film dialogue and audio description) and non-verbal (motion pictures) representations of knowledge, fosters knowledge acquisition as it provides information via multiple channels, which in turn strengthens memory retrieval. In the study we employ eye tracking methodology to examine the recognition of previously seen film material, testing whether audio description promotes recognition- rather than elimination-based decision-making in the visual recognition task. The analysis of first fixation duration and first run fixation count measures in the experimental and control groups partially confirmed our hypotheses. Children in the experimental group generally looked longer at the scenes they had seen, which supports the hypothesis that their decision was based on recognition, whereas children in the control group had longer fixations on scenes they were unfamiliar with, suggesting a decision based on elimination.