Toward Artificial Synesthesia: Linking Images and Sounds via Words

We tackle a new challenge of modeling a perceptual experience in which astimulus in one modality gives rise to an experience in a different sensorymodality, termed synesthesia. To meet the challenge, we propose a probabilisticframework based on graphical models that enables to link visual modalities andauditory modalities via natural language text. An online prototype system isdeveloped for allowing human judgement to evaluate the model’s performance.Experimental results indicate usefulness and applicability of the framework.