Welcome to Pengyu Hong's Homepage

Multimodal Human-Computer Interaction

A probabilistic approach has been developed for solving multimodal references in conversational systems. We proposed to represent multimodal information as attributed relational graphs. Particularly, we represented information from multiple modalities and contexts as attributed relational graphs. One attributed relational graphs is used to represent multimodal references. Hence, solving unknown references is converted into the problem of graph matching. We developed a probabilistic graph match algorithm to solve the graph matching problem. Our approach has three features. First, it systematically incorporates rich contexts such as conversation context, domain knowledge and visual feedback. Second, it simultaneously applies semantic constraints and temporal constraints. Third, it tolerates imprecise or ambiguous information.

This is a joint work with Joyce Chai and Michelle Zhou at the IBM T. J. Watson lab when I was a visiting scientist there in 2002.

References:

• Chai, J., Hong, P., Zhou, M., and Prasov, Z. (2004) Optimization in Multimodal Interpretation. The 42nd Annual Conference of Association of Computational Linguistics, Barcelona, Spain. July 22-24, 2004.
• Chai, J., Hong, P., and Zhou, M. X. (2004) A Probabilistic Approach to Reference Resolution in Multimodal User Interface. International Conference on Intelligent User Interfaces, Madeira, Portugal, Jan. 2004.
• Chai, J., Prasov, Z., and Hong, P. (2004) Performance Evaluation and Error Analysis for Multimodal Interpretation in Conversational Systems. In Proceedings of HLT-NAACL, Boston, MA, May 3-7, 2004.