Aaron Adler
ABSTRACT
While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. We studied the behavior of people sketching and speaking, and from this derived a set of rules for segmenting and aligning the signals from both modalities. Once the inputs are aligned, we use both modalities in interpretation. The result is a more natural interface to our system.
- A. Adler. Segmentation and Alignment of Speech and Sketching in a Design Environment. Master's Thesis, Massachusetts Institute of Technology, 2003.Google Scholar
- C. Alvarado and R. Davis. Resolving ambiguities to create a natural computer-based sketching environment. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 1365--1374, 2001. Google ScholarDigital Library
- K. D. Forbus, R. W. Ferguson, and J. M. Usher. Towards a computational model of sketching. In Proceedings of the 6th International Conference on Intelligent User Interfaces, pages 77--83. ACM Press, 2001. Google ScholarDigital Library
- T. J. Hazen, S. Seneff, and J. Polifroni. Recognition confidence scoring and its use in speech understanding systems. Computer Speech and Language, 16:49--67, 2002.Google ScholarDigital Library
- M. Johnston, S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, and P. Maloor. MATCH: An architecture for multimodal dialogue systems. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 376--383, 2002. Google ScholarDigital Library
- M. Oltmans. Understanding Naturally Conveyed Explanations of Device Behavior. Master's Thesis, Massachusetts Institute of Technology, 2001.Google Scholar
- S. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro. Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions. Human Computer Interaction, 15(4):263--322, 2000. Google ScholarDigital Library
Index Terms
- Speech and sketching for multimodal design
Recommendations
Speech and sketching for multimodal design
SIGGRAPH '07: ACM SIGGRAPH 2007 coursesWhile sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...
Speech and sketching for multimodal design
SIGGRAPH '06: ACM SIGGRAPH 2006 CoursesWhile sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...
Multimodal human discourse: gesture and speech
Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the ...
Comments