
	Where results make sense

Topic: Multimodal integration

In the News (Sat 26 Dec 09)

Northern Trust

Marvin Harrison

Peter Chernin

Gary Locke

Match Play

Penelope Cruz

Mickey Rourke

AR Rahman

Danny Boyle

Hugh Jackman

	Test bed for integration of audio and video interfaces
		Integration of the two modes of interaction is explored in the multimodal integration module at different levels of integration.
		Integration of two interaction modalities is carried out in the multimodal integration module.
		The integration module subsamples the input vectors from the previous two modules at a rate twice that of the highest frequency of change in the visual and auditory features.
	www.ifp.uiuc.edu /~berry/thesis/node31.html (1324 words)

	Multimodal integration - Wikipedia, the free encyclopedia
		Multimodal integration, also known as multisensory integration, is the integration in the brain of inputs from the various senses, to form multimodal representations or percepts.
		To take an example, a single event such as an object falling on one's foot is likely to generate input for vision, audition, tactition (the perception of pressure), and nociception (the perception of pain).
		The fact that we are able in some way to connect all of those, and to attribute the different percepts to a single cause, exemplifies multimodal integration.
	en.wikipedia.org /wiki/Multimodal_integration (112 words)

	proposal
		Providing systems with multimodal behaviour is seen as a way of allowing for more flexible and user-friendly interaction between humans and digital services: by improving the information systems' functionality, usability, and acceptability, multimodality makes it possible for human-machine communication to take place in a more natural way.
		Currently focus is on integrating dialogue systems with intelligent document processing techniques in order to develop multimodal dialogue systems that can retrieve information from unstructured documents, where the request requires that the user, in a dialogue with the system, specifies their information needs (Merkel and Jönsson, 2001).
		Multimodal human-computer interaction including speech, touch and gaze, with particular regard to the usability aspect, is one of the main research themes within the group.
	cst.dk /mumin/proposal.html (2867 words)

	Multimodal Interaction Requirements
		Multimodal interactions extend the Web user interface to allow multiple modes of interaction, offering users the choice of using their voice, or an input device such as a key pad, keyboard, mouse or stylus.
		In multimodal systems, an event is a representation of some asynchronous occurrence of interest to the multimodal system.
		Simultaneous multimodal inputs, imply that the inputs from several modalities are interpreted one after the other in the order that they where received instead of being combined before interpretation.
	www.w3.org /TR/mmi-reqs (9592 words)

	Multimodal Integration for Advanced Multimedia Interfaces (MIAMI)
		Multimodality aspects in accessing, representing and producing multimedia information by visual, acoustical and tactile/gestural systems are studied, as the basis for future multimedia systems.
		This integration is useful, both in the disambiguation of the human input to the system and in the disambiguation of the system output.
		This project is focused on a central issue of the integrated handling of information acquired by different modalities and presented in a multimedia context.
	tcw2.ppsw.rug.nl /~lambert/projects/miami (371 words)

	BattleView
		Integration of the two modes defines the multimodal integration module.
		Integration of gestures and speech occur in the multimodal integration module.
		The integration module emulates the wand functions by using a state machine to track meaningful states and by manipulating data into specific formats.
	www.ifp.uiuc.edu /~berry/thesis/node30.html (1207 words)

	XML.com: Multimodal Interaction on the Web
		Multimodal interactions are the center of multimodal systems, systems that support communication with the user through different modes, including voice, gesture, handwriting, and typing.
		In multimodal systems an event is a representation of some occurrence of interest to the multimodal system.
		EMMA is the Extensible MultiModal Annotation markup language.
	www.xml.com /pub/a/2004/01/21/multimodal.html (1465 words)

	Cognitive factors facilitate multimodal integration. Proceedings of the 8th Tübinger Perception Conference. (Eds.) ...
		Integration seems to be broken if there is a spatial discrepancy between the signals (Gepshtein et al., submitted).
		If integration breaks due to the spatial discrepancy we expect subjects’ percept to be less biased by multimodal information.
		In conclusion, multimodal signals seem to be combined if observers have high-level cognitive knowledge about the signals belonging to the same object, even when there is a spatial discrepancy.
	www.kyb.tuebingen.mpg.de /publication.html?publ=3238 (356 words)

	Multimodal Integration Patterns in Children - Xiao, Girand, Oviatt (ResearchIndex) (Site not responding. Last check: 2007-11-06)
		Abstract: Multimodal interfaces are designed with a focus on flexibility, although very few multimodal systems currently are capable of adapting to major sources of user or environmental variation.
		2: Integration and synchronization of input modes during multimodal human-computer..
		45 Integration and synchronization of input modes during multim..
	citeseer.ist.psu.edu /xiao02multimodal.html (303 words)

	Lecture notes on Multimodal Interfaces, by Antti Oulasvirta
		She was General Chair of the International Conference on Multimodal Interfaces (ICMI) in 2003, and is Founding Chair of ICMI's Advisory Board.
		Simultaneous integrators start input in one modality before input in another modality has ended, whereas sequential integrators leave a temporal gap before switching to the other.
		Furthermore, Oviatt found “entrenching” in integration patterns during increased task difficulty, what she believed was due to an attempt to restoring equilibrium (in the spirit of Gestalt theories) to an unbalanced situation.
	www.cs.helsinki.fi /u/oulasvir/multimodality (1593 words)

	CHCC Current and Past Research
		Multimodal systems offer the potential for considerable flexibility, broad utility, and use by a larger and more diverse population than ever before.
		Another objective is to examine the impact of system display characteristics on users' subsequent linguistic input to multimodal systems and to apply this information to the design of interfaces that effectively but transparently guide users' input to match system processing capabilities.
		The system will have to understand aspects of their [multimodal] discussion and decide when it ought to summarize by placing the unit under discussion at the agreed upon location on a map.
	www.cse.ogi.edu /CHCC/Research/main.html (1695 words)

	EMMA: Extensible MultiModal Annotation markup language
		The Multimodal Interaction Working Group may address in later drafts the issues of modularization and profiling, that is: which sets of annotations are to be supported by which classes of EMMA component.
		In the W3C Multimodal Interaction Framework, multimodal integration can be handled by an integration component which follows the application of speech understanding and other kinds of interpretation procedures for individual modes.
		Given the broad range of different techniques being employed for multimodal integration and the extent to which this is an ongoing research problem, standardization of the specific method or algorithm used for multimodal integration is not appropriate at this time.
	www.w3.org /TR/emma (11466 words)

	Multimodal Interfaces for Mobile Information Access
		Multimodal Access to City Help provides a testbed for our research on multimodal interaction and has driven the development of multimodal platform which enables a broad range of mobile applications.
		Multimodal Access to City Help is a working city guide and navigation system that enables mobile users to access restaurant and subway information for New York City.
		Multimodal Access to City Help also uses novel multimodal generation techniques to generate synchronized presentations of subways routes and restaurant information that combine dynamic graphics and synthetic speech.
	www.research.att.com /projects/MultimodalAccessToCityHelp.old/index.html (585 words)

	Recent Progress in the Design of Advanced Multimodal Interfaces (Site not responding. Last check: 2007-11-06)
		In the present talk, I will describe state-of-the-art research on multimodal interaction and interface design, and in particular two topics that are generating considerable activity at the moment both within our own lab and around the world.
		The second involves a recent surge of research activity on human multisensory processing and users' multimodal integration patterns during human-computer interaction, as well as implications for the design of adaptive multimodal interfaces.
		The long-term goal of research in these and related areas is the development of advanced multimodal interfaces that can support new functionality, unparalleled robustness, and flexible adaptation to individual users and real-world mobile usage contexts.
	www.cse.msu.edu /DLS/oviatt.htm (410 words)

	Multimodal dialogue systems for industrial applications (Site not responding. Last check: 2007-11-06)
		Håkan Johansson is a new graduate student that has developed a model for synergistic integration of multimodal speech and pen information (Johansson 2001a, 2001b, 2001c).
		The model consists of an algorithm for matching and integrating interpretations of inputs from different modalities, as well as of a grammar that constrains integration.
		The integrator is part of the general framework for multimodal information systems with dialogue capabilities.
	www.ida.liu.se /~arnjo/ceniit.html (2624 words)

	NIH Guide: MULTIMODAL INTEGRATION RESEARCH NETWORKS IN COGNITIVE NEUROSCIENCE
		MULTIMODAL INTEGRATION RESEARCH NETWORKS IN COGNITIVE NEUROSCIENCE Release Date: October 4, 2001 RFA: RFA-NS-02-011 National Institute of Neurological Disorders and Stroke (http://www.ninds.nih.gov/) Letter of Intent Receipt Date: December 17, 2001 Application Receipt Date: January 18, 2002 THIS RFA USES "MODULAR GRANT" AND "JUST-IN-TIME" CONCEPTS.
		One promising approach is multimodal integration - the combination of imaging and electrophysiological recording techniques to increase the precision of temporal and spatial data beyond what is possible with the use of a single technique.
		The goal of this RFA is to act as a catalyst for the formation of novel collaborations among cognitive scientists, neuroscientists, behavioral and computational neuroscientists, imaging specialists and clinical scientists.
	grants.nih.gov /grants/guide/rfa-files/RFA-NS-02-011.html (3005 words)

	Title
		This paper focus is on a central question in neuroscience, “How the brain integrates information from multiple modalities?” The effects of a meow of a cat produce two completely different sensations; however, they are used in identification of the object, whether animals or man made objects.
		The ventral temporal cortex responds to the form, color, and texture of objects, while the lateral temporal cortex is responsive to the motion of objects.
		The hypothesis proposed in this paper was that auditory-visual integration of complex objects might occur in the midtemporal cortex, between the auditory association cortex in the superior temporal gyrus (STG) and the visual association cortex in the posterior lateral temporal cortex.
	www.unr.nevada.edu /~mzappett (933 words)

	[No title] (Site not responding. Last check: 2007-11-06)
		Late fusion also allows in principle the integration of more than one modality, and the addition of modalities to a system in an easier way than is generally possible when early fusion is used.
		In this article, 10 myths about multimodal interaction are identified as currently fashionable among computationalists and are discussed from the perspective of contrary empirical evidence.
		Multimodal interaction enables the user to employ different modalities such as voice, gesture and typing for communicating with a computer.
	home.comcast.net /~pbarthelmess/Survey_Pages/Sign/FrameMult.htm (4084 words)

	Task Integration in Multimodal Speech Recognition Environments
		It was noted that the most influential factor in predicting the use of integrated multimodal speech and handwriting was contrastive functionality.
		In other words, participants were most likely to integrate the two modalities in a contrastive way to designate a shift in context or functionality, such as original input versus corrected input, or data versus command.
		A similar approach was taken while integrating a natural language interface with a graphical airborne early warning test planning tool at the Naval Research Laboratory [18].
	www.cs.umbc.edu /~mikeg/papers/report02.html (2974 words)

	References
		Crossmodal integration in the identification of consonant segments.
		Integration audition-vision par reseaux de neurones: une etude comparative des modeles d'integration appliques la perception des voyelles.
		Integration of acoustic and visual speech signal using neural networks.
	hwr.nici.kun.nl /~miami/taxonomy/node168.html (4146 words)

	Position Paper for W3C
		This note presents the work on multimodal dialog that is one of the activities of the W3C Voice Browsing Working Group. It also introduces some work at HP Labs on multimodal systems.
		Multimodal browsers allow users to interact via a combination of modalities, for instance, speech recognition and synthesis, displays, keypads and pointing devices.
		It is no longer always sufficient to wait for the first input before moving on to the next step in the dialog. A multimodal integration module combines the information (semantics) provided by the individual modalities. The integration follows temporal constraints, i.e.
	www.w3c.org /2000/09/Papers/HPLabs.html (633 words)

	Unification-based Multimodal Integration - Johnston, Cohen, McGee, Oviatt, Pittman, Smith (ResearchIndex) (Site not responding. Last check: 2007-11-06)
		This paper describes a multimodal language processing architecture which supports interfaces allowing simultaneous input from speech and gesture recognition.
		Integration of spoken and gestural input is driven by unification of typed feature structures representing the semantic contributions of the different modes.
		12: Integration and synchronization of input modes during multimodal human-computer..
	citeseer.ist.psu.edu /135215.html (496 words)

	Dr. Phil Cohen
		Multimodal Interaction under Exerted Conditions in a Natural Field Setting.
		Corradini, A. and Cohen, P.R. Multimodal Speech-Gesture Interface for Handfree Painting on a Virtual Paper using Partial Recurrent Neural Networks as Gesture Recognizer.
		McGee, D. and Cohen, P.: "Augmenting Physical Tools: Bridging the Chasm of Late Adoption with Multimodal Language," to appear in proceedings of HCI International 2001, the 9th International Conference of Human-Computer Interaction (HCI'01) in New Orleans, August 5-10, 2001.
	www.cse.ogi.edu /CHCC/Personnel/pcohen.html (2386 words)

	[No title]
		Speech and pointing is the dominant multimodal integration pattern.
		Multimodal language does not differ linguistically from unimodal language.
		Multimodal integration involves redundancy of content between modes.
	www.acm.vt.edu /~jchen/cs5984/multimodal_handout.doc (122 words)

	Journal of Vision - Integration of multimodal cues in temporal segmentation of visual motion, by Sekuler & Wong
		Integration of multimodal cues in temporal segmentation of visual motion
		To minimize individual differences, each cue's physical value was adjusted according to individual subjects' sensitivity to that cue; additionally, the effects of the separate cues were equated to one another.
		Sekuler, R., and Wong, V. Integration of multimodal cues in temporal segmentation of visual motion [Abstract].
	www.journalofvision.org /4/8/704 (308 words)

	M4 Project
		Smart rooms are environments equipped with multimodal sensors and computers, that are designed to enable their inhabitants to work more efficiently as individuals and in groups.
		In multimodal communication it is often the case that the semantic information is spread across different modalities.
		A multimodal indexing system will be at the centre of the information access tools, building on the partners' experience of text, speech, image and video retrieval.
	www.m4project.org /programme.html (2486 words)

	IDIAP : Multimodal Interaction
		The objective of the PASCAL network is to build a Europe-wide Distributed Institute which will pioneer principled methods of pattern analysis, statistical modelling and computational learning as core enabling technologies for multimodal interfaces that are capable of natural and seamless interaction with and among individual human users.
		It is proving an increasingly important tool in Machine Vision, Speech, Haptics, Brain Computer Interfaces, Information Extraction and Natural Language Processing; it provides a uniform methodology for multimodal integration; it is an invaluable tool in information extraction; while on-line learning provides the techniques needed for adaptively modelling the requirements of individual users.
		Though machine learning has such potential to improve the quality of multimodal interfaces, significant advances are needed, in both the fundamental techniques and their tailoring to the various aspects of the applications, before this vision can become a reality.
	www.idiap.ch /multimodal_interaction.php?project=25 (204 words)

	Multimodal Integration - A Biological View - Coen (ResearchIndex) (Site not responding. Last check: 2007-11-06)
		We argue that perceptual integration in multimodal systems needs to happen at all levels of the individual perceptual processes.
		integration of modalities and cross modal influence: the sharing of sensory information across modalities at various levels of perception
		Multimodal Integration in a Wider Sense - Boda Pter Pl
	citeseer.ist.psu.edu /coen01multimodal.html (679 words)

Try your search on: Qwika (all wikis)