social linkedin box blue 32
social facebook box blue 32
social twitter box blue 32
social facebook box blue 32

iit-rbcs-logo-v3

Home RBCS ■ Multisensory Mapping

Multisensory Mapping


We investigate how sensory deprived individuals compensate missing sensory channels by vicarious modalities. Our focus is on sensory enhancement and how to achieve it with novel assistive technologies, mainly aimed at the construction of cognitive maps. Our methodology serves to build hardware/software platforms to decrease the digital divide, therefore increasing social inclusion.

Group leader: Luca Brayda (click here to contact him)

Research Topics :

  • Sensory Substitution
    alt


    How much spatial knowledge depends on visual experience? We investigate the neural and behavioural correlates of tactile spatial representations. 


  • Small Area Haptic Displays
    alt

    Are we able to understand simple tactile virtual objects? We study how information can be coded, displayed and understood by humans through low-tech haptic displays.

  • Sensory supplementation
    alt

    Is it possible to improve the spatial soundscape of hearing impaired individuals? We study how binaural acoustic feedback can be used in context where hearing loss prevents proper spatial awareness.

EXTERNAL PROJECTS:

BLINDPAD

The BLINDPAD (Personal Assistive Device for BLIND and visually impaired people) project aims at making graphical content accessible to visually impaired subjects.

GLASSENSE

The GLASSENSE project (Wearable Technologies for Sensory Supplementation) aims at building novel devices to assist hearing-impaired and visually-impaired people in daily tasks such as acoustic source localization and reading.

SELECTED PUBLICATIONS:


  • *Campus C., *Brayda L., De Carli F., *Chellali R., Famà F., *Bruzzo C., Lucagrossi L. and Rodriguez G. (2012)
    Tactile exploration of virtual objects for blind and sighted people: the role of beta 1 EEG band in sensory substitution and supra-modal mental mapping
    Journal of Neurophysiology, vol. 107, (no. 10), pp. 2713-2729, 0022-3077
  • *Brayda L., *Campus C., *Chellali R., Rodriguez G. and Martinoli C. (2011)
    An investigation of search behaviour in a tactile exploration task for sighted and non-sighted adults
    International conference on Human factors in computing systems, Vancouver, Canada
  • *Campus C., *Brayda L., *Chellali R., Martinoli C. and Rodriguez G. (2011)
    A neurophysiological and behavioral investigation of tactile spatial exploration for sighted and non-sighted adults
    Human Factors and Ergonomics Society 55th Annual Meeting, September 19-23 2011, vol. 55, (no. 1), pp. 227-231, Las Vegas, Nevada, USA 2011
  • *Brayda L. and Campus C. (2012)
    Conveying perceptible virtual tactile maps with a minimalist sensory substitution device
    IEEE Haptic Audio-Visual Environments and Games, Munich, Germany
  • *Brayda L. and *Chellali R. (2012)
    Measuring Human-Robots Interactions
    International Journal of Social Robotics, pp. 1-3, 1875-4791
  • *Brayda L., *Campus C. and *Gori M. (2013)
    What you touch is what you get: self-assessing a minimalist sensory substitution device
    IEEE World Haptics Conference 2013, (no. 491-496), Daejeon, Korea, April 14-17, 2013
  • *Campus C., Lucagrossi L., *Memeo M. and *Brayda L.
    Similarity of blind and sighted subjects when constructing maps with small-area tactile displays: performance, behavioural and subjective aspects
    EuroHaptics 2014, 24th and 27th of June in Versailles, France
  • *Memeo M., *Campus C. and *Brayda L.
    Do blind subjects differ from sighted subjects when exploring virtual tactile maps?
    International Conference on Computers Helping People with special needs, pp. 12-17, Saint Denis, France, July 9-11, 2014
  • *Brayda L., *Campus C., *Memeo M. and Lucagrossi L.
    The importance of visual experience, gender and emotion in the assessment of an assistive tactile mouse
    IEEE Transactions on Haptics, 1939-1412
  • *Brayda L., *Campus C. and *Gori M.
    Predicting successful tactile mapping of virtual objects
    IEEE Transactions on Haptics, vol. 6, pp. 473-483, 1939-1412
  • Geronazzo M., Bedini A., *Brayda L., *Campus C. and Avanzini F.
    Interactive spatial sonification for non-visual exploration of virtual maps
    International Journal of Human-Computer Studies
alt Sensory Substitution
  • Touch
  • Brain
  • Haptics
  • Sensory Rehabilitation
People involved:

Luca Brayda Claudio Campus


BACKGROUND

The neural correlates of exploration and cognitive mapping in blindness remain elusive. The role of visuo-spatial pathways in blind versus sighted subjects is still under debate. In this preliminary study we investigate, as a possible estimation of the activity in the visuo-spatial pathways, the EEG patterns of blind and blindfolded sighted subjects during the active tactile construction of cognitive maps from virtual objects as compared to rest and passive tactile stimulation.

 Image titleImage titleImage title

STUDY 1: Neural correlates in the EEG beta 1 band


When the finger was only stimulated (passive stimulation) or the contour of a virtual object was touched (during active exploration), event-related spectral power and coherence perturbations were evaluated within the beta 1 band (14-18 Hz). They were then related to a subjective cognitive-load estimation required by the explorations (Perceived Levels of Difficulty).
RESULTS: We found complementary cues for sensory substitution and spatial processing in both groups: both blind and sighted subjects showed, while exploring, late power decreases and early power increases, potentially associated with motor programming and touch, respectively. The latter involved occipital areas only for blind subjects (long-term plasticity) and only during active exploration, thus supporting tactile-to-visual sensory substitution. In both groups, coherences emerged among the fronto-central, centro-parietal and occipito-temporal derivations associated with visuo-spatial processing. This seems in accordance with mental map construction involving spatial processing, sensory-motor processing and working memory. The observed involvement of the occipital regions suggests that a substitution process also occurs in sighted subjects. Only during explorations, coherence correlated positively with Perceived Levels of Difficulty for both groups and in derivations which can be related to visuo-spatial processing, supporting the existence of supra-modal spatial processing independently of vision capabilities.
 
STUDY 2 How search behaviour reflects EEG patterns


A new method for evaluating objectively the process of performing a tactile exploration with a visuo-tactile sensory substitution system is proposed. Both behavioral and neurophysiological cues are considered to evaluate the identification process of virtual objects and surrounding environments.
RESULTS: Our experiments suggest that both sighted and visually impaired users integrated spatial information and developed similar behavioural and neurophysiological patterns. The proposed method could also serve as a tool to evaluate touch-based interfaces for application in orientation and mobility programs.




 

 

 


*Campus C., *Brayda L., *Chellali R., Martinoli C. and Rodriguez G. (2011)
A neurophysiological and behavioral investigation of tactile spatial exploration for sighted and non-sighted adults
Human Factors and Ergonomics Society 55th Annual Meeting, September 19-23 2011, vol. 55, (no. 1), pp. 227-231, Las Vegas, Nevada, USA 2011

*Campus C., *Brayda L., De Carli F., *Chellali R., Famà F., *Bruzzo C., Lucagrossi L. and Rodriguez G. (2012)
Tactile exploration of virtual objects for blind and sighted people: the role of beta 1 EEG band in sensory substitution and supra-modal mental mapping
Journal of Neurophysiology, vol. 107, (no. 10), pp. 2713-2729, 0022-3077

top
alt Small Area Haptic Displays
  • Haptics
  • Sensory Rehabilitation
  • Assistive device
People involved:

Mariacarla Memeo Claudio Campus Monica Gori Luca Brayda



We have proposed TAMO (TActile MOuse), a device able to display tactile virtual reliefs. TAMO has been designed to help visually impaired people to develop cognitive maps of unknown geometrical contents.


RESULTS:


- TAMO was found to reflect users' expectations. In particular, subjects can predict their own tactile sensitivity (we can say that "WYTIWYG: What You Touch Is What You Get"). Therefore, users can self-assess their tactile performance, combining their own judgements with objective data. This is crucial since TAMO is thought to be used in home-based scenarios. Download the paper HERE
- TAMO is also used to discover how blind develop spatial exploration strategies, as compared to sighted. There is a long debate about how impaired subjects acquire and imagine maps. This is important to design correctly targeted rehabilitation devices. We found that blind subjects develop very similar spontaneous strategies as their sighted fellows. See a video of the strategies HERE and HERE.
- TAMO elicits similar brain areas in blind than in sighted people. This is important because it shows that spatial abilities can be developed independently from vision capabilities. Read the paper HERE.
- TAMO is a low-cost device. It is built with commercial-off-the-shelf components. 
 


VIDEOS:


http://youtu.be/0HjYpVXmxRA

http://www.youtube.com/watch?v=IhGfYzNrlIg


How to use TAMO to explore a virtual object

*Brayda L. and Campus C. (2012)
Conveying perceptible virtual tactile maps with a minimalist sensory substitution device
IEEE Haptic Audio-Visual Environments and Games, Munich, Germany

*Brayda L., *Campus C. and *Gori M. (2013)
What you touch is what you get: self-assessing a minimalist sensory substitution device
IEEE World Haptics Conference 2013, (no. 491-496), Daejeon, Korea, April 14-17, 2013

*Brayda L., *Campus C., *Memeo M. and Lucagrossi L.
The importance of visual experience, gender and emotion in the assessment of an assistive tactile mouse
IEEE Transactions on Haptics, 1939-1412

*Campus C., *Brayda L., De Carli F., *Chellali R., Famà F., *Bruzzo C., Lucagrossi L. and Rodriguez G. (2012)
Tactile exploration of virtual objects for blind and sighted people: the role of beta 1 EEG band in sensory substitution and supra-modal mental mapping
Journal of Neurophysiology, vol. 107, (no. 10), pp. 2713-2729, 0022-3077

*Brayda L., *Campus C. and *Gori M.
Predicting successful tactile mapping of virtual objects
IEEE Transactions on Haptics, vol. 6, pp. 473-483, 1939-1412

Geronazzo M., Bedini A., *Brayda L., *Campus C. and Avanzini F.
Interactive spatial sonification for non-visual exploration of virtual maps
International Journal of Human-Computer Studies

top
alt Sensory supplementation
  • Disabilities
  • Perception
  • Hearing
People involved:

Luca Giuliani Luca Brayda Francesco Diotalevi Victor Oliveira

Glassense enhances acoustic sources of interest in the direction of look. Two arrays of microphones mounted on a pair of glasses, powered by a pocket-size elaboration board, spatially filter the surrounding soundscape. The enhanced acoustic content (e.g. speech) can be delivered to existing acoustic prostheses of hearing impaired people. Glassense is thought to complement - not to substitute - existing hearing aids.

Image titleA prototype of the Glassense device, which hosts two microphone arrays and a microcamera

Image title

Testing Glassense in an anechoic chamberImage title

The microboard hosting the beamforming algorithmImage title

Performance of the beamforming algorithm at 1 kHz. The polar diagram indicates the direction where sound is attenuated.Image titleGlassense complements hearing aids, by sending the enhanced audio content wirelessly to existing prosthesis and to a smartphone and




*Brayda L., Traverso F, *Giuliani L., *Diotalevi F., Repetto S., Sansalone S., *Trucco A. and *Sandini G.
Spatially selective binaural hearing aids. BodySenseUX
Workshop on Full-Body and Multisensory Experience, ACM UBICOMP 2015, Osaka, Japan, September 7-11, 2015

top

INFORMATION NOTICE ON COOKIES

IIT's website uses the following types of cookies: browsing/session, analytics, functional and third party cookies. Users can choose whether or not to accept the use of cookies and access the website.
By clicking on further information, the full information notice on the types of cookies used will be displayed and you will be able to choose whether or not to accept cookies whilst browsing on the website.

Try our new site and tell us what you think
Take me there