Multi-Sensory Experiences and Superadditivity
“The Whole is Other than the Sum of the Parts”
-KURT KOFFKA
There is nothing like real-life experience is there? The smell of freshly cut grass, the warm sun on your face…Our brains need stimulation and we enjoy experiences more when they address more of our senses. To truly experience something, our brain requires more than just one of our senses being stimulated out of five: Sight, smell, hearing, touch, and taste. Instead of just using one, researchers and designers have found ways to manipulate many senses at once.
Our brains combine senses in non-linear ways. One of such nonlinear multi-sensory interaction is superadditivity. The idea is that senses work together to deliver a combined and unique message to the customer. For example, the PIXAR intro. The jumping visual of the lamp is accompanied by the squeaky sound.
This is a case where two of the senses (sight, and sound) accompany each other. In this concept, it is important to highlight that senses should not be considered individually as they do not offer the same effect as a multi-sensory interaction.
A brain in its’ essence is a multisensory organ. Therefore, changing even one sensory attribute (sound, sight, taste…) of an experience, will affect the overall product experience! As the Gestalt psychologist Kurt Koffka puts it “The whole is different than the sum of the parts”.
Now think about a scenario where the senses are added in an incongruent manner, this phenomenon is called subadditivity. Remember Tab Clear in the 1990s by Coca-Cola?
The product had a very short life cycle. Even though clear color and sweet taste separately fit with the soda concept, it did not work when combined. Customers reported that they expected a sour taste seeing a clear soda and found it very ‘strange’ to taste a sugary drink which they would expect to be dark brown.
In order to understand how this works, let’s have a look at the way senses contribute to our experiences…
HEARING
Hearing is a sense that works mechanically. Our brain interprets the physical movement of the electrical signals into vibrations which we experience as sound. Hearing is an essential asset to experience, as changing the sound of a product can change its feel (Ferringdon,1994). One of the restaurants owners give high importance to the combination of hearing and taste.
The tasting menu at The Fat Duck restaurant offers a dish called “ The Sound of the Sea” where food that is reminiscent of a beach (with what looks like foam, sand, seaweed) and is served with a mini iPod hidden inside a sea-shell. Diners are encouraged to eat while listening to ocean sounds.
The experience was found to be significantly higher when diners were drawn into the whole experience with multiple-senses. Customers who ate with farm sounds rated the experience significantly lower than the sea-sounds group (Spence & Piqueras-Fiszman, 2013).
This allows experiences to be enhanced by applying superadditivity and influencing not just what the audience hears but what they feel, taste, see and smell.
TASTE
Why it is that changing a product’s color often changes its taste? Several studies confirm that changing the color of a product also results in the experienced taste being different.
For example, in a VR experiment same taste was delivered in 3 different colors. The same taste was labeled to be sweeter in pink/red presentation, sourer in yellow and white, minty/ herby in green. Our senses are preconditioned to expect a sensation when there is a color. To date, one of the most important variables of taste is color. Should the color not match the taste, the result of the experience is often rated very negatively (Spence, 2015).
TOUCH
Haptic receptor touch is one of the least visited sense that is explored by designers and companies. Today’s haptic sense is generally limited to vibrations whether in games to alert an upcoming enemy or the 3D touch of our phones for secondary commands.
Our sense of touch is incredibly layered and complex. It analyzes a variety of mechanoreceptors in the skin that gather data from vibration, stretch, texture and many others. So, it has been very tough to replicate touch in digital environments, until recent years as studies became more frequent. On April 23, 2020, a team of researchers at Wexner Medical Center published their research on restoring sensation to the hand of a participant with a severe spinal cord injury using a brain-computer interface (BCI) system (Syam, Lakany, Ahmad & Conway, 2017)
It is possible to use a brain-machine interface to stimulate the brain to “believe” it is touching something, as BCI becomes more popular brands will also start using tactile elements in their experiences.
SMELL
Smell is one of our emotional senses that is often associated with memory. For example, when you smell food in a restaurant it might remind you of a favorite homecooked meal. Maybe smelling Play-Doh transports you back to the 1990s. This could explain why the loss of ability to smell has become an early Alzheimer’s disease symptom (Aqrabawi & Kim, 2018). They work together.
FeelReal (https://feelreal.com) is a VR device that also has integrated this into their system with aroma features. The device also brings other haptic features such as rain, heat, cold, and more. Imagine feeling the heat of the sun on your face while being immersed in a VR beach environment. Studies support the argument that smell is one of the most important stimulators for memories and that they remain important elements of even long term memories.
A product that is to be remembered in a long time will need to offer a unique scent.
SIGHT
Is seeing believing? Not necessarily… Our visual perception would be very lonely and very open to misconceptions if it was not supported by the other senses. This is only one of the reasons why AR and VR platforms gained popularity. They supported 3D environments but also sound, and sometimes tactical engagement with the use of haptic gloves, thus making the experience more believable and enjoyable for the user.
Previous studies have shown that 3D images have a bigger psychological effect on the audience by enhancing the sense of presence (IJsselsteijn, Ridder, & Vliegen, 2000, p. 225). Many studies have confirmed over the years that VR proves to bring engagement, the field of depth, and a combination of other senses that other platforms can’t supply.
But, the use of 3D alone to enhance visual perception doesn’t always mean higher engagement and satisfaction. This can be supported by several studies that have found that using a 3D or a 2D version of a film does not differ in self-reported satisfaction and emotional arousal (Rooney & Hennessy, 2014).
Intense and immersive visuals are powerful, but they are just one component of a truly immersive, multisensory experience.
Conclusion
—
It is clear that the key in manipulating the human senses is addressing them at the same time. It is not clear which specific senses play a key role in making products and design better, however sight, sound and taste seem to enhance perception and emotion; whereas touch and smell stay longer in the memory and prove to feel more ‘realistic’. A multi-sensory experience, in the concept of superadditivity, is key for a product or design to be successful and deliver the ultimate user experience.
