VR Therapy: Multisensory Experience
Introduction
Autism spectrum disorder (ASD) is “a group of complex neurodevelopment disorders characterized by repetitive and characteristic patterns of behavior and difficulties with social communication and interaction” (National Institute of Neurological Disorders and Strokes, 2015). Many of the current therapies aim to improve the social abilities of the individual. However, in the interaction with autism therapists, our client, Ashley Carrick, found that the current therapies are staff-intensive and repetitive. Each individual has a customized set of therapies that have to be conducted by specific therapists, Since there is no effective and efficient way of conducting a generalized version of autism therapy. As such, she wondered if the immersive VR environment can help share the workload of therapists, and improve the social skills of individuals with ASD. Ashley then founded TherapyOS, a company that aims to use virtual reality (VR) technology to help autism therapy. As we progress, we found that the idea of mocking up a social situation in the virtual world has limited benefits on social skills in real life. Based on expert interviews and user observation sessions with different groups, including Nishan (7 years old), children at Park School (16 – 21 years old) and Ethan (19 years old), we redefined our project. We aim to create a multisensory VR experience that is enjoyable and engaging for our user, as a first step to create a VR therapy application for people with autism.
We learned from our client and user that our design must satisfy the following requirements (learn more by clicking the Requirements button on the right) :​
-
Must be engaging and enjoyable for Ethan Ducayet, our user
-
Should incorporate creativity and freedom for the user
-
Should be intuitive and easy to use
-
Should be a multisensory experience, pairing and overlapping sensory stimulation
-
Should be highly visually stimulating and exciting
-
Should also incorporate audio and kinesthetic stimulation
-
Should allow others to observe user’s experience in the VR environment
​Based on the above requirements, we created a multisensory VR environment in Tilt Brush, a VR drawing application. The environment includes three subspaces, namely the drawing room, the interaction room and the dancing room. Each room has different elements that either stimulate or calm different senses, so as to encourage sensory integration.
​
​
​
​
​
Requirements
User & Requirements
Our user is Ethan Ducayet. He is a 19 year old recent high school graduate and local Evanston resident on the autism spectrum. Our client Ethan communicates well through speech, which is not true for all individuals on the autism spectrum. Additionally, Ethan has interest in and experience with technology. He has his own laptop, took computer science classes in high school, and had used VR headsets before this project. For these reasons, Ethan was a helpful user and was crucial to the success of our project. For more information on how we found a user, click the button on the right and go to Identifying the User section.
User
Our team outlined the following requirements for the VR experience we designed:
-
VR experience should incorporate creativity and freedom for the user
-
Rationale: Through user testing, our user told us that he preferred the VR experiences which were less structured and allowed him more freedom and gave him more control
-
-
VR experience should incorporate VR hand controllers
-
Rationale: Through user testing, we observed our user was more engaged in VR experiences that incorporated the use of controllers as opposed to just the visuals.
-
-
VR application should allow user to walk around in virtual experience
-
Rationale: Through user testing, we observed our user preferred the VR experiences that allowed him to walk around.
-
-
VR Experience should include social interaction.
-
Rationale: The ultimate goal for TherapyOS is to make a VR application that would be useful for people with autism in some way. Therefore, while making the VR experience engaging and enjoyable, we need to make it socially relevant for our user so that the insights we gain are useful for TherapyOS’s final goal.
-
-
VR application should be intuitive
-
Rationale: an intuitive application creates a more engaging user experience.
-
For more information on the requirements and their rationale, click the button on the right or go to Project Definition page.
Conclusion
Our design meets the key requirements we defined for our project (refer to Project Definition page for more information on the requirements). The VR experience was engaging and enjoyable by incorporating and emphasising components our user engrossed and reveled in during user testing. By building our prototype using Tilt Brush, the user could express his creativity freely, and these actions can be viewed in real time on the computer monitor. Another benefit of Tilt Brush is that it is relatively easy and intuitive to learn. The ease of use of our prototype was also enhanced by basing our VR renditions on real, familiar objects. Furthermore, our VR application engaged our user through visual, aural, tactile and kinesthetic means. By meeting these requirements, our design can be used as an initial prototype to test and explore how VR can be incorporated into autism therapy
In our final user testing we tried a two-person drawing game by alternating the use of headsets. This was troublesome and distracting for the users and hindered the progress and enjoyment of the game. Due to technological constraints, our current design is focused on a single user’s exploration and interaction with objects. While this good for sensory integration purposes, the potential benefits of social interaction in VR have yet to be explored. Using multiplayer VR applications would have allowed us to gain more insight into the benefits and challenges of incorporating social interactions in VR, and determine ways to make our VR application more engaging and enjoyable from a social standpoint. The sensory room we designed provides a good platform for collaborative social experiences. Such collaboration could be achieved by incorporating multiplayer gameplay in the same physical space.
Increased realism
During user testing Ethan pointed out in multiple instances that one should not be able to walk through walls (and other items) in VR. While there was one instance where Ethan found it intriguing to be able to look through VR objects to see their inner components, his overallfeedback suggests a preference for more realism in VR. This can be achieved with mixed reality, where there are real physical objects (which are also represented in VR) in the physical space the user experiences VR.
Guided interactions
In the “interaction room” of our final design, we have included an axe which the user can pick up and swing to chop wood. For a neurotypical user, actuating this motion in VR is most likely notdifficult because they are able to mimic the chopping action even without prior experience. However, for users with sensory integration difficulties, this action could be more challenging. A possible solution to making this and other kinesthetic actions more intuitive for users is to have the interactions guided by arrows or lines which the users can trace, and haptic feedback thatlead the user to perform the correct movement. Visual and tactile guides could also be used to demonstrate complex VR interactions more intuitively. These guides will be necessary for future developments intending to add educational and therapeutic value to our prototype.
Timer
From our background research, expert interviews, and user observations, we learned that individuals on the autism spectrum have strong preference for visual schedules. The inclusion of a timer was also suggested by our user. Adding a timer into the VR space allows users to have better awareness of their time spent in the activity, and can be used to limit prolonged usage to reduce potential side effects of VR.
Progressive levels of stimulation
As autism is a spectrum disorder, different individuals have different sensory needs and difficulties. Some may be much more sensitive to light, while others could be much less sensitive to sound. A future development would be to make possible the customization of levels of stimulation in each sense. For example, a user could lower brightness levels and increase haptic feedback if they are hypersensitive to light and hyposensitive to touch. How such progression should be scaled needs to be informed by extensive user testing across a broad demographic. More customizable interactions.
Future prototypes should make possible more interactions and effects. For instance, when the user drops an object, it could have an associated movement and sound it bounces on the ground. Other interactions could be less realistic but more stimulating. The tapping of a drum in VR could be accompanied by music and visuals that appear based on the rhythm and force applied by the user. These interactions should be personalized to suit the taste and sensitivity of the Individual.
Local multiplayer experience
Limitations and Future Development
Tactile sense integration
The sense of touch was not fully integrated in the most natural and intuitive way. For instance, the moving of a virtual object in our application is actuated by pressing a side button on a controller. Thus, the tactile sense was limited by the shape of the controller, and the user did not have a feel of the texture, shape, and weight of the object. Furthermore, Tilt Brush did not allow us to incorporate real objects into VR sketch. A more complete integration of touch would greatly enhance the immersion and realism of the VR experience. It would also allow for a greater variety of manipulative interactions in VR which can make the application much more engaging.
A possible method of improving the sense of touch in our design is to add trackers to real physical objects so that their presence is displayed in VR in real time. This way users have at least a real sense of touch with some components. Tactile integration with virtual objects can be achieved by
improvements to the controller. Instead of the current handheld VIVE controller, future products could use gloves that provide haptic feedback that mimics the sense of texture and resistance. The haptic feedback could be further customised based on the type of virtual object, the user’s finger movements and the force applied by the user. We also anticipate that improvements in mixed
reality and VR technology will provide novel ways of enhancing the sense of Touch.
Moreover, augmenting the tactile sense in VR is especially pertinent to future developments that intend to add educational or therapeutic value to our prototype. The focus of our project is to engage multiple senses, but for the VR experience to aid the user in improving their sensory integration skills, the sense of touch needs to be much more realistic. We draw particular attention to touch here because the visual and auditory senses are readily integrated in current VR.
Testing limitations
In building our final design, we combined components that our user liked in our previous mock-ups. While we are confident the features we included are engaging and enjoyable for our user, these were only determined by our user’s surface level preferences of what he likes. We have yet to determine from a clinical standpoint with the aid of Ethan’s therapists and teachers, what are (if any) the sensory needs and challenges Ethan faces and tailor or VR environment to them. Determining these needs and challenges would require a much longer period of study and multiple versions of VR environments to probe different aspects of Ethan’s sensory integration capabilities.
There are limitations to our prototype due to technological and testing reasons. Suggestions to resolve these limitations, and potential ways to further improve the prototype are explicated below.
Market feasibility limitations
Since we have defined our project to be the case study of a single user, further research and testing with more users of a wider demographic of the autism spectrum are necessary for developing our prototype into a commercially viable product. More market research needs to be conducted to validate the demand for VR sensory experiences akin to our prototype.
Video D1. Doodle room
Additionally, this room encourages freedom and creativity. We learned from our expert interview with Michele Ricamato that most people on the autism spectrum find art and other creative activities engaging. There are no restrictions to what Ethan draws in this room, emphasized by the message “Draw anything you want” (see Figure D1). Another message reminds Ethan that he can use different brushes and colours (see Figure D2). By giving Ethan this freedom, we encourage him to express himself through what he draws.
As most people on the autism spectrum get overwhelmed by an overload of new sensory information, the emptiness of the room can help Ethan gradually acclimatize to the sensory experience. This would also help Ethan develop control over the virtual environment. We learned from our expert interview that Ethan would feel safe by gaining control over the environment. Having control over the environment also makes the experience easier to understand.
​
Decoration room
​
The second room incorporates all elements of the first, and gives a structured activity of decorating a chair. A 3D chair sits in the middle of an empty room (see Figure D3). Ethan is instructed to “Decorate the Chair!” (see Figure D4). The purpose is to impose some constraints upon Ethan’s drawing, and give it more direction. We noticed in user testing that Ethan quickly got bored of an empty Tilt Brush environment, because there was no direction regarding what he could do. It also a common idea that too much choice can create anxiety, which psychologist Barry Schwartz explores in depth in “The Paradox of Choice- Why More is Less” (Schwartz, 2005). The exercise of decorating the chair is meant to be a stress-free environment of decorating the chair. [1] This movement can help Ethan understand the physical form of a chair, which could be extended to other objects in future developments.
​
Additionally, the motion of drawing in 3D space is a fine motor movement that stimulates proprioception. Proprioception refers to components of muscles, joints, and tendons that provide a person with a subconscious awareness of body position. It also allows us to manipulate objects using fine motor movements, such as writing with a pencil. While drawing around the chair in different orientations, the user must coordinate his hands, body posture and legs (see Video D2). Furthermore, the user cannot see his body while drawing (due to the virtual reality headset), which further necessitates a subconscious awareness of body position. In this way, the action of drawing stimulates the proprioceptive system and makes our design relevant to sensory integration therapy.
Video D2. Trace chair
Drawing Room
​
In the third room in the drawing environment, Ethan must draw a chair from imagination or copy one of the 2D images of chairs in the room (see Figure D5). The images serve as inspiration for the different possibilities of chairs that one can draw. Additionally, the text “Draw a funky chair” inspires Ethan to be creative while drawing the chair. The purpose of this room is to increase the level of abstraction of drawing a chair. We added this room because we noticed through user observation that he was quite good at drawing a car from imagination when we asked him to do so, and wanted to explore such imaginative drawing with him (see Video D3).
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
Figure D5. The drawing environment
Video D3. Draw chair
Through the three rooms, we tried to create a schedule of a warm up, followed by outlining the form of a chair and then drawing the chair by oneself. We created this schedule because Ethan’s mother informed us that Ethan likes have a fixed schedule. This is also a very common trait of people with autism. Additionally, we will have a timer to regulate the amount of time Ethan spends in each room, so he does not spend too long in the VR (which could have negative consequences), but also so that he has more direction and guidance while interacting with each room.
Figure D3. Empty decoration room
Figure D4. Instruction to decorate the chair
Video D5. Catch fish
Video D6. Axe
In the videos above, the user can catch fish from the river and perform the action of cooking them in the fire. We added this particular action because Ethan enjoys performing unrealistic actions in VR, as we learned from user testing when he mentioned that he enjoyed being able to walk inside the television and see its internal circuitry, which is an action that cannot be performed in real life. This is why we added the unrealistic action of catching a fish. Furthermore, the action of plucking a string to invoke music is meant to combine movement with auditory stimulation, to increase the level of sensory integration. However, our current design does not support such a feature, and thus will require future developments.
Dancing environment
The third environment of our design is the dancing environment, which incorporates music, vividly bright and colorful walls (and floor and ceiling), beach balls and stars (see Figure D8). The bright lights on the walls vibrate to the beat of the music. The beach balls, stars and other virtual objects can be controlled and moved. The purpose of the room is to encourage creativity while dancing, create a safe space to explore different kind of dance movements, and hyperstimulate the visual system using bright lights.
Figure D8. The dancing environment
The design incorporates music played through headphones for several reasons. Firstly, music is a creative stimulus, which we learned is a crucial element for autism therapy and engaging people on the spectrum. Additionally, we learned from interview with Ethan’s mother Joan that he is particularly fond of music. This was further verified in user testing when he rated the experience of just listening to his favorite song “Mr. Brightside” five out of five.He gave all other VR experiences a lesser score. Therefore, music is essential for engagement, enjoyability and creativity.
Another essential component of this environment is the bright lights and colors. The colorful electricity animation on the walls of the environment help to stimulate the visual system of our user, making this environment relevant to sensory integration therapy. Such therapy is generally used for people who are hyposensitive to visual stimulus, especially in sensory rooms, in order to successfully stimulate their visual system. In this environment, the electric animation vibrates to the beat of the music playing through the headphones, making the environment engaging and immersive.
Lastly, this environment encourages Ethan to move his hands and legs freely and creatively in response to the music. Such movement is also relevant to sensory therapy because it involves coordination of different parts of the body. Furthermore, Ethan can move the beach balls, stars and other objects in the environment, making the environment more interactive and engaging (see Video D7).
Figure D1. Emphasize on creativity in drawing room
Figure D2. Reminder to use different brushes
Design concept & rationale
Our design is an enjoyable and engaging virtual reality (VR) experience that stimulates the different senses of our user. The design concept of is based on a “sensory room”, which is a personalized therapeutic physical space that stimulates the visual, auditory, tactile, vestibular and proprioceptive senses. Our prototype was created using the Tilt Brush application in the HTC VIVE VR system, and makes use of the headset, two controllers and headphones (see Image). The design is a sketch within the Tilt Brush application that contains three primary environments: the drawing environment, the interaction environment and the dancing environment. The user can walk around in the virtual space and interact with the environment using the controllers, either to draw objects, move objects or use the other features of Tilt Brush.
This page has two sections: (i) Environments (ii) Software. The first section describes the environments we created in the Tilt Brush software, and the second section briefly describes the software we used and our rationale for selecting the software.
Environments
Each element of our design serves a different purpose in making the design enjoyable, engaging and relevant to sensory therapy. These elements are personalised for our user. Some of our design elements are things that Ethan enjoyed, which we learned from observation and survey. A major theme is creativity, as we realized that Ethan preferred the VR experiences that were less structured and free. Our expert interviews corroborated this finding. Other elements are techniques used in sensory integration therapy, or experiences that stimulate the senses commonly targeted in therapy. This makes our design engaging and relevant for sensory integration therapy.
From interviewing therapists at Soaring Eagle Center for Autism, we learned that a common technique used in sensory integration therapy is to begin with a calming atmosphere and gradually increase the level of stimulation. For this reason, the level of visual, auditory and kinesthetic stimulation is low in the first environment (drawing) increases in the second environment (interaction), and then skyrockets in the third environment (dancing environment), which has extremely bright visuals and dance music. This progression helps Ethan gradually adjust.
The three environments (along with their rationale) are described below.
Drawing environment
The drawing environment contains three rooms. Each room explores a different drawing experience. The rationale for creating this environment is to encourage creativity, calmness and fine motor movement.
Doodle Room
​
The first room of the drawing environment is a fairly empty environment in which Ethan can doodle and draw different lines and forms without restrictions. The purpose is to allow Ethan to physically warm up to the new experience of drawing in 3D space. After a certain amount of time in this room, we hope that Ethan will develop proprioceptive understanding of his hands, legs and the rest of his body while drawing (see Video D1). We observed in user testing that Ethan was able to comfortably draw in the VR environment after 2 to 3 minutes.
Interaction environment
The second environment of our design is the interaction environment, which contains only one room. The room has a fantastical (and natural) atmosphere that contains a river with fish, a fire, a tree with hanging fruit, some logs, an axe, a jug and a set of musical strings (see Figure D6 and D7). The user can move the virtual objects using the controller. The user listens to sounds of flowing water, calming music and chirping birds (through the headphones). The purpose of this room is to make the sensory experience enjoyable, calming and relevant to therapy.
Figure D6. Interaction environment view 1
Figure D7. Interaction environment view 2
The interaction environment is enjoyable for Ethan because it contains elements that he prefers. Firstly, he enjoys the fantastical environment of nature more than a normal bedroom (Appendix). We learned that he preferred to play Tilt Brush over Job Simulator, a game that involves a more realistic environment of a room. Furthermore, he expressed preference in user testing for the Tilt Brush sketches that were more surreal or other-wordly, such as those that involved fire, water and dragons rather daily objects, such as beds and tables. This is why we added a river, fire and a tree in the interaction environment and attempted to create a surreal atmosphere.
The sounds of flowing water, calming music and chirping birds played through the headphones help to make the user feel immersed in this natural environment. Furthermore, they help create a calming atmosphere. There are two types of sensory integration difficulties that people with autism have: hyposensitivity and hypersensitivity. Hypersensitivity is when a person feels overstimulated by a particular sense, whereas hyposensitivity is the exact opposite. Hypersensitive people respond when to calming stimuli such as soft and soothing music, which is explained on the website “www.sensory-processing-disorder.com”. For this reason, we hope that the sounds will help Ethan feel calm. Studying Ethan’s reactions to these sounds can also help therapists understand Ethan’s sensory needs better. auditory stimulation is also an integral aspect of sensory integration therapy that can make the experience more engaging for Ethan.
Furthermore, the environment is relevant to therapy because it involves moving different virtual objects using the controller, such as plucking fruits from the tree (see Video D4). People with autism often have difficulty in picking up objects and performing other motions that require coordination of different body parts. This difficulty is related to proprioception that was discussed earlier. For instance, our professor Emma DeCosta had to teach her son with autism how to pick up a cup by holding his hand and controlling his motion. By directing the user with text such as “Pick up axe and chop wood”, we hope to offer similar guidance as our professor did with her son while Ethan uses the axe in the environment (see Video D4, D5 and D6). Additionally, we noticed while observing students with autism at Park School that they used their hands to point to specific lines while reading, or to get the attention of their teacher. We also noticed in our observation with Nishan DeCosta that he enjoyed the iPad more than the cardboard VR headset because the iPad allowed him to use his hands to interact with the virtual screen, whereas the cardboard headset only offered visual interaction. For this reason, we thought it was essential for Ethan to be able to pick up objects and move them around because it is the closest approximation to tactile interaction that can be attained in virtual reality. The following videos show some of the interactions that the user can perform in the interaction environment:
Video D4. Pluck fruit
Video D7. Dance with music
Reference
Schwartz, Barry. (2005). Barry Schwartz on the paradox of choice, TED Ideas Worth Spreading. Retrieved at April 21, 2008, from the website temoa : Open Educational Resources (OER) Portal at http://www.temoa.info/node/244
Hardware and software
The HTC Vive is a modern virtual reality system that provides an immersive experience for the user. It contains a headset and two controllers, one for each hand. The headset uses technology called “room scale technology” to turn a room into 3D space via sensors attached to the four edges of the room, with the virtual world allowing the user to navigate naturally by walking around the room. The two controllers use motion tracking, allowing the user to vividly manipulate objects, interact with precision, communicate and experience immersive environments These are some of the reasons why we decided to use the HTC Vive:
​
-
Ethan enjoyed the HTC Vive experience and had no complaints regarding it except that one could walk through walls (and other objects) in the virtual environment (See User Testing). Since no current virtual reality technology can prevent this, we did not see this exception as a problem with HTC Vive but rather a problem with VR that could be included in future development (See Future Development). He also preferred this experience far more than the alternative mockup we showed him that was created using InstaVR, a VR-app making software.
-
The HTC Vive system facilitated greater interaction and engagement than our alternative mockups created using InstaVR.
-
The HTC Vive was easily accessible to us through The Garage, a centre at Northwestern. This access would help us conduct user observations and testing, and build mockups. Additionally, the computer connected to the HTC Vive displays what the user is viewing in virtual reality, which helps us get real-time feedback on how the user responds to different elements in VR.
HTC VIVE
Tilt Brush
The HTC Vive also contains Tilt Brush, which was fundamental to the creation of our design. Tilt Brush is an application that lets you paint in 3D space with virtual reality. Tilt Brush allows the user to draw and paint with a myriad of different colours and brushes. It also allows the user to add 3D models and images into the 3D space. The user can move the 3D models using the controllers. It also contains a variety of sophisticated features such as teleportation, guides and models that are explained in the Appendix XX: Tilt Brush Instruction Guide. These are some of the reasons we used Tilt Brush:
​
-
Compared to the other application “Job Simulator” we tested with Ethan, Ethan spent more time in the Tilt Brush environment and rated it higher in a survey than Job Simulator (see User Testing page). We concluded that he preferred the Tilt Brush environment.
-
The drawing feature of Tilt Brush increased creativity and engagement for the user.
-
The other features of Tilt Brush (such as teleportation) increased interactivity for the user, which can be found in Tilt Brush Instruction Guide. They also helped us create our design in the 3D space with ease.
