XR app with AI chatbot for pediatric patients who are experiencing long-term hospitalization
About
UU is an immersive mixed reality experience designed to reduce anxiety and loneliness for pediatric patients during extended hospital stays. Developed during the 2023 PICO DevJam Hackathon, it combines spatial computing with an
LLM-powered virtual companion to create playful, emotionally supportive interactions.
Impact
2023 PICO DevJam Hackathon Best MR Design and 2nd Prize
Presented at 2024 Meaningful XR Conference at Stanford University
Tools
Figma
Adobe illustrator
C4d
Unity
PICO XR
Team
PM / Product Designer - me
PM - Siyue S
Visual Designer - Siqi C
XR Developer - Xiaoan L
Developer - Alan R
Visual Designer - Shun H
My Role
Product strategy/lead
User research
UX/UI/Visual design
Design system
Design lead
Final output showcase
UU’s introduction!
Landing page
Select feature
Choose two ways of discovery
Gesture interaction
Gesture recognition
Companion feature in the daytime: Choose a story
Companion feature in the daytime: Tell a tory
Companion feature in the daytime: Interactions in the story
Companion feature at night: Idle state UU
Companion feature at night: Voice trigger UU show up
Companion feature at night: Learn and explore the universe
Discovery feature: Exploration in the hospital !
Discovery feature: Intro to MRI !
Discovery feature: Intro to nurse station !
Cognition feature: Learn about Cell !
Companion feature: Emails from friends, and other patients
Companion feature: Read emails
Discovery feature: Wall paint
Discovery feature: Wall paint selection
Discovery feature: Wall paint color palette
Discovery feature: Choose paint brush
Discovery feature: Paint brush
Discovery feature: Drawing a master piece!
Part of Design System
background
The project was inspired by the documentary film Life Matters (2019). It depicts a dream about child patients fighting against cancer in a gamified way in a Chinese public hospital. Besides the side effects of treatments, children in long-term hospitalization suffer from unimaginable torture (fear, loneliness, loss of control, and despair). I hope that through the design, I can find a solution that could help them alleviate the pain during the treatment process.
Problem
25%
In this project, I decided to focus on leukemia because it is the most common pediatric cancer, comprising 25% of all cancer types.
85%
The 5-year relative high survival rate of leukemia is 85%, which means they have a high chance of returning to normal life after hospitalization.
According to my research, child patients with Leukemia may have a lot of problems and difficulties during long-term hospitalization in Chinese public hospital:
Long treatment:
Most patients take a total treatment about 3 years to get healed
Restrained activities:
Get treatment in an unfamiliar environment with restrictions
Psychological risk:
Long-term hospitalization may cause isolation
Social demand:
Need the company of friends to share stories and get support
Solution
We designed the virtual character and user interfaces for the XR app with design system. For the 2023 PICO DevJam hackathon, we also made a prototype implemented in VR headset using unity.
Research question 1
How do pediatric patients think and feel during hospitalization?
With this question, I conducted secondary research using methods such as interviews and the examination of real medical cases. I interviewed 8 medical staff members in Chinese public hospital who are strongly associated with pediatric patients and 2 leukemia pediatric patients’ parents. Out of ethical considerations, I refrained from direct conversations with the patients.
I conducted the interview with medical staff members and patients’ parents in Mandarin Chinese. If you read Mandarin, please feel free to check the full interview and ideation notes in my Miro board.
Research question 2
What do pediatric patients with Leukemia need during the treatment process?
I went through 10+ papers related to leukemia treatment in China and come out with the Leukemia treatment experience map. Then, I approach to the touchpoint of each treatment stage for the design ideation.
Hospitalized
Remission
Induction
33d
The fear of an unknown illness needs to be calmed
The adverse reactions brought by treatment will not only bring physical and mental pain, but also have obvious adverse
effects, which need to be pacified
More intense treatment, more intense physical pain, need to be soothed
After high intensity chemotherapy and radiotherapy, the body's immunity is almost zero, requiring close care
Extremely weak, in need of close care, high sterility requirements, high diet requirements, demanding details of life
long-time Hospitalization, need to be connected with the outside world immunity is low, need to control infection
Possible positive effects of the treatment
Get pessimistic and tired of the world, think that they are incurable or painful, lose confidence in treatment, lose enthusiasm for life, and they need to regain their confidence in life
Healthy
Early Stage
Chemotherapy
Stem Cell
Transplant
Cure
Recurrent
Newly
diagnosed
7d
PRE-
treatment
7d
hrs
2wk
months
2wk
Maintenance
2yr
Consolidation
56d
Delayed
Intensification
2wk
Touchpoint
The little patient need to switch between hospital and home environment
Getting used to the treatment environment and the treatment response, needs to be distracted
Needs to be distracted from the painful treatment
Maintenance treatment requires access to the outside world, but also requires close adherence to medical advice
Transplant
Room
Touchpoint
Emotion：stubborn, impatient, fear, alienated, anxiety, isolated
Reasons：experiencing physical discomfort caused by the side effects of the drug/ stress of the diagnosed with the disease and leaving the familiar environment
Emotion: self-esteem, inferiority, anxiety,depression, reduced social adaptability
Reasons：physiological changes the use of hormones and chemo drugs
Reasons：long-term hospitalization, isolation
Emotion: High risk of mental breakdown and trauma
Reasons：unable to have friends or peers
Takeaway
Looked into the treatment, we hope to design for every stage of medical treatment and explore potential features. Then we developed our main design features based on deep-down medical research on cognitive and behavior therapy.
3 Design principles
Expand the Environment
Use XR to transform the hospital room into an engaging and explorable space.
Support Learning Through Play
Help children understand medical procedures using interactive and age-appropriate experiences.
Reduce Emotional Burden
Design a virtual companion that provides comfort, encouragement, and a sense of companionship.
From principles to experience - product journey map
To transform research insights into a cohesive product experience, I mapped the patient's treatment journey and identified opportunities where UU could provide the greatest emotional and educational support. The journey map became the foundation of the product, helping prioritize the three core features—Companion, Discovery, and Cognition—and defining how they work together across different stages of hospitalization.
Character design
Introduce virtual character UU
What kind of friends can give young patients warm company and strong support? After analyzing the characteristics of various companion characters in animation, films, television works, and our own friends, we created UU, this cute buddy.
UU
Personalities:
UU’s story:
Willing to express but better at listening
Round Body
Light and Agile
White and Clean
Bouncy and Soft
Dim Glow
Cute and Naive
Tenderly Look
Kind and Patient
“Little mysophobia”, can't stand sticky things
Adventure and Travel
Write a diary/photography/record life
Play with friends
Collect meaningful things on the journey
Magic Tricks
Teleport
Story-telling
UU is a little explorer who likes to travel around. Have been to many places, met many friends, and wrote them in the diary. This time, UU came to the earth, looking for the bravest soul.
Here, UU and the little patient met. "Let me accompany you", said UU.
Short stay turned into long-term companionship. UU wants to grow up with the child and completes a story in their life journey.
Hobbies
From Parallel Universe
Personality
Appearance
Expression
Material
Personality Traits
Color
Round Face
Empathetic Look
Always smiling
Round Head
No edges or corners
translucent
Silicon like
fluffy
Mocha like
patience
kindness
Characteristics of the companion role
Warmer tones
luminescent
Light color
Proactive
Friendly
Gentle
Being a Good Listener
Reliable
Have similar interests
Companionship
Humorous
Round Belly
Appreciated Qualities
in Friends
Characteristics of a “Reliable Friend”
When I‘m sick
I‘d like to hear my friends saying that
Communicate Effectively
Won’t get embarrassed even when no one’s talking
Soft tone
Sincere look
The Person Who Speaks To Me First
Good Temper
Won’t force me to do things that I don't like
Emotionally
stable
Quiet not
noisy
Problem
solver
Ability to Make People
Feel at Ease
Happiness maker
Similar experiences
Similar hobbies
Positive values
considerate
empathetic
Get used to the other's presence
Never worried that they would leave
Share the
happiness
Comfort and Care
Play toys
Someone’s there with me, when I wake up
Mom’s hug
dad’s shoulder
Being cared and asked patiently
do not want to talk
Diversion
during injection
Divert Attention
“I’ll always be
there for you”
Provide
Confidence
“Not feeling alone”
Virtual Friend “U”
“U” has the same pronunciation as “友”, a Chinese character stands for “friend”
Support my
decisions
“You’re already
excellent”
Encouragement
Takeaways
Translated extensive healthcare research into a cohesive product strategy and end-to-end XR experience.
Balanced emotional design, technical feasibility, and accessibility when designing for pediatric healthcare.
Used rapid prototyping and continuous iteration to evolve an ambitious concept into an award-winning XR prototype.
Identified clear next steps for clinical validation, MVP definition, and long-term product adoption.
Photos & love
