2024 · Designer & Developer
GestureSketch
A webcam-driven mid-air drawing tool for expressive therapy
Role
Designer & Developer
Team
Solo
Research Methods
Prototype Testing, Expert Review
Platform
Browser-based (Webcam)
Duration
6 weeks
Overview
Conventional drawing apps require styluses, menus, and technical confidence. For therapy clients seeking expressive creative outlets, this friction kills the flow before it starts. GestureSketch strips the interface down to a single camera and a set of natural hand gestures.
My Role
Full-stack designer and developer — research, interaction design, prototyping, and implementation.
Tools & Methods
- MediaPipe (hand landmark detection)
- p5.js (real-time stroke rendering)
- Figma
- SVG export
Objectives
Remove the Learning Curve
Let someone start drawing with zero onboarding — no styluses, no menus, no tutorial needed.
Keep the Interface Invisible
Nothing on screen should interrupt creative flow once someone starts.
Validate with Real Practitioners
Test the concept with someone who'd actually use it in a clinical setting, not just other designers.
Interaction Model
I designed a three-gesture vocabulary that maps to natural hand movements: index finger + thumb pinch to draw, thumb + ring finger to cycle colors, thumb + pinky to undo. Button backups for undo and clear sit at screen edges as a confidence-building safety net for new users.
Therapist Validation
“An art therapist reviewed the prototype and praised its ability to bridge the digital/human connection in online therapy sessions. The naturalness of mid-air gestures made the tool feel less like software and more like an expressive medium.”
Technical Architecture
MediaPipe hand-landmark model running in-browser at 60fps
p5.js canvas for real-time stroke rendering
Gesture buffer system with configurable dead-zones to prevent false triggers
Per-stroke metadata storage (color, size, timestamp)
By the Numbers
Design Solutions
Insight 1
Natural hand movement isn't pixel-precise — a raw gesture-to-draw mapping picked up every small jitter as an unwanted mark.
Recommendation 1
Dead-zones around each gesture trigger absorb jitter from natural hand movement without dulling responsiveness.
Insight 2
New users don't trust a gesture interface at first — they need to know undo will actually work before they'll draw freely.
Recommendation 2
Button backups for undo and clear sit at the screen edges as a confidence-building safety net alongside the gesture vocabulary.
Insight 3
Any UI chrome — toolbars, menus, settings — broke the sense of just drawing in the air.
Recommendation 3
Minimal UI. No toolbars, no menus, nothing that interrupts creative flow.
Insight 4
Early LLM-prompt features weren't reliable enough to ship without undermining trust in the core drawing experience.
Recommendation 4
Deferred LLM prompts to v2 and shipped a clean, fast drawing core first.
Outcomes
What Worked
An art therapist reviewed the prototype and praised its ability to bridge the digital/human connection in online therapy sessions.
The naturalness of mid-air gestures made the tool feel less like software and more like an expressive medium.
Areas for Growth
LLM-assisted prompts were cut from v1 after tuning issues — still on the roadmap for v2.
Only tested with one therapist so far — needs validation with actual therapy clients, not just the practitioner.
Learnings
Constraints Make Interfaces Feel Natural
Stripping away every piece of UI chrome was scarier than adding features would have been, but it's what made the tool feel like a medium instead of software.