GestureSketch: making it easier to start drawing

I kept coming back to one thing about art therapy: everyone I talked to believed it could help them, and almost no one actually did it.

So I ran 7 interviews and a short survey to find out where the gap was. Most people said the same thing: they didn't know how to start. They'd sit down to do something creative and feel stuck before they'd made a single mark. A lot of them were anxious about being bad at it, worried the thing they made would be judged, or would just prove they couldn't draw. And the younger people I spoke to didn't really want a person in the room for this. They wanted something private, on their own time, no one watching.

People wanted support, but the moment AI did too much, the drawing stopped being theirs. If the AI smooths every line and fixes every shape, the drawing stops really being yours, and the point of art therapy is that what you make comes from you.

So I mapped it before designing anything. One axis: how much a moment should lean on AI. The other: whether it's mechanical or personal. Getting started, loosening up and the tools were things AI could help with. The choices, the reflection and what the piece is about needed to stay with the person. A few things sit in between, where AI helps but you stay in control.

“Build the parts that make starting easier, and leave the meaning alone.”

The map pointed to three ideas, each for a different moment in a session.

I built the first one. Starting was the problem almost everyone named, so that's where I thought a working prototype would be most useful.

I did try the prompt assistant, then cut it. The therapists kept saying people need to draw without being interrupted, and the AI prompts fought the quiet the drawing was there to create.

With the concept decided, I sketched five layouts, each with the tools and the AI in a different place, and showed them to 6 people one at a time.

I was looking for where they agreed, and they agreed on a lot. Colour and stroke size belong together, not on opposite sides of the screen. Save and clear belong together too, and away from everything else, so you don't hit one when you meant the other. The prompt had to stay on the same screen, because people said they'd forget it if it opened another page. The camera window should be the biggest thing. And undo came up a lot, and some people wanted it as a gesture rather than a button to hunt for.

Most people preferred Layout 5, so I built from it and added the fixes they named.

A webcam turns your hand into a brush. Pinch thumb and index to draw, tap thumb and ring to change colour, tap thumb and pinky to undo. There's no stylus and no menus to learn.

It opens on a welcome screen that shows you the three gestures before you start, so the camera never turns on cold.

When I handed in the first version it was still a work in progress. It had latency issues, and the drawing didn't always keep up with your hand. I knew that going in. It was a bug I had in mind the whole time, so after the class ended I went back to fix it.

In v1 the same pinch distance started and stopped a line, so if your fingers drifted a little mid-stroke the line could drop out. Now a pinch has to be tight to start a line but only loosely closed to keep going. Your hand can relax once you're drawing. I also stopped recording points that sit almost on top of each other, which was making curves overshoot.

The second thing was that you couldn't tell what the camera thought you were doing. v1 gave you nothing until a line appeared, or didn't. Now there's a small badge at the top that says drawing, colour or undo as it happens, so when something goes wrong you know whether it was your hand or the tracking.

The tools moved too. Colour, brush size, undo, clear and save sit in one bar along the bottom, and the colours are swatches you can see instead of one dot you had to cycle through blind.

And I added flowers, just for fun. Make a fist and they burst up from your hand. Wave and they rain down the canvas. They don't do anything useful. I'd wanted them since the start of the project, and I like adding a bit of fun to everything I make.

The welcome screen stayed, it just shows all five gestures now.

Four decisions, each traceable to something the research or the concept tests turned up.

GestureSketch is a working prototype, not a finished product, and it's the one piece of the larger concept I actually built. It's meant to make starting feel less intimidating. But it's early, and I haven't tested it in a real therapy session with a client yet.

The closest I got was showing it to Tanak Bajaj, an art therapist in London, who saw the thing I was hoping for:

“Clients simply move their hands to express what they're feeling, and I can watch their creations unfold in real time. It really bridges the gap that screens often put between therapist and client.”

Tanak Bajaj, art therapist

That's one therapist's reaction, not a study, so whether it holds up in actual therapy is still an open question.

I'd try prompts again, but built the opposite way from the ones I cut. The version I removed interrupted you mid-drawing. A better one would wait until you asked for it, next to the canvas, so it's there when you're stuck and out of the way when you're not.

I'd add gesture-triggered shapes, so a pinch could pull in a simple circle or wave to build from when a blank canvas still feels like too much. And the real next step is proper testing with therapists and their clients, to find out whether any of this actually helps where it counts.

What cutting the prompts taught me

The thing I keep coming back to is the feature I removed. I built an AI prompt assistant that pushed suggestions on you while you were drawing, and it fought the quiet the drawing was supposed to create, so I took it out. It showed me that AI can have a place here, as long as the person decides when it speaks.