Building a touch app

This is a build-along tutorial for the Touch API. We’ll build TouchDraw: a full-screen multitouch paint app. Paint with all ten fingers on an off-white canvas, switch tools from a floating rail, pick colors from a side drawer, and two-finger pinch/rotate/scroll to zoom, spin, and pan — all with Gatecaster’s own pointer injection suppressed, so the cursor never fights you and a Liquid-Glass chrome floats over the paper.

We build it twice — once as a native SwiftUI app, once for the browser — to show one API consumed by two very different runtimes. The complete, runnable source is in the gatecaster-examples repo under games/touchdraw (native) and games/touchdraw-web (browser).

The big idea: suppress input, own the screen

Most apps let Gatecaster drive the Mac as a mouse and trackpad. TouchDraw does the opposite: it sends suppress: ["input"], which mutes Gatecaster’s synthetic pointer entirely for as long as the socket is held. No tap is ever turned into a click. That means no tap reaches a SwiftUI button — so there are no buttons. Every control you see (the tool rail, the color drawer, the Close ✕) is render only. The app receives the raw finger stream and hit-tests every control itself.

That one decision is the whole architecture, and it’s the clearest possible demonstration of consuming touch directly:

  socket (AF_UNIX, NDJSON)
        │  hello → subscribe[fingers,gestures] + suppress[input]
        ▼
  TouchClient        — connection, line framing, JSON decode, reconnect
        │  frames (fingers / gestures)
        ▼
  PaintModel         — per-finger roles, strokes, camera, hit-testing
        │  reads ↓
  Layout             — every control's rect + one hitTest (single source of truth)
        │
        ▼
  PaintView          — off-white canvas + render-only Liquid-Glass chrome

Each touch verb maps to exactly one API feature — that mapping is the design:

1. Connect & handshake

The socket lives at ~/Library/Application Support/Gatecaster/api.sock. The instant you connect, Gatecaster pushes one hello line — read it first; it also tells you which display the touchscreen is on, so you can full-screen there. Then send two commands: subscribe to the channels you want, and suppress the input you’ll handle yourself.

// On the connection becoming ready (TouchClient.swift):
send(#"{"subscribe":["fingers","gestures"]}"# + "\n")
send(#"{"suppress":["input"]}"# + "\n")
receive()   // then read newline-delimited JSON frames

// bridge.js — the browser can't open the socket, so Node does the handshake:
if (msg.type === 'hello') {
  sock.write('{"subscribe":["fingers","gestures"]}\n');
  sock.write('{"suppress":["input"]}\n');
}

2. Fingers into strokes

Subscribe to fingers and you get a frame per panel report (~120/s), each with zero or more contacts. Every contact has a stable id for the life of one touch and a phase:

{"type":"fingers","fingers":[
  {"id":3,"x":0.41,"y":0.87,"sx":791.0,"sy":945.0,"phase":"moved"}
]}

The rule that makes multitouch drawing work: key everything by id. In Brush mode a began starts a stroke, each moved extends it, ended commits it, and cancelled (palm rejection took the finger) discards it. Because each id is independent, ten fingers draw ten strokes with no extra code.

// PaintModel — strokes are stored in CANVAS space so they stay put when you
// later pan/zoom/rotate; screenToCanvas undoes the camera transform.
case .canvas where tool == .brush:
    liveStrokes[id] = Stroke(id: id, colorIndex: colorIndex, width: brushWidth,
                             points: [camera.screenToCanvas(p)])
    role[id] = .drawing
// moved:  liveStrokes[id]?.points.append(camera.screenToCanvas(p))
// ended:  if committed, !s.points.isEmpty { strokes.append(s) }   // a 1-point stroke is a tap-dab
// cancelled: drop it — that was a palm

3. Render-only chrome & per-finger roles

Because input is suppressed, the chrome can’t be tappable controls — so the model does the routing. On every began it asks one question — what did this finger land on? — and assigns the contact a role it keeps until it lifts:

let hit = layout.hitTest(p, drawerOpen: drawerOpen, menuAt: menuAt, ...)
switch hit {
case .tool(let t):   tool = t;               role[id] = .consumed   // tapped a tool
case .swatch(let i): colorIndex = i;         role[id] = .consumed   // picked a color
case .drawerHandle:  role[id] = .drawerDrag; lastPoint[id] = p       // dragging the drawer
case .canvas:        /* brush / pan / transform, per the active tool */
// …
}

A finger that started on the drawer can therefore never leak into a brush stroke — its role is fixed. This is the payoff of sx/sy being pre-mapped: hit-testing is just rectangle containment, no coordinate math.

The rectangles themselves live in one place — Layout — which both the renderer and the model read, so what you see and what you can touch can never drift apart. Its hitTest encodes a priority order (open menu → Close → drawer controls → handle → tool rail → canvas), and it’s a pure function, so it’s unit-tested with no GUI:

func hitTest(_ p: CGPoint, drawerOpen: CGFloat, menuAt: CGPoint?, ...) -> HitTarget {
    if let m = menuAt { /* a menu row, if open */ }
    if quitRect.contains(p) { return .quit }          // Close wins even under the drawer
    if drawerOpen > 0.5 { /* swatches, slider, clear */ }
    if handleRect(open: drawerOpen).contains(p) { return .drawerHandle }
    for (i, t) in Tool.allCases.enumerated() where toolRect(i).contains(p) { return .tool(t) }
    return .canvas                                     // empty paper → the active tool
}

4. Gestures → camera (anchored to the fingers)

Multi-finger motion arrives on the gestures channel after Gatecaster’s intent latch has decided what it is — you don’t re-derive pan/zoom from raw finger deltas, the driver already did. TouchDraw only listens in Transform mode, so two fingers never disturb the canvas while you’re painting.

The subtle part is where a pinch zooms. Naïvely scaling the camera zooms about the canvas origin, so the image lunges toward the top-left corner. To keep the content under the fingers, anchor the zoom at the focal point f (the centroid of the transforming fingers). The renderer is screen = offset + zoom · point, so to pin f while zoom scales by k, the outer translation must move to offset' = f − k·(f − offset):

case "pinch":
    let f = transformFocus                                    // centroid of transform fingers
    let newZoom = min(8, max(0.2, camera.zoom * (1 + (g.value ?? 0))))
    let k = newZoom / camera.zoom
    camera.offset.width  = f.x - k * (f.x - camera.offset.width)
    camera.offset.height = f.y - k * (f.y - camera.offset.height)
    camera.zoom = newZoom
case "rotate":
    let f = transformFocus                                    // spin the anchor vector (f − offset)
    let dθ = CGFloat((g.value ?? 0) * .pi / 180)
    let v = CGVector(dx: f.x - camera.offset.width, dy: f.y - camera.offset.height)
    camera.offset.width  = f.x - (cos(dθ) * v.dx - sin(dθ) * v.dy)
    camera.offset.height = f.y - (sin(dθ) * v.dx + cos(dθ) * v.dy)
    camera.rotation += dθ
case "scroll":
    camera.offset.width += g.dx ?? 0; camera.offset.height += g.dy ?? 0

5. Long-press → context menu (the “right-click”)

There’s no right mouse button on a touchscreen, so TouchDraw demonstrates a gesture for it: a Brush finger held still (over 0.5 s, moved under 12 px, few points) becomes a context menu. A perfectly still finger emits no moved frames, so you can’t wait for motion — schedule a check and guard it against the contact id being reused by a later touch:

DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) { [weak self] in
    guard self.pressGen[id] == gen, self.role[id] == .drawing,
          self.pressStart[id] != nil, (self.liveStrokes[id]?.points.count ?? 0) <= 4 else { return }
    self.liveStrokes[id] = nil       // discard the dab it would've drawn
    self.menuAt = p                  // open the menu here
}

6. On the web

A browser can’t open an AF_UNIX socket, so the web build adds a tiny bridge. It’s dependency-free Node (net + http, no npm install): it owns the Gatecaster connection, does the subscribe + suppress itself, and relays each frame to the page over Server-Sent Events — one-directional, which is all the touch stream needs.

Gatecaster ──AF_UNIX socket──▶ bridge.js ──SSE──▶ browser <canvas>

// in the page — consume the same frames the native app sees
const es = new EventSource('/events');
es.onmessage = (e) => {
  const m = JSON.parse(e.data);
  if (m.type === 'hello')         setScreen(m.screen);
  else if (m.type === 'fingers')  applyFingers(m.fingers);
  else if (m.type === 'gesture')  applyGesture(m);
};

Because the suppress lease lives with the bridge’s socket, killing the bridge (Ctrl-C) instantly returns touch to the system — the same safety guarantee the native app gets for free. The page is a port of the native model — the same per-finger roles, the same Layout hit-test, the same focal-anchored camera — so the only real difference between the two builds is the renderer (SwiftUI Canvas vs. browser <canvas>).

7. Build & run it

Clone the samples repo, then pick a runtime. Both need Gatecaster running and the launching terminal granted Accessibility + Input Monitoring.

git clone https://github.com/TheAleSch/gatecaster-examples
cd gatecaster-examples

cd games/touchdraw
swift run            # full-screen paint window on the touchscreen display
swift test           # headless: Layout hit-testing + Camera round-trip
swift build -c release   # optimized binary at .build/release/TouchDraw

cd games/touchdraw-web
node bridge.js       # then open http://localhost:8787

Go further

The three layers — client, model, renderer — are a template, not just a paint app. Add an eraser or fill tool, persist the canvas, key each finger’s color to its id for a heat-map effect, or build something else entirely on the same spine: a music controller, a multitouch visualizer, a kiosk. The Touch API reference has the full protocol; the gatecaster-examples repo has every line of TouchDraw — native  and web  — to copy from.