---
key: mechanic/spatial-interfaces--t305
type: mechanic
id: spatial-interfaces--t305
lang: en
title: Musical instruments and specialized controllers
summary: >-
  A familiar performance gesture sends commands to an app. AudioSwift’s demo
  uses musical keys to trigger notes and a trackpad to shape sound.
updatedAt: '2026-10-22'
status: active
url: 'https://gamemechanics.pro/en/knowledge/mechanic/spatial-interfaces--t305/'
jsonUrl: >-
  https://gamemechanics.pro/api/v1/knowledge/en/mechanic/spatial-interfaces--t305.json
markdownUrl: >-
  https://gamemechanics.pro/content/knowledge/en/mechanic/spatial-interfaces--t305.md
aliases:
  - Musical and specialised controllers
tags:
  - Spatial and alternative interfaces
  - Mechanics and variants
sourceIds:
  - source-3e8db7ac782a4997
  - source-fd215230c062deb7
sources:
  - id: source-3e8db7ac782a4997
    title: 'W3C: Web MIDI API'
    url: 'https://www.w3.org/TR/webmidi/'
    verifiedAt: null
  - id: source-fd215230c062deb7
    title: AudioSwift
    url: 'https://audioswiftapp.com/'
    verifiedAt: null
relations: []
placements:
  - catalogId: spatial-interfaces
    id: t305
    title: Spatial and alternative interfaces
    url: 'https://gamemechanics.pro/en/mechanics/spatial-interfaces/#t305'
    categoryIds:
      - mechanics
supersededBy: []
attributes:
  preview:
    updatedAt: '2026-10-22'
    title: Musical and specialised controllers
    description: >-
      A familiar performance gesture sends commands to an app. AudioSwift’s demo
      uses musical keys to trigger notes and a trackpad to shape sound.
    image:
      src: images/mechanics/control-audioswift-3.webp
      width: 1024
      height: 576
      game: AudioSwift
      sourceUrl: 'https://audioswiftapp.com/'
      originalUrl: >-
        https://www.dropbox.com/scl/fi/fc2ycw35tusq9f0joj5s5/04-Core-Features-XY-Pads.mp4?rlkey=hc7s0jir9zb5du0z08f9xipon&st=nltffpdt&raw=1
      sourceFrameSeconds: 3
      credit: Nigel Ríos; host apps and instruments retain their respective rights
      rights: >-
        Imagery © Nigel Ríos; host apps and instruments retain their respective
        rights. Developer-distributed screenshot or frame; all rights reserved.
        Excluded from MIT and CC BY 4.0.
      caption: 'AudioSwift XY: a trackpad shapes sound during keyboard performance'
      alt: 'AudioSwift XY: a trackpad shapes sound during keyboard performance'
      markers:
        - x: 24
          'y': 55
          label: The XY point maps two motion axes to sound parameters
    tips:
      - Show how physical controls map to notes or application actions.
    examples:
      - title: AudioSwift
        url: 'https://audioswiftapp.com/'
endsAt: null
provenance: >-
  Original bilingual editorial overviews authored for gamemechanics.pro on
  2026-10-20 and reviewed on 2026-10-21. Scenarios are hypothetical design
  examples; prototype checks describe work to perform, not completed playtests.
  Primary sources support the stated techniques and implementation context, not
  every proposed variant.
license: CC-BY-4.0
codeLicense: MIT
attribution: gamemechanics.pro contributors
catalogKey: catalog/spatial-interfaces
---
# Musical instruments and specialized controllers

id: spatial-interfaces--t305
type: mechanic
key: mechanic/spatial-interfaces--t305
lang: en
status: active
url: https://gamemechanics.pro/en/knowledge/mechanic/spatial-interfaces--t305/

A familiar performance gesture sends commands to an app. AudioSwift’s demo uses musical keys to trigger notes and a trackpad to shape sound.

## At a glance

![AudioSwift XY: a trackpad shapes sound during keyboard performance](https://gamemechanics.pro/images/mechanics/control-audioswift-3.webp)

[AudioSwift](https://audioswiftapp.com/) · Imagery © Nigel Ríos; host apps and instruments retain their respective rights. Developer-distributed screenshot or frame; all rights reserved. Excluded from MIT and CC BY 4.0.

[Source recording](https://www.dropbox.com/scl/fi/fc2ycw35tusq9f0joj5s5/04-Core-Features-XY-Pads.mp4?rlkey=hc7s0jir9zb5du0z08f9xipon&st=nltffpdt&raw=1), 3 s

1. The XY point maps two motion axes to sound parameters

- Show how physical controls map to notes or application actions.

- [AudioSwift](https://audioswiftapp.com/)

## How it works

Specialized controllers translate familiar performance gestures into game actions through explicit event mappings.

## Variants

### Mechanic variants

<a id="t305-variants"></a>

- Notes select actions through a mapping table. <a id="t305-v1"></a>
- Event strength modulates values when supported. <a id="t305-v2"></a>
- Pedals change modes or sustain effects. <a id="t305-v3"></a>
- Calibration aligns input latency with music. <a id="t305-v4"></a>

## Design example

A music controller's keys open gates in sequence while a pedal holds an opened passage.

Prototype check: Test actual event formats, simultaneously held notes and a keyboard substitute when the controller is absent.

## Context and limits

Spatial and specialized input requires calibration, clear action confirmation and a way to continue after signal loss. Each overview separates player actions, state changes and alternative rules. Examples are hypothetical design scenarios, not claims about a named game. A prototype check states an observable question for the next iteration. Existing detailed references remain separately linked from the direction map.

## Connected materials


## Sources and provenance

Original bilingual editorial overviews authored for gamemechanics.pro on 2026-10-20 and reviewed on 2026-10-21. Scenarios are hypothetical design examples; prototype checks describe work to perform, not completed playtests. Primary sources support the stated techniques and implementation context, not every proposed variant.

- [W3C: Web MIDI API](https://www.w3.org/TR/webmidi/) (id: source-3e8db7ac782a4997)
- [AudioSwift](https://audioswiftapp.com/) (id: source-fd215230c062deb7)

CC-BY-4.0 · gamemechanics.pro contributors