Virtual instrument control
Turn musical instructions into coordinated native controls, matched auditions and exact undo.
Keep this sound, but let the notes linger. Give the attack more time. Bring that last change halfway back.
These requests concern a sound already in progress. A musician wants to preserve what works while exploring what might work better. The useful model needs an instrument, a current patch and a way to make a controlled change audible.
The experience
The demonstration uses OB-Xd, a native virtual instrument, with a control surface for proposing edits, comparing audio and accepting a new patch. Chosen controls can be protected while other parts of the sound change. A subsequent instruction starts from the accepted state; undo restores the saved native state.
The interactive showcase offers recorded, tested examples and their actual rendered audio. Choosing one replays that example without a new model request. Any displayed timings belong to the captured run. The native sound and parameter changes remain inspectable together.
Listen at the same point in the phrase when comparing versions. A longer release matters after the note ends; a slower attack matters at its beginning. The comparison itself is part of the tool. Changing the listening position at the same time as the patch makes it harder to judge what the edit did.
The model and the tool
Our sound-control workbench operates on real OB-Xd and Dexed state. It keeps the current controls, the proposed edit and the saved result distinct. The model interprets the request against the available operations. The tool checks which controls may change, applies the plan and renders the instrument.
instruction + current patch + protected controls
↓
proposed edit plan
↓
checked native state → audio audition
↓
accept, refine or restore
This is model-directed instrument control. The plug-in produces the sound. The model’s decision changes the parameters through which that sound is produced.
There are two clocks in this arrangement. Audio processing must keep its own schedule; an interpretive request can take longer and return between edits. The VST3 parameter-change interface illustrates this separation in another plug-in version: the host transmits parameter changes to the processing block. Our OB-Xd demonstration uses VST2, but the design obligation is the same—an interpretation request must not become a dependency inside the audio callback.
Undo also needs a precise meaning. If a control was clipped at its maximum, applying the opposite movement would not necessarily recover its old value. Saving and restoring the previous state gives an exact operation where an approximate inverse would not. “Halfway back” is a new edit; “undo” is restoration. They deserve different actions and different tests.
A useful research connection
Wekinator, described by Fiebrink, Trueman and Cook in 2009, lets instrument designers supply examples, train mappings, play them and revise those examples in one environment. The authors describe a performance in which musicians developed their mappings while making music. Evaluation could include whether a mapping was interesting and controllable, beyond reproducing an initially imagined answer.
SynthAssist approaches a related problem through vocal imitation and feedback. A person supplies a sound, hears candidates and indicates which are closer. The system refines its search and returns synthesizer parameters. It gives the person material to react to when the desired sound is difficult to specify completely in advance.
Together these projects suggest a productive role for learned control: keep the musician in an encounter with the sound. The next useful instruction may be discovered by hearing the previous result. A more elaborate prompt box would not supply that encounter on its own.
Building it with System1
The deliverable combines the decision behavior with the tool that makes it useful. For an instrument, that means defining the editable state, permitted plans, protected controls, audition and restoration alongside the examples used to develop the model.
A meaningful test follows the whole sequence: request a compound envelope change, verify that protected filter controls stayed fixed, audition the result, accept it, make a relative follow-up and restore the prior state. Parameter preservation, interpretation quality and musical usefulness are separate observations. Each can improve without being mistaken for the others.
This is why model and tooling belong in the same project. A model can select a plausible plan that the host applies incorrectly. A host can preserve every byte while the chosen edit misses the musician’s intention. The development process needs to examine both.
Where this could lead
Imagine a composer preparing a theme for three scenes. The melody stays fixed, while the instrument needs three related behaviors: a close, dry version; a slower, unsettled entrance; a spacious ending. A tool could propose bounded variations from the same patch, preserve the controls that define its identity and make the differences available for immediate comparison. The composer would direct a family of sounds rather than begin three unrelated preset searches.
A performer could go further and commission a small vocabulary of gestures or phrases around one instrument. “Open up” might change a permitted combination of filter and envelope controls while leaving tuning and level protected. That vocabulary would become part of the instrument’s design, developed through examples and listening.
The same architecture reaches beyond sound. An animator could preserve a character’s path while asking for a more hesitant movement. A lighting designer could vary the transition into a scene while protecting the positions that keep a performer visible. In each case, a learned interpretation operates inside a tool that knows the state, the allowed changes and how to return.
The opportunity is to make the expressive behavior of a tool something its users can commission and refine—while continuing to work with the material they care about.