Control a synthesizer by voice

“Give it that bowed, brass sound, like a futuristic string section.”

That is a direction a musician might give another musician. It combines an instrumental association, an imagined manner of playing and a scene. It does not identify a filter value. A useful instrument can answer with something audible, keep what the musician likes and make a correction without undoing unrelated work.

The demo couples a decision model to OB-Xd, a real native VST instrument. The musician keeps speaking and listening while the instrument plays. A model proposes a permitted change and its scope; the tool checks what that change would actually affect, then applies it or asks for clarification. The synthesizer produces the sound. The model in this study does not listen to it.

The experience

The 118-second film is one uninterrupted working session, with ten musical edits and one actual clarification:

1. Find and refine a voice. Bowed brass becomes less brassy and more bowed while its bloom and held C minor chord stay intact.
2. Resolve an ambiguity. “Let it hang in the air, something unresolved” could concern harmony or note tails. The tool keeps the current sound and asks which. “I mean the chord. Let it feel unresolved. Keep this sound” changes E♭ to F, producing a suspended-fourth chord without changing the native patch.
3. Develop the phrase. The notes climb one after another. “More purple. Darker, with a slow shimmer” changes the sound around that pattern; “half as much shimmer” reduces only its modulation depth. Each shimmer state has five seconds of listening before the next direction.
4. Keep separate kinds of progress. The notes slow to 72 BPM. “Bring back the sound from before the purple. Keep the slower notes” retrieves the earlier patch without reverting the newer tempo or arpeggio.
5. Make a small correction. “Take a little bite off the beginning. Leave the brightness” changes attack only. “A bit too soft. Halfway back, just the beginning” makes a relative correction to that accepted edit.
6. Compare, then keep listening. A visible Compare sounds click plays the starting and current native banks over the same current notes, with fixed monitor gain. Original sound, Your current sound and Back to your sound follow the actual audio. Both complete phrases and release tails remain before the live instrument returns. This is an explicit control, not a spoken comparison or save operation.

The generated voice inputs were played in full and recognized during this take. Eleven fresh model requests produced ten checked native edits and the clarification; no model responses or earlier recordings were substituted. These authored development cases establish this sequence and its state-preservation checks. They do not establish general musical understanding or whether a musician would prefer the resulting sound.

Hosted instrument tooling

The hosted instrument keeps a session’s sound, phrase and history and returns native audio. Its OpenAPI contract defines operations to submit a direction against a confirmed revision, read its outcome, and retrieve the resulting audio. The hosted sandbox and decision model form an embeddable instrument service. This prototype uses a finite gesture palette and a prebuilt fast model.

The hosted tool renders separate eight-second auditions. The continuous film above instead runs a local native instrument beside actual network model requests. One observed hosted browser request took 11.4 seconds from submission through verified, decoded audio, including cold startup and delivery; its service-ready measurement was 2.43 seconds. Those are individual hosted observations, not the continuous film’s latency.

What does “purple” control?

Here, it names a reversible proposal, supported by the rest of the request: darker, slow shimmer. We do not infer a universal mapping between a colour and a sound. The selected recipe is one authored interpretation a person can audition and reject. The explicit clues make this a considerably narrower task than interpreting the isolated word “purple.”

A useful implementation separates three questions:

• What could the person mean? In this palette, a darker cutoff, longer release and slow filter movement form one coherent candidate. A brighter glass pluck is a different candidate. The model selects among complete gestures, including a no-change clarification outcome.
• What may the candidate change? The darker gesture preserves oscillator waveforms, attack and the complete phrase. Its actual application changed seven exposed control values. All eighty controls were verified, including the other seventy-three; every native bank byte outside the declared edits was preserved.
• Is the result artistically right? That remains a listening judgment. A hash can establish restoration. It cannot establish that a sound feels purple, suspenseful or emotionally appropriate.

Those boundaries affect the interaction. If “more anticipatory” produces an interesting chord but an unwelcome texture, a person needs to keep one and reject the other. A single undifferentiated “regenerate” action would discard the useful part of the conversation.

Represent the current material as (native sound, musical phrase). A harmony gesture transforms the phrase and retains the sound. A sound gesture transforms the patch and retains the phrase. Restoration retrieves an accepted patch snapshot and combines it with the current phrase. The film contains a case where ordinary whole-session undo cannot recover the requested combination:

Point in the session	Direction	Native sound	Phrase
Before “purple”	Bowed voice, notes already climbing	A	Suspended-fourth arpeggio, 96 BPM
Darker	Slow shimmer	B	Same arpeggio, 96 BPM
Relative correction	Half the shimmer	B′	Same arpeggio, 96 BPM
Tempo change	Slower notes, same sound	B′, exact	Same arpeggio, 72 BPM
Scoped restoration	Sound before purple, keep slower notes	A, exact	Same arpeggio, 72 BPM

One whole-session undo would remove the slower tempo and retain sound B′. Going further back to sound A would also retrieve the old tempo. Sound-only restoration instead constructs A with the newer 72-BPM phrase. No earlier complete session snapshot contains that combination. The tool needs separate sound history and current musical state; a single stack of whole-session snapshots does not supply that operation by itself.

The restored native bytes match the actual pre-purple bank, while the complete phrase retains the later tempo change. Identical native hashes concern stored instrument state, not identical audio samples: oscillator phase can differ. The final original/current comparison preserves MIDI, velocity and monitor gain where sound is the variable being judged.

What this continuous recording measures

The ten accepted changes entered the browser’s native-audio output graph 1.19–1.34 seconds after the acoustic end of speech, with a median of 1.28 seconds. That includes the original trailing silence, CPU speech recognition, actual network model request, validation and buffering. This is not a claim of subsecond speech interaction. The model made eleven requests, including the clarification follow-up; the Compare button made none.

Speech end is the last 10 ms frame at or above −45 dBFS. The response boundary is the first native sample under the accepted combined sound/phrase state, mapped to the browser clock. It is not necessarily the instant a listener perceives a change: a longer release becomes apparent when a note ends. Recognition processed complete generated voice inputs; live microphone endpoint detection was outside this run.

Slow modulation needs time to hear. The instrument’s native rate display reads 0.40 Hz, a 2.5-second cycle. The darker shimmer and its half-depth correction receive 5.13 and 5.02 seconds at full monitor level before the next spoken direction—over two cycles each. This supplies exposure to the change; it does not establish that a listener finds the texture convincing. The rest of the conversation keeps its shorter relative corrections and a full phrase after the tempo change.

All eleven original voice waveforms and both native comparison excerpts were matched against the complete decoded film. A separate segment of the final live return matched the raw native stream. The capture had zero browser audio underruns, native-stream discontinuities or late native blocks; all 20,411 blocks reconcile at shutdown. Screen/audio alignment has a 43-ms uncertainty bound. These are signal and software checks, not subjective listening or a musical-quality rating.

The film preserves the complete working sequence, full A/B and more than five seconds of live return. Only calibration pre-roll and recorder shutdown after that ending are removed; there are no internal cuts or speed changes. Its single successful sequence does not establish reliability on unseen requests. A previous spoken-comparison request was misinterpreted during development; the visible comparison control here is a distinct, explicitly scoped interaction, not a claim that general spoken comparison is solved.

The model and the tool

Our sound-control workbench operates on real OB-Xd and Dexed state. The continuous voice session uses OB-Xd. It keeps three objects distinct:

• Current patch: the instrument state from which the request starts.
• Proposed plan: a complete permitted edit selected by the model.
• Accepted state: the result after native validation, with history retained for a correction or restoration.

Protected controls constrain the available plans. The model chooses a plan; the tool checks and applies it; the native instrument produces the sound. The continuous controller does more than forward a knob number:

• Instrument-specific units: envelope corrections use the instrument’s time mapping; oscillator detune uses cents. Moving halfway through a normalized slider range would not generally halve the audible quantity. These transformations follow the pinned OB-Xd engine and audio utilities.
• Scoped history: “just the attack halfway back” refers to the accepted attack change. It preserves the newer filter setting. “Halve the extra detune” refers to the addition, rather than halving the whole current value.
• Complete plans: a compound edit is checked as one operation. Before a new state is accepted, all eighty exposed controls and every native bank byte outside the declared edits are checked.
• Current-state binding: the chosen plan must still match the issued menu, native state and accepted history. Modified, replayed, expired or stale plans are rejected.

The continuous prototype asks two questions in one model request: which permitted plan addresses this direction, and which dimensions may change? The tool derives the plan’s actual native/music effects and checks that they fit the selected scope. Both judgments come from the same model and context; they are not independent semantic proof. Their disagreement on the unresolved chord/tail request produced the useful clarification seen in the film. Candidate construction, unit conversion, native verification and scoped history are deterministic parts of the tool. They make a small decision useful without pretending the model generates an unrestricted patch or listens to the sound itself.

There are two clocks in this arrangement. Audio processing must keep its own schedule; an interpretive request can take longer and return between edits. The VST3 parameter-change interface illustrates this separation in another plug-in version: the host transmits parameter changes to the processing block. Our OB-Xd demonstration uses VST2, but the design obligation is the same—an interpretation request must not become a dependency inside the audio callback.

The running native instrument therefore lives in a separate process. Speech recognition and network work cannot hold its Python execution. Menu construction runs from an immutable snapshot outside the audio locks, then checks that snapshot again before accepting a result. The original/current comparison is prepared from exact native banks in another process while the musician listens. These scheduling choices keep interpretation outside audio processing.

Undo also needs a precise meaning. If a control was clipped at its maximum, applying the opposite movement would not necessarily recover its old value. Saving and restoring the previous state gives an exact operation where an approximate inverse would not. “Halfway back” is a new edit; “undo” is restoration. They deserve different actions and different tests.

Learning a mapping, discovering a preference

Two earlier research systems help distinguish goals that a prompt-driven instrument could otherwise collapse.

In Wekinator, Fiebrink, Trueman and Cook make the mapping itself part of the musical activity. The performer supplies input/output examples, trains, plays the result, changes examples or features, and trains again. Sections 3.1–3.2 distinguish statistical evaluation from trying the mapping in performance, including unfamiliar inputs. A creative goal may be to discover a more interesting or varied mapping rather than recover a fixed target. This matters for a commissioned vocabulary: “open up” could become a personal gesture through accepted examples, rather than a dictionary definition imposed on every player. Our prototype selects an authored vocabulary with a prebuilt model; it does not perform Wekinator’s interactive retraining.

SynthAssist, by Cartwright and Pardo, takes a different route. A vocal imitation becomes an audio query over synthesizer sounds. The 2014 demonstration compares absolute and relative feature trajectories using dynamic time warping, then updates the query and feature weights from relevance feedback. The temporal shape matters because a voice may approximate how a note develops even when it cannot reproduce the synthesizer’s timbre. Its interface lets a person audition and rate suggestions before receiving the chosen parameters. Our text-conditioned finite selection neither extracts those audio features nor performs that retrieval procedure. The useful connection is the feedback structure: a first approximation can make the next preference easier to express.

Consider three different meanings of “better.” A selected gesture can correctly satisfy “keep the chord”; its native execution can correctly preserve the promised controls; its result can still fail the musician’s intention. A study of the first needs unseen instructions and conflict cases. A study of the second needs state and rendering checks. A study of the third needs people making judgments with the material, in context. Ten accepted edits in one development performance cannot substitute for all three.

A further experiment could make that distinction concrete. Give a composer three candidate gestures for one phrase, with a shared instruction and fixed monitor gain. Record which candidate they keep, what they reject and the exact follow-up. Then compare a general text selector with a model trained on those accepted gesture, preservation and correction relationships, using new sessions rather than reusing the examples it learned. Measure constraint errors separately from which audition the composer prefers. This would test whether the commissioned vocabulary helps someone direct the instrument; it remains a proposed experiment.

Building it with SYSTEM1

The deliverable combines the decision behavior with the tool that makes it useful. For an instrument, that means defining the editable state, permitted plans, protected controls, audition and restoration alongside the examples used to develop the model.

These instrument demonstrations use a prebuilt fast model while we qualify the interaction. Custom training is a separate step. The useful training target includes the tool’s own vocabulary, native-unit transformations, eligibility conditions and accepted-history relationships. A model that recognizes “warmer” in general has not thereby learned what this particular instrument may change, what must remain protected or which prior edit “halfway back” refers to.

A meaningful test follows the whole sequence: request a compound envelope change, verify that protected filter controls stayed fixed, audition the result, accept it, make a relative follow-up and restore the prior state. Parameter preservation, interpretation quality and musical usefulness are separate observations. Each can improve without being mistaken for the others.

This is why model and tooling belong in the same project. A model can select a plausible plan that the host applies incorrectly. A host can preserve every byte while the chosen edit misses the musician’s intention. The development process needs to examine both.

Where this could lead

• One theme, several scenes. Imagine a composer preparing a theme with three related behaviors: a close, dry version; a slower, unsettled entrance; a spacious ending. Independent decisions could propose bounded changes for each scene, while shared rules preserve the controls that define the instrument’s identity. Each result would remain available for immediate comparison. The composer would direct a family of sounds rather than begin three unrelated preset searches.

• An instrument with a personal vocabulary. A performer could commission a small set of gestures or phrases. “Open up” might choose a permitted filter/envelope combination while leaving tuning and level protected. A separate stage could resolve whether the performer means the current patch or a transition into the next section. The executable plan would remain bounded even as the language becomes personal.

• A phrase that carries several decisions. “Let the lead step forward, but keep the bass steady” could first resolve the musical roles, then examine affected instruments in parallel. A shared check could reject a combined proposal that changes protected levels or disrupts a transition. The accepted result might coordinate several parameters across several instruments. This is a proposed extension: the current film controls one instrument and does not analyse a mix.

The same architecture suggests other continuous working interfaces:

• An animator could keep a motion loop playing while proposed timing changes arrive, preserving the character’s path and comparing a more hesitant movement at the same point.
• A lighting designer could audition different transitions into a scene while protecting the positions that keep a performer visible.

In each case, interpretation operates beside the ongoing material. The tool knows the state, permitted changes and how to return.