Building semantic tools—and models that learn to use them

Ask a coding model to build a searchable catalog, or a media model to assemble a film from several clips. A remarkable part of the result is what the model does not have to invent.

It inherits an accumulated vocabulary:

• Shell composition: one program’s output can become another’s input through a defined connection. The model can assemble existing programs instead of implementing each transformation. The shell pipeline specification gives that connection precise behavior.
• Media operations: FFmpeg exposes streams, filters, timestamps and ways to connect them. A cut and a change of time origin are distinct operations: trim preserves timestamps; setpts can rebase them. That distinction matters when segments are joined. The model is working with a temporal representation, not merely an array of colored pixels. FFmpeg filter documentation
• Interface structure: React provides components, data flow and state. A search field and its result list can depend on the same state instead of becoming unrelated patches to the screen. Thinking in React

These tools carry decisions made over many years: what an object is, how operations combine, what changes when they execute, and which details a caller can leave to the implementation. Model capability arrives inside that accumulated work.

This accumulated work suggests a productive question: what capabilities could appear if we deliberately built equally useful vocabularies for work that is still awkward to express—and trained models to use the tools we created?

Give an intention something to operate on

An editor says: “Keep the quotation and its attribution together. Give the opening more air, without adding a page.”

There are many possible low-level responses: move text, shrink a font, alter line breaks, increase a margin. Some satisfy one part of the instruction by quietly violating another. A more useful interface would represent the quotation as a protected object, the attribution as its dependent, and the page count as a constraint. It could then propose changes within the editor’s permitted range.

The operation needs more than a pleasant name. It needs a meaning that survives execution:

• Objects: the quotation, attribution, opening paragraph and page—not just coordinates that happen to contain them.
• Allowed changes: spacing, measure or placement, with explicit ranges.
• Preserved relations: the words remain unchanged; the attribution stays attached; reading order survives.
• A useful result: feasible alternatives, or a precise conflict that lets the editor decide what to relax.

This is what I mean by a semantic tool: an operation whose objects and consequences correspond to distinctions that matter in the work. It can contain algorithms, ordinary code, learned judgments, or a combination of them. Calling a function make_elegant does not supply the missing meaning.

Typography supplies a mature example of this move. Knuth and Plass represent a paragraph using boxes, adjustable glue and penalties for possible breaks. Their line-breaking method considers the paragraph as a whole, rather than greedily finishing each line. That representation makes a family of typographic choices available to an algorithm. Breaking Paragraphs into Lines, 1981

The optimization is exact relative to its objective. The objective remains a choice. A publisher may value compactness differently from a designer; an editor may reject a mathematically preferable layout because it gives the wrong emphasis. The proposed tool should expose that choice rather than conceal it in a supposedly universal “quality” score.

Now the first operation can become a component of another. “Balance this spread” might coordinate paragraph fitting, quotation placement, captions and image crops. A publication-level operation could coordinate several spreads while preserving section openings. Each level makes some relations explicit and delegates others. The useful vocabulary grows recursively.

Give the work a different rhythm

Now give the editor a six-page feature about a repair workshop. The source material includes a photograph of a repaired chair, a close-up of its joint, a drawing of the repair, the maker’s explanation and the owner’s account of why they kept it.

The editor asks: “Let readers notice the repair before explaining it. Open quietly, build density around how it works, and end with the chair back in use.”

That request describes an experience. A useful operation could make several arrangements of the material available for inspection. One opens with the repaired joint and very little explanation, uses the middle spread to reveal the method, then returns to the whole chair. Another begins with the owner’s reason for keeping it; the detail and method become answers to a human question. The editor can compare how each version changes the significance of the same material.

The operation has several nested levels:

• Argument and experience. Represent the parts by what they do: establish a question, explain a mechanism, supply context, offer testimony or return to an earlier image with a new meaning. A learned interpretation proposes those roles from the material and the editor’s intention. The editor can correct a reading: the owner’s quotation expresses attachment; it is not evidence that the repair is structurally sound.
• Sequence and rhythm. Represent when each role appears and how attention moves between them. “Open quietly” could mean a full image with a short line, followed by a deliberately denser explanatory spread. “Keep the doubt alive longer” could move the explanation while preserving the material that makes the question intelligible. These are alternative editorial strategies, not a universal recipe for good pacing.
• Spread composition. Turn a strategy into choices about image scale, contrast, text density, repeated motifs and relationships across the gutter. A learned model could propose a composition; a solver could find layouts that meet its selected measures and spacing rules. The page-fitting and quotation operations become components here.
• Rendered particulars. Resolve line breaks, crop bounds, alignments and linked captions. Exact algorithms can check many of these consequences. The rendered alternatives return to the editor, who judges whether the intended rhythm actually works.

Rhetorical Structure Theory offers a useful precedent at the first level. Mann and Thompson describe relations among text spans through their function and the effect an author plausibly intends for a reader. They explicitly allow ambiguous and simultaneous analyses. A functional account can therefore be precise enough to discuss without claiming that a passage has one objectively correct role. Our extension into visual sequence and interactive editing is a design proposal. Rhetorical Structure Theory: A Theory of Text Organization, 1987, Sections 2, 5–6 and 9

The new power is the ability to explore an editorial intention across these levels. “Keep the quiet opening, but let the owner’s reason arrive earlier” can regenerate a sequence while retaining a chosen compositional quality. “Show the method without turning the feature into a manual” can produce a different distribution of detail. The person is shaping how the material becomes an argument, not merely choosing a page template.

A theory supplies distinctions; the learned model interprets this material and request; the algorithms turn chosen relations into candidates. None of those alone determines the result. Together they could create a working vocabulary for a kind of editorial judgment that is otherwise scattered across many small edits.

Representations change the work itself

An external representation can do more than save keystrokes. It can change which mental task a person performs.

In Edwin Hutchins’s study of cockpit coordination, prepared marks on an instrument let pilots judge spatial relations instead of repeatedly recalling and comparing numerical speeds. The information was distributed across people, procedures and physical representations. The explanation concerned the whole system, not an isolated improvement to individual memory. How a Cockpit Remembers Its Speeds, 1995

There is a corresponding opportunity in an editor’s workspace. If the tool retains the protected text, allowed changes and reasons a layout was rejected, the person can compare alternatives directly. They no longer have to remember a sequence of coordinate edits and reconstruct what each one disturbed. A correction such as “keep this emphasis, but restore the original measure” can address the structure of the work.

The representation also determines what can be asked. A document that knows only glyph positions cannot directly answer “which captions will lose their illustrations if this section moves?” One that represents those associations can. A model may help recover the associations from an old document; once accepted, they can become explicit objects used by subsequent operations.

This is a practical reading of Engelbart’s augmentation framework: improvements to symbol manipulation can enable new working processes, which in turn make more powerful representations useful. The opportunity is not limited to accelerating an unchanged procedure. It includes work that becomes thinkable because the person can now organize and inspect it. Augmenting Human Intellect, Section II, 1962

A high-level tool can be probabilistic

Consider a survey team asking: “Which wetlands need another visit before we draw conclusions about this species?”

The raw material is a series of visits. A species might be detected, not detected, or not surveyed. Those are different observations. A zero on a completed visit is not the same thing as a blank cell, and neither directly establishes absence.

MacKenzie and colleagues developed an occupancy model that separates the probability a site is occupied from the probability of detecting the species when it is present. Repeated visits provide information about those distinct quantities. Missing visits contribute no detection observation. Their basic model assumes occupancy remains unchanged during the survey period and that detections are not false positives. It also explains why weak detection and limited sampling can leave occupancy poorly determined. Estimating Site Occupancy Rates When Detection Probabilities Are Less Than One, 2002

Here is a small illustration, using assumed probabilities rather than fitted field results. Suppose occupancy probability is 0.6 and detection probability on each visit, given occupancy, is 0.4. With conditional independence and the same occupancy state throughout:

• Three completed visits with no detection leave a probability of about 24.5% that the site is occupied.
• Two nondetections and one missing visit leave about 35.1%.
• No visits leave the original 60% assumption unchanged.

For three nondetections, the calculation is 0.6 × 0.6³ / (0.4 + 0.6 × 0.6³). The numerator is the occupied-but-missed possibility; the denominator also includes the unoccupied possibility. The missing visit supplies no extra factor. These values illustrate the model’s logic; they are not estimates for real wetlands.

A useful review_survey_coverage operation could combine the histories, sampling conditions, fitted model and its diagnostics. Its output would help a specialist distinguish “little evidence of presence” from “little opportunity to observe.” A planning operation could then propose another visit subject to travel time and the team’s objectives.

The model interpreting the request should not silently substitute a different scientific question. “Where is the species disappearing?” requires evidence about change across periods; a single-period occupancy analysis does not answer it. The tool vocabulary makes that mismatch visible and gives the conversation somewhere useful to go.

This example also breaks a misleading hierarchy. High-level tools are not necessarily vague, and probabilistic tools are not necessarily low-level. A paragraph optimizer may be deterministic. An occupancy analysis has explicit uncertainty. A learned interpretation may select either one. Each deserves the representation and evaluation appropriate to what it does.

Meaning has to survive composition

A well-formed chart can still say something false.

Suppose our survey interface maps “detected,” “not detected” and “not surveyed” to bars of increasing length. The encoding implies an order that those categories do not possess. A more attractive chart does not fix the relation it invents.

Mackinlay’s APT work separates expressiveness—whether a graphical language can represent the intended information—from effectiveness—how well a person can perceive it. Its composition approach joins compatible encodings of shared information. This is a deeper requirement than arranging independent charts neatly. Automating the Design of Graphical Presentations of Relational Information, 1986

For a new tool, that suggests questions at every join:

• Does this map refer to the same sites and survey period as the table beside it?
• Does an estimated quantity remain visibly different from a direct observation?
• Does combining two outputs preserve their units, denominators and uncertainty?
• Does the visual operation imply a rank, precision or causal relation that neither input supports?

Semiotics helps articulate why these are substantive questions. In Peirce’s account, resemblance, connection to a particular thing, and learned convention do different work. A picture may resemble a place without establishing that it records an observation there. A symbol’s familiar appearance does not supply its referent. What Is a Sign?, 1894

For an interface, the consequence is concrete: “observed,” “estimated” and “proposed” should remain distinct kinds of things as they move through tools. Their identity cannot depend on a paragraph of explanation that disappears at the next step. This is our design inference from the sources, not a claim that a philosophical taxonomy is itself an executable type system.

Make an operation reusable enough to become a language

Imagine a workshop repeatedly making enclosures for different installations. A useful operation could accept internal dimensions, a mounting pattern and a clearance rule, then construct a part. Another could generate a compatible family of parts and a material list. A third could compare proposals against available installation space.

The vocabulary progresses from geometry to a part, from a part to a system, and from a system to a decision someone can inspect. The higher operation remains connected to its lower-level meaning. Changing the material list cannot silently change the geometry; satisfying a clearance rule does not prove an unmodeled load requirement.

CAD shows how rich this vocabulary can become. FeatureScript makes custom features ordinary participants in a parametric model, with dimensional types and deterministic execution. Its language introduction describes that foundation. The current official AI-assisted feature workflow goes further: describe a custom feature, generate and refine its code, then reuse it. That is a constructive precedent for turning a local method into a continuing capability.

The opportunity extends to operations that a particular practice has not yet made explicit. An enclosure specialist might care about a recurring installation exception. An editor might have a recognizable way to preserve emphasis across formats. A research group might need a composition that keeps observation and inference separate through several transformations. The valuable unit is neither an isolated API call nor an entire profession. It is a useful piece of work with enough structure to be reused and enough openness to be revised.

McIlroy’s early argument for software components already treated families, interfaces and discoverability as central problems. The accompanying discussion also exposed how difficult it can be to choose the right units. That history is useful here: a reusable vocabulary requires invention, not just packaging. Mass Produced Software Components, 1968

Teach a model the tools you have created

There are several ways a model and a tool can meet. They should not be collapsed into one claim.

• Learn to use supplied tools. Toolformer constructs tool-use training examples by proposing calls, executing them and filtering for their contribution to a language-model objective. The APIs are supplied; the system does not invent a missing domain operation. Toolformer
• Learn API selection with documentation. Gorilla studies fine-tuning and retrieval for invoking existing APIs. Its single-call evaluation is not an evaluation of an entire professional workflow, but it makes the learned interface a concrete research object. Gorilla
• Create a reusable tool, then use it. LATM separates a tool-making model from a tool user. Its user learns from examples in context; that is different from training new model weights. Large Language Models as Tool Makers

For the editorial operation, custom training could teach a model to map varied requests and document contexts into the vocabulary we have designed. “Keep the quote intact” and “don’t break the quotation from its credit” may call for the same protected relation. “Shorten the quote” requests different authority. “Make the opening more open” may need clarification or a choice between spacing and content density.

The training examples should therefore contain more than successful command names. They need the relevant document context, the intended operation, its operands, the allowed effect and useful corrections. Negative cases matter because an operation can be available yet inappropriate. Two requests with similar wording can require different changes; two very different requests can require the same structural operation.

The compound operation can itself contain several kinds of decision. In a proposed “balance this spread” tool, a learned interpreter might identify which emphasis the editor wants to preserve. Exact document relations would collect the linked captions and quotations. A layout solver could generate feasible alternatives; another learned judgment could compare them against the stated emphasis. The editor would inspect the alternatives before accepting a change. A publication-level tool could then use that whole operation as one component.

Each boundary needs its own examples and failure behavior. An uncertain caption association should return a question, not become a confident constraint merely because a later solver is exact. A model trained to compare layouts needs examples of the relevant editorial judgment; training it to select command names would not teach that task. This proposed composition makes the training target more specific while leaving the person able to revise what matters.

The tool implementation provides something valuable for evaluation: observable consequences. Did the quotation remain unchanged? Did its attribution move with it? Did the operation touch only permitted objects? Did the resulting page satisfy the chosen constraints? A syntactically valid call is only one part of success.

That leaves genuine judgment for the learned model and the person. Which layout serves the argument? Is a surprising request a new use of an existing operation, or evidence that the vocabulary needs another one? Training should make the model better at those distinctions without pretending that the current set of operations exhausts the field.

Let the vocabulary and the model develop together

DreamCoder offers a particularly relevant research precedent. It learns reusable program components and a neural policy that guides program search. Solutions yield new abstractions; the growing library supplies replayed and generated problems for training the policy. Earlier learned functions can become parts of later ones. Its evaluations examine the contribution of both library learning and neural guidance. DreamCoder, PLDI 2021

The starting conditions matter: an expressive base language and a corpus of related tasks are supplied. The authors identify learning richer types, idioms and domain expertise as continuing challenges. This is evidence for a joint-learning mechanism, not proof that a model can infer an entire profession from a few examples.

A professional system could make that loop deliberately inspectable. Repeated accepted edits might suggest a new compound operation. The expert could examine the common structure before it becomes part of the library. A new version would need new examples and tests: the same command name can acquire different consequences when its implementation or assumptions change.

It would be a mistake to promote every repeated sequence. Three edits might look similar while one preserves wording, another rewrites it and a third changes the evidence being shown. Compression is useful only if the new abstraction preserves the distinctions that matter. Sometimes the right discovery is that a supposed single operation should be split.

To test the proposal, we should compare systems rather than merely display a convincing result: generic operations versus a domain vocabulary, and a prompted model versus a model trained on that vocabulary. Hold out whole projects and unfamiliar combinations, not just paraphrases of training requests. Measure task completion, unintended effects, corrections and the time until a person can continue useful work. The result may show that a better tool matters more than more training—or that both are needed.

What could become possible to ask for?

The most interesting tools may arise where a person already recognizes a meaningful change but cannot yet ask software to perform it coherently:

• An editor could revise the emphasis of a publication across formats while retaining quotations, caption relationships and deliberate exceptions.
• A survey team could move from “show the observations” to “help us decide what to observe next,” with sampling assumptions and uncertainty carried through the decision.
• A workshop could turn a recurring installation method into a configurable part family, then teach a model to recognize which customer constraints call for which variation.

Each possibility changes who can act. A specialist can make a method available to others. A colleague can invoke it without reconstructing every low-level step. Both can inspect the result and discover where the method needs to grow.

The aim of SYSTEM1 is to streamline that path: from an idea about useful work, through its architecture, to the tools, data and custom models that make it a continuing capability. The broader research question comes first. What operational language would let people do valuable work they can already imagine—and what new work would become imaginable once they had it?