Deixis and the Prague Correspondence: Stepping toward General Visual Programming by Demonstration with Direct Manipulation
Gavriel Loria
Presented at the 12th Workshop on Live Programming (LIVE 2026). https://doi.org/10.5281/zenodo.24409597
The design of general-purpose visual programming languages where one programs by demonstration is an ongoing challenge, particularly when limiting escape to text-based editing. The author reframes this challenge by coining the Prague Correspondence Problem: Can we draw a rigorous relationship between instructions that execute in a virtual machine, direct manipulations or gestures that communicate those instructions to a virtual machine, and visual representations of these instructions executing that the machine may communicate back?
Deixis is an in-progress prototype of a live visual coding environment. A user can directly demonstrate the execution of a graph-rewrite algorithm the way they want it to run. The nodes are symbols and include first-class operators, making Deixis visually homoiconic. The edges are spatial relations between nodes determined by matching on and transforming node arrangements. Its visual virtual instruction set architecture includes a base set of graph-rewrite instructions, such as creation, moving, and evaluation. Each is associated with an interaction and a visual that is concurrent with execution. A user can demonstrate programming factorial.
It remains unclear what models exist for conceptually and practically unifying computing with associated visuals and direct manipulation—where direct manipulation is execution, and where its corresponding visual representation is execution as well. Deixis suggests a path toward another model by using ideas from linguistics, such as imbuing spatial arrangements with syntactic structure. Deixis aims to advance the Prague Correspondence Problem by making visual structures interactive and executable.
https://www.dorianzgraggen.com/external/live2026
The headline
I discovered a different possible path toward unifying interactions, execution, and visuals in a programming-by-demonstration prototype, where you can match and manipulate spatial arrangements to record graph-rewriting algorithms.
Building on the past
- John McCarthy’s LISP (1960) introduced code as data. Deixis builds on this in the visual programming language domain.
- David Canfield Smith’s Pygmalion (1975), found in Allen Cypher’s Watch What I Do (1993), showed us how to program factorial by demonstration, using 6! as an example. My 2019 prototype demonstrated programming factorial and used the 6! example—chosen independently, a happy coincidence—so Pygmalion claims priority on those. However, Deixis is explicitly making use of spatial relationships between objects. Also, in Deixis, the staircase shape example illustrates the full call stack of 6!.
- David Canfield Smith et al.’s KidSim (1994), later Cocoa and Stagecast Creator (1999), uses graphical rewrite rules written by demonstration to compute up to Turing equivalence (in Creator). However, Deixis is meant to be tailored for general-purpose programming, not just visual simulation.
- Ken Kahn’s ToonTalk (1995–) not only is general-purpose programming by demonstration, but also unifies code, coding, visuals, and interactions, and may be one of the conceptually closest precedents to Deixis. However, it doesn’t pursue ideas from linguistics like spatial arrangement as syntax.
- Michael Homer’s Tiled Grace (2013) features snapping together boxes of text in a visual programming environment, as does Resnick et al.’s Scratch (2007–2009). Tiled Grace also features allowing the user to move between the visual editor and text code. However, Deixis is using spatial syntax, graph rewriting, and directly demonstrating execution.
- Jonathan Edwards’s Subtext (2005) eschews conventional textual syntax for a direct-manipulation WYSIWYG visual programming environment. It also claims “the representation of a program is the same thing as its execution,” to which Deixis also aspires. However, while there is copying in Deixis, Deixis is based around graph rewrite as the “essence of programming” rather than copying.
- Ravi Chugh et al.’s Sketch-n-Sketch (2015–2020) has visual programming but is fundamentally bimodal, where direct manipulation changes text code, whereas Deixis works toward a unified view with directly demonstrating execution.
- Cyrus Omar et al.’s Hazelnut (2017–2019) features a rigorous structured editor calculus with a cursor and holes, which can be future inspiration for more rigorously defining the operation of Deixis, as can Gérard Huet’s “The Zipper” (1997), Ehrig et al.’s Fundamentals of Algebraic Graph Transformation (2006), and Barry Jay’s Tree Calculus (2021).
- Deixis is an immediate successor and reimplementation of Gavriel Loria’s Visual virtual programming machine for real-time interactive creation, playback, execution, inspection and manipulation of programming elements (US11294645B2, priority 2019, granted 2022), but eschews inferring programs from demonstrated examples, and, at least for now, focuses on graphical relations between objects as patterns; I hope programming with syntactically connected trees and graphs will be reintroduced into Deixis.
- Lu Wilson’s CellPond (2023), Marcel Goethals’s Subsequently (2024), and Marek Rogalski’s Automat (2025) are contemporary examples working in the same space as visual coding via direct manipulation meriting mention. My 2019 prior art precedes these.
- Pablo Donato’s Flower Calculus (2024), along with its Flower Prover, shares the spirit of the Prague Correspondence with Deixis, but pursues proofs rather than computation.
- I am and Deixis is inspired by linguistics associated with or spiritually downstream of the Prague Linguistic Circle, dating back to 1926, including in particular Lucien Tesnière’s dependency grammar in Éléments de syntaxe structurale (1959). I am appreciative of the Prague Linguistic Circle’s 100-year anniversary.
- I was first introduced to grammar foundations resembling dependency grammar in a course by my high school English teacher Dr. Rembert Herbert.
Keeping a critical eye
At a low level, Deixis has the following limitations:
- Writing one conditional branch from a clean snapshot of state is implemented, but writing the other branch from a clean snapshot is not fully implemented.
- The ability to demonstrate negative spatial relations, among other more precise spatial relations, for example, when there is not an object above this one then do something, is not implemented.
- Directly demonstrating a new variable-arity rule is not yet implemented.
At a medium level, Deixis has the following limitations:
- I believe it’s achievable, but a truly unified view probably shouldn’t need buttons in the header like Evaluate, Record, etc. Everything should be at least possibly gestural.
- I believe it’s achievable, but my 2019 prototype included syntactic joining and detaching of objects as instructions in the instruction set architecture (ISA), which this prototype doesn’t have yet.
- I believe it’s achievable, but Deixis doesn’t yet have the formal steps of parsing spatial arrangements into graphs, rewriting those graphs, and drawing the graphs back into spatial arrangements. Rather, Deixis has spatial arrangements and rearrangements or adjustments based on ISA instructions, which are technically already equivalent to graphs and graph rewrites.
- I believe it’s achievable, but Deixis doesn’t yet have a working example of auto-triggering sequences of rules.
- I haven’t proven theoretic properties about the current or near-future ISA, such as whether it achieves Turing completeness.
- Factorial is just one example of many things a user should be able to write. Also, the current interaction flow, where, for example, you have to select five objects to get the next half of an iterative step of factorial to execute, is cumbersome at best.
- “Visual homoiconicity” doesn’t yet extend to gestures, visual representations, and spatial rules being manipulable in the canvas medium, though user-defined operators like “!” are.
At a high level, Deixis has the following limitations:
- I hope that Deixis can one day accept large categories of existing text-based programs and transpile them to and from Deixis, allowing the user to make sense of external programs.
- I have little scientific evidence that my choices of gestures are truly intuitive, even for an experienced programmer, and the environment is not built for other users at this point.
- There is also no evidence yet that a user will be able to grasp very large programs in this medium, though this is a long-term goal. However, we may refer to Prague-school linguistics for inspiration here—computations may be abridged or shorthand, but expanded upon or unabridged when interrogated. A small example of this in the 2019 prototype is local undo and redo, where a 3 may still have its history of having been computed by evaluating 1+2. Deixis has a more limited version of one-step undo/provenance for some objects, accessible by button click. Deixis also has the spatial arrangement of objects to visually show the computation that produces a result and to suggest what is a detail, as shown in the staircase demo.
- Deixis does not solve the Prague Correspondence Problem as currently stated. But it explores what visual representations, interactions, and machine instructions can be, which I believe helps advance our understanding of the Problem and what a solution might entail.