Published October 17, 2026 | Version v1

Deixis and the Prague Correspondence: Stepping toward General Visual Programming by Demonstration with Direct Manipulation

Authors/Creators

Description

Presentation and extended abstract from LIVE 2026.

Abstract

The design of general-purpose visual programming languages where one programs by demonstration is an ongoing challenge, particularly when limiting escape to text-based editing. The author reframes this challenge by coining the Prague Correspondence Problem: Can we draw a rigorous relationship between instructions that execute in a virtual machine, direct manipulations or gestures that communicate those instructions to a virtual machine, and visual representations of these instructions executing that the machine may communicate back?

Deixis is an in-progress prototype of a live visual coding environment. A user can directly demonstrate the execution of a graph-rewrite algorithm the way they want it to run. The nodes are symbols and include first-class operators, making Deixis visually homoiconic. The edges are spatial relations between nodes determined by matching on and transforming node arrangements. Its visual virtual instruction set architecture includes a base set of graph-rewrite instructions, such as creation, moving, and evaluation. Each is associated with an interaction and a visual that is concurrent with execution. A user can demonstrate programming factorial.

It remains unclear what models exist for conceptually and practically unifying computing with associated visuals and direct manipulation—where direct manipulation is execution, and where its corresponding visual representation is execution as well. Deixis suggests a path toward another model by using ideas from linguistics, such as imbuing spatial arrangements with syntactic structure. Deixis aims to advance the Prague Correspondence Problem by making visual structures interactive and executable.

Files

Deixis and the Prague Correspondence_ Stepping toward General Visual Programming by Demonstration with Direct Manipulation.mp4

Files (216.8 MB)