Project architecture and evolution notes

Runtime flow

.eftl document
  -> VS Code language client (`src/extension.ts`)
  -> LSP server (`src/server.ts`)
  -> tokenizer (`src/tokenizer.ts`)
  -> stack parser (`src/parser.ts`)
  -> diagnostics and per-document variable definitions

Separately, VS Code loads the TextMate grammar, language configuration, and snippets directly from package.json contributions.

Current responsibilities

src/extension.ts: starts out/server.js over IPC for file-backed EFTL documents.
src/tokenizer.ts: recognizes a fixed, mostly uppercase set of tags, expressions, comments, and plain text; stores one-based line and column plus a character length.
src/parser.ts: checks stack-balanced open and close token kinds. It declares an AstNode interface but does not currently build an AST or enforce IF/WHILE child ordering.
src/server.ts: tokenizes and parses on document changes, validates VAR type, collects variable declarations, publishes diagnostics, and implements same-document go-to-definition by word matching.
syntaxes/eftl.tmLanguage.json: highlights a broader syntax independently of the TypeScript parser.
language-configuration.json: defines comment, bracket, auto-close, surrounding-pair, and folding behavior.
snippets/eftl.json and examples/sample.eftl: provide authoring examples and useful regression inputs.

Baseline gaps to re-check

These observations describe the repository when this skill was created and are not permanent requirements:

Documented LOG, FOR, IS_EMPTY, and IS_NOT_EMPTY constructs are not tokenized or structurally parsed.
The specification describes case-insensitive tags; tokenization currently uses exact uppercase matches.
An unrecognized [ currently makes scanText() return without consuming input, so lowercase, unknown, or incomplete tags can trap tokenization in an infinite loop.
Prefix checks such as [VAR need a tag-name boundary; otherwise longer unknown names can be misclassified.
WHILE attributes and the attribute form of TRIM are not recognized by the current exact scanners.
Parser validation is balance-only; it does not enforce root scope, IF branch ordering/cardinality, WHILE shape, or allowed bodies.
Attribute parsing uses regular expressions in the server and validates only VAR type.
Source positions combine one-based line/column with token string length. Multiline diagnostic end ranges and variable-definition selections require care.
eftl.maxNumberOfProblems is declared but not consumed by the server.
No automated test script is declared.
npm run compile passes at the baseline. npm run lint fails before linting because ESLint 9 cannot find a flat eslint.config.* file.

Preferred direction

Evolve incrementally toward:

a source-span model based on offsets with reliable LSP conversion;
a context-aware scanner that can report malformed input and recover;
structured attribute parsing with ranges;
an AST or equivalent parse structure that represents documented relationships;
separate semantic passes for declarations, references, types, and TAG payloads;
table-driven language metadata shared where doing so reduces drift;
core unit tests plus a small number of LSP integration tests.

Do not perform this redesign wholesale for an unrelated small fix. Introduce seams that make the next supported construct easier and safer.

Completion criteria for a language construct

A construct is complete only when applicable layers agree on:

accepted spellings and delimiters;
attribute names and body form;
nesting and ordering;
malformed-input recovery;
diagnostic message, severity, and exact range;
highlighting and folding;
snippet/example syntax;
focused positive, negative, boundary, and regression tests.

3.8 KiB Raw Blame History