# Project architecture and evolution notes

## Runtime flow

```text
.eftl document
  -> VS Code language client (`src/extension.ts`)
  -> LSP server (`src/server.ts`)
  -> tokenizer (`src/tokenizer.ts`)
  -> stack parser (`src/parser.ts`)
  -> diagnostics and per-document variable definitions
```

Separately, VS Code loads the TextMate grammar, language configuration, and snippets directly from `package.json` contributions.

## Current responsibilities

- `src/extension.ts`: starts `out/server.js` over IPC for file-backed EFTL documents.
- `src/tokenizer.ts`: recognizes a fixed, mostly uppercase set of tags, expressions, comments, and plain text; stores one-based line and column plus a character length.
- `src/parser.ts`: checks stack-balanced open and close token kinds. It declares an `AstNode` interface but does not currently build an AST or enforce IF/WHILE child ordering.
- `src/server.ts`: tokenizes and parses on document changes, validates VAR `type`, collects variable declarations, publishes diagnostics, and implements same-document go-to-definition by word matching.
- `syntaxes/eftl.tmLanguage.json`: highlights a broader syntax independently of the TypeScript parser.
- `language-configuration.json`: defines comment, bracket, auto-close, surrounding-pair, and folding behavior.
- `snippets/eftl.json` and `examples/sample.eftl`: provide authoring examples and useful regression inputs.

## Baseline gaps to re-check

These observations describe the repository when this skill was created and are not permanent requirements:

- Documented `LOG`, `FOR`, `IS_EMPTY`, and `IS_NOT_EMPTY` constructs are not tokenized or structurally parsed.
- The specification describes case-insensitive tags; tokenization currently uses exact uppercase matches.
- An unrecognized `[` currently makes `scanText()` return without consuming input, so lowercase, unknown, or incomplete tags can trap tokenization in an infinite loop.
- Prefix checks such as `[VAR` need a tag-name boundary; otherwise longer unknown names can be misclassified.
- WHILE attributes and the attribute form of TRIM are not recognized by the current exact scanners.
- Parser validation is balance-only; it does not enforce root scope, IF branch ordering/cardinality, WHILE shape, or allowed bodies.
- Attribute parsing uses regular expressions in the server and validates only VAR type.
- Source positions combine one-based line/column with token string length. Multiline diagnostic end ranges and variable-definition selections require care.
- `eftl.maxNumberOfProblems` is declared but not consumed by the server.
- No automated test script is declared.
- `npm run compile` passes at the baseline. `npm run lint` fails before linting because ESLint 9 cannot find a flat `eslint.config.*` file.

## Preferred direction

Evolve incrementally toward:

1. a source-span model based on offsets with reliable LSP conversion;
2. a context-aware scanner that can report malformed input and recover;
3. structured attribute parsing with ranges;
4. an AST or equivalent parse structure that represents documented relationships;
5. separate semantic passes for declarations, references, types, and TAG payloads;
6. table-driven language metadata shared where doing so reduces drift;
7. core unit tests plus a small number of LSP integration tests.

Do not perform this redesign wholesale for an unrelated small fix. Introduce seams that make the next supported construct easier and safer.

## Completion criteria for a language construct

A construct is complete only when applicable layers agree on:

- accepted spellings and delimiters;
- attribute names and body form;
- nesting and ordering;
- malformed-input recovery;
- diagnostic message, severity, and exact range;
- highlighting and folding;
- snippet/example syntax;
- focused positive, negative, boundary, and regression tests.