concept 5 · code analysis

How Nemesis reads code: AST, pipeline and semantic detection

Understanding intent takes more than searching text. The code is parsed into a tree (AST) and its nodes visited; a layered pipeline runs cheap to expensive, entropy flags obfuscated payloads and taint follows data from source to sink.

In 30 seconds

tree-sitter turns the code into a syntax tree (AST). The traversal walks that tree and, at each node, calls a set of visitors; each visitor is a method that recognizes a semantic pattern. This is part of a layered pipeline: byte, entropy, regex, manifest, ide, exfil, AST and finally the recursive decoder. Shannon entropy exposes strings that look like encoded payloads, and taint tracking links a sensitive source to a dangerous sink within the file.

Content flows cheap to expensive: byte/Unicode, entropy, regex, manifest, IDE config, exfil chain, AST visitors and the recursive decoder that reveals and rescans; the result is the severity. — Pipeline from cheapest to most expensive; the decoder rescans what was hidden. Taint links source to dangerous destination.

From scratch: AST, Visitor and pipeline

A lexer breaks text into tokens; the parser builds from them an abstract syntax tree, where each node is a language construct (a function call, a string, an assignment). Visiting the AST means walking that tree applying a function to each node: the Visitor pattern. Important: a visitor is a detection method, not a unit of coverage. Entropy measures unpredictability; natural text and code have medium entropy, compressed or encrypted data has high entropy. Taint tracking is flow analysis: it marks values coming from an untrusted source and checks whether they reach a dangerous point.

Why it matters

Attackers hide intent. An eval of a concatenated string, a secret read and sent to the network, a command assembled in pieces: none of that is caught by naive substring search without a mountain of false positives. Reading the structure, the scanner distinguishes "the word exec appears in a comment" from "there is an exec call over a value decoded at runtime". That is the difference between a false alarm and real detection.

Source is parsed by tree-sitter into a syntax tree; a visitor walks the nodes and recognizes the decode-then-eval pattern even with renamed variables; the result feeds compute_severity. — Regex sees text; the visitor sees structure. Renaming the variable does not change the tree: the *decode → exec* pattern is still there.

How it correlates

The layers chain from cheap to expensive: the byte scan runs without a parser; entropy and regex are fast; the AST only runs on supported languages; and the decoder closes by decoding what was hidden and rescanning it. Taint is a full content pass coupled to the AST. The severity that comes out feeds the decision model, and many visitors in this pipeline are exactly the ones hunting the attacks of group 8.

How Nemesis applies it

The AST traversal calls the visitors at each node (here, JavaScript), and couples the taint pass over the whole content:

📄 .nemesis/nemesis-defender/src/scanner/ast_scanner.rs:58-79 · Visitor por nó + taint

violations.extend(visitors::taint_tracker::scan_js_content(content));
// ... em cada nó:
violations.extend(visitors::decode_exec::visit_js_node(node, source));
violations.extend(visitors::dynamic_cmd::visit_js_node(node, source));
violations.extend(visitors::url_in_exec::visit_js_node(node, source));
violations.extend(visitors::unicode_steg::visit_js_node(node, source));
violations.extend(visitors::prompt_injection::visit_js_node(node, source));
// ...recursão nos filhos

The real pipeline is orchestrated in scan_content, in effective order (byte with four sub-scans, then entropy, regex, manifest, ide, exfil, AST and decoder):

📄 .nemesis/nemesis-defender/src/lib.rs:242-288 · execução real do pipeline

let byte_violations = scanner::byte_scanner::scan_bidi(content);      // + pua, homoglyph, zero_width
let entropy_violations = scanner::entropy::scan_high_entropy(content);
let regex_violations = scanner::regex_layer::scan(content, &language);
let manifest_violations = scanner::manifest_scanner::scan(path, content);
let ide_violations = visitors::ide_config_poisoning::scan_ide_config(path, content);
let exfil_chain_violations = visitors::exfil_chain::scan_content(path, content);
// Layer 5 AST (só linguagens suportadas) · Layer 6 decoder recursivo

Entropy has a tuned threshold to catch payloads without flagging legitimate CSS:

📄 .nemesis/nemesis-defender/src/scanner/entropy.rs:14-16 · limiar de Shannon

const ENTROPY_THRESHOLD: f64 = 5.5;   // > 5.5 bits/char = suspeito
const MIN_STRING_LEN: usize = 20;
const MAX_STRING_LEN: usize = 500;

And taint runs in three passes: collect sources, propagate through assignments and detect the source reaching an exec or network sink:

📄 .nemesis/nemesis-defender/src/visitors/taint_tracker.rs:251-318 · fonte → propagação → sink

// Pass 1: coletar fontes contaminadas
// Pass 2: propagar taint por atribuições
// Pass 3: detectar var contaminada alcançando EXEC_SINK / NETWORK_SINK

Self-check

▸Why read the AST instead of just searching substrings?

To distinguish intent from mention: exec in a comment is harmless; exec over a value decoded at runtime is an attack. The structure gives that context and kills the false positive.

▸Is the visitor the unit of coverage in Nemesis?

No. A visitor is a detection method (semantic intent via traversal). Coverage is the sum of layers (the coefficient); counting visitors undercounts the protection.

▸What does taint tracking need to fire?

A flow: a tainted source that, propagated through assignments, reaches an exec or network sink. An isolated source or an isolated sink does not trigger it.

From scratch: AST, Visitor and pipeline

Why it matters

How it correlates

How Nemesis applies it

Self-check

Further reading