SelectionToken Struct Reference

Owns one token and its source position. More...

#include <SelectionToken.h>

Public Attributes
SelectionTokenType	type
std::string	text
int	pos

Detailed Description

Owns one token and its source position.

Token text is copied from the source expression. pos is a zero-based byte offset, not a Unicode code-point index. The terminal token owns an empty string and records the source byte length.

Member Data Documentation

pos

int SelectionToken::pos

Stores the zero-based source byte offset as an int.

text

std::string SelectionToken::text

Owns the exact source bytes for this token.

type

SelectionTokenType SelectionToken::type

Stores the parser category assigned by SelectionTokenizer.