Struct orzir_core::TokenStream

pub struct TokenStream<'a> { /* private fields */ }

A tokenizer for the IR program.

This is used for the Parse trait to parse the IR program.

Implementations

impl<'a> TokenStream<'a>

pub fn new(slice: &'a str) -> Self

pub fn checkpoint(&mut self)

Set a checkpoint.

pub fn commit(&mut self)

Pop the last checkpoint and commit the changes.

pub fn rollback(&mut self)

Rollback to the last checkpoint.

This will also reset the buffered token.

pub fn peek(&mut self) -> ParseResult<&Token>

Peek the next token.

This will return the buffered token if exists, otherwise buffer the next

pub fn consume(&mut self) -> ParseResult<Token>

Consume the next token.

pub fn consume_if(&mut self, kind: TokenKind) -> ParseResult<Option<Token>>

pub fn expect(&mut self, kind: TokenKind) -> ParseResult<()>

pub fn curr_pos(&mut self) -> ParseResult<Pos>