Whenever you start wondering how to create your syntax or rules on expression, it’s probably time to check out this powerful tool you may be interested in. The allows you to build your own grammar rules, validate, and parse the result at the end. Chevrotain What does this library structurally consist of? of all, it’s . Lexer is the class that basically goes through input text and match token to each symbol or sequence of symbols (words), it found. To create an instance of the Lexer class, I need to define all tokens that I want to use. The token should have a unique name, the pattern - means regex. First Lexer For example token will match the exact word in the text. ‘hackernoon‘ const hackernoon = createToken({
  name: 'hackernoon',
  pattern: /hackernoon/
}) Also, there is a possibility to tokens. For this purpose, I need to define one more token with a reserved pattern value . In this case, Lexer will not use it to match some characters. And in other tokens that I want to categorize, I add my new token to a property ‘categories’. Like in the example below, I introduce and add this token to and . categorize Lexer.NA MathOperator MathPlus MathMinus export const MathOperator = createToken({
    name: Token.MathOperator,
    pattern: Lexer.NA,
});

export const MathPlus = createToken({
    name: Token.MathPlus,
    pattern: /\+/,
    categories: MathOperator,
});

export const MathMinus = createToken({
    name: Token.MathMinus,
    pattern: /-/,
    categories: MathOperator,
});

export const Number = createToken({ 
    name: Token.Number, 
    pattern: /[+-]?\d*\.?\d+(?:[Ee][+-]?\d+)?/
});

export const Whitespace = createToken({
    name: Token.WhiteSpace,
    pattern: /\s+/,
    group: Lexer.SKIPPED,
}); When all tokens are defined, I need to create a new instance of the Lexer class and put there an array of these tokens. const allTokens = [
  WhiteSpace,
  Number,
  MathPlus,
  MathMinus
]

const mathLexer = new Lexer(allTokens) And from this moment the Lexer is ready to tokenize input strings. I need to invoke the function ‘tokenize’ of the Lexer, and as a result, I will receive an object with tokens. With them, I can go to the next step. const lexingResult = mathLexer.tokenize(text); is having next interface . Where errors array can be very helpful to understand for the user what is wrong in the expression because it contains coordinates to the problem: line and column. lexingResult ILexingResult export interface ILexingResult {
  tokens: IToken[]
  groups: {
    [groupName: string]: IToken[]
  }
  errors: ILexingError[]
} The important thing is . It is a tool to describe the rules of your expression. For example, in math expression, you are expecting that the first variable will be a the second should be some operator and the third must be a as well. So rule will describe such behavior. First things first I create my own class the inherited from . That will accept tokens list in the constructor, and the constructor will contain all required rules and after them the to check all written rules. second Parser Number +/- Number MathParser CstParser this.performSelfAnalysis() class MathParser extends CstParser {
    public mathCond: RuleFn;
    private valueOrParen: RuleFn;
    private parentheses: RuleFn;
    private value: RuleFn;
    private mathOp: RuleFn;

    constructor(tokens: TokenType[]) {
      super(tokens);
      const $ = this;

      // Rules here
      
      this.performSelfAnalysis(); // Must be after all rules
    }
} The rule is the main entrance rule that will be invoked to perform the parsing. Basically, in my example, I assume that expression should be starting from some number or from parentheses. You are wondering how to read this rule? It actually tells if at least one number or parentheses with a number or parentheses with some operation (the operation described in keyword), then the expression is valid. mathCond 123 (123) (123 + 456) MANY Of course, as you can see there could be more  parentheses and math operations inside one parentheses, in these terms - recursion. And says that expression can repeat zero or more times. As like the example with - it repeats one time if it will be like then 2 repeats. this can be read: 1 repeat in parentheses and one after them. MANY (123 + 456) (123 + 456 - 789) (123 + 456) - 789 mathCond this.mathCond = $.RULE('mathCond', () => {
    $.SUBRULE($.valueOrParen, { LABEL: 'lhs' });
    $.MANY(() => {
        $.SUBRULE1($.mathOp)
        $.SUBRULE2($.valueOrParen, { LABEL: 'rhs' });
    });
}); The rule is expecting some Number or other parentheses with math expression. valueOrParen this.valueOrParen = $.RULE('valueOrParen', () => {
    $.OR([
        { ALT: () => $.SUBRULE($.value) },
        { ALT: () => $.SUBRULE($.parentheses) },
    ]);
}); In the next rule parentheses, I want to cover the case with braces and inside braces, recursion happens. and are two more tokens. is the main rule function. () LeftParen RightParen mathCond this.parentheses = $.RULE('parentheses', () => {
    $.CONSUME(LeftParen);
    $.SUBRULE($.mathCond);
    $.CONSUME(RightParen);
}); is the rule which looks for OR . mathOp + - this.mathOp = $.RULE('mathOp', () => {
    $.OR([
        { ALT: () => $.CONSUME(MathPlus) },
        { ALT: () => $.CONSUME(MathMinus) },
    ]);
}); The rule is created just to show that it can be extended with other specific types. value this.value = $.RULE('value', () => {
    $.OR([
        { ALT: () => $.CONSUME(Number) }
    ]);
}); Once the parser is ready, create it and call the main function. const mathParser = new MathParser(allTokens);
mathParser.mathCond(); As for Lexer, there are errors that help users, here as well, after calling the main function parser may contain errors with Token where the error happened. mathParser.errors So this library helps you to verify the string that the user typed and actually evaluate the expression. And even more, you can try a custom to create your own expression tree. Visitor

Lexer

Super

More Features of the RTK Query

The RTK Query, API Splitting Approach, and the Nuances of "Making Life Simpler"

Nominated for 2022 - HackerNoon Contributor of the Year - Redux

Nominated for 2022 - No No No Nodejs

Nominated for 2022 - HackerNoon Contributor of the Year - Coding Skills

Nominated for 2022 - HackerNoon Contributor of the Year - Typescript

Nominated for 2022 - HackerNoon Contributor of the Year - Api

Too Long; Didn't Read

Make resilience your competitive advantage

Creating Your Own Grammar Rules

Creating Your Own Grammar Rules

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Build a Solana Crowdfunding App using React

The Noonification: Reduce Javascript: Master the Basics (1/11/2023)

The Noonification: Satoshi Might Be a Bitcoin ETF Skeptic (1/13/2024)

The Noonification: Small Puddle of Freedom (11/25/2022)

The Noonification: How to Use AI for Your B2B Marketing (11/11/2022)

Build a Solana Crowdfunding App using React

The Noonification: Reduce Javascript: Master the Basics (1/11/2023)

The Noonification: Satoshi Might Be a Bitcoin ETF Skeptic (1/13/2024)

The Noonification: Small Puddle of Freedom (11/25/2022)

The Noonification: How to Use AI for Your B2B Marketing (11/11/2022)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps