In this part of creating your own programming language, we will improve our lexical analysis with Regex lookaheads and lookbehinds (please check out and ). Part I Part II Basically, lookaheads and lookbehinds allow you to create your own and sub-expressions (anchors). With their help, you can set up a condition that will be met at the beginning and at the end of the line or won’t, and this condition will not be a part of the “matched” expression. ^ $ Lookbehind can “look” back and should be placed at the beginning of the regular expression. Lookahead “looks” forward and accordingly should be placed at the end. There are four types of lookaheads and lookbehinds with corresponding regex syntax: Positive lookahead: (?=pattern) Negative lookahead (?!pattern) Positive lookbehind (?<=pattern) Negative lookbehind (?<!pattern) Let’s dive into code and see how we can improve lexical analysis for our programming language. We will go through the declared with the corresponding regex value to define an appropriate lexeme: Token types Keyword lexeme if|then|end|print|input|struct|arg Let’s suppose we defined a variable starting with a Keyword: print_value = 5
print print_value During the lexical analysis we will capture this variable by the lexeme instead of being parsed as a variable. To read it properly, we need to be sure that after the keyword is placed either a space or the end of the line. It could be done with the help of a positive lookahead in the end of the expression: print_value Keyword Keyword (if|then|end|print|input|struct|arg)(?=\\s|$) Logical lexeme true|false The same rule is applied to the lexeme: Logical true_value = true To read the lexeme properly, it should be separated either by space or by the end of the line: Logical (true|false)(?=\\s|$) Numeric lexeme [0-9]+ For the values, we will introduce the ability to read floating-point numbers with a decimal point and numbers with a negative/positive sign in front of the number: Numeric negative_value = -123
float_value = 123.456
leading_dot_float_value = .123
ending_dot_float_value = 123. First, to match optional or we add a character class: + - [+-]?[0-9]+ Then to read a float number, we introduce a dot character with arbitrary numbers set after it: [+-]?[0-9]+[.]?[0-9]* There could be a special case when our number starts with a leading dot, so we add an extra alternation: [+-]?([0-9]+[.]?[0-9]*|[.][0-9]+) We can also replace an alternation by applying a positive lookahead with a requirement to contain maximum 1 dot and at least 1 number after an arbitrary sign declaration. In this case, our main expression for the float number will have a greedy quantifier: * [+-]?((?=[.]?[0-9])[0-9]*[.]?[0-9]*) Lastly, to map floating-point numbers, we modify to accept type as a generic type: NumericValue Double Value public class NumericValue extends Value<Double> {
    public NumericValue(Double value) {
        super(value);
    }

    @Override
    public String toString() {
        if ((getValue() % 1) == 0) //print integer without fractions
            return String.valueOf(getValue().intValue());
        return super.toString();
    }
} Operator ([+]|[-]|[*]|[/]|[>]|[<]|[=]{1,2}|[!]|[:]{2}|[(]|[)]|new) struct Object
	arg number
end

new_value = new Object [ 5 ] For the operator, we as well need to introduce a positive lookahead separating value with a space or with the end of the line: new ([+]|[-]|[*]|[/]|[>]|[<]|[=]{1,2}|[!]|[:]{2}|[(]|[)]|new(?=\s|$)) In this part we slightly improved our lexical analyzer with positive lookaheads. Now our toy language is a little bit smarter and able to parse more complex variable expressions.

Super

2022 - HackerNoon Contributor of the Year - Algorithms

2022 - HackerNoon Contributor of the Year - Coding Skills

2022 - HackerNoon Contributor of the Year - Functional Programming

2022 - HackerNoon Contributor of the Year - Java

2022 - HackerNoon Contributor of the Year - Programming

2022 - HackerNoon Contributor of the Year - Sql

2022 - Rawr For Ror

Coding Guru of 2022

Nominated for 2022 - Rawr For Ror

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for Coding Guru of 2022

Nominated for 2022 - HackerNoon Contributor of the Year - Coding Skills

Nominated for 2022 - HackerNoon Contributor of the Year - Java

Nominated for 2022 - HackerNoon Contributor of the Year - Algorithms

Building Your Own Programming Language From Scratch

Parallel Merge Sort with Fork/Join Framework

3 Ways to Deal With Hibernate N+1 Problem

Too Long; Didn't Read

Make resilience your competitive advantage

Build Your Own Programming Language Part III: Improving Lexical Analysis with Regex Lookaheads

Build Your Own Programming Language Part III: Improving Lexical Analysis with Regex Lookaheads

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Building Your Own Programming Language From Scratch: Part II - Dijkstra's Two-Stack Algorithm

How I Built a Config Framework That Scales Across 15+ Microservices

Object Creation: The Issues I Faced and the Patterns That Helped Me

Optimizing the Ever-Growing Balance in an 11-Year-Old Game

Tracking my Computer Science Relearning Journey

Building Your Own Programming Language From Scratch: Part II - Dijkstra's Two-Stack Algorithm

How I Built a Config Framework That Scales Across 15+ Microservices

Object Creation: The Issues I Faced and the Patterns That Helped Me

Optimizing the Ever-Growing Balance in an 11-Year-Old Game

Tracking my Computer Science Relearning Journey

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps