Get the most recent info and news about Lexer on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

How retailers unify all their customer data, understand their customer, create segments, acquire new customers, and grow lifetime value.

Lexer

lexer

Based at the old Lexer Plantation house, Frightmare Manor combines a dark history with the fun of Halloween. With four different haunted houses to go through, Frightmare Manor is where True Terror lives.

True Terror lives in an old Talbott Plantation

Domingo last Monday, QCPD director Brig. Gen. Redrico Maranan said on Saturday. The six suspects in police custody are Justine Artates, Mark Leo Andales, Lexer Angelo Manarpies, Kyle Michael de Castro, John Xavier Arcosa and John Arvin Kaylle Diocena.

Frat leaders tagged in PCCR hazing death

Officers of the Tau Gamma Phi fraternity were among those identified as suspects in the killing of Philippine College of Criminology student Ahdlryn Lery Bravante in Quezon City, police said yesterday.

A fourth year student of the Philippine College of Criminology died from at least 60 blows to the body during initiation rites by suspected members of the Tau Gamma Phi fraternity, the Quezon City Police District said yesterday.

PCCR hazing: Student suffered 60 blows – QCPD

The Armed Forces of the Philippines says the 2016 Arbitral Award is no longer up for debate after China claimed ownership of Pag-asa Island in the West Philippine Sea.

INQToday: AFP says Arbitral Award not up for debate: China’s claim over WPS invalid

This event is now in the past - Lexer / Mayes / Noble + Semay Wu + Miman at The Lit & Phil on Friday 06 October 2023 at 19:20.

Lexer / Mayes / Noble + Semay Wu + Miman

With 28 blue chip clients in Australia and the United States, Lexer is a Software as a Service (‘SaaS’) marketing technology (‘MarTech’) company which offers an enterprise grade platform that allows businesses to unify private, public and third ...

Lexer Fund

In the of this series on how to build a WebAssembly programming language, we constructed a lexer. In this post, we’ll cover the next phase of our compiler, parsing. Parsing is the portion of our compiler that takes the token stream generated by the lexer and converts it into an (AST). last post abstract syntax tree An AST is a tree-like data structure that organizes the tokens into a logical hierarchy that can more easily be translated into machine code. Thankfully, because wispy is an S-expression language, our code is essentially an AST. Take the following stream of tokens: already “(“, “add”, “3”, “(“, “sub”, “2”, “1”, “)”, “)” Each set of parentheses represents a subtree, where the first token is the operator node and the following tokens are its operands. If we run into another opening parenthesis before the current set is closed, we know it represents an operand that itself is a subtree. The above stream of tokens would be organized into a tree that looks like this: If you're interested in writing a parser for a more complex C-like syntax, see my previous series. Building A Programming Language More About AST As we did with the lexer, we'll start by defining our types. These types will define the structure of our AST. Each type represents a “Node”, the circle from our diagram in the intro. Here are the basic nodes. We'll gloss over them, as they aren't a lot different from the tokens we defined in the lexer: // src/types/ast.mts

export type IntNode = {
  type: "int";
  value: number;
};

export type FloatNode = {
  type: "float";
  value: number;
};

export type IdentifierNode = {
  type: "identifier";
  identifier: string;
};

export type TypedIdentifierNode = {
  type: "typed-identifier";
  // Note that here, we break down the identifier into its components
  identifier: string;
  typeIdentifier: string;
}; A new concept to the AST is the . A is an expression made up of a group of other nodes. BlockNode BlockNode For example, is a block of three nodes: (add 1 2) An identifier that evaluates to a function, . add An Int that simply evaluates to the number . 1 An Int that simply evaluates to the number . 2 How the block itself gets evaluated is up to the compiler. We'll get to that in the next post. Here's the definition: // src/types/ast.mts

export type BlockNode = {
  type: "block";
  expressions: AstNode[];
}; Finally, we define the . Like the type from the lexer, is a discriminated union that can be one of any other node we previously defined: AstNode Token AstNode export type AstNode = IntNode | FloatNode | IdentifierNode | TypedIdentifierNode | BlockNode; You may have noticed that has an array of s, Since a , s can contain child . In other words, is a recursive type. This ability to recursively represent children that can have children is the foundation of our AST. It's where the tree in AST is allowed to form. BlockNode AstNode AstNode can be BlockNode BlockNode BlockNodes BlockNode At this point is finished and should look like . src/types/ast.mts this file Now export the types from as we did with the token types: src/types/index.mts // src/types/index.mts
export * from "./token.mjs";
export * from "./ast.mjs"; Constructing the AST Now that we've defined the AST, it's time to build one. Create a new file and add all the imports we'll use: src/parser.mts // src/parser.mts
import {
  Token,
  IdentifierNode,
  TypedIdentifierNode,
  IdentifierToken,
  TypedIdentifierToken,
  FloatToken,
  FloatNode,
  IntToken,
  IntNode,
  AstNode,
  BlockNode,
} from "./types/index.mjs"; Now we can define our top level parse function. The parse function takes the tokens generated by the lexer and returns a that acts as our tree’s root. BlockNode // src/parser.mts
export const parse = (tokens: Token[]): BlockNode => {
  const blocks: BlockNode[] = [];

  // This loop is run as long as there are tokens to consume
  while (tokens.length) {
    // consumeTokenTree converts an array of tokens into a tree of tokens, more on that later.
    const tree = consumeTokenTree(tokens);

    // parseBlock turns our new tree of tokens into an actual BlockNode, recursively. More on that later as well.
    blocks.push(parseBlock(tree));
  }

  // Finally we return the top level BlockNode
  return {
    type: "block",
    expressions: blocks,
  };
}; Next, we define the function. converts a flat array of tokens, into a tree of tokens. consumeTokenTree consumeTokenTree Given this wispy expression: (add (sub 3 1) (sub 5 2)) The lexer will produce this array of tokens: // Note: I've simplified the Token format to just be strings to keep things short
["(", "add", "(", "sub", "3", "1", ")", "(", "sub", "5", "2", ")", ")"]; will take that flat array and turn it into a tree. This is as simple as putting every token in between a set of bracket tokens into an array. So our token array from above becomes this token tree: consumeTokenTree () ["add", [, "sub", "3", "1"], ["sub", "5", "2"]]; Here's the actual definition of : consumeTokenTree // src/parser.mts

// This is token besides for the bracket tokens
export type NonBracketToken = Exclude<Token, "parenthesis" | "square-bracket">;

// The token tree is made of NonBracketTokens and other TokenTrees
export type TokenTree = (NonBracketToken | TokenTree)[];

const consumeTokenTree = (tokens: Token[]): TokenTree => {
  const tree: TokenTree = [];

  // Ensures the first token is a left bracket and then discards it, defined below this function.
  consumeLeftBracket(tokens);

  while (tokens.length) {
    // Preview the next token
    const token = tokens[0];

    // Check to see if the next token is a left bracket.
    if (token.type === "bracket" && getBracketDirection(token) === "left") {
      // If it is, we just ran into a sub-TokenTree. So we can simply call this function within
      // itself. Gotta love recursion.
      tree.push(consumeTokenTree(tokens));
      continue;
    }

    // Check to see if the next token is a right bracket
    if (token.type === "bracket" && getBracketDirection(token) === "right") {
      // If it is, we just found the end of the tree on our current level
      tree.shift(); // Discard the right bracket
      break; // Break the loop
    }

    // If the token isn't a bracket, it can simply be added to the tree on this level
    tree.push(token);

    // Consume / discard the token from the main tokens array
    tokens.shift();
  }

  // Return the tree. Don't forget to check out the helper functions below!
  return tree;
};

const consumeLeftBracket = (tokens: Token[]) => {
  const bracketDirection = getBracketDirection(tokens[0]);

  if (bracketDirection !== "left") {
    throw new Error("Expected left bracket");
  }

  return tokens.shift();
};

const getBracketDirection = (token: Token): "left" | "right" => {
  if (token.type !== "bracket") {
    throw new Error(`Expected bracket, got ${token.type}`);
  }

  // If we match a left bracket return left
  if (/[\(\[]/.test(token.value)) return "left";

  // Otherwise return right
  return "right";
}; Now that we have a token tree, we need to turn it into a block. To do so, we create a function that takes the tree as its input and returns a : parseBlock BlockNode 1const parseBlock = (block?: TokenTree): BlockNode => {
  return {
    type: "block",
    // This is where the recursive magic happens
    expressions: block.map(parseExpression),
  };
}; As you may have noticed, maps each item of the tree with a yet to be written function. takes either a or a and transforms it to its corresponding type. parseBlock parseExpression parseExpression TokenTree NonBracketToken AstNode Here's the definition: const parseExpression = (expression?: TokenTree | NonBracketToken): AstNode => {
  // If the expression is an Array, we were passed another TokenTree, so we can
  // pass the expression back to the parseBlock function
  if (expression instanceof Array) {
    return parseBlock(expression);
  }

  // The mapping here is pretty straight forward. Match the token type and pass the
  // expression on to a more specific expression parser.
  if (isTokenType(expression, "identifier")) return parseIdentifier(expression);
  if (isTokenType(expression, "typed-identifier")) return parseTypedIdentifier(expression);
  if (isTokenType(expression, "float")) return parseFloatToken(expression);
  if (isTokenType(expression, "int")) return parseIntToken(expression);

  throw new Error(`Unrecognized expression ${JSON.stringify(expression)}`);
}; Let's define the function. This function is pretty neat and demonstrates one of the most powerful features of TypeScript, . Simply put, tests the expression and narrows down the type to a specific . This allows TypeScript to be certain we are passing the correct tokens to their corresponding parser functions down the line. isTokenType custom type guards isTokenType TokenType Here's the definition: export const isTokenType = <T extends Token["type"]>(
  item: TokenTree | NonBracketToken | undefined,
  type: T
): item is Extract<Token, { type: T }> => {
  return isToken(item) && item.type === type;
};

const isToken = (item?: TokenTree | NonBracketToken): item is NonBracketToken => {
  return !(item instanceof Array);
}; There's a lot happening there, so let's walk through it. First up, we have a generic definition, . This is essentially saying that T must be one of the possible values of the field. Typescript is smart enough to know that means T must be one of . <T extends Token["type"]> Token.type "int" | "float" | "identifier" | "typed-identifier" | "bracket The next interesting piece of code is the return type predicate . This predicate tells TypeScript that if the return value of is true, then must be the whose type matches the string passed as the parameter. item is Extract<Token, { type: T }> isTokenType item Token type In practice that means that if we were to pass an unknown to , typescript will be able to correctly narrow the value to a more specific token, like . Token isTokenType IntToken Now that we have our custom type guard defined, we can define the actual token parsers. The first three are simple; they essentially just return a copy or slightly modified copy of the token: const parseFloatToken = (float: FloatToken): FloatNode => ({ ...float });

const parseIntToken = (int: IntToken): IntNode => ({ ...int });

const parseIdentifier = (identifier: IdentifierToken): IdentifierNode => {
  return {
    type: "identifier",
    identifier: identifier.value,
  };
}; The final parser is the . Remember that a typed identifier takes the form . Parsing it is as simple as splitting the string by the colon. The first value of the returned array is the , the second is the . Here's the definition: parseTypedIdentifier identifier:type identifier type const parseTypedIdentifier = (identifier: TypedIdentifierToken): TypedIdentifierNode => {
  const vals = identifier.value.split(":");

  return {
    type: "typed-identifier",
    identifier: vals[0],
    typeIdentifier: vals[1],
  };
}; Here’s the finished file That's all the code required for a working parser. Before we move on, let's update the main file to view the output of the parser: src/index.mts // src/index.mts
#!/usr/bin/env node
import { readFileSync } from "fs";
import { lex } from "./lexer.mjs";
import { parse } from "./parser.mjs";

const file = process.argv[2];
const input = readFileSync(file, "utf8");

const tokens = lex(input);
const ast = parse(tokens);
console.log(JSON.stringify(tokens, undefined, 2)); Build and run project: npx tsc
wispy example.wispy If all goes well, the output should look like . this With that, the parser is finished. We can now convert the stream of tokens from the lexer into an AST. In the next post, we can get into the juicy bits: generating and running machine-readable code. today to see how Courier works to make product notifications delightful! Request a demo

How to Build a WebAssembly Language for Fun and Profit: Parsing

Cómo construir un lenguaje WebAssembly para divertirse y obtener ganancias: análisis

Comment créer un langage WebAssembly pour le plaisir et le profit : analyse

मनोरंजन और लाभ के लिए WebAssembly भाषा कैसे बनाएं: पार्सिंग

楽しみと利益のために WebAssembly 言語を構築する方法: 構文解析

Como construir uma linguagem WebAssembly para diversão e lucro: análise

Cách xây dựng ngôn ngữ WebAssembly để thú vị và lợi nhuận: Phân tích cú pháp

如何为乐趣和利润构建 WebAssembly 语言：解析

Lexical Analysis

You have probably read that JavaScript is an interpreted language, while others argue that it's not an interpreted language and that it's actually a compiled language. This can be confusing, I was often confused because I received conflicting answers whenever I read books or articles about JavaScript being an interpreted or compiled language. But the truth is that it depends on how the language is implemented. Let me explain. To clarify this point, let's first examine what are compiler and interpreter also the difference between them Compiler A compiler is a software tool that translates human-readable source code into machine-executable code. The process of compilation involves several stages, including lexical analysis syntax analysis (parsing) semantic analysis code generation optimization During lexical analysis, the compiler breaks down the source code into a series of tokens, which are individual units of meaning. For example, in the statement , the tokens would be , , , , and . The lexer also assigns a type to each token, such as "identifier", "operator", or "literal". "x = 5 + 3;" "x" "=" "5" "+" "3" Then, during syntax analysis (parsing), the compiler checks the order and structure of the tokens to ensure that they form a valid program. This involves analyzing the grammar of the programming language and constructing a syntax tree that represents the structure of the program. After the syntax analysis is complete, the compiler performs semantic analysis to check for logical errors in the code. This involves verifying that the program follows the rules of the language, such as type checking, scoping rules, and function calls. Once the compiler has verified the correctness of the program, it generates machine code or bytecode that can be executed by the computer. This involves translating the syntax tree and semantic information into low-level instructions that can be executed by the computer's hardware. Finally, the compiler performs optimization to improve the efficiency of the generated code. This process involves analyzing the code and making changes to improve the memory usage. Overall, the compiler is an essential tool for translating human-readable source code into machine-executable code. The compilation process involves several stages, each of which is critical for producing correct and efficient code. Interpreter An interpreter is a program that reads and executes code written in a high-level programming language directly, without requiring an intermediate compilation step. Unlike a compiler, which translates the entire source code into machine code before execution, an interpreter reads and executes the source code line by line, translating and executing each statement as it is encountered. When an interpreter runs a program, it first performs lexical analysis and parsing of the source code to generate an abstract syntax tree (AST) representing the program's structure. The interpreter then walks through the AST, executing each node in turn. During execution, the interpreter may generate and manipulate additional data structures to represent the state of the program, such as a symbol table for tracking variable values. One advantage of using an interpreter is that it allows for more dynamic and interactive development, as programmers can test and modify their code on the fly without having to recompile the entire program. However, interpreting code can be slower than executing compiled code, since the interpreter has to perform additional work to translate and execute each statement. Overall, an interpreter provides a convenient way to execute code in a high-level language, but may not be as performant as compiled code in certain situations. Compiler vs. Interpreter in a simple way There is a lot of technical information about compilers and interpreters, so let's make it simple: And like everything else in life, let's try to understand it with a cup of tea. Let's say you want to make a cup of black tea, you go to Google and search for the ingredients list, this is what you came up with: Black tea leaves Water Sugar (optional) Milk (optional) There are 2 ways to make the tea, the Compiler or the Interpreter way. The compiler will first, before doing any steeping, organize all the ingredients in front of him, the specific amounts of every single ingredient, only then, will he steep all the ready components of the tea. The interpreter will take his cup and will start by reading the ingredients, line by line. he will boil the water, add the tea leaves, steep them for a few minutes, add sugar or milk if desired, and then strain the tea into the cup. The build (preparation) time of the compiler will be longer than the interpreter's. However, the run (steeping) time will be much shorter. Now that you know the difference let’s talk about JavaScript. Is JavaScript Compiled or an Interpreted Language? When I first started learning JavaScript, I, like many others, was taught that it was an interpreted language, and I believed it for a long time. However, after conducting further research, I have come to realize that this is not entirely accurate. While many people still consider JavaScript to be an interpreted language, there is evidence to suggest that it is actually a compiled language. One example of this can be seen in the way that the JavaScript engine handles bugs. When you encounter a syntax error in your JavaScript code, the engine will typically alert you to the problem before it even begins executing the code. This suggests that the code is compiled and validated before it is executed, which is a characteristic of compiled languages. You can try this for yourself by opening the following HTML code in your browser and checking the console: console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World!); The last line of code is missing a closing quotation mark, which should trigger a syntax error. However, if JavaScript were truly an interpreted language, the console would have already printed the first nine lines before throwing an error. Instead, the program crashes immediately, Uncaught SyntaxError: Invalid or unexpected token Thats indicating that it was compiled and validated before execution. While there is still debate over whether JavaScript is truly a compiled language or not, it's clear that it's not entirely accurate to call it an interpreted language. Another way to prove that JavaScript is not an entirely interpreted language is through the concept of . For example, consider the following code: hoisting max(1, 2);
// 2
function max(num1, num2){
  return num1 > num2 ? num1 : num2;
} In traditional programming languages, a function must be defined before it can be called. However, in JavaScript, the function can be called before it is defined. This behavior is known as , and it's a fundamental aspect of how JavaScript works. hoisting Note: We will explore more about `hoisting` in other blog for now this explanation is enough. How does the JavaScript engine know about the function before it reaches the declaration? The only reasonable answer to this question is that the code must first be compiled before execution. max So, JavaScript is a compiled language, right? Well, it's complicated. In the past, every programming language was fairly easy to categorize as either compiled or interpreted. However, with the modern approach of running source code, a sort of "in-between" area has been created. How does JavaScript actually get translated? JavaScript has come a long way since its inception as a simple scripting language. In the early days, JavaScript engines were only interpreters, which executed code line by line. However, as JavaScript became more popular and its use cases expanded, performance issues emerged. Interpreting code on-the-fly resulted in a loss of performance, especially for complex applications. This led to the development of new engines that use a just-in-time (JIT) compiler. A JIT compiler is different from traditional compilers, such as those used for C++. Traditional compilers have plenty of time to optimize the code during compilation, but JIT compilers must compile code just before it's executed. As soon as the JavaScript code is about to run, it gets compiled into executable bytecode. Despite the differences in how JavaScript is compiled compared to other compiled languages, the process still follows some of the same rules as traditional compilers. The JavaScript code is parsed before execution, which makes it look like a parsed language. However, the code is converted to binary form before execution. To better understand this process, let's take a closer look at how code execution works behind the scenes: First, the code is transpiled using tools like Babel or Webpack. The transpiled code is given to the engine, which parses it to an Abstract Syntax Tree (AST). The AST is then converted to bytecode that is understood by the machine. This is an Intermediate Representation (IR), which is further optimized by the JIT compiler. After optimization, the JS Virtual Machine (VM) executes the code. Some people argue that the JS VM is "interpreting" the bytecode, but this is not entirely accurate. If we were to use that definition, we would also have to say that Java, which is another JVM-driven language, is also interpreted. Thus, we can conclude that JavaScript is executed in three phases: Parsing Compiling Executing JavaScript code is initially interpreted before any execution begins. For example, consider the following code: console.log("Hello");
for(var i = 0;i<4;i++) {
    console.log(Hello);
} The output shows the interpreter behavior in the above code example, where first is printed to the console, and then an error is reported. This output strongly supports the fact that JavaScript is an interpreted language. Hello While it may seem like JavaScript code is being executed line by line, this is only true during the parsing phase. In reality, the entire code is compiled at once to convert it into machine-readable code before execution. Therefore, JavaScript is a just-in-time compiled language that uses an interpreter in its first phase. Note: JavaScript can operate in an interpreted manner in older browsers. However, every modern browser currently supports "JIT", so JavaScript code is always compiled. Whether JavaScript is compiled or interpreted depends on the environment in which it is run. If it runs in older browsers, it's interpreted. If it runs in modern browsers, it's compiled. Explaining JIT in JavaScript Now that we have a basic understanding of how JavaScript works and what JIT compilers are, let's delve deeper into how JIT compilers work and why they're important. But don't worry, we won't get too technical with jargon. Instead, we'll explore JIT compilers in simple terms to understand how they work, why they're important, and the reason they were introduced in the first place. JIT (just-in-time) compilation is a technique used by modern JavaScript engines to improve the performance of JavaScript code. Unlike traditional compilers that optimize code during compilation, JIT compilers take a middle-ground approach. They initially interpret the code and then selectively compile parts of it that are used repeatedly. This approach balances fast start-up time with improved performance. When a JavaScript engine encounters a function, it can either interpret the code on-the-fly or compile the code before execution. Interpreting the code can be faster to start execution but slower in terms of overall performance, as the code is translated line-by-line each time the function is executed. Compiling the code before execution can be slower to start, but it can result in faster execution due to machine code optimizations. In this approach, the code is compiled into machine code, which is then executed by the processor. JIT compilers take a middle-ground approach, where they initially interpret the code and then selectively compile parts of it that are used repeatedly. As a result, JIT compilers offer a balance of fast start-up time and improved performance. To understand how JIT compilers work, let's consider an example: function add(a, b) {
  return a + b;
}

for (let i = 0; i < 100000000; i++) {
  add(i, i + 1);
} In this code, we have a simple function that adds two numbers and a loop that calls this function million times with different inputs. add 100 When the JavaScript engine first encounters this code, it will interpret it on-the-fly and execute it. However, as the loop runs repeatedly, the engine detects that the function is being called repeatedly and decides to compile it to machine code to improve performance. add The compiler analyzes the code and optimizes it for the specific inputs being used in the loop. JIT For example, it may decide to inline the function, which means replacing the function call with the actual addition code, eliminating the overhead of calling a function. Additionally, it may also perform other optimizations such as constant folding or dead-code elimination, which remove unnecessary computations. add Overall, JIT compilers help improve the performance of JavaScript code by selectively compiling parts of it that are used repeatedly, while still allowing for fast start-up times. Now that we have a better understanding of how the JIT compiler works, let's delve into the issues we previously discussed and their underlying causes. Why the Pre-execution syntax error alerts Let's explore the reason behind syntax error alerts in JavaScript using the same example we discussed earlier. console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World from javascript!");
 console.log("Hello World!); In the case of the code example provided, the JavaScript engine compiles the code before execution, and the syntax error is detected during the compilation phase. This is because the engine uses the compiler that optimizes code on the fly as it's executed. When the engine encounters an error during the compilation phase, it immediately throws an error without executing any code. JIT Therefore, the fact that JavaScript detects syntax errors during the compilation phase suggests that it's more like a compiled language than an interpreted one. What is the story of hoisting? Let's now understand the hoisting in the context of engine using the code we used before: JIT max(1, 2);
// 2
function max(num1, num2){
  return num1 > num2 ? num1 : num2;
} So, how does the JavaScript engine know about the function before it reaches the declaration? max The answer lies in the Just-In-Time ( ) compilation process that occurs behind the scenes when JavaScript code is executed in modern browsers. JIT During the optimization step, the compiler can analyze the code to determine which functions are likely to be called frequently and which variables are likely to be accessed repeatedly. By identifying these patterns, the compiler can optimize the code for faster execution, which can result in significant performance gains. JIT In the case of hoisting, the compiler can recognize the pattern of a function being declared at the top of its scope and optimize the code accordingly. Conclusion In summary, while JavaScript is commonly thought of as an interpreted language, it is actually a Just-In-Time compiled language. Modern JavaScript engines use a compiler to optimize the code for execution. This compiler allows JavaScript to be executed much faster and in an efficient way than traditional interpreted languages. While the compiling of Javascript works in a different way, if compared to other compiled languages, it still follows some rules that reflect the process of compiling. JIT JavaScript code is parsed before execution, which makes it look like a parsed language, but the code is actually converted to binary(machine code that is directly executed by the computer's hardware) form before execution. This conversion process involves several steps, including transpiling the code, parsing it to an Abstract Syntax Tree ( ), converting it to bytecode, and optimizing it with a compiler. AST JIT So, to answer the question the answer is that it is a bit of both. It is interpreted in older browsers, but in modern browsers, it is compiled with the help of a compiler. "Is javascript compiled or interpreted language?" JIT Also published . here

Is Javascript a Compiled or an Interpreted Language?

183 Stories To Learn About Engineering

Recently, Air2phin, a scheduling system migration tool, announced its open source. With Air2phin, users can migrate the scheduling system from Airflow to Apache DolphinScheduler in two steps, which brings great convenience to users who need to migrate the scheduling system. What is Air2phin? Air2phin is a scheduling system migration tool, which aims to convert Apache Airflow DAGs files into Apache DolphinScheduler Python SDK definition files, to migrate the scheduling system (Workflow orchestration) from Airflow to DolphinScheduler. It is a multi-rule-based AST converter that uses LibCST to parse and convert Airflow's DAG code. All its rules are defined using Yaml files and provide certain custom rule extension capabilities. Recently, Air2phin has released version 0.0.12, which provides rich functions and can better help users complete the migration from Airflow to Apache DolphinScheduler. Note: AST is the abbreviation of Abstract Syntax Tree (Abstract Syntax Tree), which is a data structure that represents the grammatical structure of code in a tree structure. In a compiler, the AST is produced by a lexer and a parser. The lexical analyzer converts the source code into a stream of tokens, and the parser converts the stream of tokens into an abstract syntax tree. AST is a tree structure, which consists of a series of nodes, each node represents a grammatical structure in the code (such as expressions, statements, functions, classes, etc.), and the relationship between nodes represents the embedding of grammatical structures. set of relationships. Why open-source Air2phin? Some people may ask, why do I need a migration tool? This is because with the development of business, the original workflow orchestration system used by enterprises or organizations can no longer meet the current needs, and the workflow orchestration system needs to be migrated to a new platform or updated to a new version. After research, many users need to migrate the scheduling system from the open-source workflow orchestration system Airflow to Apache DolphinScheduler. During the migration process, since data processing tasks may involve dependencies between multiple systems, the migration process needs to be completed without affecting business operations. At this time, the dispatching system migration tool can play an important role. It can reduce manual intervention, complete the migration work between the two dispatching systems as automatically as possible, and can be compatible with multiple versions between multiple systems. Intervention completes the migration. To this end, WhaleOps has specially developed the open-source migration tool Air2phin, which allows users to migrate the scheduling system from Airflow to Apache DolphinScheduler in two steps, bringing great convenience to users. To let everyone better understand the importance of Air2phin, let's start with the background knowledge of the scheduling system and understand the benefits of migrating the scheduling system from Airflow to Apache DolphinScheduler. Why migrate from Airflow to DolphinScheduler? What is a workflow orchestration system? A workflow orchestration system manages data flow in a manner that respects orchestration rules and business logic. Workflow orchestration tools allow users to convert multiple related tasks into workflows that can be scheduled, run, and observed, helping companies better manage and control business processes, thereby improving business efficiency. Workflow orchestration is one of the indispensable components in data processing. It is responsible for executing data processing tasks according to predefined rules and logic to ensure that the data processing process is executed smoothly as expected. Popular workflow orchestration systems include Apache DolphinScheduler, Apache Airflow, Apache Oozie, Azkaban, etc. What is Airflow? Among them, Apache Airflow is an open-source workflow orchestration system, which can help users create, schedule and monitor complex workflows. Originally developed by Airbnb and open-sourced in 2016, Airflow is now maintained by the Apache Software Foundation. Written in the Python language, Airflow is highly scalable and flexible and supports a variety of task types, such as calculations, data processing, notifications, interactions, and more. Airflow's workflow is defined by writing Python scripts, and its functionality can be extended using operators and hooks provided by Airflow, as well as custom operators and hooks. However, it has flaws that cannot be ignored, such as the need for in-depth secondary development, the high cost of upgrading from the community version; the high iteration cost of Python technology stack maintenance; scheduler loop scanning Dag folder delays reduces the performance; and poor stability in production environments and so on. Apache DolphinScheduler, born under the business needs of the new data era, is an open-source distributed workflow scheduling system, which makes up for the weaknesses of previous scheduling systems, and aims to provide enterprise users with a reliable, efficient, and easy-to-use workflow scheduling platform. Multiple task types are supported, such as computing, data processing, ETL, etc. Compared with Airflow, DolphinScheduler adopts a distributed architecture and provides a variety of task types. Users can define dependencies between tasks, set task priorities and schedule policies, etc. It uses a visual interface to create and manage workflows, which is in stark contrast to Airflow, making it easier to operate and more friendly to non-programmers. After research and comparison, for many users, migrating the scheduling system to Apache DolphinScheduler is a better choice to reduce costs and increase efficiency. How to install and use Air2phin Air2phin is a python package, which can be installed through the Python package installation tool pip, see air2phin getting start for details. python -m pip install --upgrade air2phin A simple example We use a simple example to illustrate how to use Air2phin. We have intercepted part of the code in airflow tutorial.py as an example of Air2phin conversion to illustrate how Air2phin can be converted into dolphinscheduler python sdk step by step. picture Figure 1: Part of the code in airflow tutorial.py Figure 2: How Air2phin is converted into dolphinscheduler python sdk step by step Suppose you save part of the content of airflow tutorial.py to the file tutorial_part.py, and want to convert it into the definition of dolphinscheduler python sdk, you only need one line of command to complete it. The result is shown in Figure 2. Because the command adds the --inplace parameter, Air2phin will directly overwrite the original file. If you don't need to overwrite the original problem, you don't need to use the --inplace parameter, and Air2phin will add a tutorial_part-air2phin.py file to save the converted content. air2phin migrate --inplace tutorial_part.py We can see that this conversion triggers multiple conversion rules, including Convert airflow.DAG to pydolphinscheduler.core.process_definition.ProcessDefinition, this rule is in the third line (import statement) and the sixth line DAG context Convert airflow.operators.bash.BashOperator to pydolphinscheduler.tasks.shell.Shell, this rule is used in tasks t1, t2 In addition to the corresponding class conversion, we need to convert the attributes of the class, such as converting airflow.DAG.schedule_interval to ProcessDefinition.schedule, and modify the content of some values, such as converting timedelta(days=1) to' 0 0 0 * * ? *' Finally, we only need to install pydolphinscheduler and run the converted file through python to complete the workflow migration. For details, see pydolphinscheduler usage (https://dolphinscheduler.apache.org/python/main/start.html# installing-pydolphinscheduler). # Install apache-dolphinscheduler python -m pip install apache-dolphinscheduler # Submit workflow to dolphinscheduler python tutorial_part.py When running python tutorial_part.py, we must ensure that the dolphinscheduler API and python gateway service have been started, and the corresponding ports have been opened. For details, see Starting python gateway service. So far, we have explained how Air2phin completes the migration through a simple example. Working principle of Air2phin How do Airflow and dolphinscheduler python sdk work? Before understanding how Air2phin works, it is very important to understand how Airflow and dolphinscheduler python sdk work. It will help us better understand the migration steps of Air2phin and deal with problems more calmly. The information related to Airflow workflow is stored in the DAG file, and then the DAG file is placed in the specified directory of Airflow, and the Scheduler of Airflow will scan and analyze the DAG file of Airflow at a certain interval, so the DAG file is passively Scanned and updated. How Airflow works: Similar to Airflow, all workflow-related information is defined through Python files, but dolphinscheduler python sdk submits workflow information through artificial triggering and runs the command python workflow file name to complete the active Task submission. dolphinscheduler python sdk: Air2phin workflow After understanding the submit/discover workflow mechanism, it will be more conducive to our understanding of the working principle of Air2phin. Because both Airflow's DAG file and DolphinScheduler's Python sdk definition file are written in Python, most of Air2phin's code deals with the differences between the two, and finally converts Airflow's code into dolphinscheduler python sdk and definitions. Air2phin uses LibCST (https://libcst.readthedocs.io/en/latest/) to implement the abstract syntax tree parsing of airflow python DAG code, and then through LibCST's Transformer (https://libcst.readthedocs.io/en/ latest/tutorial.html#Build-Visitor-or-Transformer) combined with the transformation rules and finally transformed into the definition of dolphinscheduler python sdk. The overall workflow of Air2phin is as follows: Get the original Airflow DAG content from stdin or a file Load all transformation rules from yaml file Parse Airflow DAG content into a CST tree via LibCST Transform dolphinscheduler python sdk definitions via LibCST Transformer Air2phin Best Practices Migrate entire folders instead of individual files When users want to migrate Airflow to DolphinScheduler, most of them want to migrate the whole system instead of a single file. Air2phin provides the ability to migrate the entire folder by just changing the path from the file path to a folder. # Migrate the entire ~/airflow/dags folderair2phin migrate --inplace ~/airflow/dags Add custom rules Some of the Airflow users self-define Hooks or Operator. User-defined Operators cannot be converted through Air2phin's built-in conversion rules, so users need to add custom rules and tell Air2phin the location of the rules. For example, suppose we have an operator called MyCustomOperator that inherits most of the functions of PostgresOperator, with only a different name. Its definition is as follows: from airflow.providers.postgres.operators.postgres import PostgresOperator class MyCustomOperator(PostgresOperator): def __init__( self, *, sql: str | Iterable[str], my_custom_conn_id: str = 'postgres_default', autocommit: bool = False, parameters: Iterable | Mapping | None = None, database: str | None = None, runtime_parameters: Mapping | None = None, **kwargs, ) -> None: super().__init__( sql=sql, postgres_conn_id=my_custom_conn_id, autocommit=autocommit, parameters=parameters, database=database, runtime_parameters=runtime_parameters, **kwargs, ) It is used in multiple DAGs in Airflow in the following way: from custom.my_custom_operator import MyCustomOperator with DAG( dag_id='my_custom_dag', default_args = default_args, schedule_interval='@once', start_date=days_ago(2), tags=['example'], ) as dag: t1 = MyCustomOperator( task_id='my_custom_task', sql='select * from table', my_custom_conn_id='my_custom_conn_id', ) Now we need to convert this Operator. We can customize a conversion rule and name it MyCustomOperator.yaml. The content is as follows. The main content is the definition of migration.module and migration.parameter, which determine the conversion rule: name: MyCustomOperator description: The configuration for migrating airflow custom operator MyCustomOperator to DolphinScheduler SQL task. migration: module: - action: replace src: custom.my_custom_operator.MyCustomOperator dest: pydolphinscheduler.tasks.sql.Sql parameter: - action: replace src: task_id dest: name - action: replace src: my_custom_conn_id dest: datasource_name Then use the --custom-rules parameter to specify conversion custom parameters, and the conversion of custom rules can be applied: # Specify the custom rule path as /path/to/MyCustomOperator.yaml air2phin migrate --inplace --custom-rules /path/to/MyCustomOperator.yaml ~/airflow/dags Make Air2phin run faster By default, Air2phin runs the conversion of DAG files in one process. When you have many DAG files, Air2phin conversion is very time-consuming. We provide a parameter to start multi-process to run Air2phin conversion --multiprocess, which can be specified as the user machine's CPU amount to reduce conversion time: # Specify air2phin to start 12 processes to convert at the same time air2phin migrate --inplace --custom-rules /path/to/MyCustomOperator.yaml --multiprocess 12 ~/airflow/dags FAQ Q: Why did you choose to parse Airflow DAG files instead of databases? A: Because the completed workflow information is only in the Airflow DAG file, the Airflow database only has basic workflow information, no task definition information, and no task relationship. We choose to complete it by parsing the Airflow DAG file instead of the database transform. Q: Why do I need to transfer through the dolphinscheduler python sdk instead of submitting to DolphinScheduler myself? A: Because Airflow DAG is defined by Python, there are many Python features in Airflow DAG, we don't want to convert these features into structured data (the conversion may cause information loss), it happens that DolphinScheduler already has Python sdk, so It is easier to convert directly through LibCST. Q: Why use LibCST instead of python's built-in AST? A: Because LibCST is more suitable for us, the comment information will be lost when the Python built-in AST library is parsed into AST, but we hope to keep some information. And LibCST provides more visitors to ensure that we can implement replacement more conveniently. Reference: air2phin (https://github.com/WhaleOps/air2phin) Also Published Here

How to Migrate from Airflow to Dolphinscheduler in Two Steps

Hello everyone, in the last article I introduced the Turtle Graphics Android App project with implementation details and resources about the scripting language, editor, generating documentation…etc, after I published the app and got more than 2000 downloads a few times and good ratings and feedback, I decided to add support for code formatting and in this article, I will talk in detail how a simple code formatter works and how I implemented it in the Turtle Graphics app. As programmers, Code formatters are an essential tool in our day-to-day jobs, They make it easier to read the code if it is formatted, but did you ask yourself how it works? Before talking about Code formatter, let’s first talk about how Compilers represent your code from text to data structure to do the process on it such as type checking. Let’s start our story from your file that contains a simple hello world example fun main() {
   print("Hello, World!")
} The first step is to read this text file and convert it into a list of tokens, A token is a class that represents a keyword, number, bracket, string, …etc with this position in the source code for example data class Token (
   val kind : TokenKind,
   val literal : String,
   val line : Int,
) We can also save the file name, column start and end so when we want to report an error we can provide useful info about the position for example Error in File Main Line 10: Missing semicolon :D This step is called scanner, lexer or tokenizer and at the end, we will end up with a List of tokens for example { FUN_KEYWORD, "fun", 1 }
{ IDENTIFIER, "main", 1 }
{ LEFT_PAREN, "(", 1 }
{ RRIGHT_PAREN, ")", 1 }
{ LEFT_BRACE, "{", 1 }
{ IDENTIFIER, "print", 2 }
{ LEFT_PAREN, "(", 2 }
{ STRING, "Hello, World!", 2 }
{ RRIGHT_PAREN, ")", 2 }
{ RIGHT_BRACE, "}", 3 } The result is a list of tokens val tokens : List<Token> = tokenizer(input) Note that in this step we can check for some errors such as unterminated string or char, unsupported symbols …etc After this step, you will forget your text file and deal with this list of tokens, and now we should convert some tokens into nodes depending on our language grammar for when we saw FUN_KEYWORD that means we will build a function declaration node and we expect name, paren, parameters …etc In this step, we need a data structure to represent the program in a way we can traverse and validate it later and it is called Abstract Syntax Tree (AST), each node in AST represent statement such as If, While, Function declaration, var declaration …etc or expressions such as assignments, unary …etc, each node store required information to use them later in the next steps for example Function Declaration data class Function (
   var name : String,
   var arguments : List<Argument>,
   var body : List<Statement>
) Variable Declaration data class Var (
   var name : String
   var value : Expression
) This step is called parsing and we will end up with an AST object that we can use later to traverse all nodes. var astNode = parse(tokens) If the language statically types such as Java, C, Go …etc we will go to the Type Checker step, the goal for this step is to check that the user use type correctly for example, if the user declares a variable with int type it should store only integers on it, the if condition must be a boolean type or an integer in a language like C …etc After this step, we will end up with the same AST node but now we know that it is valid and we can now compile it to any target or evaluate it, But also we can do the formatting, static analysis, optimization, check code style …etc For example suppose that we want all developers to declare variables without using _ inside the name, to check that we will traverse our AST node to find all Var nodes and check them fun checkVarDeclaration(node : Var) {
   if (node.name.contains("_") {
      reportError("Ops your variable name ${node.name} contains _")
   }
} But now we need to format it, so how to do that? It's the same we traverse our AST and for each node, we will write it back to text but formatted for example fun formatVarDeclaration(node : Var) : String {
   var builder = StringBuilder()
   builder.append(indentation)
   builder.append("var ")
   builder.append(node.name)
   builder.append(" = ")
   builder.append(formatValue(node.value))
   builder.append("\n")   
   return builder.toString()
} In this simple method, we rewrite the node to string but with correct indentations and add a new line after it so now 2 variables are declared in the same line, the value also is formatted using another function you can use Visitor design pattern to make it easy to handle all nodes. At the end of this step, we end up with a string that represents the same input file but formatted and then we write it back to the file. This is the basic implementation of code formatter, a real production code formatter must handle more cases, for example, what if the code is not valid?, should I format only valid code? should we read the whole program every time we want to format or compile the code? Now back to Turtle graphics, In this project i already did all the required steps before and has a ready AST, so i just rewrite it with code as you saw above ^_^ i read it from the UI format it and write it back to UI in my case If you are interested and want to read more I suggest Read at least one Compiler book such as Crafting interpreters Read about Language Server Protocol (LSP) Watch Typescript Compiler explained by the Author Anders Hejlsberg Think if you have Your program as AST what else you can do with it I hope you enjoyed my article and you can find me on GitHub Linkedin Twitter Enjoy Programming 😋.

How is a Code Formatter Implemented in Turtle Graphics

Welcome to other chapters of Let’s Understand Chrome V8 In this article, I will talk about how the Parse, AST, and Token work together, and walk you through the compilation as well as watch and see the detail. Specifically, you will see how to scan tokens, and how Parse consumes tokens for generating an AST tree. 1. Parse_Info Below is the Parse_Info class. It’s the kernel class of the Parser, that manages JavaScript source code and an AST tree. Simply, the input of the Parser is JavaScript code and the output is an AST tree. But note that from the moment V8 loaded the JavaScript source code, the code has been encoded as UTF-16 already. Let’s step into the Parse_Info. 1.  // A container for the inputs, configuration options, and outputs of parsing.
2.  class V8_EXPORT_PRIVATE ParseInfo {
3.  public:
4.  //omit...
5.  AstValueFactory* ast_value_factory() const {
6.    DCHECK(ast_value_factory_.get());
7.    return ast_value_factory_.get();
8.  }
9.  const AstRawString* function_name() const { return function_name_; }
10.  void set_function_name(const AstRawString* function_name) {
11.    function_name_ = function_name;
12.  }
13.  FunctionLiteral* literal() const { return literal_; }
14.  void set_literal(FunctionLiteral* literal) { literal_ = literal; }
15.  private:
16.   //------------- Inputs to parsing and scope analysis -----------------------
17.   const UnoptimizedCompileFlags flags_;
18.   UnoptimizedCompileState* state_;
19.   std::unique_ptr<Zone> zone_;
20.     v8::Extension* extension_;
21.     DeclarationScope* script_scope_;
22.     uintptr_t stack_limit_;
23.     int parameters_end_pos_;
24.     int max_function_literal_id_;
25.     //----------- Inputs+Outputs of parsing and scope analysis -----------------
26.     std::unique_ptr<Utf16CharacterStream> character_stream_;
27.     std::unique_ptr<ConsumedPreparseData> consumed_preparse_data_;
28.     std::unique_ptr<AstValueFactory> ast_value_factory_;
29.     const AstRawString* function_name_;
30.     RuntimeCallStats* runtime_call_stats_;
31.     SourceRangeMap* source_range_map_;  // Used when block coverage is enabled.
32.     //----------- Output of parsing and scope analysis ------------------------
33.     FunctionLiteral* literal_;
34.     bool allow_eval_cache_ : 1;
35.   #if V8_ENABLE_WEBASSEMBLY
36.     bool contains_asm_module_ : 1;
37.   #endif  // V8_ENABLE_WEBASSEMBLY
38.     LanguageMode language_mode_ : 1;
39.   }; In the above code, line 5 is the AST factory method; Line 13 the literal returns the AST tree; Line 17 the flags_ shows the reason why the function is not optimized; Line 33 the literal_ is responsible for holding the completed AST tree. As I mentioned the AST is an output of the parser, so if you want to watch the workflow of the parser in detail, you just debug it and keep watching the member variable literal_, in this way you are going to see the AST generation and how the factory method is called. 2. AST Tree As I mentioned above, the variable literal_ represents an AST tree, which its declaration class is the FunctionLiteral. 1.  class FunctionLiteral final : public Expression {
2.   public:
3.    template <typename IsolateT>//省略很多代码...............
4.    MaybeHandle<String> GetName(IsolateT* isolate) const {  }
5.    const AstConsString* raw_name() const { return raw_name_; }
6.    DeclarationScope* scope() const { return scope_; }
7.    ZonePtrList<Statement>* body() { return &body_; }
8.    bool is_anonymous_expression() const {
9.      return syntax_kind() == FunctionSyntaxKind::kAnonymousExpression;
10.    }
11.    V8_EXPORT_PRIVATE LanguageMode language_mode() const;
12.    void add_expected_properties(int number_properties) {}
13.    std::unique_ptr<char[]> GetDebugName() const;
14.    Handle<String> GetInferredName(Isolate* isolate) {  }
15.    Handle<String> GetInferredName(LocalIsolate* isolate) const {}
16.    const AstConsString* raw_inferred_name() { return raw_inferred_name_; }
17.    FunctionSyntaxKind syntax_kind() const {}
18.    FunctionKind kind() const;
19.    bool IsAnonymousFunctionDefinition() const {  }
20.    int suspend_count() { return suspend_count_; }
21.    int function_literal_id() const { return function_literal_id_; }
22.    void set_function_literal_id(int function_literal_id) {  }
23.   private:
24.    const AstConsString* raw_name_;
25.    DeclarationScope* scope_;
26.    ZonePtrList<Statement> body_;
27.    AstConsString* raw_inferred_name_;
28.    Handle<String> inferred_name_;
29.    ProducedPreparseData* produced_preparse_data_;
30.  }; I would emphasize that the AST granularity is one function, in other words, a JavaScript function corresponds to only one AST tree. In the above code, line 26 member variable body_ holds the AST body, go through the body_ if you wonder about the details. Lines 4–15 set language mode as strict or sloppy. Let’s look more at line 18 FunctionKind. 1.  enum FunctionKind : uint8_t {
2.    // BEGIN constructable functions
3.    kNormalFunction,kModule,kAsyncModule,kBaseConstructor,kDefaultBaseConstructor,
4.    kDefaultDerivedConstructor,
5.    kDerivedConstructor,
6.  //omit....
7.    kLastFunctionKind = kClassStaticInitializerFunction,
8.  }; You should see that the function kinds are not the types in the JavaScript books because the function kinds just represent the function kinds in a compiler. See compiler books for details. 3. AST Node Type The following macro templates are non-exhaustive node type lists. 1.  #define DECLARATION_NODE_LIST(V) \
2.    V(VariableDeclaration)         \
3.    V(FunctionDeclaration)
4.  #define ITERATION_NODE_LIST(V) \
5.    V(DoWhileStatement)          \
6.    V(WhileStatement)            \
7.    V(ForStatement)              \
8.    V(ForInStatement)            \
9.    V(ForOfStatement)
10.  #define BREAKABLE_NODE_LIST(V) \
11.    V(Block)                     \
12.    V(SwitchStatement)
13.  #define STATEMENT_NODE_LIST(V)       \
14.    ITERATION_NODE_LIST(V)             \
15.    BREAKABLE_NODE_LIST(V)             \
16.    V(ExpressionStatement)             \
17.    V(EmptyStatement)                  \
18.    V(SloppyBlockFunctionStatement)    \
19.    V(IfStatement)                     \
20.    V(InitializeClassStaticElementsStatement)//omit
21.  #define LITERAL_NODE_LIST(V) \
22.    V(RegExpLiteral)           \
23.    V(ObjectLiteral)           \
24.    V(ArrayLiteral)
25.  #define EXPRESSION_NODE_LIST(V) \
26.    LITERAL_NODE_LIST(V)          \
27.    V(Assignment)                 \
28.    V(Await)                      \
29.    V(BinaryOperation)            \
30.    V(NaryOperation)              \
31.    V(Call)                       \
32.    V(YieldStar)//omit
33.  #define FAILURE_NODE_LIST(V) V(FailureExpression)
34.  #define AST_NODE_LIST(V)                        \
35.    DECLARATION_NODE_LIST(V)                      \
36.    STATEMENT_NODE_LIST(V)                        \
37.    EXPRESSION_NODE_LIST(V) In the above code, see line 34, it organizes all the templates together to generate an AST tree for your JavaScript function in the parser which is a part of the compiler pipeline. By the macro AST_NODE_LIST, we should see that there are three types in an AST, and every type also has more subtypes. All of these types correspond to our JavaScript code and represent semantics accurately. 4. Token Type The following macro templates are non-exhaustive token lists. 1.  #define TOKEN_LIST(T, K)                                           \
2.    T(TEMPLATE_SPAN, nullptr, 0)                                     \
3.    T(TEMPLATE_TAIL, nullptr, 0)                                     \
4.    /* BEGIN Property */                                             \
5.    T(PERIOD, ".", 0)                                                \
6.    T(LBRACK, "[", 0)                                                \
7.    /* END Property */                                               \
8.    /* END Member */                                                 \
9.    T(QUESTION_PERIOD, "?.", 0)                                      \
10.    T(LPAREN, "(", 0)                                                \
11.    /* END PropertyOrCall */                                         \
12.    T(RPAREN, ")", 0)                                                \
13.    T(RBRACK, "]", 0)                                                \
14.    T(LBRACE, "{", 0)                                                \
15.    T(COLON, ":", 0)                                                 \
16.    T(ELLIPSIS, "...", 0)                                            \
17.    T(CONDITIONAL, "?", 3)                                           \
18.  //omit. The token is used in the scanner which is a part of the V8 compiler, the scanner is responsible for converting the JavaScript code to tokens. Let’s say a simple example, see line 5, the token type is PERIOD and corresponds to the dot in JavaScript code, which means that the scanner will convert every dot that it sees to a PERIOD token. Samely, lines 10 and 12 are light paren and right paren, which are used to describe the and in JavaScript code. So in every line, in every macro which starts with a T, the first parameter is the token type, the second is lexical corresponding to the type, and the last number is the token priority, the number is smaller, the token is higher, and will be converted early. ( ) 5. Finite-state Automation In most compilers, the lexer and parser use the FSA(finite-state automation) to analyze the source code. In V8, it uses the switch-case to implement the FSA, let’s have a look at the lexer FSA, namely the token scanner. 1.  V8_INLINE Token::Value Scanner::ScanSingleToken() {
2.   Token::Value token;
3.   do {
4.     next().location.beg_pos = source_pos();
5.     if (V8_LIKELY(static_cast<unsigned>(c0_) <= kMaxAscii)) {
6.       token = one_char_tokens[c0_];
7.       switch (token) {
8.         case Token::LPAREN:
9.         case Token::RPAREN:
10.          case Token::LBRACE:
11.          case Token::RBRACE:
12.          case Token::LBRACK:
13.          case Token::RBRACK:
14.          case Token::COLON:
15.          case Token::SEMICOLON:
16.          case Token::COMMA:
17.          case Token::BIT_NOT:
18.          case Token::ILLEGAL:
19.            // One character tokens.
20.            return Select(token);
21.          case Token::CONDITIONAL:
22.            // ? ?. ?? ??=
23.            Advance();
24.            if (c0_ == '.') {
25.              Advance();
26.              if (!IsDecimalDigit(c0_)) return Token::QUESTION_PERIOD;
27.              PushBack('.');
28.            } else if (c0_ == '?') {
29.              return Select('=', Token::ASSIGN_NULLISH, Token::NULLISH);
30.            }
31.            return Token::CONDITIONAL;
32.  //omit
33.          case Token::WHITESPACE:
34.            token = SkipWhiteSpace();
35.            continue;
36.          case Token::NUMBER:
37.            return ScanNumber(false);
38.          case Token::IDENTIFIER:
39.            return ScanIdentifierOrKeyword();
40.          default:
41.            UNREACHABLE();
42.        }
43.      }
44.      if (IsIdentifierStart(c0_) ||
45.          (CombineSurrogatePair() && IsIdentifierStart(c0_))) {
46.        return ScanIdentifierOrKeyword();
47.      }
48.    } while (token == Token::WHITESPACE);
49.    return token;
50.  } The ScanSingleToken() is responsible for generating a token. In lines 8–40, every case is a token. The following code is the parser FSA, which consumes tokens and generates nodes for an AST tree. 1.  ParserBase<Impl>::ParseStatementListItem() {
2.    switch (peek()) {
3.      case Token::FUNCTION:
4.        return ParseHoistableDeclaration(nullptr, false);
5.      case Token::CLASS:
6.        Consume(Token::CLASS);
7.        return ParseClassDeclaration(nullptr, false);
8.      case Token::VAR:
9.      case Token::CONST:
10.        return ParseVariableStatement(kStatementListItem, nullptr);
11.      case Token::LET:
12.        if (IsNextLetKeyword()) {
13.          return ParseVariableStatement(kStatementListItem, nullptr);
14.        }
15.        break;
16.      case Token::ASYNC:
17.        if (PeekAhead() == Token::FUNCTION &&
18.            !scanner()->HasLineTerminatorAfterNext()) {
19.          Consume(Token::ASYNC);
20.          return ParseAsyncFunctionDeclaration(nullptr, false);
21.        }
22.        break;
23.      default:
24.        break;
25.    }
26.    return ParseStatement(nullptr, nullptr, kAllowLabelledFunctionStatement);
27.  } In end, I would argue that FSA is an important fundamental in a compiler, which is worth examining in a little more depth. Okay, that wraps it up for this share. I’ll see you guys next time, take care! Please reach out to me if you have any issues. : qq9123013 : WeChat Email v8blink@outlook.com This story was first published here.

Understaning Chrome V8 - Chapter 23:  Compiler Workflow Parse, AST, and Token 

How to Build a WebAssembly Language for Fun and Profit

(wasm) is a high-performance assembly-like format optimized for the web. Code targeting WebAssembly can run at near-native speeds while still benefiting from the safe environment of a browser VM. Wasm has opened up a whole new world of demanding desktop-class apps that can comfortably run in the browser. For example, AutoCAD was able to to the web using wasm. Cases like AutoCAD’s have made it clear that wasm will be a major disruptive force in how web apps are developed. WebAssembly port decades of code To facilitate the adoption of wasm, WebAssembly team developed a powerful compiler toolchain library called binaryen. Binaryen does a huge amount of heavy lifting for compiler authors. It offers dead code removal, code size reduction, and various performance optimizations out of the box. As someone who has long been interested in programming languages, this piqued my interest. Writing compiled languages has always been a daunting task. What I found is binaryen made it incredibly fun and easy to build new languages that are shockingly speedy. That's why I decided to create this guide and provide a simple overview designed to help get your feet wet in building languages and exploring the inner workings of wasm. Here's a quick taste of the lisp inspired language we'll build, wispy: (fn fib:i32 [val:i32]
  (if (lt_i32 val 2)
    val
    (add_i32 (fib (sub_i32 val 1)) (fib (sub_i32 val 2)))))

(fn main:i32 [] (fib 15)) This simple function calculates values of the , a sequence of numbers that appears in surprising places through mathematics and nature. It's one of my favorite illustrations of how elegantly patterns of the universe can be described in code. fibonacci sequence This guide is designed for intermediate to advanced software developers looking for a fun side project to challenge themselves with. By the end, we’ll have built a working compiler and runtime for wispy. The guide will be broken down into three articles: (this article): the process of converting the characters of code into meaningful units called tokens Setup and Lexing the process of converting the tokens into a logical tree known as an AST. Parsing: the process of converting the AST into the binary instructions run by our computer Compiling (or code generation): Setup In this guide, we will be using TypeScript and NodeJS. The concepts are highly portable, so feel free to use the environment you're most comfortable with. Our only major dependency, , has a simple C API. You are welcome to skip ahead to the next section if you're using a different language. binaryen Requirements: NodeJS v16+ Git Quick Start git clone git@github.com:drew-y/wispy.git
cd wispy
git checkout quickstart
npm i Manual Setup I've included manual setup instructions as an alternative to the quick start, in case you want to know exactly how the project was set up or just like doing things from scratch. If you've already done the quick start, skip to the next section. Open a terminal window and make a new directory: mkdir wispy
cd wispy Initialize package.json: npm init -y # Be sure to have NodeJS 16+ installed Install the project dependencies: npm i @types/node binaryen typescript Add these two fields to the package.json "type": "module", // Binaryen uses the new module format so we must follow suit
"bin": {
    "wispy": "dist/index.mjs" // This will allow us to easily run the compiler from our terminal
}, Create a tsconfig file: npx tsc init . 6. Set the following fields in tsconfig.json: "module": "ES2022",
"rootDir": "./src",
"moduleResolution": "node",
"outDir": "./dist" Lexing Lexing is the process of digesting each individual character of our program into a set of tokens. A token is a group of characters that take on a special meaning when put together. Take the following snippet of wispy: (add 1 2) There are five unique tokens in that snippet (, add, 1, 2 and ). The lexer's job is simply to identify and list those tokens in order. Lexing is typically the first step in turning human readable code into something closer to what a computer can understand. Defining Our Tokens We'll start by defining our tokens in a new file: #mts extension is important, it tells typescript to create a corresponding mjs file, so Node knows to use modules

mkdirp -p src/types/token.mts First up is the IntToken. This token represents whole numbers like 1045: // src/types/token.mts
export type IntToken = { type: "int"; value: number }; Next up is the FloatToken. This token represents numbers that may have a decimal, like 1.8: // src/types/token.mts
export type FloatToken = { type: "float"; value: number };

/** Previously defined tokens omitted for brevity */ Now, let's define some identifier tokens. In wispy, an identifier can represent either the name of a function, or the name of a function parameter. We have two types of identifier tokens, a standard IdentifierToken and a TypedIdentifierToken. ' An IdentifierToken is used in the body of a function to refer to the function's parameters or to call another function. A TypedIdentifierToken is used when defining a function or a parameter. Typed identifiers take the form identifier:type. For example, val:i32 defines a parameter that is a 32-bit integer. When defining a function, the type represents the function's return type. For example, fib:i32 is a function that returns a 32-bit integer. Here are the definitions: // src/types/token.mts
export type IdentifierToken = { type: "identifier"; value: string };
export type TypedIdentifierToken = { type: "typed-identifier"; value: string };

/** Previously defined tokens omitted for brevity */ Up next is BracketToken. Wispy uses , like lisp. So brackets are very important. To keep things simple we allow two kinds of brackets () and []. To keep things even more simple the compiler will treat () and [] as interchangeable. In actual use we will only use [] to define parameters. S-expression syntax // src/types/token.mts
export type BracketToken = { type: "bracket"; value: Bracket };
export type Bracket = "(" | ")" | "[" | "]";

/** Previously defined tokens omitted for brevity */ Finally, we define the top level Token type: // src/types/token.mts
export type Token = BracketToken | IntToken | FloatToken | IdentifierToken | TypedIdentifierToken;
/** Previously defined tokens omitted for brevity */ Token is a . Discriminated Unions are an incredibly powerful programming language construct. They represent a value that can be one of many types. In our case, a Token can be any one of the more specific token types we defined earlier, such as IntToken or FloatToken. You'll notice that each of these tokens have a unique type field, such as type: "int" in the case of IntToken. This is the discriminator. Down the line you can pass a Token to a function and that function can use the type field to figure out which specific token it's working with. discriminated union ' At this point src/types/token.mts is finished and should look . like a file To make our new types easily accessible, export them from a new index.mts file: // src/types/index.mts
export * from "./token.mjs"; The Lex Function Now that we have our tokens defined we can write the actual lex function. The lex function will take a string (i.e. a .wispy file) and output an array of tokens (Token[]): Make a new lex file: mkdirp -p src/lexer.mts Define the lex function: // src/lexer.mts
import { Bracket, Token } from "./types/index.mjs";

export const lex = (input: string): Token[] => {
  const chars = input
    // Remove any leading or trailing whitespace for simplicity
    .trim()
    // Break up the file into single characters
    .split("");

  // This array stores our tokens
  const tokens: Token[] = [];

  // The loop continues as long as we have characters to consume
  while (chars.length) {
    // Here, a word is an unidentified token. It is usually any single group of non-whitespace
    // characters such as 123 or 123.4 or im_a_function
    const word = consumeNextWord(chars); // We'll define this function later

    // We ran out of tokens. Break out of the loop.
    if (word === undefined) break;

    const token = identifyToken(word); // We'll define this function later

    // Add the token to our store
    tokens.push(token);

}

    // Return the tokens
    return tokens;
}; Next, we define the consumeNextWord function: // src/lexer.mts

/** previous function(s) omitted for brevity */

const consumeNextWord = (chars: string[]): string | undefined => {
  const token: string[] = [];

while (chars.length) {
  // Save a preview of the current character without modifying the array
  const char = chars[0];

  // No more characters to read
  if (char === undefined) break;

  // Whitespace characters terminate the token (we'll define the isWhitespace function later)
  if (isWhitespace(char) && token.length) {
  chars.shift(); // Remove the whitespace so it doesn't get included in the next token
  break;
}

  // Discard leading whitespace characters
  if (isWhitespace(char)) {
    chars.shift();
    continue;
}

  // Terminator tokens signify the end of the current token (if any). (we'll define the isTerminatorToken function later)
  if (isTerminatorToken(char) && token.length) break;

  // Add the character to the token and discard it from the input
  token.push(char);
  chars.shift();

  // If the only token we've received so far is a single character token, that's our whole token.
  if (isTerminatorToken(char)) break;

}

  // If we have characters for our token, join them into a single word. Otherwise, return undefined to signal to the lexer
  // that we are finished processing tokens.
  return token.length ? token.join("") : undefined;
}; Now we'll define our identifyToken function. As the name suggests, this function takes a word and figures out what token that word represents. // src/lexer.mts

/** previous function(s) omitted for brevity */

const identifyToken = (word: string): Token => {
// Don't worry we'll get to all the is helper functions in a bit
  if (isInt(word)) return { type: "int", value: parseInt(word) };
  if (isFloat(word)) return { type: "float", value: parseFloat(word) };
  if (isIdentifier(word)) return { type: "identifier", value: word };
  if (isBracket(word)) return { type: "bracket", value: word };
  if (isTypedIdentifier(word)) return { type: "typed-identifier", value: word };

  throw new Error(Unknown token: ${word});
}; Finally, we define our helper functions. These functions all take a string and return true if the string passes their test, false otherwise. Most are written using regex. If you're unfamiliar with regex, I highly recommend as a resource to learn more. In a nutshell, regex is an expression syntax that's used to extract meaningful information from text. In our case, we'll use it to match words against tokens. regexone const isInt = (word: string) => /^[0-9]+$/.test(word);

const isFloat = (word: string) => /^[0-9]+.[0-9]+$/.test(word);

const isIdentifier = (word: string) => /^[a-zA-Z_][a-zA-Z0-9_-]*$/.test(word);

const isTypedIdentifier = (word: string) =>
  /^[a-zA-Z_][a-zA-Z0-9_-]:[a-zA-Z_][a-zA-Z0-9_-]$/.test(word);

const isBracket = (word: string): word is Bracket => /[()[]]/.test(word);

/** Brackets are the only terminator tokens for now */
const isTerminatorToken = (word: string): word is Bracket => isBracket(word);

// Not sure why I didn't use regex here ¯_(ツ)_/¯
const isWhitespace = (char: string) => char === " " || char ===" || char === "\t"; At this point, src/lexer.mts is finished and should look something like this file. Running the Lexer It's time to actually run the lexer. Start by making a new file src/index.mts: #!/usr/bin/env node

// src/index.mts

import { readFileSync } from "fs";
const file = process.argv[2];
const input = readFileSync(file, "utf8");
const tokens = lex(input);
console.log(JSON.stringify(tokens, undefined, 2)); Next, create an example.wispy file in the project root to compile. (fn fib:i32 [val:i32]
  (if (lt_i32 val 2)
    val
    (add_i32 (fib (sub_i32 val 1)) (fib (sub_i32 val 2)))))

(fn main:i32 [] (fib 15)) Now build the lexer: npx tsc
npm link # This will make wispy available to run as its own command Finally, run the lexer: wispy example.wispy
# Note, if npm link failed you can call our compiler directly with this as an alternative:
node dist/index.mjs example.wispy If everything goes well, wispy should output something like this: [
{
    "type": "bracket",
    "value": "("
},
{
    "type": "identifier",
    "value": "fn"
},
{
    "type": "typed-identifier",
    "value": "fib:i32"
},
{
    "type": "bracket",
    "value": "["
},
{
    "type": "typed-identifier",
    "value": "val:i32"
},
  // Omitting the rest for brevity
] With that, we have a working lexer. We can break our code down into tokens. This is a good place to break for now. In the next article, we’ll move onto parsing these tokens into a logical tree that will ultimately be converted to wasm.

How to Go About Building a WebAssembly Language for Fun and Profit 

What Are the Top Startups in Oceania?

Whenever you start wondering how to create your syntax or rules on expression, it’s probably time to check out this powerful tool you may be interested in. The allows you to build your own grammar rules, validate, and parse the result at the end. Chevrotain What does this library structurally consist of? of all, it’s . Lexer is the class that basically goes through input text and match token to each symbol or sequence of symbols (words), it found. To create an instance of the Lexer class, I need to define all tokens that I want to use. The token should have a unique name, the pattern - means regex. First Lexer For example token will match the exact word in the text. ‘hackernoon‘ const hackernoon = createToken({
  name: 'hackernoon',
  pattern: /hackernoon/
}) Also, there is a possibility to tokens. For this purpose, I need to define one more token with a reserved pattern value . In this case, Lexer will not use it to match some characters. And in other tokens that I want to categorize, I add my new token to a property ‘categories’. Like in the example below, I introduce and add this token to and . categorize Lexer.NA MathOperator MathPlus MathMinus export const MathOperator = createToken({
    name: Token.MathOperator,
    pattern: Lexer.NA,
});

export const MathPlus = createToken({
    name: Token.MathPlus,
    pattern: /\+/,
    categories: MathOperator,
});

export const MathMinus = createToken({
    name: Token.MathMinus,
    pattern: /-/,
    categories: MathOperator,
});

export const Number = createToken({ 
    name: Token.Number, 
    pattern: /[+-]?\d*\.?\d+(?:[Ee][+-]?\d+)?/
});

export const Whitespace = createToken({
    name: Token.WhiteSpace,
    pattern: /\s+/,
    group: Lexer.SKIPPED,
}); When all tokens are defined, I need to create a new instance of the Lexer class and put there an array of these tokens. const allTokens = [
  WhiteSpace,
  Number,
  MathPlus,
  MathMinus
]

const mathLexer = new Lexer(allTokens) And from this moment the Lexer is ready to tokenize input strings. I need to invoke the function ‘tokenize’ of the Lexer, and as a result, I will receive an object with tokens. With them, I can go to the next step. const lexingResult = mathLexer.tokenize(text); is having next interface . Where errors array can be very helpful to understand for the user what is wrong in the expression because it contains coordinates to the problem: line and column. lexingResult ILexingResult export interface ILexingResult {
  tokens: IToken[]
  groups: {
    [groupName: string]: IToken[]
  }
  errors: ILexingError[]
} The important thing is . It is a tool to describe the rules of your expression. For example, in math expression, you are expecting that the first variable will be a the second should be some operator and the third must be a as well. So rule will describe such behavior. First things first I create my own class the inherited from . That will accept tokens list in the constructor, and the constructor will contain all required rules and after them the to check all written rules. second Parser Number +/- Number MathParser CstParser this.performSelfAnalysis() class MathParser extends CstParser {
    public mathCond: RuleFn;
    private valueOrParen: RuleFn;
    private parentheses: RuleFn;
    private value: RuleFn;
    private mathOp: RuleFn;

    constructor(tokens: TokenType[]) {
      super(tokens);
      const $ = this;

      // Rules here
      
      this.performSelfAnalysis(); // Must be after all rules
    }
} The rule is the main entrance rule that will be invoked to perform the parsing. Basically, in my example, I assume that expression should be starting from some number or from parentheses. You are wondering how to read this rule? It actually tells if at least one number or parentheses with a number or parentheses with some operation (the operation described in keyword), then the expression is valid. mathCond 123 (123) (123 + 456) MANY Of course, as you can see there could be more  parentheses and math operations inside one parentheses, in these terms - recursion. And says that expression can repeat zero or more times. As like the example with - it repeats one time if it will be like then 2 repeats. this can be read: 1 repeat in parentheses and one after them. MANY (123 + 456) (123 + 456 - 789) (123 + 456) - 789 mathCond this.mathCond = $.RULE('mathCond', () => {
    $.SUBRULE($.valueOrParen, { LABEL: 'lhs' });
    $.MANY(() => {
        $.SUBRULE1($.mathOp)
        $.SUBRULE2($.valueOrParen, { LABEL: 'rhs' });
    });
}); The rule is expecting some Number or other parentheses with math expression. valueOrParen this.valueOrParen = $.RULE('valueOrParen', () => {
    $.OR([
        { ALT: () => $.SUBRULE($.value) },
        { ALT: () => $.SUBRULE($.parentheses) },
    ]);
}); In the next rule parentheses, I want to cover the case with braces and inside braces, recursion happens. and are two more tokens. is the main rule function. () LeftParen RightParen mathCond this.parentheses = $.RULE('parentheses', () => {
    $.CONSUME(LeftParen);
    $.SUBRULE($.mathCond);
    $.CONSUME(RightParen);
}); is the rule which looks for OR . mathOp + - this.mathOp = $.RULE('mathOp', () => {
    $.OR([
        { ALT: () => $.CONSUME(MathPlus) },
        { ALT: () => $.CONSUME(MathMinus) },
    ]);
}); The rule is created just to show that it can be extended with other specific types. value this.value = $.RULE('value', () => {
    $.OR([
        { ALT: () => $.CONSUME(Number) }
    ]);
}); Once the parser is ready, create it and call the main function. const mathParser = new MathParser(allTokens);
mathParser.mathCond(); As for Lexer, there are errors that help users, here as well, after calling the main function parser may contain errors with Token where the error happened. mathParser.errors So this library helps you to verify the string that the user typed and actually evaluate the expression. And even more, you can try a custom to create your own expression tree. Visitor

Creating Your Own Grammar Rules

Golang is full of tools to help us on developing securer, reliable, and useful apps. And there is a category that I would like to talk about: . Static Analysis through Linters What is a linter? Linter is a tool that analyzes source code without the need to compile/run your app or install any dependencies. It will perform many checks in the static code (the code that you write) of your app. It is useful to help software developers ensure coding styles, identify tech debt, small issues, bugs, and suspicious constructs. Helping you and your team in the entire development flow. Linters , but let us take a look at the Golang ecosystem. are available for many languages : Disclaimer The author of the story is the CTO at SourceLevel First things first: how do linters analyze code? Most linters analyzes the result of two phases: Lexer Also known as tokenizing/scanning is the phase in which we convert the source code statements into . So each keyword, constant, variable in our code will produce a token. tokens Parser It will take the tokens produced in the previous phase and try to determine whether these statements are semantically correct. Golang packages In Golang we have , , , and (Abstract Syntax Tree) packages. Let's jump straight to a practical example by checking this simple snippet: scanner token parser ast package main

func main() {
	println("Hello, SourceLevel!")
} Okay, nothing new here. Now we'll use Golang standard library packages to visualize the generated by the code above: ast import (
	"go/ast"
	"go/parser"
	"go/token"
)

func main() {
	// src is the input for which we want to print the AST.
	src := `our-hello-world-code`

	// Create the AST by parsing src.
	fset := token.NewFileSet() // positions are relative to fset
	f, err := parser.ParseFile(fset, "", src, 0)
	if err != nil {
		panic(err)
	}

	// Print the AST.
	ast.Print(fset, f)
} Now let's run this code and look at the generated AST: 0  *ast.File {
 1  .  Package: 2:1
 2  .  Name: *ast.Ident {
 3  .  .  NamePos: 2:9
 4  .  .  Name: "main"
 5  .  }
 6  .  Decls: []ast.Decl (len = 1) {
 7  .  .  0: *ast.FuncDecl {
 8  .  .  .  Name: *ast.Ident {
 9  .  .  .  .  // Name content
16  .  .  .  }
17  .  .  .  Type: *ast.FuncType {
18  .  .  .  .  // Type content
23  .  .  .  }
24  .  .  .  Body: *ast.BlockStmt {
25  .  .  .  .  // Body content
47  .  .  .  }
48  .  .  }
49  .  }
50  .  Scope: *ast.Scope {
51  .  .  Objects: map[string]*ast.Object (len = 1) {
52  .  .  .  "main": *(obj @ 11)
53  .  .  }
54  .  }
55  .  Unresolved: []*ast.Ident (len = 1) {
56  .  .  0: *(obj @ 29)
57  .  }
58  } As you can see, the AST describes the previous block in a struct called which is compound by the following structure: ast.File type File struct {
	Doc        *CommentGroup   // associated documentation; or nil
	Package    token.Pos       // position of "package" keyword
	Name       *Ident          // package name
	Decls      []Decl          // top-level declarations; or nil
	Scope      *Scope          // package scope (this file only)
	Imports    []*ImportSpec   // imports in this file
	Unresolved []*Ident        // unresolved identifiers in this file
	Comments   []*CommentGroup // list of all comments in the source file
} To understand more about lexical scanning and how this struct is filled, I would recommend . Rob Pike talk Using AST is possible to check the formatting, code complexity, bug risk, unused variables, and a lot more. Code Formatting To format code in Golang, we can use the package, which is already present in the installation, so you can run it to automatically indent and format your code. Note that it uses tabs for indentation and blanks for alignment. gofmt Here is a simple snippet from unformatted: Go by Examples package main

import "fmt"
func intSeq() func() int {
    i := 0

    return func() int {
        i++
        return i
    }
}

func main() {

    nextInt := intSeq()

		fmt.Println(nextInt())
    fmt.Println(nextInt())
   	   fmt.Println(nextInt())

    newInts := intSeq()
    fmt.Println(newInts())

} Then it will be formatted this way: package main

import "fmt"

func intSeq() func() int {
	i := 0

	return func() int {
		i++
		return i
	}
}

func main() {

	nextInt := intSeq()

	fmt.Println(nextInt())
	fmt.Println(nextInt())
	fmt.Println(nextInt())

	newInts := intSeq()
	fmt.Println(newInts())

} So we can observe that earned an extra linebreak but the empty line after function declaration is still there. So we can assume that we shouldn’t transfer the responsibility of keeping your code readable to the : consider it as a helper on accomplishing readable and maintainable code. import main gofmt It’s highly recommended to run before you commit your changes, you can even for that. If you want to overwrite the changes instead of printing them, you should use . gofmt configure a precommit hook gofmt -w Simplify option has a as Simplify command, when running with this option it considers the following: gofmt -s An array, slice, or map composite literal of the form: []T{T{}, T{}} will be simplified to: []T{{}, {}} A slice expression of the form: s[a:len(s)] will be simplified to: s[a:] A range of the form: for x, _ = range v {...} will be simplified to: for x = range v {...} Note that for this example, if you think that variable is important for other collaborators, maybe instead of just dropping it with I would recommend using instead. _ _meaningfulName A range of the form: for _ = range v {...} will be simplified to: for range v {...} Note that it could be incompatible with earlier versions of Go. Check unused imports On some occasions, we can find ourselves trying different packages during implementation and just give up on using them. By using package]( ) we can identify which packages are being imported and unreferenced in our code and also add missing ones: [goimports https://pkg.go.dev/golang.org/x/tools/cmd/goimports go install golang.org/x/tools/cmd/goimports@latest Then use it by running with option to specify a path, in our case we’re doing a recursive search in the project: -l go imports -l ./..
../my-project/vendor/github.com/robfig/cron/doc.go So it identified that is unreferenced in our code and it’s safe to remove it from our code. cron/doc Code Complexity Linters can be also used to identify how complex your implementation is, using some methodologies as an example, let’s start by exploring ABC Metrics. ABC Metrics It’s common nowadays to refer to how large a codebase is by referring to the LoC (Lines of Code) it contains. To have an alternate metric to LoC, Jerry Fitzpatrick proposed a concept called , which are compounded by the following: ABC Metric , , , , , , , , , , and (A) Assignment counts: = *= /= %= += <<= >>= &= ^= ++ -- Function is called (B) Branch counts when: Booleans or logic test ( , , , , , , , and ) (C) Conditionals counts: ? < > <= >= != else case This metric should be used as a “score” to decrease, consider it as just an indicator of your codebase or current file being analyzed. Caution: not To have this indicator in Golang, you can use package]( ): [abcgo https://github.com/droptheplot/abcgo $ go get -u github.com/droptheplot/abcgo
$ (cd $GOPATH/src/github.com/droptheplot/abcgo && go install) Give the following Golang snippet: package main

import (
	"fmt"
	"os"

	"my_app/persistence"
	service "my_app/services"

	flag "github.com/ogier/pflag"
)

// flags
var (
	filepath string
)

func main() {
	flag.Parse()

	if flag.NFlag() == 0 {
		printUsage()
	}

	persistence.Prepare()
	service.Compare(filepath)
}

func init() {
	flag.StringVarP(&filepath, "filepath", "f", "", "Load CSV to lookup for data")
}

func printUsage() {
	fmt.Printf("Usage: %s [options]\n", os.Args[0])
	fmt.Println("Options:")
	flag.PrintDefaults()
	os.Exit(1)
} Then let’s analyze this example using : abcgo $ abcgo -path main.go
Source            Func         Score   A   B   C
/tmp/main.go:18   main         5       0   5   1
/tmp/main.go:29   init         1       0   1   0
/tmp/main.go:33   printUsage   4       0   4   0 As you can see, it will print the based on each found in the file. This metric can help new collaborators identify files that a pair programming session would be required during the onboarding period. Score function Cyclomatic Complexity , on the other hand, besides the complex name, has a simple explanation: it calculates how many paths your code has. Cyclomatic Complexity It is useful to indicate that you may break your implementation in separate abstractions or give some and insights. code smells To analyze our Golang code let use package]( ): [gocyclo https://github.com/fzipp/gocyclo $ go install github.com/fzipp/gocyclo/cmd/gocyclo@latest Then let’s check the same piece of code that we’ve analyzed in the ABC Metrics section: $ gocyclo main.go
2 main main main.go:18:1
1 main printUsage main.go:33:1
1 main init main.go:29:1 It also breaks the output based on the function name, so we can see that the function has 2 paths since we’re using conditional there. main if Style and Patterns Checking To verify code style and patterns in your codebase, Golang already came with installed. Which was a linter that offer no customization but it was performing recommended checks from the Golang development team. It was archived in mid-2021 and it is being recommended be used as a replacement. [golint](https://github.com/golang/lint) Staticcheck Golint vs Staticcheck vs revive Before Staticcheck was recommended, we had , which for me sounds more like a community alternative linter. revive As revive states how different it is from archived : golint Allows us to enable or disable rules using a configuration file. Allows us to configure the linting rules with a TOML file. 2x faster running the same rules as golint. Provides functionality for disabling a specific rule or the entire linter for a file or a range of lines. allows this only for generated files. golint Optional type checking. Most rules in golint do not require type checking. If you disable them in the config file, revive will run over 6x faster than golint. Provides multiple formatters which let us customize the output. Allows us to customize the return code for the entire linter or based on the failure of only some rules. Everyone can extend it easily with custom rules or formatters. provides more rules compared to . Revive golint Testing revive linter I think the extra point goes for at the point of creating custom rules or formatters. Wanna try it? revive $ go install github.com/mgechev/revive@latest Then you can run it with the following command: $ revive -exclude vendor/... -formatter friendly ./... I often exclude my directory since my dependencies are there. vendor If you want to customize the checks to be used, you can supply a configuration file: # Ignores files with "GENERATED" header, similar to golint
ignoreGeneratedHeader = true

# Sets the default severity to "warning"
severity = "warning"

# Sets the default failure confidence. The semantics behind this property
# is that revive ignores all failures with a confidence level below 0.8.
confidence = 0.8

# Sets the error code for failures with severity "error"
errorCode = 0

# Sets the error code for failures with severity "warning"
warningCode = 0

# Configuration of the `cyclomatic` rule. Here we specify that
# the rule should fail if it detects code with higher complexity than 10.
[rule.cyclomatic]
  arguments = [10]

# Sets the severity of the `package-comments` rule to "error".
[rule.package-comments]
  severity = "error" Then you should pass it on running : revive $ revive -exclude vendor/... -config revive.toml -formatter friendly ./... What else? As I’ve shown, you can use linters for many possibilities, you can also focus on: Performance Unused code Reports Outdated packages Code without tests (no coverage) Magic number detector Feel free to try new linters that I didn’t mention here, I’d recommend the . archived repository awesome-go-linters Where to start? To start, consider using before each commit or whenever you remember to run, then try . Which linters are you using? gofmt revive : Disclaimer The author of the story is the CTO at . SourceLevel

Golang Linters: How they Analyze Code, Formatting, Complexity, and Comparison

In this document, we will showcase the capabilities of each framework and see how they map to each other and provide a foundation for how one would go about . migrating from react-i18next to Transifex Native Initialization Both frameworks require you to initialize a global object that will serve as a "gateway" between your application and the ability to localize it. In react-i18next, this looks like this (example taken from the official website): i18n ; { initReactI18next } ; resources = { : { : { : ,
    },
  }, : { : { : ,
    },
  },
};

i18n
  .use(initReactI18next)
  .init({
    resources, : , : { : },
  }); import from "i18next" import from "react-i18next" const en translation 'Hello world' 'Hello world' el translation 'Hello world' 'Καλημέρα κόσμε' lng 'en' interpolation escapeValue false This snippet doesn't tell the whole story. The translated content (the 'resources' field) can originate from a variety of sources and can also be added after the initialization. Also, the 'init' method accepts many more arguments and allows you to customize the experience to a great extent. With Transifex Native, you lose some of this flexibility, given that the source of truth for your content is always the content on transifex.com, and in return, you end up with the simpler: { tx } ;

tx.init({ : ,
}); import from '@transifex/native' token '...' There are other initialization options in Transifex Native that allow you to: Limit the content you will receive from Transifex with the 'filterTags' field What to do when a translation is missing with the 'missingPolicy' field What to do when the translation cannot be rendered (because, for example, the translator made a formatting mistake) with the 'errorPolicy' field Keys react-i18next react-i18next is key-based. This means that if your component code looks like this: { { t } = useTranslation(); ;
} ( ) function Paragraph const return {t('a_key')} < > p </ > p Then you need to have the exact same key in your 'resources': resources = { : { : { : ,
	},
  }, : { : { : ,
	},
  },
}; const en translation 'a_key' 'Hello world' el translation 'a_key' 'Καλημέρα κόσμε' If the key is not found in the resources, then the key itself will be rendered in the application. For this reason, it makes sense to use the source string itself as the key: resources = { : { : { : ,
	},
  }, : { : { : ,
	},
  },
}; { { t } = useTranslation(); ;
} const en translation 'Hello world' 'Hello world' el translation 'Hello world' 'Καλημέρα κόσμε' // i18n.use(...).init(...) ( ) function Paragraph const return {t('Hello world')} < > p </ > p However, this gets a little difficult to manage if you have plurals. Let's assume you have the following component: { [count, setCount] = useState( ); { t } = useTranslation(); ( <p>
    	  {t('You have {{count}} messages', { count })}
  	</p> <p>
    	  <button onClick={() => setCount(count - 1)}>-</button>
    	  <button onClick={() => setCount(count + 1)}>+</button>
  	</p> ( ) function Paragraph const 2 const return <> );
} </> And the following resources: resources = { : { : { : , : ,
    },
  }, : { : { : , : ,
    },
  },
}; const en translation 'You have {{count}} messages_one' 'You have one message' 'You have {{count}} messages_other' 'You have {{count}} messages' el translation 'You have {{count}} messages_one' 'Έχετε ένα μήνυμα' 'You have {{count}} messages_other' 'Έχετε {{count}} μηνύματα' Then the application will work as expected. When the 'count' becomes 1, the application will render "You have one message". If the keys are not in the resources, however, or if they are not formatted properly, you"ll get "You have 1 messages". Transifex Native With Transifex Native, the default behaviour is to not worry about keys at all. The string itself becomes the key. In case of plurals, the fact that the strings are formatted with ICU MessageFormat means that it contains all the information needed to render in both the source and target languages: { [count, setCount] = useState( ); ( <p>
    	  <T
          _str="{count, plural, one {You have one message} other {You have # messages}}"
          count={count} />
  	</p>
  	<p>
        <button onClick={() => setCount(count - 1)}>-</button>
        <button onClick={() => setCount(count + 1)}>+</button>
  	</p>
    </>
  );
} ( ) function Paragraph const 2 return <> you want to use keys, however, probably because you are using Transifex Native with a design tool like Figma or , You can supply the '_key' prop: If Sketch { ( ) function Paragraph return ;
} < = = /> T _key "a_key" _str "Hello world" In both cases, the source language is editable by using the "edit source strings" feature of transifex.com. t-function react-i18next Most of the translation methods offered by react-i18next boil down to offering you a javascript function called 't'. Using this function will render the argument in the correct language and make sure the component will re-render when the selected language changes. 'useTranslation' hook { useTranslation } ; { { t } = useTranslation(); ;
} import from 'react-i18next' ( ) function Paragraph const return {t('Some text')} < > p </ > p 'withTranslation' (Higher-Order Component) { withTranslation } ; { ;
} Paragraph = withTranslation()(_Paragraph); { import from 'react-i18next' ( ) function _Paragraph { t } return {t('Some text')} < > p </ > p const ( ) function App return ;
} < /> Paragraph 'Translation' (Render Prop) { Translation } ; { ( <p>{t('Some text')}</p> );
} import from 'react-i18next' ( ) function Paragraph return {(t) => < > Translation } </ > Translation 'I18nProvider' context { useContext } ; { I18nContext, I18nextProvider } ;

i18n.use(...).init({...});

ReactDOM.render( <App /> , .getElementById( ),
); { <p>{t('Some text')}</p> import from 'react' import from 'react-i18next' < = > I18nextProvider i18n {i18n} </ > I18nextProvider document 'root' ( ) function App return ;
}

function Paragraph() {
  const { i18n: { t } } = useContext(I18nContext);
  return < /> Paragraph ;
} Transifex Native With Transifex Native you can achieve the same result by using the 'useT' hook: { useT } ; { t = useT(); ;
} import from '@transifex/react' ( ) function Paragraph const return {t('Some text')} < > p </ > p or by using the preferable 'T-component': { T } ; { <T _str="Some text" /></p> import from '@transifex/react' ( ) function Paragraph return < > p ;
} Interpolation and Plurals react-i18next react-i18next allows you to interpolate dynamic values into your translations by using the '{{name}}' syntax, as demonstrated here: { useTranslation } ; { { t } = useTranslation(); ;
} import from 'react-i18next' ( ) function Paragraph const return {t('Hello {{name}}', { name: 'Bob' })} < > p </ > p or { useTranslation, Trans } ; { { t } = useTranslation(); ( <Trans t={t}>
          Hello <strong>{{name}}</strong>
      </Trans> );
} import from 'react-i18next' ( ) function Paragraph const return < > p </ > p In order to support plurals, you have to pay very close attention to your keys: resources = { : { : { : , : },
  }, : { : { : , : } },
};
i18n.use(...).init({ ..., resources }); { count = ; { t } = useTranslation(); ;
} const en translation messages_one 'You have 1 message' messages_other 'You have {{count}} messages' el translation messages_one 'Έχετε 1 μήνυμα' messages_other 'Έχετε {{count}} μηνύματα' ( ) function Paragraph const 3 const return {t('messages', { count })} < > p </ > p Or { count = ; { t } = useTranslation(); ( <Trans t={t} i18nkey="messages" count={count}>
        You have {{count}} messages
      </Trans> );
} ( ) function Paragraph const 3 const return < > p </ > p Transifex Native Transifex Native uses ICU MessageFormat natively (pun intended). This gives you a solid background to work with interpolation: { T } ; { <T _str="Hello {name}" name="Bob" /></p> <p>
      <T
        _str="{cnt, plural, one {You have 1 message} other {You have # messages}}"
        cnt={count} />
    </p>
  );
} import from '@transifex/react' ( ) function Paragraph return < > p ;
}
ICU MessageFormat also offers you industry standard capabilities for plurals:
function Messages() {
  const count = 3;
  return ( And also for select statements and number formatting: <T _str= gender={gender} /> "It's a {gender, select, male {boy} female {girl}}" < = = /> T _str "Today is {today, date}" today {today} Translation with Formatted Text react-i18next When you want to translate HTML content, react-i18next offers the '<Trans>' component: { useTranslation, Trans } ; { { t } = useTranslation(); ( <p>Some <strong>bold</strong> text</p> );
} import from 'react-i18next' ( ) function Paragraph const return < = > Trans t {t} </ > Trans In order for this to work, the source text in i18n's resources must have the form <1>Some <2>bold</2> text</1> Which you have to generate by hand. Transifex Native With Transifex Native you have the option to use the 'UT'-component: { UT } ; { <p>Some <strong>bold</strong> text</p> import from '@transifex/react' ( ) function Paragraph return < = UT _str " " />;
} Or to interpolate the outer template with react elements (that can be also translated): { T } ; { ( <p>Some {strong} text</p> <strong><T _str="bold" /></strong>} />
  );
} import from '@transifex/react' ( ) function Paragraph return < = T _str " " = strong { With both ways, what your translators will be asked to work on will have the exact same form as the argument you used for '_str'. Language Selection Both frameworks allow you to dynamically set the currently active language. With react-i18next you can do: { ( <button onClick={() => i18n.changeLanguage('en')}>English</button> <button onClick={() => i18n.changeLanguage('el')}>Greek</button> ( ) function Languages return <> );
} </> And with Transifex Native you can do the similar: { ( <button onClick={() => tx.setCurrentLocale('en')}>English</button> <button onClick={() => tx.setCurrentLocale('el')}>Greek</button> ( ) function Languages return <> );
} </> What Transifex Native can offer you above this is due to the fact that transifex.com is your source-of-truth not only for your translation content but also for your available languages. Taking this into account, you can do: { useLanguages } ; { languages = useLanguages(); ( <button
                key={code}
                onClick={() => tx.setCurrentLocale(code)}>
              {name}
            </button> import from '@transifex/react' ( ) function Languages const return {languages.map(({ code, name }) => ( <> ))} );
} </> Or the more direct: { LanguagePicker } ; { import from '@transifex/react' ( ) function Languages return ;
} < = /> LanguagePicker className "pretty" Which will render a language selector dropdown. After this, languages added on transifex.com will be reflected in your application without requiring you to publish a new release. String Extraction react-i18next There are some third-party tools to help with extracting strings from your code in the i18next ecosystem. One that can work with react applications is . Assuming you have the following application: i18next-parser { { t } = useTranslation(); ( <p>
    	  {t('Hello world 1')}
  	</p> <p>
    	  <Trans t={t}>
      	Hello <strong>world</strong> 2
    	  </Trans>
  	</p> <p>
    	  {t('Hello {{count}} worlds', { count: 3 })}
  	</p> export default ( ) function App const return <> );
} </> And use this configuration: .exports = { : , : { : [ ],
  },
}; module useKeysAsDefaultValue true lexers js 'JsxLexer' Then, if you run i18next-parser, you will end up with the following resource file: { : , : , : , : } "Hello world 1" "Hello world 1" "Hello <1>world</1> 2" "Hello <1>world</1> 2" "Hello {{count}} worlds_one" "Hello {{count}} worlds" "Hello {{count}} worlds_other" "Hello {{count}} worlds" ΅΅΅Which is a good starting point and even takes care of the unintuitive key generation for the '<Trans>' component and of plurals. After that, you will of course have to worry about how to generate translated versions of these files (hint: you should upload them to Transifex) and how to import these files into the application when it's running. Transifex Native With Transifex Native, you don't have to worry about files at all. You simply have to run: TRANSIFEX_TOKEN=...
export TRANSIFEX_SECRET=...
npx txjs-cli push src export After that: Your content will be available on transifex.com for your translators to work on Any ready translations will be available to your application during runtime Namespaces / Lazy loading react-i18next and i18next in general offers extensive support for compartmentalizing your translated content and, via plugins, loading these compartmentalized namespaces into your application via HTTP after the initial booting of the application. Transifex Native has limited support for namespacing via tags. You can add a tag to a translatable string with the '_tags' property: <T _str= _tags= /> "Hello world" "main" or by specifying tags during the extraction cli execution: npx txjs-cli push --append-tags=helpdesk src/helpdesk
npx txjs-cli push --append-tags=dashboard src/dashboard Then, when initializing the 'tx' object, you can specify which tags you want to filter against: tx.init({ : ..., : }); token filterTags 'android' This is useful in case you are using the same transifex.com project for different platforms and you want each platform to only pull translations that are relevant. As of now, we don't support lazy loading of translations but we have plans to implement this in the near future. Migration Guide Preparing the code For minimal inconvenience, you should replace the invocation of react-i18next's 't' function with Transifex Native's 't' function. From: { useTranslation } ; { { t } = useTranslation(); ;
} import from 'react-i18next' ( ) function Paragraph const return {t('Some text')} < > p </ > p to { useT } ; { t = useT(); ;
} import from '@transifex/react' ( ) function Paragraph const return {t('Some text')} < > p </ > p However, it may be preferable to use the T-component: { T} ; { <T _str="Some text" /></p> import from '@transifex/react' ( ) function Paragraph return < > p ;
} Simple variable interpolation is done with single curly brackets instead of double. From: { useTranslation } ; { { t } = useTranslation(); ;
} import from 'react-i18next' ( ) function Paragraph const return {t('Hello {{username}}', { username: 'Bob' })} < > p </ > p to: { T } ; { <T _str="Hello {username}" username="Bob" /></p> import from '@transifex/react' ( ) function Paragraph return < > p ;
} For formatted content, your best bet is to replace '<Trans>' with <UT />'. From: { useTranslation, Trans } ; { { t } = useTranslation(); ( <p>Some <strong>bold</strong> text</p> );
} import from 'react-i18next' ( ) function Paragraph const return < = > Trans t {t} </ > Trans To: { UT } ; { <p>Some <strong>bold</strong> text</p> import from '@transifex/react' ( ) function Paragraph return < = UT _str " " />;
} Migrating the translated content First, you'll have to upload your current translations to transifex.com on a file-based project. After creating the project, you can use the transifex-client to help with uploading the resources (one namespace per resource): # Install the client
wget https: tar xf tx-linux-amd64.tar.gz tx
rm tx-linux-amd64.tar.gz
mv tx ~ translations/ <lang>.json \
  --type=KEYVALUEJSON \
  locales/helpdesk/en.json

# Push the content to transifex
tx push --source --translation --all //github.com/transifex/cli/releases/download/v0.3.0/tx-linux-amd64.tar.gz /bin

# Set up the mapping to transifex.com
tx init
tx add \
  --organization=... \
  --project=... \
  --resource=translations \
  --file-filter=locales/ .json \
  --type=KEYVALUEJSON \
  locales/translations/en.json
tx add \
  --organization=... \
  --project=... \
  --resource=helpdesk \
  --file-filter=locales/helpdesk/ < > lang Next, you need to create a Native project. Make sure you keep the public and secret tokens for later. Before pushing the content to the new project, you need to: Add the target languages to the new project Add both projects to the same TM group Now in your local project, use the extraction cli to push content to the native project in transifex.com (use tags for namespacing as discussed above: npm install --save-dev @transifex/cli
npx txjs-cli push --token=... --secret=... src Because we put both projects in the same TM group, existing translations will be used to fill up the new ones. In case you had to make minor changes to some of the strings while migrating the code, take a look in the editor to fill these in. Even if they were not filled up automatically, partial TM matches should make this job easy. Finally, make sure you initialize the 'tx' object when your application boots up and you should be good to go! Once everything is up and running, you can delete the old file-based project. Wrapping Up Wanna share a quick React localization guide with a fellow developer? ! Find it on this page This post was co-authored by Konstantinos Bairaktaris and Mike Giannakopoulos Find Transifex on our: , , , and . Website Facebook Twitter LinkedIn

React Internationalization: Transifex Native Vs. react-i18next

Parsing SQL queries provides superpowers for monitoring data health. This post describes how to get started on parsing  SQL for data observability. Query history of a data warehouse is a rich source of information to glean how data is used in your organization. Many aspects of data observability can be tracked by analyzing query history. For example, query history analysis can extract: Popular tables and columns Unused tables and columns Column-level lineage Freshness These statistics also help to automate common data engineering tasks like: Backup and Disaster Recovery Triage Data Quality issues Track sensitive data and how they are used. Challenges and Approaches SQL language is an ISO/IEC standard and the latest version is . However, every database implements the standard differently, uses different function names for the same operation, and has extensions to access specific custom features. Therefore, there isn’t one SQL parser for dialects of all popular databases and data warehouses. SQL2016 Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. is a popular Python package that uses regular expressions to parse SQL. sqlparse An alternate approach is to implement the SQL grammar using parser generators like . There are similar open-source parser generators in other popular languages. There are multiple projects that maintain parsers for popular open-source databases like MySQL and Postgres. For other open-source databases, the grammar can be extracted from the open-source project. For commercial databases, the only option is to reverse engineer the complete grammar. There are SQL parser/optimizer platforms like that help to reduce the effort to implement the SQL dialect of your choice. ANTLR Apache Calcite Open Source Parsers Some popular open-source databases and data warehouses are: MySQL/MariaDB parser is a MySQL parser in Go. Pingcap in phpmyadmin is a validating SQL lexer and parser with a focus on MySQL dialect. SQL Parser Postgres extracts the parser (written in C) from the Postgres project and packages it as a stand-alone library. This library is wrapped in other languages by other projects like: libpg_query : Python pglast : Ruby pg_query : Golang pg_query_go : in Node and in the browser JS psql-parser pg-query-emscripten : Rust pg_query.rs Multiple Engines implements Apache Hive, Presto/Trino and Vertica dialects. queryparser implements BigQuery, Spanner, and Dataflow dialects. zetasql Generic Parsers : Python sqlparse : Rust sqlparser-rs : Python mo-sql-parseing Platforms Parser/Optimizer platforms implement the common SQL language features and allow customization as a first-class feature of the platform. Two popular open-source projects are: is a popular parser/optimizer that is used in popular databases and query engines like , , and . Apache Calcite Apache Hive BlazingSQL many others can parse multiple SQL dialects like MySQL, Postgres and Oracle. The grammar can be modified to support other SQL dialects. JSQLParser Apache Calcite allows customizations at various points of the parsing process. Parser rules can be to support custom syntax. changed Conventions such as quotes vs double quotes, case sensitivity. Add optimizer rules. Apache Calcite also provides visitors for traversing the SQL execution plan. Visitor pattern is an algorithm to traverse a SQL plan. Practical tips to Getting Started There are many abandoned open-source SQL parsers. The first filter is to use a project that will be supported in the future. For popular databases such as Postgres and MySQL/MariaDB, there are parsers available in multiple programming languages. What if there is no parser for your database? Most teams do not create a parser from scratch. A popular option is to use the Postgres parser and then add custom SQL syntax. AWS Redshift, Vertica, and DuckDB are examples. Use a Postgres SQL parser to parse the query history of these databases to parse the majority of the queries. Many queries will fail to parse such as UNLOAD in AWS Redshift. If it is important to also parse the variants, consider modifying the projects to accept the custom grammar OR use a platform like Apache Calcite. Conclusion There is a demand for SQL parsers to build reports on database or data warehouse usage. There are a number of good open-source projects. However, there is a steep learning curve to use these projects and in many cases, a project may not fit your specific requirements. Struggling with parsing query history? Get in touch Also published . on Tokern

14 Open Source SQL Parsers

14 analizadores SQL de código abierto

is a process of converting into a . A data structure type can be any suitable representation of the information engraved in the . Parsing formatted text data structure source text type is a common and standard choice for XML parsing, HTML parsing, JSON parsing, and any programming language parsing. The output tree is called or . In an HTML context, it is called the (DOM). Tree Parse Tree Abstract Syntax Tree Document Object Model A CSV file parsing can result in a of a . List List of values or List of Record objects Type is a choice for natural language parsing. Graph A piece of program that does parsing is called . Parser How it works Parser analyses against the format* prescribed. If source text does not match against format error is thrown or returned. source text If source text does not match against format error is thrown or returned. If matches then “data structure” is returned. *format is coded inside the parser. Format is the DNA of a parser. Small Case Study Consider an example of Date parsing from a string (source) in format to Date object: DD-MM-YYYY { day; month; year;
} class Date int int int Implementation Note For Date parsing, I would be using ¹ ( for short). Regex can be matched against a string. It also helps in extracting part of the source text Regular Expression regex if it is matched. Note: This is going to be a small-scale illustration of parsing with “Regex.” This might be a fair approach for one-two lines input text but not in every case . You may have to write a parser by hand (most complex) or use Parser Generator tools (moderately complex). Code The parsing and extracting the Date element is as: String date = ; Pattern dateMatcher = Pattern.compile( ); Matcher matcher = dateMatcher.matcher(date); (matcher.find()) { day = Integer.parseInt(matcher.group( )); month = Integer.parseInt(matcher.group( )); year = Integer.parseInt(matcher.group( )); Date(day, month, year);
} { DateParseException(date + )
} "20-05-2012" // 1. defining a regular expression "(\\d{2})-(\\d{2})-(\\d{4})" // 2. matching a regular expression // 3. if pattern matches if // extract day int 1 int 2 int 3 // 4. validate date : skipped code return new else // throw Error throw new " is not valid" The code is written in Java. How does this code work: 1. Date formatted as DD-MM-YYYY can be matched using regex: (\d{ })-(\d{ })-(\d{ })
# where \d matches any digit between to , 
# {n} means how many times the last character type is expected
# () is called capturing group and enclosed part string that would be extracted.
# Each group is numbered, the first group is assigned , the second one is given number , and so on 2 2 4 0 9 of "1" "2" 2. The given date as the string is matched against the defined regex. 3. If the match is successful, then the day, month, and year are extracted from the source string using provided by Regular Expression API. It is a standard construct in any Regular Expression API in any language. Group Construct 4. The extracted date parts can be validated; related code is skipped for you to exercise. This is an illustration of Regular Expression based parsing, which is limited to format, which can be defined by regular expressions. It is of parsing. A programming language or format like XML parsing is complex in nature. You can refer to the book named “ ” to get an idea about a full-scaled parsing. a basic example Crafting Interpreters You can also read about “ ” Lezer: Code Mirror’s Parsing System Phases of Parsing can be scoped as a composition of Scanning and Syntactic Analysis or just Syntactic Analysis. Parsing Scanning is a process of converting a stream of characters into Scanning tokens. A token represents a “concept” introduced by format. Logically, a token can be considered as a label assigned to one or more characters. From a processing point of view: A is an object, can contain , location information, and more. token lexeme³ In Java language: if, while, int are examples of tokens. In date parsing, tokens are defined by Regex; (day, month), (separator), (year) are tokens. Note: Day and month are the same token type. A token is defined by “the pattern of characters” not by locations of characters. \d{2} — \d{4} Scanning is also called or . A part of the program which does scanning is called Tokenization Lexical Analysis Lexer or Scanner or Tokenizer. Syntactic Analysis analyses the structure formed as keeping tokens in order as their positions. It also validates and extracts engraved data to create the preferred Data Structure. Syntactic Analysis In date parsing example: “day is followed by month and year.” The order is checked by Regex Engine. Extraction is done based on order and matches. Errors reported by this phase are called . Going back to Date parsing example, is an invalid Date; however, is a valid Date because rule says so. It may seem absurd, but parsers generally validate Syntactic Errors 99-JAN-2021 99–99–9999 (\d{2})-(\d{2})-(\d{4}) Syntactic Correctness. Closing Notes The scale of parsing determines the inclusion or exclusion of Scanning as part of it: It is beneficial for big-scale parsing like language (natural or programming) parsing to keep Scanning, and Syntactic Analysis separated. In this case, Syntactic Analysis is called parsing. For small-scale parsing like Date, it may not be beneficial to distinct Lexer and Syntactic Analysis. In this case, it is called . Scannerless Parsing Many times, parsing tasks are delegated to parser code generators like or . These tools require a set of rules or grammar and generate parser code. Antlr Lex The generated parsers produce a tree upon parsing, which may not be the desired data structure; however, these libraries provide sufficient APIs to convert the constructed tree to the desired data structure. There is more in Parser world; the type of parsing styles: top-down or bottom-up, Leftmost or Rightmost derivation. These design styles limit parsing capabilities of the parser. You may visit the following link to get a synoptic view of these design choices and their comparison: https://tomassetti.me/guide-parsing-algorithms-terminology/#tablesParsingAlgorithms Footnotes is an algebraic expression² representing a group of strings. Regular Expression An algebraic expression utilizes symbols and operators. is a raw string version of Token. A Lexeme Previously published at https://themightyprogrammer.dev/article/parsing

Convert Formatted Text Into a Data Structure Using Parsing

Introduction Sooner or later every software engineer will come across a situation that requires some kind of text parsing. The text might contain information in semi-structured form that need to get parsed and saved into data structures. Different methods can be used to tackle the text parsing problem. One approach is to develop custom code using a line split approach and then parse the separate elements. Another one can be to use a built-in scanner library function available in many programming languages. If you are interested to dive deep into the topic you can find a , covering everything from regular expressions, grammars and lexers to parsing trees and algorithms. very detailed guide here Deterministic vs Non-deterministic, what's the difference? In this article, I compare two parsing methods with a focus on their performance. The first method uses for parsing and data extraction. You can find a regular expression engine in any programming language. Great examples of regular expression library capabilities are included in perl, python, php and java. regular expressions A regular expression can be modeled by a . FSMs are quite flexible and can be used for any task that has distinct states of processing or operation. In parsing text using regular expressions, a class of FSMs is of particular interest, the Non-deterministic FSM or as they are also known the , You can easily and then use the NDFA to parse the text input. Finite State Machine (FSM) Non-deterministic Finite Automata (NDFA) convert a regular expression into a NDFA Using an NDFA in practice usually means a lot of backtracking. The algorithm has to follow every possible transition from a state to another and when a possible path fails it goes back and continues with another path. In theory, there are methods available that convert a NDFA to a Deterministic Finite Automaton (DFA). This conversion can be quite tricky when your parser is designed using regular expressions. In this article, I opted for a DFA designed from scratch as the second method to parse the text and extract the data. This allowed me to compare the two methods, the NDFA and the DFA, in terms of performance. Case study - An .ini file parser To compare the two methods, the case study of an configuration file parser and reader was used. The input file can have configuration sections that are denoted in a single line using brackets and section names, for example: ini-like [sectionA] The section line can also have a comment part: [sectionA]          ; is a comment this Comments can also exist on their own in a line: ; line contains only comment this Each section can contain multiple lines with settings in a key-value fashion. The key name must be a string and the value can take various types. The supported types are string, integer, path, array and boolean. Some examples are given below: keyname1 = keyname2 = keyname3 = .ssh/file.conf                           
keyname4 = value1,value2,value3                           
keyname5 = "A string value" 35363262 /root/ true Each setting can have overrides that are loaded instead of the main value when the user enters the flag during loading. For example, you can have multiple values for key name config_path depending on the deployment system production, staging, testing: config_path<production> = lib/config/prod.cfg               
config_path<testing> = lib/config/prod.cfg               
config_path<staging> = lib/config/prod.cfg /var/ /var/ /var/ For a more comprehensive example of the accepted input, you can check the github repository containing the code used in this article. A sample input file can be found here. Non-deterministic parser using Regular Expressions Firstly, the regular expression parser was put in action and its performance measured. The graph detailing the NDFA that was created from the regular expression is shown in graph 1. This graph implies backtracking when a path fails until all possible paths in the graph from Start to End are exhausted. In each one of the nodes, a regular expression match is tried and if it succeeds the corresponding data is extracted if applicable. Comment node expects the following regular expression to match \s*;.* Section node has a data group and expects \s*\[([\\w_]+)\]\s* For a full definition of the regular expressions used in this NDFA you can check the class that implements it in the source code. Deterministic parser using Finite State Machine Next, a parser based on an FSM implementation was build and tested on the same input data. The FSM graph is shown in graph 2. In this case, transitions between states are deterministic and depend only on the current input character. There is no backtracking and each text line is processed character by character using the FSM. States that are tagged with the isFinal flag are valid terminal states. If the input text is fully processed while the FSM is in a final state, the parsing is completed successfully. The color legend shows the states extract different data types from the input text. For example, light-green state 3 extracts the group section name. For a full definition of the FSM and its implementation in Java you can check the . source code here Performance results So what are the results of the performance comparison between the two methods? To find out, a set of text parsing operations on a large synthetic file was done. The parsing was repeated multiple times for each method and the average time was measured. The resulting data from text parsing was put onto memory in HashMap. You can see the results of the comparison for different file sizes in graph 3. The blue line shows the parsing time of NDFA for different sizes of the input file. The green line shows the parsing time of DFA for the same input size. The input size is counted in total configuration lines and the x axis scale is logarithmic. The DFA scales well and approximates a O(logN) complexity, whereas NDFA approximates a O(N) complexity. At N = 2.4M lines DFA is 4-5x faster than NDFA. This makes DFA parser an excellent parser candidate when text parsing speed is important. Conclusion In this article, I have reviewed and compared the performance of different types of Finite Automata for the task of text parsing into a data structure in memory. A Non-Deterministic Finite Automaton based on Java's Regular Expression engine was and tested. built Next, a Deterministic Finite Automaton was using a custom-built Java code FSM implementation. The DFA was found at least 4 times faster than the NDFA and having logarithmic time complexity as the input size increases. You can find the . built full code implementation used in the article in my Github repository About the author currently works as a Senior Software Architect / Technical Lead at . He likes to design and implement large-scale distributed application backends and single-page web applications, embedded with Machine Learning (ML) components. From time to time he might also get his hands dirty with Infrastructure as Code (IaC) and DevOps tasks or mini IoT projects. Spiros Dimopoulos Behavioral Signals

High Performance Text Parsing Using Finite State Machines (FSM)

Packages play a big role in the scene. How big and which packages?
 To find it out we took the data from the API and filtered out the
 Laravel packages. Laravel Packagist.org How did we identify the Laravel packages? By looking at . We 
classified a package as belonging to the Laravel ecosystem if , , or fields contained the string . 
We also considered it to be a package if it has or any of the subtree splits as a dependancy. And finally — the key in field is also a trigger. composer.json name description keywords type laravel Laravel laravel/framework illuminate/* laravel extra While these criteria do not guarantee a package to be Laravel-only, they do
 show that the package was made with Laravel in mind. The most popular packages By naively looking at the , 
we see quite a bit of unexciting results — core packages and packages with a 
plenty of uses outside Laravel projects. list of the most downloaded 
packages | Package                 | Monthly downloads |
|-------------------------|-------------------|
| nesbot/carbon           | |
| laravel/framework       | |
| laravel/tinker          | |
| fideloper/proxy         | |
| nunomaduro/collision    | |
| intervention/image      | |
| facade/ignition         | |
| facade/flare-client-php | |
| composer/installers     | |
| illuminate/support      | | 4084214 2987907 2455355 2376631 1510258 1099989 966800 934651 803222 756189 Perhaps a more interesting toplist would be one with these "false positives"
 removed. We manually filtered out the top 30 packages that are made for Laravel
 projects and are installed deliberately. We also loosely classified the purposes 
of those packages. |    | Package                            | Monthly | Total    | Purpose |
|----|------------------------------------|---------|----------|---------|
| | maatwebsite/excel                  | | | port    |
| | laravelcollective/html             | | | helper  |
| | barryvdh/laravel-ide-helper        | | | dev     |
| | barryvdh/laravel-debugbar          | | | dev     |
| | laravel/passport                   | | | auth    |
| | beyondcode/laravel-dump-server     | | | dev     |
| | sentry/sentry-laravel              | | | integ   |
| | laravel/ui                         | | | auth    |
| | barryvdh/laravel-cors              | | | helper  |
| | laravel/socialite                  | | | auth    |
| | laravel/slack-notification-channel | | | integ   |
| | tymon/jwt-auth                     | | | auth    |
| | barryvdh/laravel-dompdf            | | | port    |
| | laravel/helpers                    | | | helper  |
| | spatie/laravel-permission          | | | auth    |
| | laravel/horizon                    | | | integ   |
| | laravel/dusk                       | | | dev     |
| | laravel/nexmo-notification-channel | | | integ   |
| | laravel/telescope                  | | | dev     |
| | bugsnag/bugsnag-laravel            | | | dev     |
| | yajra/laravel-datatables-oracle    | | | integ   |
| | aws/aws-sdk-php-laravel            | | | integ   |
| | orchestra/testbench-core           | | | dev     |
| | orchestra/testbench                | | | dev     |
| | symplify/package-builder           | | | dev     |
| | laravel/scout                      | | | integ   |
| | barryvdh/laravel-snappy            | | | port    |
| | webpatser/laravel-uuid             | | | helper  |
| | spatie/laravel-medialibrary        | | | helper  |
| | spatie/laravel-activitylog         | | | dev     | 1 680466 17769696 2 678048 21785383 3 644297 18918427 4 625080 19404288 5 552633 10099903 6 530901 11035024 7 455879 8440070 8 433850 1599042 9 400920 10823255 10 394471 10166784 11 385921 7172937 12 359693 9107092 13 349572 8006731 14 337956 1692185 15 308453 5224001 16 304864 4599598 17 256261 4542482 18 238192 6080124 19 227966 2289576 20 223167 6544766 21 212985 5644721 22 212260 7070121 23 199780 1864659 24 195269 2279889 25 193661 1484783 26 177270 3391042 27 156741 3651684 28 153883 4774711 29 149290 2293823 30 148932 2387978 Explanation of purposes: — authentication and authorization auth — development and debugging tools dev — support and simplifaction of work helper — integration with other services integ — import and/or export in various forma port The development tools are most prevalent on this list followed by integration. We expeceted to see more of the packages at the very top, but it turns out that
 maatwebsite/excel and the tools by are more universally used. Spatie Barry vd. Heuvel Vendor overview In total there were 47337 Laravel packages identified from 23167 different developers. The distribution is quite even with lots and lots of small players. Half of packages (23668) were made by the top 3927 developers which is far more
 than would be if was obeyed here. 15042 of the developers have only
 one package published. Price's law If we sort package developers by number of packages, we get to see some quite
 unheard names. Here are the most prolific package developers along with the
 download numbers summed over their packages. |    | Developer                     | Count | Daily | Monthly | Total    |
|----|-------------------------------|-------|-------|---------|----------|
| | socialiteproviders            | | | | |
| | bantenprov                    | | | | |
| | spatie                        | | | | |
| | railken                       | | | | |
| | laravel-enso                  | | | | |
| | amethyst                      | | | | |
| | faustbrian                    | | | | |
| | artisanry                     | | | | |
| | sukohi                        | | | | |
| | anomaly                       | | | | |
| | sanatorium                    | | | | |
| | laravel-notification-channels | | | | |
| | hnhdigital-os                 | | | | |
| | arcanedev                     | | | | |
| | inetstudio                    | | | | |
| | larakit                       | | | | |
| | orchestra                     | | | | |
| | abuseio                       | | | | |
| | genealabs                     | | | | |
| | smorken                       | | | | | 1 159 10645 256409 5826725 2 132 0 5 3160 3 124 89124 2746665 43095685 4 119 0 11082 313129 5 95 1519 29456 879635 6 92 1232 8118 100819 7 85 152 2639 52698 8 78 129 1190 11938 9 75 58 1773 53805 10 59 198 6095 974489 11 59 0 0 20 12 54 3107 99536 1621420 13 54 38 1571 20304 14 53 3875 127702 3293360 15 53 241 3870 77009 16 49 1 410 18540 17 48 21569 544696 7606461 18 45 1 951 78289 19 43 3219 87234 1215349 20 41 36 985 14277 Apparently there are quite a few developers who make a lot of components and
 put them up without much advertisement. To see more of the better known names
 and how they compare, let us sort the list by total downloads. |    | Developer         | Count | Daily  | Monthly | Total     |
|----|-------------------|-------|--------|---------|-----------|
| | laravel           | | | | |
| | illuminate        | | | | |
| | nesbot            | | | | |
| | barryvdh          | | | | |
| | spatie            | | | | |
| | fideloper         | | | | |
| | intervention      | | | | |
| | composer          | | | | |
| | league            | | | | |
| | nunomaduro        | | | | |
| | laravelcollective | | | | |
| | maatwebsite       | | | | |
| | cocur             | | | | |
| | jenssegers        | | | | |
| | beyondcode        | | | | |
| | laracasts         | | | | |
| | graham-campbell   | | | | |
| | facade            | | | | |
| | yajra             | | | | |
| | tymon             | | | | | 1 39 364375 9832728 225867755 2 33 244509 7210832 197044965 3 1 179106 4084214 112631927 4 19 82289 2246578 63115434 5 124 89124 2746665 43095685 6 3 100748 2376631 41313378 7 7 46259 1260102 33891504 8 1 10603 803222 26065561 9 7 18996 731742 24433630 10 19 68333 1734552 23394056 11 6 32460 710099 23285426 12 6 27955 717140 18258360 13 1 19246 491761 16609476 14 15 24263 544700 16468485 15 29 14221 677724 12648233 16 11 10352 317232 12119703 17 36 11092 291114 10184667 18 5 67794 1915933 9860242 19 19 17794 446360 9792222 20 2 11437 359693 9107116 If one sorts this list by monthly dowloads instead, the picture is slightly different. |    | Developer         | Count | Daily  | Monthly | Total     |
|----|-------------------|-------|--------|---------|-----------|
| | laravel           | | | | |
| | illuminate        | | | | |
| | nesbot            | | | | |
| | spatie            | | | | |
| | fideloper         | | | | |
| | barryvdh          | | | | |
| | facade            | | | | |
| | nunomaduro        | | | | |
| | intervention      | | | | |
| | composer          | | | | | 1 39 364375 9832728 225867755 2 33 244509 7210832 197044965 3 1 179106 4084214 112631927 4 124 89124 2746665 43095685 5 3 100748 2376631 41313378 6 19 82289 2246578 63115434 7 5 67794 1915933 9860242 8 19 68333 1734552 23394056 9 7 46259 1260102 33891504 10 1 10603 803222 26065561 These distributions are extremely top heavy. Regardless if you order by daily,
 monthly or total downloads, more than half of all downloads go to top 6 package
 developers. For example, out of 5.65 million daily package downloads, 2.85 million
 are split among , , , , and . laravel illuminate nesbot spatie fideloper barryvdh We can use something a kin to to assess the depth of the scene.
 There are only 317 developers that get 317 or more daily downloads. 1118 vendors
 get 1118 or more monthly downloads and the same number is 3003 for total numbers. Hirsch index Licensing choices Our licensing overview is not very precise as various strings are used for the same
 license, for example , , , , even and many more for the same license. gpl-3.0 gpl 3.0 gpl3 gplv3 gnu general
 public license version 3 Luckily, the main conclusions are clear as the string is the license of 
37502 Laravel packages. 6304 packages are unlicensed, More than 500  
packages are licensed under each of Apache 2.0 and GPL 3.0. And there are also few hundred
 packages with BSD 2, BSD 3 and GPL 2.0. mit 79% of Laravel packages are MIT-licensed, 14% are unlicensed and the other
 licenses are each used in used in 2% of projects or less. This is probably in part
 influenced by Laravel which itself is MIT licensed. Among all of the packages
 the prevalence of the MIT license is considerably smaller — 60%. Outside Laravel The number of packages for different frameworks shows the involvment of the
 community in the development of the framework tools and how much they are 
willing to share their code with others. We also counted the abandoned packages
 and their percentage to see which communities are thriving and which are limping. |    | Framework     | Packages | Abandoned | Aband. % |
|----|---------------|----------|-----------|----------|
| | Laravel       | | | , %     |
| | Symfony       | | | , %     |
| | Yii           | | | , %     |
| | ZendFramework | | | , %    |
| | WordPress     | | | , %     |
| | Nette         | | | , %     |
| | CakePHP       | | | , %     |
| | Slim          | | | , %     |
| | Laminas       | | | , %     |
| | Phalcon       | | | , %     |
| | Codeigniter   | | | , %    |
| | Aura          | | | , %     |
| | Kohana        | | | , %    |
| | FuelPHP       | | | , %     |
| | PHPixie       | | | , %     | 1 47337 2618 5 5 2 39428 3362 8 5 3 15354 552 3 6 4 7459 790 10 6 5 6446 480 7 4 6 3460 342 9 9 7 2607 173 6 6 8 1411 64 4 5 9 811 17 2 0 10 726 40 5 5 11 663 91 13 7 12 450 32 7 1 13 341 43 12 6 14 148 12 8 1 15 73 1 1 4 On the other hand, the total download numbers are reflective of how much the 
packages are important in the development of everyday projects. Please keep in mind that these download numbers represent all of the packages related to a framework not just the core. It’s just like the number of packages listed above. The criteria for classifying a package as belonging to certain framework are . One package can belong to ecosystems of multiple frameworks. here |    | Framework     | Monthly   | Total      |
|----|---------------|-----------|------------|
| | Symfony       | | |
| | Laravel       | | |
| | ZendFramework | | |
| | Nette         | | |
| | Laminas       | | |
| | Yii           | | |
| | CakePHP       | | |
| | WordPress     | | |
| | Slim          | | |
| | FuelPHP       | | |
| | Codeigniter   | | |
| | Kohana        | | |
| | Aura          | | |
| | Phalcon       | | |
| | PHPixie       | | | 1 366029776 8945189030 2 56523199 1315681505 3 50789083 1460984224 4 14137793 295710951 5 9540512 134960483 6 5662063 200378585 7 4067892 112091915 8 3871828 96962490 9 2123558 46196613 10 930846 32043126 11 919926 30474042 12 912106 31918403 13 467771 18063339 14 137762 4322810 15 5353 572100 It should be noted that the top position doesn't mean Symfony projects download
 the most packages as some of the Symfony core is used by Laravel and many of the
 Symfony packages can also be used in Laravel projects. It is also noteworthy that Zend packages have almost as many monthly downloads
 as Laravel even though the project is no longer maintained. As a farewell we shall leave you with the list of most downloaded PHP packages overall. |    | Package                           | Monthly | Total     |
|----|-----------------------------------|---------|-----------|
| | symfony/polyfill-mbstring         | | |
| | psr/log                           | | |
| | symfony/polyfill-ctype            | | |
| | psr/container                     | | |
| | symfony/ | | |
| | guzzlehttp/guzzle                 | | |
| | symfony/finder                    | | |
| | psr/http-message                  | | |
| | guzzlehttp/promises               | | |
| | guzzlehttp/psr7                   | | |
| | doctrine/instantiator             | | |
| | symfony/process                   | | |
| | phpdocumentor/reflection-docblock | | |
| | monolog/monolog                   | | |
| | symfony/event-dispatcher          | | |
| | webmozart/assert                  | | |
| | doctrine/lexer                    | | |
| | ralouphie/getallheaders           | | |
| | doctrine/inflector                | | |
| | phpdocumentor/type-resolver       | | | 1 7943600 210645235 2 7768451 238663129 3 7284741 110601284 4 6847606 119345271 5 console 6713344 199367074 6 6607580 168514253 7 6486878 176261290 8 6384401 164860087 9 6372505 149984782 10 6350221 161806561 11 6199120 180684872 12 6167368 169254030 13 6140913 165150240 14 6109702 196673732 15 6098870 190433052 16 5905460 144395517 17 5895066 157678399 18 5861396 66293788 19 5743593 171588858 20 5706391 136101740 The data displayed in this article was compiled mainly by Gatis Šūpulnieks in April and 
May of 2020 as a part of his thesis "Laravel package development and usage". Relevant sources: Raw data https://github.com/Larzs/packagist-packages-data Related article by Jason McCreary https://jasonmccreary.me/articles/laravel-numbers/ Author of this article Juris Evertovskis Compiler of data Gatis Šūpulnieks The package made by Gatis https://github.com/GlaivePro/Invytr Our newest micropackage https://github.com/crudly/encrypted Our silliest and most popular package: https://github.com/GlaivePro/Hidevara Previously published at https://medium.com/@juris_9772/laravel-package-ecosystem-4afd53fad192

Explaining Laravel Package Ecosystem

I love pet projects, they are great excuse to use libraries and other technologies that you can’t use at work. Lately I’ve been working on a bigger pet project that needs to parse Go files, I’ve used before to make this kind of things but unfortunately, Go target has poor performance. So I began to search for alternatives written in pure Go and came across with , which took a different approach on creating parsers with Go, but before we’re going to understand how this library is different from others, let’s cover some basic concepts about parsing. ANTLR ANTLR’s this one Sorry, Parsing? For the sake of “formality” here is what Wikipedia has to say: Parsing, syntax analysis, or syntactic analysis is the process of analyzing a string of symbols, either in natural language, computer languages or data structures, conforming to the rules of a formal grammar. The term parsing comes from Latin pars (orationis), meaning part (of speech). Now, let’s try to break it down by using an example, if you use Go your’e probably familiar with go.mod files; but if you aren’t using it, go.mod files are used to manage your project dependencies. Here is a simple go.mod file: module github.com/matang28/ -latest require (
  github.com/some/dependency1 v1 github.com/some/dependency2 v5 )

replace github.com/some/dependency1 => github.com/some/dependency1@dev latest go go 1.13 .2 .3 .1 .0 // indirect before we can parse this file, we need two things: A stream of symbols that will be obtained from the file itself. A grammar definition that will be used by the parser to validate the syntax. The first one is formally called a Lexer (or lexing). Lexers are all about turning strings into a list of predefined symbols that our parser uses, any symbol (or token) used by our language should be identified by the Lexer. In our case we need to capture the following tokens: Keywords: , , , . module go require replace Strings such as: , . github.com/some/dependency1 1.13 Version string such as: , . v1.2.3 v1.4.21-haijkjasd9ijasd Other misc symbols such as: , whitespaces, line-terminators and tabs. => , // , ( , ) The second one will give these tokens a syntactic meaning, you can see that files follows some basic rules: go.mod You have to declare your own module name with the directive. module You may declare you go version using the directive. go You may list your dependencies with the directive. require You may replace certain dependencies with the directive. replace You may exclude certain dependencies with the directive. exclude These rules are often called the grammar of the language (the go.mod language) and there is a more formal ways to represent them, one of them is the EBNF standard: grammar GoMod; sourceFile 		: moduleDecl stmt* EOF;

stmt 			: (versionDecl | reqDecl | repDecl | excDecl); moduleDecl 		: STRING; versionDecl 	: STRING; reqDecl 		: dependency
        		| dependency* ; repDecl 		: STRING dependency; excDecl 		: dependency;

dependency 		: STRING version comment?;

version 		: VERSION | ;

comment 		: STRING;

VERSION 		: [v]STRING+;
STRING 			: [a-zA-Z0- _+.@/-]+;
WS  			  : [ \n\r\t\u000C]+ -> skip; // A source file should be structured in the following way: // The module directive: 'module' // The version directive: 'go' // The require directive, it can be either a single line: // require github.com/some/dep1 OR multi line: // require ( //   github.com/some/dep1 //   github.com/some/dep2 // ) 'require' 'require (' ')' // The replace directive: 'replace' '=>' // The exclude directive: 'exclude' 'latest' '//' 9 Now back to the definition, a Parser takes a stream of tokens (e.g the tokenization of our text file) and will try to match each token against the grammar, if our file doesn’t follow the grammar we will get a parse error but, if our file does follow the grammar we know for sure that the input file is valid (in term of syntax) and we can get the parse tree which we can traverse throughout the matched grammar rules. Parsing is all about making this magic happen… taking a stream of tokens and building a parse tree from it (iff the syntax matches the grammar). Whats so different? At the beginning of the article I’ve said that one library was the spark to create this pet project, why is that? Well parse trees aren’t the most comfortable data structure to extract data from, “walking” on a parse tree can be pretty confusing. Most of the tools available today uses code generation techniques to generate lexers and parsers from a grammar file and, in order to be as generic as possible most of them give you a listener interface which will get notified when the parser enters or exits each grammar rule. Participle uses different approach, one that’s more common for Go developers. It uses Go’s structs system to represent the grammar (or more precisely the parts that we want to capture from the grammar), it also utilizes “struct tags” to define the grammar rules. Wow! that was a mouthful, let’s learn by doing. At first we need some structs that represents a standard `go.mod` file: GoModFile {
	Module Statements []Statement
} Statement {
	GoVersion    * Requirements []Dependency
	Replacements []Replacement
	Excludes     []Dependency
} Dependency {
	ModuleName Version Comment    * } Replacement {
	FromModule ToModule   Dependency
} // The root level object that represents a go.mod file type struct string type struct string // A struct that represents a go.mod dependency type struct string string string // A struct that represents a replace directive type struct string In order to write a parser, we need a stream of tokens. Participle got us covered by giving us a regex-based lexer (their docs says that they’ve been working on an EBNF-style one that could be really useful since this is the most cumbersome part of Participle IMO). Let’s see that in action: ( ) iniLexer = lexer.Must(lexer.Regexp( + + + + ,
)) import "github.com/alecthomas/participle/lexer" // The lexer uses named regex groups to tokenize the input: var /* We want to ignore these characters, so no name needed */ `([\s\n\r\t]+)` /* Parentheses [(,)]*/ `|(?P<Parentheses>[])` /* Arrow [=>]*/ `|(?P<Arrow>(=>))` /* Version [v STRING] */ `|(?P<Version>[v][a-zA-Z0-9_\+\.@\-\/]+)` /* String [a-z,A-Z,0-9,_,+,.,@,-,/]+  */ `|(?P<String>[a-zA-Z0-9_\+\.@\-\/]+)` Now we can move on to define our grammar, think of it as a combination between the EBNF file and our Go structs representation. Here are some of the basic syntax operators Participle has to offer: Use quotes to match a symbol for example: will try to match the symbol . "module" module Use to capture expression (i.e named group defined in our lexer), for example will match the symbol and extract it into the struct field. @ @String String Use to let the underlaying struct to match the input, this is useful when you have multiple alternatives for a rule. @@ Use the common regex operators to capture groups: to match zero or more, to match at least one, to match zero or one, etc... * + ? GoModFile {
	Module Statements []Statement } Statement {
	GoVersion    * Requirements []Dependency Replacements []Replacement Excludes     []Dependency } Dependency {
	ModuleName Version Comment    * } Replacement {
	FromModule ToModule   Dependency } // The root level object that represents a go.mod file type struct string `"module" @String` `@@*` type struct string `( "go" @String )` `| (("require" "(" @@* ")") | ("require" @@))` `| (("replace" "(" @@* ")") | ("replace" @@))` `| (("exclude" "(" @@* ")") | ("exclude" @@))` // A struct that represents a go.mod dependency type struct string `@String` string `(@Version | @"latest")` string `("//" @String)?` // A struct that represents a replace directive type struct string `@String "=>"` `@@` All we have to do now is to build the parser from our grammar: {
	p, err := participle.Build(&GoModFile{},
		participle.Lexer(iniLexer),
	) err != { , err
	}

	ast := &GoModFile{}
	err = p.ParseString(source, ast) ast, err
} func Parse (source ) string (*GoModFile, error) if nil return nil return And we’re done, we can parse `go.mod` files now! Let’s write a simple test to make sure everything works as expected: {
	file, err := Parse( )
	assert.Nil(t, err)
	assert.NotNil(t, file)

	assert.EqualValues(t, , file.Module)
	assert.EqualValues(t, , *file.GoVersion)
	assert.EqualValues(t, , (file.Requirements))

	assert.EqualValues(t, , file.Requirements[ ].ModuleName)
	assert.EqualValues(t, , file.Requirements[ ].Version)
	assert.EqualValues(t, , *file.Requirements[ ].Comment)

	assert.EqualValues(t, , file.Requirements[ ].ModuleName)
	assert.EqualValues(t, , file.Requirements[ ].Version)
	assert.Nil(t, file.Requirements[ ].Comment)

	assert.Nil(t, file.Replacements)
} func TestParse_WithMultipleRequirements (t *testing.T) `
  module github.com/matang28/go-latest
  go 1.12
  require (
  	github.com/bla1/bla1 v1.23.1 // indirect
  	github.com/bla2/bla2 v2.25.8-20190701-fuasdjhasd8
  )
` "github.com/matang28/go-latest" "1.12" 2 len "github.com/bla1/bla1" 0 "v1.23.1" 0 "indirect" 0 "github.com/bla2/bla2" 1 "v2.25.8-20190701-fuasdjhasd8" 1 1 Where is the pet project? Picking files for the examples wasn’t accidental, the tool I’ve decided to build is a simple automation tool. If you use Go in your organization you probably know that editing file manually can be tedious. go.mod go.mod to the rescue! will scan for files recursively, patching every dependency that matches your query to latest . go-latest go-latest go.mod For example, for this file tree: .
├── go.mod
├── subModule1
│   └── go.mod
└── subModule2
    └── go.mod If I want to patch dependencies that matches my organization name to their latest version I’ll type: $> go-latest ”organization.com” . See?! Iv’e told you, pet projects are fun :) and I don’t care that we’ve could solve this problem by using simple regex expressions because this is the essence of pet projects! For more info about go-latest . checkout it’s Github page And as always, thanks for reading... Picture Credits (by the order of their appearance): Photo by on Avi Richards Unsplash Photo by on Camylla Battani Unsplash Photo by on Caleb Woods Unsplash Photo by on Curology Unsplash Photo by on Josh Rakower Unsplash

Stories, News & Mentions about LEXER

Lexer’s Stories on HackerNoon

Lexer's News & Mentions