Let's Explore ARK Core v3: Tooling [Part 6]

This is Part 6 of the Let’s Explore ARK Core series which documents the development of the next major release of the ARK Core Codebase alongside some tips & tricks on how to get started with contributing and building your next idea today.

Introduction

In the previous article, we explored how the infrastructure of Core has changed, how it affects testing and what the benefits for developers are by making these changes. Today, we'll take a look at some of the major changes in regards to tooling and how and why certain dependencies were chosen and integrated or in some cases removed and replaced.

ARK Core v3.0 Github Repository

Replacing oclif with our own pluggable CLI

When we started to move away from running Core from source via git we had to find an easy way to integrate a solution that would get us up and running quickly. There are 2 big players in the JavaScript world when it comes to building CLIs, the most popular by far is Commander.js because it follows the spirit of JavaScript by not enforcing any structure on the user besides how they register their command with the CLI.

The other is oclif, a CLI framework developed by the folks over at Heroku. It has superb TypeScript support and enforces a structure that developers have to follow to develop their commands and plugins.

At the time this choice was made, it was clear that going with oclif was the right way as we had migrated to TypeScript just before implementing the CLI so the strictness it enforced on our codebase was a welcomed change as it would force every developer to do things in a structured way. Our CLI integration is almost exactly 1 year old and our needs have now outgrown oclif in terms of control and extensibility.

Extensibility

When we initially started to build the CLI we only had a very limited set of features we required. Things like checking for updates, suggesting a command if the user has a typo in their command, and obviously the ability to quickly add new commands.

oclif made all of those easy with oclif plugins and oclif hooks. The way plugins work in oclif makes it very easy to add any functionality you need on the fly and the most common scenarios are already covered by official and community plugins.

Hooks allow you to intercept into the starting process of the CLI and perform things like update checks before the actual CLI starts working and executes any commands.

All of those things worked great and still do but we needed more control over how the CLI could be extended and interacted with to allow other developers to implement new commands for our CLI through their own plugins.

To solve this problem, we set out to build our own small CLI framework which allows us to provide tight integration into Core to provide the best experience we can whilst giving developers full control over how the CLI works.

import { Commands, Container } from "@arkecosystem/core-cli";
import { Networks } from "@arkecosystem/crypto";
import Joi from "@hapi/joi";
import { parseFileSync } from "envfile";
import { existsSync } from "fs-extra";

/**
 * @export
 * @class Command
 * @extends {Commands.Command}
 */
@Container.injectable()
export class Command extends Commands.Command {
    /**
     * The console command signature.
     *
     * @type {string}
     * @memberof Command
     */
    public signature: string = "env:get";

    /**
     * The console command description.
     *
     * @type {string}
     * @memberof Command
     */
    public description: string = "Get the value of an environment variable.";

    /**
     * Configure the console command.
     *
     * @returns {void}
     * @memberof Command
     */
    public configure(): void {
        this.definition
            .setFlag("token", "The name of the token.", Joi.string().default("ark"))
            .setFlag("network", "The name of the network.", Joi.string().valid(...Object.keys(Networks)))
            .setFlag("key", "The name of the environment variable that you wish to get the value of.", Joi.string());
    }

    /**
     * Execute the console command.
     *
     * @returns {Promise<void>}
     * @memberof Command
     */
    public async execute(): Promise<void> {
        const envFile: string = this.app.getCorePath("config", ".env");

        if (!existsSync(envFile)) {
            this.components.fatal(`No environment file found at ${envFile}.`);
        }

        const env: object = parseFileSync(envFile);
        const key: string = this.getFlag("key");

        if (!env[key]) {
            this.components.fatal(`The "${key}" doesn't exist.`);
        }

        this.components.log(env[key]);
    }
}

With the ability to create plugins for both Core and the CLI, we hope to see plugins that leverage both which could, for example, be that the transaction pool plugin exposes the transaction pool for Core to use and also some CLI plugins to interact with it, such as flushing all stored transactions.

Testability

Testing commands has been a major pain point with oclif. It does provide some tooling to ease testing as can be seen at oclif testing but they are targeted specifically for MochaJS. We use Jest and don't intend to move away from it anytime soon so all of our toolings should play nicely with it. If it doesn't, we'll need to resolve that problem.

The result of those problems is that the new CLI has been built from the ground up with TDD. Applying TDD to this problem was an obvious choice because it would reveal any testing problems and unnecessary complexity while building out the tooling.

This has greatly helped to reveal some older bugs in the existing CLI and to make testing of commands and plugins a lot easier.

The result of this process is a new package called core-cli which is responsible for all tooling that revolves around building a CLI. Accompanying to this package there are a few helpers that have been added to core-test-framework which make it a breeze to manipulate and execute commands in our test suite.

import { Console } from "@arkecosystem/core-test-framework";
import { Command } from "@packages/core/src/commands/env-get";
import { ensureDirSync, ensureFileSync, writeFileSync } from "fs-extra";
import { dirSync, setGracefulCleanup } from "tmp";

let cli;
beforeEach(() => {
    process.env.CORE_PATH_CONFIG = dirSync().name;

    cli = new Console();
});

afterAll(() => setGracefulCleanup());

describe("GetCommand", () => {
    it("should get the value of an environment variable", async () => {
        writeFileSync(`${process.env.CORE_PATH_CONFIG}/.env`, "CORE_LOG_LEVEL=emergency");

        let message: string;
        jest.spyOn(console, "log").mockImplementationOnce(m => (message = m));

        await cli.withFlags({ key: "CORE_LOG_LEVEL" }).execute(Command);

        expect(message).toBe("emergency");
    });

    it("should fail to get the value of a non-existent environment variable", async () => {
        ensureFileSync(`${process.env.CORE_PATH_CONFIG}/.env`);

        await expect(cli.withFlags({ key: "FAKE_KEY" }).execute(Command)).rejects.toThrow(
            'The "FAKE_KEY" doesn\'t exist.',
        );
    });

    it("should fail if the environment configuration doesn't exist", async () => {
        ensureDirSync(`${process.env.CORE_PATH_CONFIG}/jestnet`);

        await expect(cli.withFlags({ key: "FAKE_KEY" }).execute(Command)).rejects.toThrow(
            `No environment file found at ${process.env.CORE_PATH_CONFIG}/.env`,
        );
    });
});

As you can see, testing anything related to the CLI is now a breeze and isn't coupled to any specific testing tools or framework. This will hopefully make it easier for us and other developers to ensure that the CLI is working as intended and not performing any unwanted actions on a user's system.

Replacing pg-promise with TypeORM

When Core 2.0 was initially developed, we were using JavaScript so PG Promise was an obvious choice for our database layer as it was well maintained and the most popular package for PostgreSQL integrations into node.js applications. The other choice was Sequelize but was discarded later on as it had major performance issues and a lot of known bugs and unresolved issues.

The problem with it is that it locks us into a single database engine without the ability to swap it out for something like SQLite while running tests or on devices with lower hardware specifications.

After the move to TypeScript in early 2019, we started to look at TypeScript alternatives to enforce more strictness on the structure and to provide a cleaner interface for developers that want to interact with the database.

The result of that search is TypeORM. As the name suggests is an ORM that supports a wide variety of platforms and database engines which made it a perfect fit for us to achieve our goals of having a database-agnostic backend. The integration of TypeORM was fairly straightforward and the result is an integration that provides developers with more access to the internals that are responsible for the database.

With TypeORM, we will be able to support more platforms for development that have fewer resources available and continue to remove dependencies that are required to run Core. Overall this will help us to lower the barrier of entry and make development easier by removing the need for Docker.

Replacing various third-party dependencies

Core 2.0 heavily relied on third-party packages even for the most basic tasks, simply for convenience. Lodash and dozens and dozens of other small packages are littered all over the codebase and there just is no chance to know what all of those dependencies do and a lot of them will have poor performance when used with larger datasets.

This is where ARK’s Utils comes in. This package provides performance-oriented implementations of commonly used functions in TypeScript. The aim of this package is to free us from the plethora of third-party dependencies that we use in all our projects and replace them with alternative implementations with a focus on performance.

Lodash has been completely removed from Core, smaller packages have been replaced by native functions or smaller alternative implementations and many more functions are being added.

This ensures that all of our developers know what our dependencies are doing and that performance adjustments can be made as needed without having to rely on others or waiting for a PR to be merged.

Relying less on third-party dependencies also means that we can tailor everything specifically to our use-cases and don't need to worry about what people might want to do and thus ending up with a very generic solution that has performance issues that will bite us sooner or later.

What’s next?

This concludes Part 6 of the Let’s Explore Core series. This is the last part of this series and we hoped you enjoyed and learned something new.

I want to help with development

That's great news! If you want to help out, our GitHub repositories are wide open, but that is not all, we also have special Monthly Development GitHub Bounties on-going where you can earn money for every valid and merged Pull-Request. 

To learn more about the program please read our Bounty Program Guidelines blog post and to learn more about ARK Core and blockchain visit our Learning Hub.