A CLI for Customizable, Scalable, Low-overhead Cloud Use

DynaCLI is an open source, cloud-friendly Python library for converting pure Python functions into Linux Shell commands on the fly.

DynaCLI is ideal for automating routine development (including testing, deployment) and administrative (e.g. new user registration) tasks in a modern cloud software environment because it supports converting a virtually unlimited set of functions into OS Shell commands with minimal run-time and maintenance overhead.

The team here at BST LABS is committed to developing and supporting open source tools. We’ve found them to be valuable in our own development and deployment of software on the Cloud — using the Cloud AI Operating System — and hope you’ll find them useful as well.

In this interview with Asher Sterkin, General Manager and Head of Technology for BST LABS, we explain why the DynaCLI library is a more flexible CLI platform, what users can expect out-of-the-box, and how you can participate in the open source project. DynaCLI is available on GitHub.

DynaCLI flexibly converts Python functions into Linux shell commands

What’s the problem with existing CLI libraries?

Most libraries for command line interfaces assume building one monolithic tool, with all commands inside, deployed at client workstations. First, this is inefficient from a runtime overhead perspective. The irony is that every specific invocation of a monolithic CLI still runs only one command, so many CPU cycles are wasted for nothing. Also, with the traditional approach, when the overall number of commands gets above 10, the startup time of a CLI tool will be tens of seconds which is an annoying delay.

The second problem is the maintenance required with a traditional library approach. This problem is minor when the number of commands and client workstations is small, the toolset is very stable, and when there is no difference in the set of commands available for different types of users (e.g., developers and administrators). However, maintenance demands expand quickly with sophisticated use.

How did you come across this challenge?

BST LABS discovered the need for a new CLI approach while working on the Cloud AI Operating System. CAIOS essentially is a development framework that makes the Cloud easier, more powerful and faster to use.

With CAIOS, we anticipated a large number of commands, a large number of client workstations, and different access rights for administrators and developers. Further, user organizations should be able to customize the CLI to meet their needs. For that reason, we wanted the basic framework to have a minimal and very stable local footprint; all commands to come dynamically from cloud storage; different access rights to be available for administrators and developers at a minimum; and a seamless way for customers to extend CAIOS CLI with their own commands. In sum, DynaCLI was written to support the CAIOS ecosystem because no other tool could provide it with the capabilities it needed. All CAIOS commands are DynaCLI commands. The CAIOS CLI tool is just an executable DynaCLI entry point.

How does it work?

DynaCLI does not build all CLI commands upfront but rather forms an argument parser dynamically, thus supporting an unlimited, open-ended CLI structure. You can have dozens of packages and modules whose functions are to be converted to CLI without defining the CLI commands beforehand.

DynaCLI acts as a simple preprocessor for the standard argparse Python library. It scans the sys.argv array to process command-line arguments one by one, uses the importlib.import_module function to bring in feature packages/modules and command modules, uses the inspect.signature to understand command function arguments and the re.match function to extract help string per argument.

Let’s browse the creation and use of a Python function that generates a QR code reflecting a person’s vaccination status. In my country, this official digital document is known as a Green Pass or, informally, as a green badge.

First you can see the definition of the Python function, displayed via ‘-h’ help.

Next, here is the function being used via the command line interface:

Please share some use cases. When would a project need commands that continually evolve?

CAIOS is an open system. Let’s imagine there is a new cloud resource supported and you need to get some commands to get its status or configure it. For example, take the schema migration of an SQL database where the DBA wants to do something with the production DB. You need a new command — possibly even a new package of commands (say, for a new database which was not supported before) — or there is a new feature in an existing resource. In the traditional approach, you need to rebuild the existing CLI tool to incorporate these new commands and re-install the new version of the CLI tool at every client workstation. With DynaCLI, you just drop new commands at the right place in cloud storage and they’re immediately available for everybody (See Medium articles on creating a Serverless Cloud Import System).

Another use case would be when a CAIOS user organisation, or an IT vendor building software with CAIOS, wants to add a command or command package which serves their specific needs. In the traditional approach, they will need to develop a separate CLI tool and to install it at all client workstations alongside CAIOS CLI. With DynaCLI, they just need to drop these commands at their cloud group storage (needing to ensure there are no name clashes), then these commands will be available to all client workstations.

What can users expect out of the box? How fast is it to set up?

With CAIOS, DynaCLI comes ready for use out-of-the-box. For other applications, the DynaCLI entry point script is just a couple of lines of Python code, which could be prepared immediately. See, as an example, the ‘To do’ Tutorial in DynaCLI docs on GitHub.

How does this approach differ from other tools?

Unlike other existing solutions such as Click and Typer, there is no need for any function decorators. Further, unlike all existing solutions, including those built on top of standard argparse like Google python-fire, DynaCLI does not build all command parsers upfront, but rather builds dynamically a single command parser based on the command line inputs.

In the project’s GitHub repository docs, there is a further comparison between DynaCLI and the above-mentioned tools.

Is the user locked into an approach involving DynaCLI if they start using this CLI?

There’s no lock-in. Unlike other libraries, DynaCLI is completely frameworkless — individual commands and packages do not depend on it whatsoever.

What pre-reqs does a technologist need to know to use DynaCLI or to contribute to the open source?

Just good, old Python 3.9+.

What were the technical challenges creating the library?

We wanted to closely follow the Python import and function invocation conventions, which are rich and powerful. Sometimes, handling special cases requires a very deep understanding of how it’s supposed to work and out-of-the-box thinking about how to implement it correctly and economically.

At its core, DynaCLI is just a Python package structure interpreter that happens to be used for converting Python functions into commands. We might find other usages for this core technology. To keep things simple, however, we decided to give up on some features, which to the best of our knowledge could be implemented using basic features. We will continue keeping track of what may be missing and will consider extending the feature set where appropriate.

How would you like to see the open source project expand with community input?

We want the DynaCLI core to be very stable, that is, seldom modified. The most interesting scenario of extending the library would be to build advanced features on top of its core. For example, is it really necessary to include command chaining or even print out in the core? We hope the answer will be “no” — that they could be built on top of DynaCLI. Of course, bug fixes, optimizations and code quality enhancements always are welcome contributions from the community, too.

What’s the main insight a reader should take away about DynaCLI?

DynaCLI is a part of a broader “cloud-native Python runtime” effort being pursued by BST LABS and others. For developers who continue working on their private laptops, DynaCLI would be just yet another Python CLI library with some cool engineering tricks. For those who are looking for efficient ways to conduct Python development in cloud-native fashion, we hope DynaCLI will be a no-brainer for you.

BST LABS is making it easier for organisations to realize the full potential of cloud computing through a range of open source and commercial offerings. We are best known for CAIOS, the Cloud AI Operating System, a development platform featuring Infrastructure-from-Code technology. BST LABS is a software engineering unit of BlackSwan Technologies.

Also Published Here