NPM as a build tool for a Python Project

Sounds crazy isn’t it?

Don’t get me wrong. Python is a powerful language. I love python. But I always felt python was missing something. It’s like the pretty girl that you never got to date.

In my honest opinion,

Compared to a project that uses NodeJs with npm, setting up the working environment for a medium to large scale Python project is really really really hard !!!

Let’s swim through a few ways that we can use npm to aid python to solve this problem.

1. Executing Scripts

Every NodeJs project has a **package.json** file. You can define everything starting from the version to dependencies to scripts to custom configurations, all in one single file. Running a script is as easy as,

npm run <script-name>

I will skip the details of npm. You can read more about the power of npm in this scotch.io article.

How does this compare to Python?

One standard option in python for setting up a project is the setuptools package. Basic metadata configurations such as name, version, and author are trivial. It also offers many sophisticated options such as cmdclass and scripts. But even a simple action such as removing some auto generated files is quite hard.

Let’s say for example, that you need to remove all the files in the coverage/ directory after running the test suite. In python, if you want to automate this task, you need to do several things.

1. Add an entry under cmdclass in setup.py(where setuptools is being used)
2. Write a new Python class extending distutils.command base class
3. Write a python code which will use subprocess.Popen to execute the desired command(s).

That’s a lot of work compared to adding a one line entry "clean_cov": "rm -rf coverage/*" to the package.json file. Another benefit here is that, you can chain npm commands to couple tasks.

"clean_cov": "rm -rf coverage/*","test": "

You can also add pre and post scripts to define the flow. With that the above code can be written as,

"test": "

On the other hand, I don’t feel that comfortable integrating some build commands to the source code of the project.

2. Configuring git hooks

I personally consider pre-commit and pre-push git hooks as vital elements of a git repository. These are the unsung heroes who avoid accidental commits to a repository which helps to maintain a clean repository with clean commits.

In the project I am working on, we planned to have hooks corresponding to the following checks.

1. Run pylint checks and evaluate the pylint score of the target files
2. Run pytest and make sure all tests pass
3. Avoid commits to master and develop branches
4. Make sure there are no forbidden words
5. Create a source distribution and ensure there are no build errors

That’s just 5 of them, there are more checks. Obviously, running these checks manually is not an option. So I need to configure hooks.

I found several pip packages including git-pylint-hook, git-pre-push-hook and pre-commit which seemed to do what I wanted. But none of the libraries I could find had proper documentation. Customizing the plugins seemed like a nightmare.

I even felt that manually writing few shell scripts is easier than finding suitable python plugins.

Now let’s see how this can be done with the presence of a package.json file.

I found these pre-commit and pre-push libraries for the job. Since I already have custom scripts in the package for running the above mentioned checks, all I had to do was install the libraries and add the relevant entries to the package.json file. For example, I have the pre-push hooks configured as follows,

"pre-push": ["git-branch","forbidden-words","pylint","test","build"]

Now that was easy right???. I should also highlight here that the pre and post scripts will also work here. For our project however, we preferred specifying the checks explicitly.

An interesting library I found during while configuring these hooks is the lint-staged library. I suggest you to have a look at it too.

3. Watch Mode

To explain why I needed this feature, I should first mention that I am using the intelliJ PyCharm as the IDE for writing python code. A missing feature in pycharm, which I would love to see in the near future is real-time feedback on pylint issues in the IDE. For example, eslint & flow integrates with Webstorm seamlessly.

Until they add that feature to pycharm, my best option was to configure pylint as an external plugin and run it manually before updating a python file.

However, with the presence of my savior, the package.json,this became easier. The npm-watch library(and other similar libraries) provide us with the option of watching a specific file type in a given directory for changes and then executing an npm script.

For example, we can have an entry as follows for running pylint whenever a .py changes.

"watch": {"pylint": {"patterns": ["app/**/*","utils/**/*"],"extensions": "py","quiet": true }}

Final Thoughts

Although I really enjoyed having a package.json file in the python project repo, I received mixed feed back for this approach. It was frowned upon by some fellow developers saying that this approach was not pythonic.

I personally didn’t find an issue with it not being pythonic because I believe that engineers should be technology agnostic and that all these languages are just mere tools for getting some work done. Nevertheless to respect the needs of all team mates, I configured all the basic stuff mentioned in this repo without using npm. My next article will probably be based on that experience.

How do you feel about this approach?

Do you have any pythonic alternatives?

Feel free to enlighten me and all other readers with a comment!!!