Senior Software Engineer
Sounds crazy isn’t it?
Don’t get me wrong. Python is a powerful language. I love python. But I always felt python was missing something. It’s like the pretty girl that you never got to date.
In my honest opinion,
Compared to a project that uses NodeJs with npm, setting up the working environment for a medium to large scale Python project is really really really hard !!!
Let’s swim through a few ways that we can use npm to aid python to solve this problem.
Every NodeJs project has a
package.json file. You can define everything starting from the version to dependencies to scripts to custom configurations, all in one single file. Running a script is as easy as,
npm run <script-name>
I will skip the details of npm. You can read more about the power of npm in this scotch.io article.
How does this compare to Python?
One standard option in python for setting up a project is the setuptools package. Basic metadata configurations such as name, version, and author are trivial. It also offers many sophisticated options such as
scripts. But even a simple action such as removing some auto generated files is quite hard.
Let’s say for example, that you need to remove all the files in the
coverage/ directory after running the test suite. In python, if you want to automate this task, you need to do several things.
setuptoolsis being used)
subprocess.Popento execute the desired command(s).
That’s a lot of work compared to adding a one line entry
"clean_cov": "rm -rf coverage/*" to the
package.json file. Another benefit here is that, you can chain npm commands to couple tasks.
"clean_cov": "rm -rf coverage/*",python setup.py test
"test_and_clean": "npm run test && npm run clean_cov"
You can also add
post scripts to define the flow. With that the above code can be written as,
"test": "python setup.py test
"posttest": "rm -rf coverage/*",
On the other hand, I don’t feel that comfortable integrating some build commands to the source code of the project.
I personally consider pre-commit and pre-push git hooks as vital elements of a git repository. These are the unsung heroes who avoid accidental commits to a repository which helps to maintain a clean repository with clean commits.
In the project I am working on, we planned to have hooks corresponding to the following checks.
That’s just 5 of them, there are more checks. Obviously, running these checks manually is not an option. So I need to configure hooks.
I found several pip packages including git-pylint-hook, git-pre-push-hook and pre-commit which seemed to do what I wanted. But none of the libraries I could find had proper documentation. Customizing the plugins seemed like a nightmare.
I even felt that manually writing few shell scripts is easier than finding suitable python plugins.
Now let’s see how this can be done with the presence of a
I found these pre-commit and pre-push libraries for the job. Since I already have custom scripts in the package for running the above mentioned checks, all I had to do was install the libraries and add the relevant entries to the
package.json file. For example, I have the pre-push hooks configured as follows,
Now that was easy right???. I should also highlight here that the
post scripts will also work here. For our project however, we preferred specifying the checks explicitly.
An interesting library I found during while configuring these hooks is the lint-staged library. I suggest you to have a look at it too.
To explain why I needed this feature, I should first mention that I am using the intelliJ PyCharm as the IDE for writing python code. A missing feature in pycharm, which I would love to see in the near future is real-time feedback on pylint issues in the IDE. For example, eslint & flow integrates with Webstorm seamlessly.
Until they add that feature to pycharm, my best option was to configure pylint as an external plugin and run it manually before updating a python file.
However, with the presence of my savior, the
package.json,this became easier. The npm-watch library(and other similar libraries) provide us with the option of watching a specific file type in a given directory for changes and then executing an npm script.
For example, we can have an entry as follows for running pylint whenever a
Although I really enjoyed having a
package.json file in the python project repo, I received mixed feed back for this approach. It was frowned upon by some fellow developers saying that this approach was not pythonic.
I personally didn’t find an issue with it not being pythonic because I believe that engineers should be technology agnostic and that all these languages are just mere tools for getting some work done. Nevertheless to respect the needs of all team mates, I configured all the basic stuff mentioned in this repo without using npm. My next article will probably be based on that experience.