A further addendum to my very personal history of programming
How much wood could a woodchuck chuck?
Let me describe to you a recent programming job I did for a friend.
He asked me if I could create a web application that took a list of names and sent each name to a third-party web service which returned a list of results that could be imported into a spreadsheet. It bears mentioning that he already had an application that did this, but it was so difficult to use (another failed enterprise IT project) that he asked me for my help in creating a replacement. This is a classic example of shadow IT at its best.
Requirements (from his IT department) were that it be written in the Python programming language, and that any libraries or code snippets that I did not write myself used the MIT license (a type of open source license).
I am not a graphic designer, so the best way for me to make sure the app would be pleasant to look at was to search Google and Github and find an MIT licensed UI template. In about an hour I had one I liked, and by the end of the day I had a pretty looking web app that didn’t do anything. All the buttons and entry fields were there, they just weren’t wired up to anything.
Because my friends’ IT department requested that “the code that did the work” be written in Python, I could not do it in the browser with JavaScript (a relief to be honest). I searched Github for a Python web server and found very nice one named Bottle. I used the modified UI template as my web content, and by the end of the second day, I had a fully functional Python-based web server with a pretty little UI that promised everything but delivered nothing.
That was a good thing. Because it enabled me to have a discussion with my friend based on an actual on-screen UI. Which forced us to examine some of his assumptions and expectations and clarified some requirements before I wrote any code for them.
To manage the tables of results (before they got exported to the spreadsheet) I used what is likely the most widely-used open source library for tables in a web UI: a library named datatables.js. This is where the fun started.
Both datatables.js and the UI template I was using depended upon some CSS and JavaScript code libraries (Bootstrap and JQuery); just not the same ones.
Predictably, datatables.js would not work with the Bootstrap and JQuery that I had already setup for the UI template, and when I used the Bootstrap and JQuery from DataTables, the UI went to hell in a handbasket. It took me the better part of a day to sort out the dependencies, and import versions of Bootstrap and JQuery, and import other CSS and JavaScript in just the right order, so that everything worked the way it should. It has now been three days and only about two or three hours have been spent on the creative work of implementing the UI (what entry fields, buttons etc. and how to lay them out to be useable). The rest has been mostly in getting the different “moving parts” to work with each other.
My stack
So far, I have in the way of moving parts:
- Bootstrap CSS library for styling of HTML
- startbootstrap-landing-page template by Blackrock Digital LLC
- jQuery JavaScript library for HTML for DOM manipulation and event handling, using low version to guarantee compatibility with DataTables
- DataTables Plug-in for jQuery to add advanced interaction controls to HTML tables
- Python for server-side scripting
- Bottle, a Python-based WSGI Web Framework
Next I had to decide on the best way to perform the steps of:
- Sending a line from the list to the web service
- Getting the result and adding it to a table
- Loading the table data into datatables.js
There are at least a half a dozen ways to do this. The key challenge or decision is the best way to hand off the data from the Python virtual machine (which does the work of getting the response from the web service), to the JavaScript interpreter in the web browser (which does the work of displaying the datatable).
Do I collect all of the results in Python first and save them as a file, and then read the file into the datatable?
Do I build the datatable line by line in JavaScript with JavaScript calling Python over and over again until it gets nothing (end of list)?
Or is it better for datatables.js to call a handwritten JavaScript function that calls a Python function that returns a JSON object that gets passed back up the chain?
In spite of the fact that datatables.js is one of most widely-used JavaScript libraries in the world, the documentation for many of its functions is a little sketchy, and more than a little inconsistent. Following the documentation, I was simply unable to get it to work even after a few days (as I keep telling you: I am not such a great programmer). I turned to the programmer’s best friend: StackExchange. This is a website where programmers ask each other for help. My problem was a common one, and I quickly found an existing question and answer that enabled me to get things working. As usual, it was a matter of getting the quotation marks, commas, and semi-colons in right places. Really.
I have now spent a full week on this. In addition to the three hours of UI, I have added another two hours of real programming (figuring out how to split up the list into different categories of request and handle each type of request differently).
Out of a full week of work: only five hours spent “writing code” the way Mel did. Thirty-five hours spent getting different libraries to work with each other in much the same way fitters² used to file away to make parts fit.
I spent another five hours packaging the code with instructions to make sure the IT team could install it without issues, and the IT department will spend dozens, if not hundreds, of hours monitoring it operationally and applying upgrades and security patches.
I have spent at most 5% of my time actually writing code.
On my good days
I actually understand the code I am including. Other days, I just hack away it until it works without any (obvious) errors.
I know that I am not alone in this. In fact I am the rule, not the exception. The reality is: that frameworks have become more important than the languages they are based upon. In my next article, I will trace the evolution of this trend, starting with PHP-Nuke and Swing.
<- Previous article in series
[1] Prior to the 1950’s most parts used in manufacturing would not fit together “out of the box” and had to be individually fitted to each other by filing away at the parts to create a smooth fit. Fitter was the job title of the person who did the filing.
This article is an excerpt from my upcoming book The Chaos Factory which explains why most companies and government can’t write software that “just works”, and how it can be fixed.