Automating My Personal Note-Taking Compilation System

The Rochester Institute of Technology (RIT) has one of the largest populations of deaf, NTID (National Technical Institute for the Deaf), and hard-of-hearing students in the world. As a result, there are a myriad of services available such as captioning, ASL interpreting, and note taking. As a student at RIT, I work as a NTID note taker for some of my classes. Generally, I take notes and upload the NTID-specific notes to the RIT note taker website, but I also upload all of my notes to my personal website so that I can access them from anywhere if I need to study.

A snapshot of the webpage to access my personal notes.

I take my notes using the LaTeX typesetting system, so the notes need to be compiled into a PDF via the pdflatex command. This is slightly annoying for me since I need to SSH into the server hosting my website every time I update my notes in order to recompile them. Since I’m a programmer, I’m lazy and I don’t want to do that. Let’s automate this process. In this techsploration, I’m going to discuss how I did that.

My Workflow

I store my notes in a separate repository on my GitHub. I use a gulpfile to automatically recompile my notes when I’m taking them during class so that I can view the updates live in Atom (my text editor of choice) using Atom’s pdf-view package. Here’s a screenshot of that workflow.

My note-taking workflow

This works great for me since I can see my notes updating live every time I save the document.

Writing My Own Gulp Plugin

Previously, I used a makefile to compile the .tex files since I only wanted to compile changed files. I invoked the make command from a gulpfile so that I could use gulp’s file watcher to automatically do it whenever I saved the file, but this turned out to be convoluted and overly contrived. While I was writing this post, I realized using a gulpfile to invoke a makefile to compile .tex files into PDF files was stupid. I decided to take out the makefile and have the compilation and change monitoring done entirely in the gulpfile.

After some searching I found two existing gulp plugins that did LaTeX compilation: gulp-latex and gulp-pdflatex. The first one, gulp-latex, errored inexplicably for me and I gave up debugging it after fiddling with its options for a few days. I tried gulp-pdflatex, and it worked fantastically as long as my .tex file didn’t have any syntax errors.

Looking at the source code, I found that gulp-pdflatex relied on a package called node-pdflatex to do the compilation. Both packages were written by the same author. The package didn’t seem to return error data, instead choosing to just emit the message “Error during LaTeX compilation” if compilation failed.

A snippet of the compiled code for node-pdflatex, which was natively written in TypeScript.

Damn it.

It worked well when the .tex files had no syntax errors, but it wasn’t very informative when there was one. To compile .tex files, the package invokes the pdflatex command in a subprocess using the child_process node module. It stores the files outputted by pdflatex in a temporary directory and then returns either a Stream or Buffer object with the content of the outputted PDF file.

This was relatively easy to reverse engineer, so I wrote my own version of gulp-pdflatex for my note taking system, adding functionality that would append the stdout and stderr of pdflatex to the error. Instead of separating the functionality into two packages, I combined it into one package and published it as a gulp plugin named gulp-pdflatex2. In retrospect, I probably should’ve forked gulp-pdflatex and sent a pull request. ¯\_(ツ)_/¯

As for only compiling changed files, that was easy to do with the gulp-changed plugin.

Automating Compilation On My Website

I had displayed my notes on my website by cloning my notes repository into my website’s static folder. I just needed a way to automatically pull and invoke the gulpfile’s compilation task.

GitHub has a feature known as a webhook which allows an external service to detect when events such as pushing to a repository have occurred. I set up a webhook on my notes repository to detect when I pushed an update to it so that it could send a POST request to my personal website when that happened.

The webhook management page for my notes repository.

Now I just needed a handler on my web server for this. To authenticate requests from GitHub’s webhook, you need to compute a hash for the request using a mutually known secret token.

The snippet on top, auth_route.js, is a bit of middleware that computes the hash of the request. The snippet on the bottom, router.js, compares the two and invokes the commands:

git pullgulp cleangulp latex

This pulls any changes I’ve made to the notes repository, cleans out log files and extraneous files, and recompiles any changed files. Here’s a link to the gulpfile in the notes repository to see the specifics of each gulp task.

And that’s it! My personal website will now pull and recompile my LaTeX notes every time I push an update to them.

That’s all I have for now. Thanks for reading! Please leave a response if you’ve got a better way of doing this or have any ideas for improvements and hit the ❤ button down below if you enjoyed this article :)

Follow me on Twitter: @omgimanerd