Building Homebrew Taps for Private GitHub Repos

I started a new job in July 2022. While the developer experience is great, one of the things that I missed at my new job was a CLI to rule them all (like I had at my old job). I thought of building an easily extensible CLI tool which I did with go and cobra. To handle distribution to the engineers, we decided to use both go install and homebrew.

Homebrew is a very popular package manager for macOS and Linux users. Users can easily install new packages with brew install [PACKAGE]. Homebrew handles the downloading and the installation of the packages to the appropriate directories and then creates symlinks to /usr/local (on Intel machines).

To create a Homebrew package, we create Ruby classes called Formulas. A Formula looks like this:

class Wget < Formula
  homepage "https://www.gnu.org/software/wget/"
  url "https://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz"
  sha256 "52126be8cf1bddd7536886e74c053ad7d0ed2aa89b4b630f76785bac21695fcd"

  def install
    system "./configure", "--prefix=#{prefix}"
    system "make", "install"
  end
end

While adding an open-source package is fairly straightforward, doing the same with a private package involves a bit of hacking.

First Steps

The first thing to do is create a Github release, if you use go, I highly recommend using goreleaser. I wrote a tutorial on how to do that here. It truly makes releases in go easy. If you're using another language, you should check out how to manage releases in GitHub here.

Create a Homebrew Tap

The next thing to do is to create a Homebrew tap. A Homebrew tap is a GitHub repository that houses our Formulas. To add non-Homebrew core taps, we need to add the taps locally to homebrew with brew tap username/package.

For us to use the single argument form of brew tap, we need to follow the naming convention and name our GitHub repo homebrew-tap since the GitHub repo must start with homebrew-.

Create a Formula

If you have a release already in Github, we can go ahead with creating the Formula. In our homebrew-tap repo, we'll create a folder called Formula. In the Formula directory, we'll create a file called mytool.rb.

Now, in our Formula, we cannot use the standard way of downloading the packages since our package is hosted in a private repo. To do this we simply add a new download strategy like such:

require "formula"
require_relative "../custom_download_strategy.rb"

class Wget < Formula
  homepage "https://www.gnu.org/software/wget/"
  url "https://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy 
  sha256 "52126be8cf1bddd7536886e74c053ad7d0ed2aa89b4b630f76785bac21695fcd"

  def install
    system "./configure", "--prefix=#{prefix}"
    system "make", "install"
  end

  test do
      system "#{bin}/wget --help"
  end
end

Note the addition of , :using => GitHubPrivateRepositoryReleaseDownloadStrategy on the third line of the class.

In the root folder of homebrew-tap, we'll create a new file called custom_download_strategy.rb. Here, we'll add some code to support our download strategy.

require "download_strategy"

class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"

  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
    set_github_token
  end

  def parse_url_pattern
    unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
    end

    _, @owner, @repo, @filepath = *match
  end

  def download_url
    "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    curl_download download_url, to: temporary_path, timeout: timeout
  end

  def set_github_token
    @github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
    unless @github_token
      raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
    end

    validate_github_repository_access!
  end

  def validate_github_repository_access!
    # Test access to the repository
    GitHub.repository(@owner, @repo)
  rescue GitHub::HTTPNotFoundError
    # We only handle HTTPNotFoundError here,
    # becase AuthenticationFailedError is handled within util/github.
    message = <<~EOS
      HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
      This token may not have permission to access the repository or the url of formula may be incorrect.
    EOS
    raise CurlDownloadStrategyError, message
  end
end

# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add
# `:using => GitHubPrivateRepositoryReleaseDownloadStrategy` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def parse_url_pattern
    url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
    unless @url =~ url_pattern
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
    end

    _, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
  end

  def download_url
    "https://#{@github_token}@api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    # HTTP request header `Accept: application/octet-stream` is required.
    # Without this, the GitHub API will respond with metadata, not binary.
    curl_download download_url, "--header", "Accept: application/octet-stream", to: temporary_path, timeout: timeout
  end

  def asset_id
    @asset_id ||= resolve_asset_id
  end

  def resolve_asset_id
    release_metadata = fetch_release_metadata
    assets = release_metadata["assets"].select { |a| a["name"] == @filename }
    raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?

    assets.first["id"]
  end

  def fetch_release_metadata
    GitHub.get_release(@owner, @repo, @tag)
  end
end

We should have the following folder structure:

├── homebrew-tap
│   ├── Formula
│   │   ├── mytool.rb
└── custom_download_strategy.rb

Usage

We're almost done. To install our tool via Homebrew, we need to export a Github token that provides access to our private repos. A new token can be created here. The token needs to have repo permissions. We need to export the token as HOMEBREW_GITHUB_API_TOKEN. To export the token, we run:

export HOMEBREW_GITHUB_API_TOKEN=<GITHUB_TOKEN>
# if you're using fish shell like me, run `set -x HOMEBREW_GITHUB_API_TOKEN <GITHUB_TOKEN>`

Then, we can run the command:

brew install username/homebrew-tap/mytool

This automatically adds the homebrew tap and installs our tool.

Summary

• Create a GitHub release with the binaries. This can be done very easily with goreleaser in the case of go.
• Create a GitHub repo called homebrew-tap. An example is here.
  • Create a Formula folder.
  • In the root of the repo, create a file called custom_download_strategy.rb. The contents are shown above
  • In the Formula folder, create a ruby file with the name of your tool like mytool.rb
• Create a GitHub token and give repo permissions to the token.
• Export the token with export HOMEBREW_GITHUB_API_TOKEN=<GITHUB_TOKEN>
• Install the tool with brew using brew install username/homebrew-tap/mytool.

Also published here.