Few things are more satisfying to me than one elegant line of Bash that automates hours of tedious work. As part of some recent explorations into automatically re-creating my laptop with Bash scripts (post to come!), I wanted to find a way to easily clone my GitHub-hosted repositories to a new machine. After a bit of digging around, I wrote a one-liner that did just that. Then, in the spirit of not putting all our eggs in the same basket, I wrote another one-liner to automatically create and push to GitLab-hosted backups as well. Here they are. A Bash one-liner to clone all your GitHub repositories Caveat: you’ll need a list of the GitHub repositories you want to clone. The good thing about that is it gives you full agency to choose just the repositories you want on your machine, instead of going in whole-hog. You can easily clone GitHub repositories without entering your password each time by using HTTPS with your or, my preferred method, by . For brevity I’ll assume we’re going with the latter, and our SSH keys are set up. 15-minute cached credentials connecting to GitHub with SSH Given a list of GitHub URLs in the file , like this: gh-repos.txt git@github.com:username/first-repository.git
git@github.com:username/second-repository.git
git@github.com:username/third-repository.git We run: xargs -n1 git < gh-repos.txt clone This clones all the repositories on the list into the current folder. This same one-liner works for GitLab repositories as well, if you substitute the appropriate URLs. What’s going on here? There are two halves to this one-liner: the input, counterintuitively on the right side, and the part that makes stuff happen, on the left. We could make the order of these parts more intuitive (maybe?) by writing the same command like this: <gh-repos.txt xargs -n1 git clone To run a command for each line of our input, , we use . The tool reads items from input and executes any commands it finds (it will if it doesn’t find any). By default, it assumes that items are separated by spaces; new lines also works and makes our list easier to read. The flag tells to use argument, or in our case, one line, per command. We build our command with , which then executes for each line. Ta-da. gh-repos.txt xargs -n1 xargs echo -n1 xargs 1 git clone xargs A Bash one-liner to create and push many repositories on GitLab GitLab, unlike GitHub, lets us do this nifty thing where we don’t have to use the website to make a new repository first. We can . The newly created repository defaults to being set as Private, so if we want to make it Public on GitLab, we’ll have to do that manually later. create a new GitLab repository from our terminal The GitLab docs tell us to push to create a new project using , but I don’t find this to be very convenient for using GitLab as a backup. As I work with my repositories in the future, I’d like to run one command that pushes to both GitHub GitLab without additional effort on my part. git push --set-upstream and To make this Bash one-liner work, we’ll also need a list of repository URLs for GitLab (ones that don’t exist yet). We can easily do this by copying our GitHub repository list, opening it up with Vim, and doing a : search-and-replace cp gh-repos.txt gl-repos.txt
vim gl-repos.txt
:%s/\<github\>/gitlab/g
:wq This produces , which looks like: gl-repos.txt git@gitlab.com:username/first-repository.git
git@gitlab.com:username/second-repository.git
git@gitlab.com:username/third-repository.git We can create these repositories on GitLab, add the URLs as remotes, and push our code to the new repositories by running: awk -F gl-repos.txt '\/|(\.git)' '{system("cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push")}' Hang tight and I’ll explain it; for now, take note that should be the full path to the directory containing our GitHub repositories. ~/FULL/PATH/ We do have to make note of a couple assumptions: The name of the directory on your local machine that contains the repository is the same as the name of the repository in the URL (this will be the case if it was cloned with the one-liner above); Each repository is currently checked out to the branch you want pushed, ie. . master The one-liner could be expanded to handle these assumptions, but it is the humble opinion of the author that at that point, we really ought to be writing a Bash script. What’s going on here? Our Bash one-liner uses each line (or URL) in the file as input. With , it splits off the name of the directory containing the repository on our local machine, and uses these pieces of information to build our larger command. If we were to the output of , we’d see: gl-repos.txt awk print awk ~/FULL/PATH/first-repository && git remote -url origin --add git@gitlab.com:username/first-repository.git && git push ~/FULL/PATH/second-repository && git remote -url origin --add git@gitlab.com:username/second-repository.git && git push ~/FULL/PATH/third-repository && git remote -url origin --add git@gitlab.com:username/third-repository.git && git push cd set cd set cd set Let’s look at how we build this command. Splitting strings with awk The tool can split input based on . The default separator is a whitespace character, but we can change this by passing the flag. Besides single characters, we can also use a . Since our repository URLs have a set format, we can grab the repository names by asking for the substring between the slash character and the end of the URL, . awk field separators -F regular expression field separator / .git One way to accomplish this is with our regex : \/|(\.git) is an escaped / character; \/ means “or”, telling awk to match either expression; | is the capture group at the end of our URL that matches “.git”, with an escaped character. This is a bit of a cheat, as “.git” isn’t strictly splitting anything (there’s nothing on the other side) but it’s an easy way for us to take this bit off. (\.git) . Once we’ve told where to split, we can grab the right substring with the . We refer to our fields with a character, then by the field’s column number. In our example, we want the second field, . Here’s what all the substrings look like: awk field operator $ $2 1: git@gitlab.com:username
2: first-repository To use the whole string, or in our case, the whole URL, we use the field operator . To write the command, we just substitute the field operators for the repository name and URL. Running this with as we’re building it can help to make sure we’ve got all the spaces right. $0 print awk -F gl-repos.txt '\/|(\.git)' '{print "cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push"}' Running the command We build our command inside the parenthesis of . By using this as the output of , each command will run as soon as it is built and output. The function creates a that executes our command, then returns once the command is completed. In plain English, this lets us perform the Git commands on each repository, one-by-one, without breaking from our main process in which is doing things with our input file. Here’s our final command again, all put together. system() awk system() child process awk awk -F gl-repos.txt '\/|(\.git)' '{system("cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push")}' Using our backups By adding the GitLab URLs as remotes, we’ve simplified the process of pushing to both externally hosted repositories. If we run in one of our repository directories, we’ll see: git remote -v origin  git@github.com:username/first-repository.git (fetch)
origin  git@github.com:username/first-repository.git (push)
origin  git@gitlab.com:username/first-repository.git (push) Now, simply running without arguments will push the current branch to both remote repositories. git push We should also note that will generally only try to pull from the remote repository you originally cloned from (the URL marked in our example above). Pulling from multiple Git repositories at the same time is possible, but complicated, and beyond the scope of this post. Here’s an to help get you started, if you’re curious. The may also be helpful. git pull (fetch) explanation of pushing and pulling to multiple remotes Git documentation on remotes To elaborate on the succinctness of Bash one-liners Bash one-liners, when understood, can be fun and handy shortcuts. At the very least, being aware of tools like and can help to automate and alleviate a lot of tediousness in our work. However, there are some downsides. xargs awk In terms of an easy-to-understand, maintainable, and approachable tool, Bash one-liners suck. They’re usually more complicated to write than a Bash script using or loops, and certainly more complicated to read. It’s likely that when we write them, we’ll miss a single quote or closing parenthesis somewhere; and as I hope this post demonstrates, they can take quite a bit of explaining, too. So why use them? if while Imagine reading a recipe for baking a cake, step by step. You understand the methods and ingredients, and gather your supplies. Then, as you think about it, you begin to realize that if you just throw all the ingredients at the oven in precisely the right order, a cake will instantly materialize. You try it, and it works! That would be pretty satisfying, wouldn’t it?

Fetch

Instantly

The Basics of Secure Application Architectures - Separation, Configuration, and Access

Easter Eggs And More

Too Long; Didn't Read

Make resilience your competitive advantage

How to write Bash one-liners for cloning and managing GitHub and GitLab repositories

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Basics of Secure Application Architectures - Separation, Configuration, and Access

AWK for the win

20,000 Leagues Under Your Shell

47 Stories To Learn About Bash

A Basic Guide to MySQL Tests Automation

The Basics of Secure Application Architectures - Separation, Configuration, and Access

AWK for the win

20,000 Leagues Under Your Shell

47 Stories To Learn About Bash

A Basic Guide to MySQL Tests Automation

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps