Publishing private NPM packages to Nexus

We’ve all worked on projects which give us an opportunity to build reusable components. Most of the times these components end up in the shared folder of the project. This folder then gets copy-pasted into several projects which over time become a nightmare to update as we cannot easily have multiple versions of a component and maintaining the same code base on multiple branches as versions is kind of a hacky solution to this problem. In this article, we will take a look at the Nexus Repository Manager (a.k.a. Nexus), which is an open source repository manager provided by . We will discuss how to create a private repository (backed by Amazon S3) and combine it with the public NPM registry to provide a full-fledged solution that can keep our private repositories private, allow version control, and cache repositories as well. Sonatype Towards the end, we will create two application, one of which will be pushed to Nexus and the other would consume from Nexus. Throughout the article, we will only be dealing with Nexus deployed locally. Evaluating Need Before we jump into the details, it’s worth considering the need for something like Nexus vs the private repositories offered by npmjs (for a price). Personally I feel that the repositories hosted on npmjs is a easier and cleaner solution if your organization does NOT have any cloud infrastructure in place. Hosting Nexus, regular backups, health checks are all an overhead if not already built into your cloud infrastructure. Let us assume for now that all of that is in place and carry on. Table of Contents: Running Nexus Repository Manager locally Understanding Volumes Creating a Blob Creating Hosted Repository Creating NPM proxy and group Pushing Binaries to Nexus Pulling Binaries from Nexus Conclusion Running Nexus Repository Manager locally Thanks to the increased popularity of Docker, running any software nowadays is as easy as finding the right (docker) image. Luckily, Sonatype provides the docker image for Nexus which can be easily pulled locally using the following pull command. docker pull sonatype/nexus Once the image is pulled. To run, just execute the following command below: docker run --rm -it -p 8081:8081/tcp sonatype/nexus3:latest This brings up the container running our instance of Nexus. To try it out, open http://localhost:8081 in the browser, and, the default credentials are admin/admin123 There are a lot of different features available in Nexus. At this time, we will only work with the ones needed to set up our private repository for NPM but the concepts can be easily applied to other forms of projects as well (maven, NuGet, docker etc). Basic Concepts Nexus exposes a group of repositories which is internally a combination of our private repository (a.k.a hosted repository in Nexus terms) and a proxy to the public NPM registry. Proxy to the public registry is necessary because we still need a way to access all the publicly available repositories from the NPM registry. Whenever we consume these public packages, they get cached in the proxy which we will see towards the end of the article. Hence, when we want to install a new private or public NPM package, we point our projects registry to the group (using .npmrc) so that it can install any package necessary (either with npm or yarn). And, to create or to update an existing repository point the publish action to the hosted repository (using publishConfig option in package.json). This is why we see a default set of group/hosted/proxy repositories combination created for us when we load Nexus for the first time. group = hosted + proxy. Read from group and write to hosted repository User Management When bootstrapping Nexus, a default user is created for us (and for everyone who uses Nexus). So pretty much every Nexus user around the world knows what the default username and password are. This is why we should delete/disable the default user once we create a new admin user. We have only two roles by default but we can always create more roles (which is a combination of privileges) and add them to the users as we see fit. Although this is not mandatory, it is highly recommended to create custom roles and only assign it to users based on their need. Next, let us prevent unauthorized users from accessing our server, click on Anonymous under Security and uncheck the option to allow accessing the server: Now that we have the new admin called npmuser created and prevented the anonymous users from accessing our repositories, we are ready to move onto the next step. Understanding Volumes Whenever we run docker images as containers, all the information enclosed is stateless i.e. if our container restarts for any reason, the data could get lost. For example, in our case, we have already created one admin user in our Nexus, if we were to restart the container, the user we created would not be available on restart since it is a brand new container which started from scratch. To bypass this problem (and other similar ones), Docker allows us to mount volumes to the container to which it can persist data. And in case of restarts, it can retain the information as long as the new container is remounted to the same volume as it did earlier. First, let us create a directory in which we will be placing all the nexus-data generated with respect to this example. mkdir nexus-data This is the folder which we will be using as a temporary volume for our Nexus image. We now need to provide the path to our volume while issuing the run command for the container: docker run --rm -it -p 8081:8081 -v /Users/../../nexus-data:/nexus-data sonatype/nexus3 The highlighted part above is the one which makes all the difference, we are specifying the complete path to our nexus-data directory and we are mounting it to the default data directory called nexus-data within the Nexus container. Additional properties and configurations are listed . here Once we run the command listed, we see different folders created under the nexus-data folder that we created. Any and all changes that we make in Nexus now would get synced back to this folder. If you are curious, create the npmuser with admin role again, stop the container and restart it. The newly created user would be persisted as expected. Creating a Blob We are now ready to create a Blob store which is a logical partition that we want to enforce for our different project types i.e. we want to segregate our NPM binaries and maven binaries to avoid any conflicts (name or otherwise). Internally, Nexus just creates different folders for each Blob store that we create. To create a Blob, go to the Settings page > Repository > Blob Stores > Create blob store This blob, once created, show up as expected in our volume: Any package that we upload now to that blob would get persisted in the volume under the folder associated with it. We are also going to use AWS S3 based blobs which is an alternative to File-based blobs provided by Nexus. To configure a Blob which is based on S3, simply select S3 from the Type dropdown and fill out the required information. A sample of the same is shown below: This ends up creating some default files in your S3 bucket: Any and all content that is pushed to the repository using this blob will get updated and managed by S3, however, the configuration which we provided to Nexus regarding our S3 bucket still remains on the volume that we mounted earlier. This is why it is important to ensure that we have regular backups of the volume. We want to be less dependent on the volume for multiple reasons: In the production cluster, we would have to keep backing up the volume attached with the container for Disaster Recovery (DR). Although this cannot be completely avoided (since we need to persist users, custom roles and other stateful information) we can reduce the backups to those that cannot be synced with the cloud such as S3. Backing up to S3 allows us to add additional rules on the AWS S3 bucket which can move our old packages to cold storage if needed. With that, we are ready to move on to the next stage of the process, creating repositories. Creating Hosted Repository As discussed earlier, Hosted repositories are the private repositories which we create to hold our private packages. What makes these repositories private is the inability to read the contents of these repositories without an authToken. We will see this in an example towards the end of the article. To create a hosted repository, go to the Settings page > Repository > Repositories > Create Repository. When we click on Create Repository, Nexus is kind enough to provide us with recipes which define how a certain type of repository needs to be configured, in our case, we are only concerned with the npm related recipes. Let us first select the npm(hosted) option, since that is what we want to start with. It requests two things from us, the name of the hosted repository and the blob in which the data needs to be persisted for that repository. Click on the Create Repository button to finish the creation of the repository. And that is it. We have created our private (hosted) repository for all our npm projects. Creating NPM proxy and group Now that we have the private repository set up, we are ready to create the npm proxy which proxies all our read requests to the public NPM registry. We can finish off our changes by combining the hosted and proxy repositories into a group. Creating Proxy Repository In create repository screen, select npm (proxy) which brings us to the configure page, we want to proxy to NPM public registry which is at the URL https://registry.npmjs.org. We only enter the 3 mandatory fields here: Name — npm-proxy Proxy Location — https://registry.npmjs.org Blob Store (for cache storage, configuration etc) — NPM-S3 (created earlier) This would create the proxy repository as expected. Creating Group Repository Creating a group repository as described earlier is to combine the hosted and the proxy repository which makes the reads much easier. Let us create the npm (group) repository similar to hosted and proxy repositories. This accepts similar configuration as it did earlier such as name, blob store etc. We are selecting both the npm-private and npm-proxy repositories and making them active members of this group. In future, if we have more compatible repositories, they would show up under the Available list which can be selected and moved over to the Members section. Pushing Binaries to Nexus Now that we are ready with the necessary repositories, we can now consume these in our projects as needed. Let us first create a sample NodeJS project with a blank index.js file and push it to our hosted repository: mkdir npm-app1 && cd npm-app1 npm init -y touch index.js To publish this project, we need to update our package.json file with the publishConfig which points to our new hosted repository. https://medium.com/media/ea254e88b78324f8d2b89b3f0ab6501d/href If we try to publish the project now with the npm publish command, we see the following error as expected saying that we need authentication: Let us add the auth using the same credentials as the Nexus dashboard (as recommended in the error message): Note: Notice the registry url contains the repository name Let us try that again! Now we see a different error, saying that it cannot make a Put request: This happens because of the way security works in Nexus, the concept of realms determine how any user can interact with Nexus. The default realm that is applied is called the Local Authentication and Local Authorization realm whose responsibility as per the documentation is as follows: They allow the repository manager to manage security setup without additional external systems. We need to add additional realms to enable npm publish feature. To enable the additional realms, go to Settings > Security > Realms. Add the npm Bearer Token Realm and save the changes. Additional information about the realms is available . here Now we are ready to retry our publish command and that works as expected: To verify, we can browse through our repository and see the package as expected. Select the Browse option from the top nav bar > Browse > npm-private to see the packages as shown below: We can also verify this to be uploaded in S3 under the folder called content, although it would not be legible to us as it is converted into a blob while storing. Pulling Binaries from Nexus Now that we have published the binary to Nexus, let us create another project from which we will try to consume npm-app1. mkdir npm-app2 && cd npm-app2 npm init -y Before we try to do npm install to retrieve npm-app1 from our repository, let us remove the credentials that we added when we did npm login earlier. We are doing this to avoid using those credentials to retrieve the package. We ideally would want to replicate the experience of a new developer who is working with our application. To remove these credentials, simply run the following command: npm logout --registry=http://localhost:8081/repository/npm-private/ Now, let us try to install the npm-app1 from our new npm-app2 project: npm i -S npm-app1 As expected, we see a 404 error since npm-app1 was not found on the public registry to which npm points by default. To fix this, we need to add a .npmrc file locally to the project. This .npmrc file contains the registry to which we need to point to pull our packages and any additional credentials needed. touch .npmrc We first add the registry to which we need to point to be able to retrieve our package registry=http://localhost:8081/repository/npm-group/ Notice that we are pointing to the npm-group and not the npm-private repository as we want to access both the private and the proxied npm public packages. We can see that the project is located but cannot be installed as we are not authorized. The simplest solution is to add the authToken needed to make this work. As a best practice, I recommend creating a new user who has a read-only role to all the repositories. We do this as we do not want to add admin auth token for Nexus to the source code. For this, create the role with all the privileges related to read and browse , the easiest way to do this is to provide the role with nx-repository-view-*-*-browse and nx-repository-view-*-*-read privileges Then create the user with this role: Next, to get the auth token of this user, perform an npm login pointing to our npm-group repository: Performing the login adds the credentials to the .npmrc file at the root of your computer. Open the global .npmrc file and locate the line which resembles the following: //localhost:8081/repository/npm-group/:_authToken=NpmToken.bb495270–9831–3046–8c24-a2978853d3a1 The _authToken is the last bit of the line printed above. We can now add the _authToken to our .npmrc file in npm-app2 project. We do this to avoid having to log in on every machine which clones this repository. Let us log out before we proceed further: npm logout --registry=http://localhost:8081/repository/npm-group/ Final form of the .npmrc file in the project looks as follows: https://medium.com/media/4816f23288de89d0241de2bec1db4c81/href With that, we should now be able to install and use any and all packages as shown below: And we can see caching of the proxied packages (lodash in this case) in action when we browse through the repositories: Conclusion This article only covered some basics on how to use Nexus as a repository manager. Although we run the container and mount the volume to a local directory, it is highly recommended that you try out the above on a cloud provider of your choice. Keep in mind that in the case of cloud providers (or self-hosting) you would need to back-up the volume for disaster recovery. If you enjoyed this blog be sure to give it a few claps, read more or follow me on LinkedIn .