744 reads

Gradle, Bazel and gRPC: A Song of Ice and Fire

by Alexey SoshinAugust 17th, 2019

Too Long; Didn't Read

Bazel is not aware of the fact that a Bazel plugin can’t work with Bazel. Instead of hardcoding our dependencies, we get them from command line. We use KrotoPlus plugin to create a wrapper around the Bazel project. We then use that wrapper to run Bazel in the same directory your project is in. Bazel will not be able to work with a well defined project structure. We need to use this wrapper to build Bazel and then generate classes based on that.

Company Mentioned

featured image - Gradle, Bazel and gRPC: A Song of Ice and Fire

Really sorry for the cheesy title. No Game of Thrones jokes below. Promise.

So, you’re using Bazel. Maybe because you had a monorepo before. Or maybe you just decided that you have too many modules in your Android project. Or just because it was hard to manage all those microservices you had in your neatly separated repos on GitHub. It doesn’t matter. Now you stuck with Bazel.

One of the things you’ve lost the moment you switched to Bazel was rich ecosystem of plugins that tools like Maven and Gradle provided.

Moreover, Gradle also allowed you writing DSLs and even custom tasks in Kotlin. All that goodness gone for good? Maybe not.

Let’s see how we can take one Gradle plugin, KrotoPlus in our case, and make it work with Bazel.

Before we start, let’s have a few words about KrotoPlus plugin and gRPC in general.

All gRPC libraries take

.proto

files as input and output three kinds of classes in your language of choice:

Messages
Clients
Server interfaces

Messages are just your plain data objects, for the sake of this discussion. They would have some fields, and those fields would have types. That’s all you need to know for now.

Clients, also called stubs, are what you use to call your gRPC service. To create a client you would usually specify the host and port of your gRPC server, and then you would call a method, passing it your message of choice.

Servers implement generated server interfaces and then begin to listen on the correct host and port to be able to serve clients.

What you need to take out of this is the follows:

gRPC Library: (proto files) => (generated classes)

Gradle likes to work with a well defined project structure, namely:

your-gradle-project
  src
    main
      proto

When you run your Gradle plugin and it “simply works”, that’s because Gradle was smart enough to recognise your Source Sets and apply plugins to them.

So, in case you had some

.proto

files in your

proto

directory, our Gradle plugin would detect them, and then generate some Java and Kotlin files in build directory:

your-gradle-project
  build <-- your generated files will appear there
  src
    main
      proto

Bazel, on the other hand, knows nothing of that neat structure.

Moreover, your

proto

files would be located outside of

your-gradle-project

directory, like so:

some-proto-project
   proto <-- some proto files
...
your-gradle-project
  src
    main
      proto <-- empty
...
other-proto-project
  api
    proto <-- more proto files

What we would like to do is to use our Gradle project sitting under Bazel as a function, that would get path to

.proto

files, as if they were part of its Source Set, and then generate classes based on those files.

There are a few ways to achieve the behavior with Gradle, but here’s one of them:

import com.google.protobuf.gradle.*
...
protobuf {
    ...
    generatedFilesBaseDir = "$buildDir/generated-sources"
    ...
    dependencies {
        protobuf(files(project.properties["protoDir"].toString()))
    }
    generateProtoTasks {
        ...
    }
}

The important part is the

dependencies

block. Here, instead of hardcoding our dependencies as we usually do, we get them from command line:

project.properties["protoDir"].toString()

What that means is that when we run our Gradle wrapper, we now pass relative path to the

.proto

files:

./gradlew generateProto -PprotoDir=../path/to/protos

Our Gradle is good and ready to go now. Next, let’s start working with Bazel.

One of your main tools when you want to hack with Bazel will be genrule.

This is your basic way to express that you would like to run something in Bash shell.

But what’s that “something”? It should be

gradlew

, same wrapper we used before. But the problem is, Bazel is not aware of it.

So, let’s create a file called

BUILD

in the same directory your

gradlew

is, and add the following block:

filegroup(
    name = "gradle",
    srcs = ["gradlew"],
)

That way we tell Bazel: hey, that’s a new file you should be aware of.

But if you try to run it like that, Gradle would complain that it can’t find its configuration.

That’s because we only told Bazel to copy

gradlew

to its sandbox, not anything else.

Let’s fix that by adding another

filegroup

that would hold the configurations:

filegroup(
    name = "gradle_config",
    srcs = ["build.gradle.kts", "krotoPlusConfig.asciipb", "settings.gradle"]
            + glob(["gradle/**/*"])
            + glob(["scripts/**/*"]),
    visibility = ["//visibility:public"]
)

As you can see, we grab

build.gradle.kts

, as well as configurations files for KrotoPlus, and Gradle wrapper itself, which is located under

/gradle

directory.

Now, to the rule itself. At first, it will look intimidating, but we’ll break it line by line.

genrule(
    name = "run_gradle",
    cmd = """
    ./$(location gradlew) -p $$(dirname ./$(location gradlew)) generateProto -PprotoDir=../proto/ &&
    cp -R $$(dirname ./$(location gradlew))/build/generated-sources  $(@D)
    """,
    outs = ["build/generated-sources"],
    message = "Generating protos",
    srcs = [],
    tools = [":gradle"] + [":gradle_config"],
)

```
name
```
is the label you want to refer to when specifying dependency of your library.
```
cmd
```
is the command that would run inside the shell. We’ll break it down later
```
outs
```
is what this command produces. Bazel tries to track every output of your script, so you need to tell it what you expect to change in the filesystem after you execute cmd. If directory build/generated-sources is not changed, Bazel will complain.
```
message
```
is what will be displayed while you wait
```
srcs
```
are the input files you would like to run your script on. Ideally, those would be your
```
.proto
```
files. But in this case, we’ll leave them empty, as we use
```
-PprotoDir
```
instead
```
tools
```
is the list of files you need to use to produce the output. In our case, it’s the Gradle Wrapper and all the configurations.

Now let’s go back to

cmd

and understand what’s going on inside.

When you start Bazel, its root is at

WORKSPACE

file location, not the directory where

gradlew

is located. So, we use

$(location gradlew)

to expand label to its relative location. In our example, that would be something like

./your-gradle-project

Once we located the Gradle Wrapper executable, we need to tell it where to find its configuration. We do that with

-p

, that’s standard Gradle flag. We know that this directory is again relative to our

WORKSPACE

, hence

./$(location gradlew)

. But this time, it must be a directory, and location will expand to path to file, so we use

$(dirname)

. But Bazel already uses

for

$(location)

. So we end up with screening the command:

$$(dirname)

Once we located the configuration, we need to tell Gradle which task to run. That’s

generateProto

And now we need to point Gradle to the location of our

.proto

files. But the context switches here, from Bazel to Gradle, so the path is relative to

gradlew

location now:

-PprotoDir=../proto/

Now Gradle and KrotoPlus work hard to generate our gRPC classes. The problem is, though, they’re generated in the Gradle directory, and not inside Bazel sandbox.

So, we need to copy them.

cp -R

is a standard shell command, and we already covered what

$$dirname()

stands for. The only part you’re not familiar with yet, is

$(@D)

That’s the output directory we specified with

outs

, and as a result, copies generated files where we want them to be.

Conclusions and next steps

Combining Bazel and Gradle to generate files is possible, if a bit cumbersome.

If you want to go down that path, your next steps should be:

Remember to cleanup after yourself.
```
rm -rf $$gradleDir/build
```
would be nice
You probably want to make a function out of this
```
genrule
```
. Syntax stays almost the same, but you need to prefix
```
genrule
```
with
```
native
```
when inside a function:
```
native.genrule
```
In the long run, using relative paths as we did with
```
-PprotoDir
```
isn’t the right choice. Although it’s useful for testing KrotoPlus plugins. Better choice would be to rely on labels.

Hopefully, if you’re already using Bazel, this article will set you on the right track.