Yes, Karate is not *true* BDD

It’s hardly been two months since Karate was open-sourced, and especially thanks to blog posts by Joe Colantonio and T. J. Maher — the response has been fantastic. Thanks Joe and T. J. for giving this brand-new open-source project — a much-needed boost in visibility !

On to the topic at hand. I’d like to explain how Karate relates to Cucumber and BDD. I expect this to be a very frequently asked question as Karate gets more widely adopted — and this blog post is intended to be the definitive reference that I can point folks to from now on.

Why Cucumber

EDIT / Jan-2019: As of version 0.9.0, Karate is no longer based on Cucumber (see why) — but retains the syntax. And if you are contemplating using BDD for API testing — you should really read what I posted on Stack Overflow.

The decision to build Karate on top of Cucumber-JVM and use the Gherkin standard — is turning out to be a great one. And here’s why:

• Free IDE plug-ins exist for both Eclipse and IntelliJ. Scripts look much more compelling with syntax-coloring, and best of all — you can ‘right-click’ and run a test without needing to write a single line of Java code. I really think that this is a huge deal, and makes web-API-testing more accessible to non-programmers. Even GitHub has built-in support for syntax-coloring, just look at the documentation — and I wouldn’t trade this away in a hurry.
• Behind the scenes, Cucumber-JVM provides ready-to-use integration with unit-test frameworks (such as JUnit and TestNG), has nice reports out of the box, and again — Karate gets all this for free. Standing on the shoulders of giants seems almost unfair sometimes. But that’s the beauty of open-source.
• Cucumber-JVM simplifies the whole process of parsing a script-file and handing control to custom Java code — and Karate can just focus on getting-right the ‘glue’ code that does the serious business of HTTP calls and JSON / XML data-assertions.
• And Karate gets the “BDD flavored” Given-When-Then syntax for free. More on this in a moment, but there should be absolutely no question that this makes tests so much more readable. Just look at the Karate ‘hello-world’ example and try to imagine it without the BDD syntax. It simply, just ain’t the same.

So is Karate BDD ?

Karate certainly gives you the option of using the BDD keywords such as Given, When, Then and And. And people familar with Cucumber already know that it doesn’t matter whether you use Given or When or Then, it is just “syntax sugar” and behind the scenes it is completely ignored.

But back to the question. I’d rather not lecture you on how BDD should actually be applied in practice because that is something best articulated by the creators and some opinionated users of Cucumber. Consider the following blog-posts a “must-read” if you are contemplating using Cucumber — or even BDD for that matter. Even if you think that you are doing BDD the Right Way™, I urge you to read these articles. You may be surprised.

1. The Worlds Most Misunderstood Collaboration Tool — by Aslak Hellesøy
2. You’re Cuking it Wrong — by Jonas Nicklas
3. Why Bother With Cucumber Testing? — by Jack Kinsella (which is a critical response to #2)

I’ll like to reproduce a quote from link #3 here below:

Cucumber has its uses, principally as a high level analysis tool on large, polyglot projects. That said, few programmers work in this kind of position, and acceptance tests beyond a list of the method names of regular integration tests seems wasteful. Cucumber, as used by the majority of Rails programmers, is no more than a clumsy wrapper over basic integration tests.

Bingo. I’m sure this will resonate with a lot of folks who have used some flavor of BDD at some point, to write tests for an existing system. And all the while wondering — wouldn’t a plain-old integration-test have sufficed ?

If Karate is a “mis-use” of Cucumber, so be it !

Karate unashamedly embraces the fact that it is for integration-testing of HTTP web-services. And it includes a whole lot of capabilities to simplify the business of doing just that. It competes directly with the likes of REST-assured in the web-service API testing-tool space — for example, take a look at this detailed head-to-head comparison of Karate vs REST-assured.

Karate makes sense especially for “platform” teams creating and maintaining web-services, and where product-owner involvement in acceptance-testing is not the highest priority.

FAQ

Once in a while, I run into a “BDD enthusiast” who takes one look at Karate and exclaims that it is an abomination upon this world. I’ll end this blog-post with a “Frequently Asked Questions” section that attempts to reason with some of these people, but I fear that in many cases this is a lost cause — because of deep-set, cognitive-dissonance. But anyhow, here goes.

Aren’t you just using Cucumber for the BDD hype ?

No. There are very good reasons, see the “Why Cucumber” section above.

This defeats the whole purpose of BDD and Gherkin ! There’s so much implementation detail in the scripts. BDD should be readable, executable specifications !

If there’s one thing that Karate does well — it is that it’s a “true DSL” for HTTP, JSON and XML. The whole point of a DSL is to simplify what would have taken multiple lines of code in a normal programming language — into one-liners.

But yes, if you are concerned that Karate looks like a programming-language, it certainly is, and by design. It is worth repeating that Karate is not for teams wishing to practice true BDD.

Who are the target users for Karate ? It clearly isn’t for non-technical business people. It looks like an abstraction, so will this be suitable for technical folks ?

Yes, Karate is for technical folks who need to test web-services and it is laser-focused on making this as simple as possible. The motivation for creating Karate in the first place was to avoid using Java code for web-services, as there is a significant impedance-mismatch. This deserves a whole other blog-post which I hope to do in the future.

Karate also adds some features missing in Cucumber — which arguably don’t belong in a BDD context, but are much-needed from an integration-testing perspective.

There is an argument to be made that some languages (e.g. Python and Ruby) can reduce the impedance-mismatch to some extent, but that is debatable — especially since Karate has JSON and XML baked “natively” into the syntax. Anyway, we are already seeing developers as well as quality-engineers create API test-automation at velocities much higher than before. One of the great things about Karate is that it is language-neutral, without sacrificing power or expressiveness for what it is designed to do.

I’m still not convinced. You are also polluting the world of BDD with Karate and you will lead people new to BDD down the wrong path.

Well, think of this blog post as my way of trying to guide users down the right path. I’ll also be prominently linking to this blog — from the Karate documentation.

That said, look at the cited blog posts and the fact that over the years, legions of programmers have fallen into the trap of mis-using Cucumber for integration testing. This is likely to continue for a while, even if Karate did not exist.

And in closing, don’t knock it till you try it ! I’m by no means un-biased, but I urge you to give Karate a try, or ask your team to evaluate it. I am confident that you will find some good things to say about it over time.