Implementing Client Load Balancing With Spring Cloud: A Guide

Introduction It is common for micro-service systems to run more than one instance of each service. This is needed to enforce resiliency. It is therefore important to distribute the load between those instances. The component that does this is the load balancer. Spring provides a Spring Cloud Load Balancer library. In this article, you will learn how to use it to implement client-side load balancing in a Spring Boot project. Client and Server Side Load Balancing We talk about client-side load balancing when one micro-service calls another service deployed with multiple instances and distributes the load on those instances without relying on external servers to do the job. Conversely, in the server-side mode, the balancing feature is delegated to a separate server that dispatches the incoming requests. In this article, we will discuss an example based on the client-side scenario. Load Balancing Algorithms There are several ways to implement load balancing. We list here some of the possible algorithms: Round robin: the instances are chosen one after the other sequentially, in a circular way (after having called the last instance in the sequence, we restart from the first). Random choice: the instance is chosen randomly. Weighted: the choice is made by a weight assigned to each node, based on some quantity (CPU or memory load, for example). Same instance: the same instance previously called is chosen if it's available. Spring Cloud provides easily configurable implementations for all the above scenarios. Spring Cloud Load Balancer Starter Supposing you work with Maven, to integrate Spring Cloud Load Balancer in your Spring Boot project, you should first define the release train in the dependency management section: org.springframework.cloud spring-cloud-dependencies 2023.0.0 pom import Then you should include the starter named spring-cloud-starter-loadbalancer in the list of dependencies: org.springframework.cloud spring-cloud-starter-loadbalancer ... Client Load Balancing Configuration We can configure our component using the application.yaml file. The @LoadBalancerClients annotation activates the load balancer feature and defines a configuration class by the defaultConfiguration parameter. @SpringBootApplication @EnableFeignClients(defaultConfiguration = BookClientConfiguration.class) @LoadBalancerClients(defaultConfiguration = LoadBalancerConfiguration.class) public class AppMain { public static void main(String[] args) { SpringApplication.run(AppMain.class, args); } } The configuration class defines a bean of type ServiceInstanceListSupplier and allows us to set the specific balancing algorithm we want to use. In the example below we use the weighted algorithm. This algorithm chooses the service based on a weight assigned to each node. @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withWeighted() .build(context); } } Testing Client Load Balancing We will show an example using two simple micro-services; one that acts as a server and the other as a client. We imagine the client as a book service of a library application that calls an author service. We will implement this demonstration using a Junit Test. You can find the example in the link at the bottom of this article. The client will call the server through OpenFeign. We will implement a test case simulating the calls on two server instances using Hoverfly, an API simulation tool. The example uses the following versions of Java, Spring Boot, and Spring Cloud: Spring Boot: 3.2.1 Spring Cloud: 2023.0.0 Java 17 To use Hoverfly in our Junit test, we have to include the following dependency: io.specto hoverfly-java-junit5 test We will configure the load balancer in the client application with the withSameInstancePreference algorithm. That means that it will always prefer the previously selected instance if available. You can implement that behavior with a configuration class like the following: @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withSameInstancePreference() .build(context); } } We want to test the client component independently from the external environment. To do so, we disable the discovery server client feature in the application.yaml file by setting the eureka.client.enabled property to be false. We then statically define two author service instances, on ports 8091 and 8092: spring: application: name: book-service cloud: discovery: client: simple: instances: author-service: - service-id: author-service uri: http://author-service:8091 - service-id: author-service uri: http://author-service:8092 eureka: client: enabled: false We annotate our test class with @SpringBootTest, which will start the client's application context. To use the port configured in the application.yaml file, we set the webEnvironment parameter to the value of SpringBootTest.WebEnvironment.DEFINED_PORT. We also annotate it with @ExtendWith(HoverflyExtension.class), to integrate Hoverfly in the running environment. Using the Hoverfly Domain-Specific Language, we simulate two instances of the server application, exposing the endpoint /authors/getInstanceLB. We set a different latency for the two, by the endDelay method. On the client, we define a /library/getAuthorServiceInstanceLB endpoint, that forwards the call through the load balancer and directs it to one instance or the other of the getInstanceLB REST service. We will perform 10 calls to /library/getAuthorServiceInstanceLB in a for loop. Since we have configured the two instances with very different delays we expect most of the calls to land on the service with the least delay. We can see the implementation of the test in the code below: @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT) @ExtendWith(HoverflyExtension.class) class LoadBalancingTest { private static Logger logger = LoggerFactory.getLogger(LoadBalancingTest.class); @Autowired private TestRestTemplate restTemplate; @Test public void testLoadBalancing(Hoverfly hoverfly) { hoverfly.simulate(dsl( service("http://author-service:8091").andDelay(10000, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8091", "application/json")), service("http://author-service:8092").andDelay(50, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8092", "application/json")))); int a = 0, b = 0; for (int i = 0; i < 10; i++) { String result = restTemplate.getForObject("http://localhost:8080/library/getAuthorServiceInstanceLB", String.class); if (result.contains("8091")) { ++a; } else if (result.contains("8092")) { ++b; } logger.info("Result: ", result); } logger.info("Result: author-service:8091={}, author-service:8092={}", a, b); } } If we run the test, we can see all the calls targeting the instance with a 20 millisecond delay. You can change the values by setting a lower range between the two delays to see how the outcome changes. Conclusion Client load balancing is an important part of micro-services systems. It guarantees system resilience when one or more of the nodes serving the application are down. In this article, we have shown how it can be implemented by using Spring Cloud Load Balancer. You can find the source code of the example of this article on GitHub. Introduction It is common for micro-service systems to run more than one instance of each service. This is needed to enforce resiliency. It is therefore important to distribute the load between those instances. The component that does this is the load balancer . Spring provides a Spring Cloud Load Balancer library. In this article, you will learn how to use it to implement client-side load balancing in a Spring Boot project. load balancer Spring Cloud Load Balancer Spring Boot Client and Server Side Load Balancing We talk about client-side load balancing when one micro-service calls another service deployed with multiple instances and distributes the load on those instances without relying on external servers to do the job. Conversely, in the server-side mode, the balancing feature is delegated to a separate server that dispatches the incoming requests. In this article, we will discuss an example based on the client-side scenario. Load Balancing Algorithms There are several ways to implement load balancing. We list here some of the possible algorithms: Round robin: the instances are chosen one after the other sequentially, in a circular way (after having called the last instance in the sequence, we restart from the first). Round robin: the instances are chosen one after the other sequentially, in a circular way (after having called the last instance in the sequence, we restart from the first). Random choice: the instance is chosen randomly. Random choice: the instance is chosen randomly. Weighted: the choice is made by a weight assigned to each node, based on some quantity (CPU or memory load, for example). Weighted: the choice is made by a weight assigned to each node, based on some quantity (CPU or memory load, for example). Same instance: the same instance previously called is chosen if it's available. Same instance: the same instance previously called is chosen if it's available. Same instance: the same instance previously called is chosen if it's available. Spring Cloud provides easily configurable implementations for all the above scenarios. Spring Cloud Load Balancer Starter Supposing you work with Maven , to integrate Spring Cloud Load Balancer in your Spring Boot project, you should first define the release train in the dependency management section: Maven org.springframework.cloud spring-cloud-dependencies 2023.0.0 pom import org.springframework.cloud spring-cloud-dependencies 2023.0.0 pom import Then you should include the starter named spring-cloud-starter-loadbalancer in the list of dependencies: spring-cloud-starter-loadbalancer org.springframework.cloud spring-cloud-starter-loadbalancer ... org.springframework.cloud spring-cloud-starter-loadbalancer ... Client Load Balancing Configuration We can configure our component using the application.yaml file. The @LoadBalancerClients annotation activates the load balancer feature and defines a configuration class by the defaultConfiguration parameter. application.yaml @LoadBalancerClients defaultConfiguration @SpringBootApplication @EnableFeignClients(defaultConfiguration = BookClientConfiguration.class) @LoadBalancerClients(defaultConfiguration = LoadBalancerConfiguration.class) public class AppMain { public static void main(String[] args) { SpringApplication.run(AppMain.class, args); } } @SpringBootApplication @EnableFeignClients(defaultConfiguration = BookClientConfiguration.class) @LoadBalancerClients(defaultConfiguration = LoadBalancerConfiguration.class) public class AppMain { public static void main(String[] args) { SpringApplication.run(AppMain.class, args); } } The configuration class defines a bean of type ServiceInstanceListSupplier and allows us to set the specific balancing algorithm we want to use. In the example below we use the weighted algorithm. This algorithm chooses the service based on a weight assigned to each node. ServiceInstanceListSupplier weighted @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withWeighted() .build(context); } } @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withWeighted() .build(context); } } Testing Client Load Balancing We will show an example using two simple micro-services; one that acts as a server and the other as a client. We imagine the client as a book service of a library application that calls an author service . We will implement this demonstration using a Junit Test. You can find the example in the link at the bottom of this article. book service library author service Junit The client will call the server through OpenFeign. We will implement a test case simulating the calls on two server instances using Hoverfly, an API simulation tool. The example uses the following versions of Java , Spring Boot , and Spring Cloud : Hoverfly , Hoverfly API Java Java Spring Boot Spring Cloud Spring Cloud : Spring Boot: 3.2.1 Spring Cloud: 2023.0.0 Java 17 Spring Boot: 3.2.1 Spring Boot: 3.2.1 Spring Cloud: 2023.0.0 Spring Cloud: 2023.0.0 Java 17 Java 17 To use Hoverfly in our Junit test, we have to include the following dependency: Hoverfly io.specto hoverfly-java-junit5 test io.specto hoverfly-java-junit5 test We will configure the load balancer in the client application with the withSameInstancePreference algorithm. That means that it will always prefer the previously selected instance if available. You can implement that behavior with a configuration class like the following: withSameInstancePreference @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withSameInstancePreference() .build(context); } } @Configuration public class LoadBalancerConfiguration { @Bean public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(ConfigurableApplicationContext context) { return ServiceInstanceListSupplier.builder() .withBlockingDiscoveryClient() .withSameInstancePreference() .build(context); } } We want to test the client component independently from the external environment. To do so, we disable the discovery server client feature in the application.yaml file by setting the eureka.client.enabled property to be false . We then statically define two author service instances, on ports 8091 and 8092: eureka.client.enabled false spring: application: name: book-service cloud: discovery: client: simple: instances: author-service: - service-id: author-service uri: http://author-service:8091 - service-id: author-service uri: http://author-service:8092 eureka: client: enabled: false spring: application: name: book-service cloud: discovery: client: simple: instances: author-service: - service-id: author-service uri: http://author-service:8091 - service-id: author-service uri: http://author-service:8092 eureka: client: enabled: false We annotate our test class with @SpringBootTest , which will start the client's application context. To use the port configured in the application.yaml file, we set the webEnvironment parameter to the value of SpringBootTest.WebEnvironment.DEFINED_PORT. We also annotate it with @ExtendWith(HoverflyExtension.class) , to integrate Hoverfly in the running environment. @SpringBootTest application.yaml webEnvironment SpringBootTest.WebEnvironment.DEFINED_PORT. @ExtendWith(HoverflyExtension.class) Hoverfly Using the Hoverfly Domain-Specific Languag e, we simulate two instances of the server application, exposing the endpoint /authors/getInstanceLB . We set a different latency for the two, by the endDelay method. Hoverfly Domain-Specific Languag /authors/getInstanceLB endDelay On the client, we define a /library/getAuthorServiceInstanceLB endpoint, that forwards the call through the load balancer and directs it to one instance or the other of the getInstanceLB REST service. /library/getAuthorServiceInstanceLB getInstanceLB REST We will perform 10 calls to /library/getAuthorServiceInstanceLB in a for loop. Since we have configured the two instances with very different delays we expect most of the calls to land on the service with the least delay. We can see the implementation of the test in the code below: /library/getAuthorServiceInstanceLB @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT) @ExtendWith(HoverflyExtension.class) class LoadBalancingTest { private static Logger logger = LoggerFactory.getLogger(LoadBalancingTest.class); @Autowired private TestRestTemplate restTemplate; @Test public void testLoadBalancing(Hoverfly hoverfly) { hoverfly.simulate(dsl( service("http://author-service:8091").andDelay(10000, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8091", "application/json")), service("http://author-service:8092").andDelay(50, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8092", "application/json")))); int a = 0, b = 0; for (int i = 0; i < 10; i++) { String result = restTemplate.getForObject("http://localhost:8080/library/getAuthorServiceInstanceLB", String.class); if (result.contains("8091")) { ++a; } else if (result.contains("8092")) { ++b; } logger.info("Result: ", result); } logger.info("Result: author-service:8091={}, author-service:8092={}", a, b); } } @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT) @ExtendWith(HoverflyExtension.class) class LoadBalancingTest { private static Logger logger = LoggerFactory.getLogger(LoadBalancingTest.class); @Autowired private TestRestTemplate restTemplate; @Test public void testLoadBalancing(Hoverfly hoverfly) { hoverfly.simulate(dsl( service("http://author-service:8091").andDelay(10000, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8091", "application/json")), service("http://author-service:8092").andDelay(50, TimeUnit.MILLISECONDS) .forAll() .get(startsWith("/authors/getInstanceLB")) .willReturn(success("author-service:8092", "application/json")))); int a = 0, b = 0; for (int i = 0; i < 10; i++) { String result = restTemplate.getForObject("http://localhost:8080/library/getAuthorServiceInstanceLB", String.class); if (result.contains("8091")) { ++a; } else if (result.contains("8092")) { ++b; } logger.info("Result: ", result); } logger.info("Result: author-service:8091={}, author-service:8092={}", a, b); } } If we run the test, we can see all the calls targeting the instance with a 20 millisecond delay. You can change the values by setting a lower range between the two delays to see how the outcome changes. Conclusion Client load balancing is an important part of micro-services systems. It guarantees system resilience when one or more of the nodes serving the application are down. In this article, we have shown how it can be implemented by using Spring Cloud Load Balancer. You can find the source code of the example of this article on GitHub . GitHub