In many cases, custom metrics may be useful to make more ad-hoc configuration of autoscaling in AWS. For example, a custom metric can allow an autoscaling group to react more rapidly on spikes or to take into account the health state of hosts for concurrency based metrics. Let's see how to set up a custom metric and autoscaling policy for an autoscaling group with an application load balancer and target group using AWS Console.
Prerequisites: autoscaling group with the target group and the application load balancer are working, you can request to load balancer and receive responses from target instances. Detailed logging is enabled for the application load balancer.
Let's create ActiveConnectionCount per Healthy Host metric described in previous posts.
1) To create custom metric go to CloudWatch in AWS Console, click on Alarms -> Create Alarm.
2) In Create New Alarm screen click on Select Metric.
3) In Search field put ActiveConnectionCount and click on ApplicationELB
-> Per AppELB Metrics, select one corresponding to your load
balancer (if you have more than one load balancer - metrics will be shown
one per load balancer).
-> Per AppELB Metrics, select one corresponding to your load
balancer (if you have more than one load balancer - metrics will be shown
one per load balancer).
4) Click on All left to the Search field to return to the root of Metric hierarchy and remove ActiveConnectionCount from Search field.
5) In Search field put HealthyHostCount , then click ApplicationELB -> Per AppELB, per TG Metrics.
6) Select HealthyHostCount metric corresponding to the same load balancer that you selected on step 3.
7) Go to Graphed metrics tab.
8) Select Start with empty expression in the drop-down menu.
9) In Details field of Expression1 put m1/m2.
10) Select period 1 minute. Unselect m1 and m2, leave only e1 selected.
11) Click on Select metric.
12) Put Metric name and description, i.e. ConnectionsPerHosts.
13) Set number of datapoints.
14) If you don't have notification list click New List and create one with your email.
15) Select notification list.
16) Click on Create Alarm.
17) You can see AutoScaling action, but it is better to set up the policy from Autoscaling group screen.
18) On the Autoscaling groups screen go to Scaling Tab and click Add policy.
19) Select metric you've created, it should be visible, but may appear with some delay, in my case sometimes it took hours to appear, so don't hurry to decide that something is wrong with the custom metric. Next click Create button in the top right of the tab.
So, in such a manner, creating different Alarms, it is possible to configure very flexible policies.
The metric created - ActiveConnectionCount per Healthy Hosts - may be helpful to scale-in the application, that may stop to respond to health-checks if it is fully loaded. In general, this metric may be used both to rapidly react on spikes, regardless of application type and may be more effective than CPU based and RequestPerHost metrics.
Of course, in the most cases the policy should be configured programmatically and not from AWS console, but the described approach is useful to experiment with different parameters and select the best metric.
Previously published at https://georgiy.dev/2020-04-10-aws-autoscaling-3.html