You must read this before releasing your Golang API into production. Based on our true story at Kurio, how we struggling for every release because we are not doing it in the right ways. laptop on fire taken from google image search A few weeks ago, we are in just fixing our weird and undetected bug in our main services. We have tried so many ways to debugging and fixing it. The bug is not with the business logic. Because it has been running on production for a few weeks. But we always saved by our auto-scaling mechanism, so it’s just like running well. Kurio Until later, we figured out it, it is because our code, we are not doing it well. Architecture Fyi, we are using a microservices pattern in our architecture. We have one gateway API — we call it — that serve API to our user (mobile, and web). Because of its role similar to API gateway, so its task is only handled request from the user, then call the required services, and build the response to user. This , written fully in Golang. The reason to choose golang is another story that I won’t tell it here. main API main API If drawn in a picture, our system will look more like this. Kurio architecture The Problem We have struggled for a long time with our main API, that always down and returning a long response to our mobile-apps and sometimes causing our API can’t be accessed. Our API dashboard monitor just turns to red — to be honest, when our API dashboard monitor goes red, it was a dangerous thing and causing stress and panic and crazy to us the engineer 😈. Other things are, our CPU and memory usage are getting higher. If this happens we just restart it manually and wait until it runs again. Our API response time up-to 86 seconds for a single request. graph our API response time, and doing restart manually for safety. This bug really makes us frustrated, because we don’t have any log that specifically telling about this bug. We just have that the response time is so long. The CPU and memory usage increasing. It just like a nightmare. Phase 1 : Using Customized http.Client One thing we learn and really really learn when developing this service is, don’t trust the default config. We use a customized , instead using the default one from the http’s package, http.Client client:=http.Client{} //default We add some config based on our need. Because we need to reuse the connection, we make some configuration in our transport and control max-idle reusable connection. keepAliveTimeout:= 600 * time.Secondtimeout:= 2 * time.SeconddefaultTransport := &http.Transport{Dial: (&net.Dialer{KeepAlive: keepAliveTimeout,}).Dial,MaxIdleConns: 100,MaxIdleConnsPerHost: 100, } client:= &http.Client{Transport: defaultTransport,Timeout: timeout,} This configuration can help us to reduce the maximum of time to be used to calling another service. Phase 2: Avoiding Memory Leak From Unclosed Response’s Body What we learn from this phase is: If we want to reuse our connection pool to another service, we must read the response body, and close it. Because our is just calling another service, we make a fatal mistake. Our suppose to reuse the available connection from so whatever happens, we must read the response body, even we don’t need it. And also we must close the response body. Both of this used to avoid memory leak in our server. main API main API http.Client We forgot to close our response body in our code. And this things can cause a huge disaster for our production. The solution is: we close the response body and read it even we don’t need the data. req, err:= http.NewRequest("GET","http://example.com?q=one",nil)if err != nil {return err} resp, err:= client.Do(req)//=================================================// CLOSE THE RESPONSE BODY//=================================================if resp != nil {defer resp.Body.Close() // MUST CLOSED THIS}if err != nil {return err} //=================================================// READ THE BODY EVEN THE DATA IS NOT IMPORTANT// THIS MUST TO DO, TO AVOID MEMORY LEAK WHEN REUSING HTTP// CONNECTION//=================================================_, err = io.Copy(ioutil.Discard, resp.Body) // WE READ THE BODY if err != nil {return err} We fix this after reading a great article here: and here: http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in-go-golang/ http://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/ Phase 1 and Phase 2 and with the help of auto scaling success to reduce this bug. Well, to be honest, it never happens anymore for even 3 months since the last year: 2017. Phase 3: Timeout Control In Golang Channel After a few months run well, this bug happens again. At the first week of January 2018, one of our service that called by our , let says: is down. For some reasons, it cannot be accessed. main API So when our is down, our is going fired again. The API dashboard going red again, the API response time is going higher and slower. Our CPU and memory usage going very high even using the autoscaling. content service main API Again, we trying to find the root problem again. Well, after re-running the we run well again. content service For that’s case, we are curious, why this happens. Because we think, we have set the timeout deadline in , so in that case, this will never happen. http.Client Searching for the potential issue in our code, then we found some dangerous code. For more simple, the code more look like this * ps: this function is just an example, but similar in pattern with ours our code before fixed If we look the above code, it’s like nothing wrong. But this function is the most accessed, and have the heaviest call in our . Because this function will do 3 API call with huge processing. main API To improve this we make a new approachment using timeout-control on the channel. Because with the above style code — which using will wait until all the process done — we must wait for all the API call must finish so we can process and return the response to the user. WaitGroup This is one of our big mistakes. This code can make a huge disaster when one of our services died. Because there will be a long waiting until the died service recovered. With 5K call/s, of course, this is a disaster. First attempt solution: We modify it with adding a timeout. So our user will not wait for so long, they will just get an internal server error. trying to add timeout Phase 4: Timeout Control Using Context After doing our problem still not fully solved. Our still consume high CPU and memory. Phase 3, main API This happens because, even we already returning to our user, but our goroutine has still existed. What we want is, if we already return the response, then all the resource is also cleared, no exception, including goroutine and API call that running in the background. Internal Server Error Later after reading this article: http://dahernan.github.io/2015/02/04/context-and-cancellation-of-goroutines/ We found some interesting feature that we do not realize yet in golang. That was using to help cancellation in go-routine. Context Instead of using to use the timeout, we move to . With this new approachment, our service is more reliable. time.After context.Context Then we change our code structure again by adding context to function that related. final fixed code. using context So then we use for every goroutine call in our code. This helps us to release memory and cancel the goroutine call. Context In addition, for more controlled and reliable, we also pass the context to our HTTP request. using context on HTTP request With all of this setting and timeout control, our system is more safe and controllable. Lesson learned: Never used the default option in production.Just never used the default option. If you’re building a big concurrent A, just never use the default option. Read a lot, tried a lot, failed a lot, and gain a lot.We learn a lot from this experience, this experience only gained in real cases and real user. And I’m so happy being part of it when fixing this bug. * last updated: 18 January 2018: fixing some of the typos If you think this worth enough to read, give me a 👏 also share this on your twitter, facebook etc, so other people can read this. If you have a question or something, you could put a response below or just email me.
