NYMGO

nymgo

Goroutines are a key feature of the Go programming language, allowing for efficient concurrent programming. However, improper use of goroutines can lead to leaks, where goroutines are left running indefinitely, consuming memory and other resources. This article will guide you through identifying and fixing goroutine leaks, ensuring your Go applications run smoothly and efficiently. Understanding Goroutine Leaks A goroutine leak occurs when goroutines that are no longer needed are not properly terminated. This can happen due to several reasons: Blocking Operations: Goroutines waiting on channels, mutexes, or other synchronization primitives that never release.
Infinite Loops: Goroutines stuck in loops that never exit.
Dangling Goroutines: Goroutines that are no longer referenced but still running. Detecting Goroutine Leaks Detecting goroutine leaks can be challenging, but there are several techniques and tools you can use: 1. Monitoring Goroutine Count Regularly monitoring the number of running goroutines can help identify leaks. You can use the runtime package to get the current goroutine count: package main

import (
	"fmt"
	"runtime"
	"time"
)

func monitorGoroutines() {
	for {
		time.Sleep(5 * time.Second)
		fmt.Printf("Number of goroutines: %d\n", runtime.NumGoroutine())
	}
}

func main() {
	go monitorGoroutines()
	// Your application code here
} 2. Profiling with pprof The pprof package provides profiling tools that can help identify goroutine leaks. You can generate and examine profiles to find goroutines that are not terminating. package main

import (
	"net/http"
	_ "net/http/pprof"
)

func main() {
	go func() {
		http.ListenAndServe("localhost:6060", nil)
	}()
	// Your application code here
} Run your application and navigate to http://localhost:6060/debug/pprof/goroutine?debug=2 to see a detailed goroutine profile. 3. Static Analysis Tools Static analysis tools like golangci-lint can help identify common patterns that lead to goroutine leaks. golangci-lint run 4. Monitoring: Install Prometheus: Follow the official Prometheus documentation to install Prometheus.
Export Go Metrics: Use the prometheus-go-client to export Go runtime metrics. package main

import (
	"net/http"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	go func() {
		http.Handle("/metrics", promhttp.Handler())
		http.ListenAndServe(":2112", nil)
	}()
	// Your application code here
} Configure Prometheus: Add a job to your prometheus.yml to scrape metrics from your Go application. scrape_configs:
  - job_name: 'go_app'
    static_configs:
      - targets: ['localhost:2112'] 4.0 Visualize Metrics using Prometheus: Use Prometheus UI to visualize the number of goroutines over time and set up alerts if the count exceeds a certain threshold. 4.1 Visualize metrics using Grafana: Use Grafana to visualize the number of goroutines over time and set up alerts if the count exceeds a certain threshold. Grafana can integrate with Prometheus to provide rich dashboards and alerting capabilities. Install Grafana: Follow the official Grafana documentation to install Grafana.
Add Prometheus Data Source: In the Grafana UI, add Prometheus as a data source by navigating to Configuration -> Data Sources -> Add a data source, and select Prometheus.
Import Dashboard: You can use pre-built dashboards or create your own. For example, import a Go Processes dashboard from Grafana's community dashboards: Go Processes Dashboard.
Create Alerts: Set up alerts in Grafana to notify you when the number of goroutines exceeds a predefined threshold. Here's an example of a Grafana dashboard visualizing goroutine metrics: Fixing Goroutine Leaks Once you've identified a goroutine leak, the next step is to fix it. Here are some common strategies: 1. Properly Closing Channels Ensure channels are closed to unblock goroutines waiting on them. package main

import (
	"fmt"
	"time"
)

func worker(ch <-chan int) {
	for val := range ch {
		fmt.Println(val)
	}
	fmt.Println("Worker done")
}

func main() {
	ch := make(chan int)
	go worker(ch)

	time.Sleep(2 * time.Second)
	close(ch)
	time.Sleep(1 * time.Second) // Give worker time to finish
} 2. Using Context for Cancellation The context package is a powerful way to manage goroutine lifecycles, allowing you to cancel goroutines when they are no longer needed. package main

import (
	"context"
	"fmt"
	"time"
)

func worker(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			fmt.Println("Worker stopped")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	go worker(ctx)

	time.Sleep(2 * time.Second)
	cancel()
	time.Sleep(1 * time.Second) // Give worker time to finish
} 3. Avoiding Infinite Loops Ensure loops within goroutines have proper exit conditions. package main

import (
	"fmt"
	"time"
)

func worker(done chan bool) {
	for {
		select {
		case <-done:
			fmt.Println("Worker done")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	done := make(chan bool)
	go worker(done)

	time.Sleep(2 * time.Second)
	done <- true
	time.Sleep(1 * time.Second) // Give worker time to finish
} Conclusion Goroutine leaks can cause your Go applications to use too much memory and run poorly. To avoid this, you can monitor the number of goroutines and use tools like pprof for profiling, and analyze your code with tools like golangci-lint. Prometheus and Grafana can help you keep an eye on goroutines and get alerts when something goes wrong. By following these steps, you can find and fix goroutine leaks, keeping your applications running smoothly and efficiently. Goroutines are a key feature of the Go programming language, allowing for efficient concurrent programming. However, improper use of goroutines can lead to leaks, where goroutines are left running indefinitely, consuming memory and other resources. This article will guide you through identifying and fixing goroutine leaks, ensuring your Go applications run smoothly and efficiently. Goroutines Understanding Goroutine Leaks A goroutine leak occurs when goroutines that are no longer needed are not properly terminated. This can happen due to several reasons: Blocking Operations: Goroutines waiting on channels, mutexes, or other synchronization primitives that never release. Infinite Loops: Goroutines stuck in loops that never exit. Dangling Goroutines: Goroutines that are no longer referenced but still running. Blocking Operations : Goroutines waiting on channels, mutexes, or other synchronization primitives that never release. Blocking Operations Infinite Loops : Goroutines stuck in loops that never exit. Infinite Loops Dangling Goroutines : Goroutines that are no longer referenced but still running. Dangling Goroutines Detecting Goroutine Leaks Detecting goroutine leaks can be challenging, but there are several techniques and tools you can use: 1. Monitoring Goroutine Count Monitoring Goroutine Count Regularly monitoring the number of running goroutines can help identify leaks. You can use the runtime package to get the current goroutine count: runtime package main

import (
	"fmt"
	"runtime"
	"time"
)

func monitorGoroutines() {
	for {
		time.Sleep(5 * time.Second)
		fmt.Printf("Number of goroutines: %d\n", runtime.NumGoroutine())
	}
}

func main() {
	go monitorGoroutines()
	// Your application code here
} package main

import (
	"fmt"
	"runtime"
	"time"
)

func monitorGoroutines() {
	for {
		time.Sleep(5 * time.Second)
		fmt.Printf("Number of goroutines: %d\n", runtime.NumGoroutine())
	}
}

func main() {
	go monitorGoroutines()
	// Your application code here
} 2. Profiling with pprof Profiling with pprof pprof The pprof package provides profiling tools that can help identify goroutine leaks. You can generate and examine profiles to find goroutines that are not terminating. pprof package main

import (
	"net/http"
	_ "net/http/pprof"
)

func main() {
	go func() {
		http.ListenAndServe("localhost:6060", nil)
	}()
	// Your application code here
} package main

import (
	"net/http"
	_ "net/http/pprof"
)

func main() {
	go func() {
		http.ListenAndServe("localhost:6060", nil)
	}()
	// Your application code here
} Run your application and navigate to http://localhost:6060/debug/pprof/goroutine?debug=2 to see a detailed goroutine profile. http://localhost:6060/debug/pprof/goroutine?debug=2 3. Static Analysis Tools Static Analysis Tools Static analysis tools like golangci-lint can help identify common patterns that lead to goroutine leaks. golangci-lint golangci-lint run golangci-lint run 4. Monitoring: Install Prometheus: Follow the official Prometheus documentation to install Prometheus. Export Go Metrics: Use the prometheus-go-client to export Go runtime metrics. Install Prometheus : Follow the official Prometheus documentation to install Prometheus. Install Prometheus official Prometheus documentation Export Go Metrics : Use the prometheus-go-client to export Go runtime metrics. Export Go Metrics prometheus-go-client package main

import (
	"net/http"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	go func() {
		http.Handle("/metrics", promhttp.Handler())
		http.ListenAndServe(":2112", nil)
	}()
	// Your application code here
} package main

import (
	"net/http"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	go func() {
		http.Handle("/metrics", promhttp.Handler())
		http.ListenAndServe(":2112", nil)
	}()
	// Your application code here
} Configure Prometheus: Add a job to your prometheus.yml to scrape metrics from your Go application. Configure Prometheus : Add a job to your prometheus.yml to scrape metrics from your Go application. Configure Prometheus prometheus.yml scrape_configs:
  - job_name: 'go_app'
    static_configs:
      - targets: ['localhost:2112'] scrape_configs:
  - job_name: 'go_app'
    static_configs:
      - targets: ['localhost:2112'] 4.0 Visualize Metrics using Prometheus : Use Prometheus UI to visualize the number of goroutines over time and set up alerts if the count exceeds a certain threshold. Visualize Metrics using Prometheus 4.1 Visualize metrics using Grafana: Use Grafana to visualize the number of goroutines over time and set up alerts if the count exceeds a certain threshold. Grafana can integrate with Prometheus to provide rich dashboards and alerting capabilities. Visualize metrics using Grafana: Install Grafana: Follow the official Grafana documentation to install Grafana. Add Prometheus Data Source: In the Grafana UI, add Prometheus as a data source by navigating to Configuration -> Data Sources -> Add a data source, and select Prometheus. Import Dashboard: You can use pre-built dashboards or create your own. For example, import a Go Processes dashboard from Grafana's community dashboards: Go Processes Dashboard. Create Alerts: Set up alerts in Grafana to notify you when the number of goroutines exceeds a predefined threshold. Install Grafana : Follow the official Grafana documentation to install Grafana. Install Grafana official Grafana documentation Add Prometheus Data Source : In the Grafana UI, add Prometheus as a data source by navigating to Configuration -> Data Sources -> Add a data source, and select Prometheus. Add Prometheus Data Source Import Dashboard : You can use pre-built dashboards or create your own. For example, import a Go Processes dashboard from Grafana's community dashboards: Go Processes Dashboard . Import Dashboard Go Processes Dashboard Create Alerts : Set up alerts in Grafana to notify you when the number of goroutines exceeds a predefined threshold. Create Alerts Here's an example of a Grafana dashboard visualizing goroutine metrics: Fixing Goroutine Leaks Once you've identified a goroutine leak, the next step is to fix it. Here are some common strategies: 1. Properly Closing Channels Properly Closing Channels Ensure channels are closed to unblock goroutines waiting on them. package main

import (
	"fmt"
	"time"
)

func worker(ch <-chan int) {
	for val := range ch {
		fmt.Println(val)
	}
	fmt.Println("Worker done")
}

func main() {
	ch := make(chan int)
	go worker(ch)

	time.Sleep(2 * time.Second)
	close(ch)
	time.Sleep(1 * time.Second) // Give worker time to finish
} package main

import (
	"fmt"
	"time"
)

func worker(ch <-chan int) {
	for val := range ch {
		fmt.Println(val)
	}
	fmt.Println("Worker done")
}

func main() {
	ch := make(chan int)
	go worker(ch)

	time.Sleep(2 * time.Second)
	close(ch)
	time.Sleep(1 * time.Second) // Give worker time to finish
} 2. Using Context for Cancellation Using Context for Cancellation Context The context package is a powerful way to manage goroutine lifecycles, allowing you to cancel goroutines when they are no longer needed. context package main

import (
	"context"
	"fmt"
	"time"
)

func worker(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			fmt.Println("Worker stopped")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	go worker(ctx)

	time.Sleep(2 * time.Second)
	cancel()
	time.Sleep(1 * time.Second) // Give worker time to finish
} package main

import (
	"context"
	"fmt"
	"time"
)

func worker(ctx context.Context) {
	for {
		select {
		case <-ctx.Done():
			fmt.Println("Worker stopped")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	go worker(ctx)

	time.Sleep(2 * time.Second)
	cancel()
	time.Sleep(1 * time.Second) // Give worker time to finish
} 3. Avoiding Infinite Loops Avoiding Infinite Loops Ensure loops within goroutines have proper exit conditions. package main

import (
	"fmt"
	"time"
)

func worker(done chan bool) {
	for {
		select {
		case <-done:
			fmt.Println("Worker done")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	done := make(chan bool)
	go worker(done)

	time.Sleep(2 * time.Second)
	done <- true
	time.Sleep(1 * time.Second) // Give worker time to finish
} package main

import (
	"fmt"
	"time"
)

func worker(done chan bool) {
	for {
		select {
		case <-done:
			fmt.Println("Worker done")
			return
		default:
			fmt.Println("Working...")
			time.Sleep(500 * time.Millisecond)
		}
	}
}

func main() {
	done := make(chan bool)
	go worker(done)

	time.Sleep(2 * time.Second)
	done <- true
	time.Sleep(1 * time.Second) // Give worker time to finish
} Conclusion Goroutine leaks can cause your Go applications to use too much memory and run poorly. To avoid this, you can monitor the number of goroutines and use tools like pprof for profiling, and analyze your code with tools like golangci-lint . pprof golangci-lint Prometheus and Grafana can help you keep an eye on goroutines and get alerts when something goes wrong. By following these steps, you can find and fix goroutine leaks, keeping your applications running smoothly and efficiently.

How to Find and Fix Goroutine Leaks in Go

Throughout my career, I have primarily focused on developing backend systems using . Despite dabbling in front-end work, I always found myself gravitating back toward development. In order to expand my skill set, I have experimented with writing backend applications using different programming languages such as JavaScript/Typescript and Rust. I have tested various technologies and frameworks, some of which I found more enjoyable than others. Nonetheless, there remains one language that I have yet to explore: Python API Go. When I first started using Golang, it felt a little unfamiliar due to its simplicity, unique error handling, and some unusual design choices like struct tags. However, even before getting fully accustomed to these "quirks", I quickly realized several things about Go that I genuinely appreciate. Interestingly, these same features can be seen as challenges in my primary programming language, Python. In this article, I will share my initial thoughts on using Go as a Python engineer. I will highlight the aspects of Go that I appreciate, which may not be as prominent in Python. These include: built-in tools packaging backend focus setting up Docker performance By the end of this article, you will understand why Go is an excellent option for backend applications and how it compares to Python in terms of convenience. Tooling When it comes to building software, the main task is writing code. However, there are several potential problems that can arise during this process. These can include: incorrect formatting bugs common code issues As engineers and humans, it's impossible to guarantee that our work is completely flawless. This is where automated tools like linters, unit tests, and static analysis can be incredibly useful in providing additional assistance. When working with Python, formatting can be a challenge. While the most popular tool for formatting is , it requires installation and configuration. However, configuration can be difficult as there are multiple ways to do it, with being the default option according to , but not all tools support it. black pyproject.toml PEP 518 In terms of linting, prior to the release of in 2022, Python developers had to rely on and its various plugins, which were inconvenient to configure. Thankfully, now supports many features and can be configured via the file. ruff flake8 ruff pyproject.toml Similarly, if you want more powerful testing capabilities than what is offered by the built-in , you'll need to install and configure additional tools and addons. It's important to ensure that all of these components work together properly in your CI system as well. unittest When working with Go, the development experience can be quite different. Upon installation, you will have access to a wide range of tools to aid your local development. Compared to Python, formatting rules in Go are less strict. For example, there is no fixed line length. However, Go has a built-in formatting tool that can be accessed through the command, eliminating the need for additional installation or configuration. go fmt Another useful tool in Go is the command, which helps to identify common coding mistakes such as unreachable code or useless comparisons. In addition, external linters like can be used to detect bugs and performance issues with ease. go vet staticcheck For testing purposes, Go provides a command that automatically discovers tests within your application and supports features such as caching and code coverage. However, if you require more advanced testing capabilities such as suites or mocking, you will need to install a toolkit like . Overall, while Go provides a highly effective testing experience, it's worth noting that writing tests in Python using is arguably one of the most enjoyable testing experiences I have encountered across all programming languages. go test testify pytest It's also important to remember that there's no need to use type-checking tools such as with Go, as it's a compiled and statically-typed language. mypy In summary, once you've installed Go, you'll have access to the necessary tools to support your code-writing efforts. The next step is to focus on packaging and managing dependencies. Packaging Managing dependencies is crucial when building software. While Golang has an extensive standard library, more complex projects often require external packages. Luckily, Go provides excellent tools for effectively managing dependencies. Go projects come with two files: and . The file is particularly crucial as it lists all the direct and indirect dependencies along with their versions. The file is automatically generated to serve as a checksum of the dependencies. go.mod go.sum go.mod go.sum To install a package, you use the command. The good news is that this command saves the installed packages in the directory, so there is no need to manually create virtual environments or similar things. go get <package> $GOPATH/pkg/mod In addition, you can use the command to ensure consistency between the file and the source code in the module. This command will add any missing module requirements needed to build the current module's packages and dependencies. If there are any unused dependencies, will remove them from . go mod tidy go.mod go mod tidy go.mod Finally, publishing packages with Go is incredibly simple. To install them, you typically just need to use the repository link, such as . So, to publish your own package, all you have to do is properly set up your module in the file, create a public repository, and request the package at . After a short period of time, your package will be available in the Go module index. That's it. go get github.com/tobias-piotr/gouth0 go.mod https://pkg.go.dev/ Now compare this with Python. By default, Python doesn't even have a checksum file. If you want more advanced features, you'll need to install a tool like , which has many external dependencies. Managing project dependencies can be done through a file, but you'll need to create and maintain it yourself. Compared to Go, publishing packages to PyPI can also feel inconvenient. poetry Without further criticism of Python packaging, Go's packaging feels much simpler and more convenient. The next chapter will cover how Go's setup is beneficial when working with Docker. Docker When it comes to building web applications in the modern world, Docker is usually involved. It simplifies the process of setting up a project locally and deploying it to production. Creating a Dockerfile for a Golang project is remarkably easy for two primary reasons: the Go tooling mentioned earlier and the fact that Go is a compiled language. Additionally, the combination of these two factors leads to a very standardized method of setting up a project. This means that if you copy a Dockerfile from one Golang project to another, there is a high probability that it will work without any issues. This is quite different from the situation in Python. With all of this in mind, take a look at a Dockerfile from one of my Go projects: FROM golang:1.20.6-alpine AS builder

EXPOSE 8080

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -o /main

# Local development
FROM builder as dev

ENV DEBUG=true

# Install air for hot reloading
RUN go install github.com/cosmtrek/air@latest

ENTRYPOINT ["air"]

# Production
FROM gcr.io/distroless/base-debian11 AS prod

WORKDIR /

COPY --from=builder /main /main

USER nonroot:nonroot

ENTRYPOINT ["/main"] Without diving too deep into the Docker syntax, this config simply downloads dependencies, compiles code, and runs the binary file. It's important to note that Go, being a compiled language, improves this process. For example, you can use a tool like to use the binary file and enable hot reloading while developing locally. Additionally, in production, a smaller base image can be used because no OS dependencies are required to start the binary file. This significantly reduces the image size of your app. air Starting a new project in Go is great because it's easy to set up a fully working and optimized Dockerfile. With tooling, packaging, and containerizing covered, we can finally focus on the language's features. Backend Focus When I first started working with Go, I had doubts about its suitability for backend development. However, it didn't take long for me to realize that Go is actually designed with backend development in mind. One of the main things that make Go well-suited for backend development is its standard library. This library includes a variety of useful packages for tasks such as encoding, cryptography, and database management. Most importantly, it also includes packages for building HTTP servers, which means you can create a semi-sophisticated API without needing to install any additional dependencies. For instance, I was able to develop an authentication API for interacting with Auth0 using just one external package (for structured logging). You can view the code here: . goloxy On top of that, there are many amazing packages available for developing web applications, including: web frameworks: Gin, Fiber, chi, Echo validation: validate, ozzo-validation, Valgo structured logging: zap, logrus, zerolog database interactions: GORM, sqlx, sqlc, Ent My preferred combination is Echo/chi as a web framework, Valgo for validation, zap for structured logging, sqlx for database interactions, and Fx for dependency injection. This setup has been effective for most of my projects, though it may vary depending on specific requirements. To highlight a difference with Python, configuring structured logging can be a challenge, even with a great framework like FastAPI. In an Echo app, I configure the zap logger in the following way: e.Use(middleware.RequestLoggerWithConfig(middleware.RequestLoggerConfig{
	LogProtocol:  true,
	LogMethod:    true,
	LogURI:       true,
	LogStatus:    true,
	LogRequestID: true,
	LogRemoteIP:  true,
	LogLatency:   true,
	LogError:     true,
	LogValuesFunc: func(c echo.Context, v middleware.RequestLoggerValues) error {
		logger.Info(
			fmt.Sprintf("%s %s %s %v", v.Protocol, v.Method, v.URI, status),
			zap.String("protocol", v.Protocol),
			zap.String("method", v.Method),
			zap.String("uri", v.URI),
			zap.Int("status", status),
			zap.String("request_id", v.RequestID),
			zap.String("remote_ip", v.RemoteIP),
			zap.String("latency", v.Latency.String()),
			zap.Error(v.Error),
		)
		return nil
	},
})) In addition to creating a logger instance, which is usually just a few lines, this is all that is required to have complete control over the logger for each request. An example of setting up structured logging in a FastAPI app can be found using the package here: (adding it between the backticks, because it is way too long to be embedded). This is surely overwhelming, even when you are fully aware of what's going on there. structlog https://gist.github.com/nymous/f138c7f06062b7c43c060bf03759c29e To summarize, Go has excellent built-in and externally installable solutions for developing web apps. However, implementation is just the beginning. The next chapter will cover the performance of Golang code. Performance When examining the most popular uses of Go, examples include proxies, infrastructure management, and "typical" web development. One of the reasons for its popularity is the exceptional performance of Go, especially when compared to Python, which is known to have weaker performance. There are numerous factors that contribute to Go's superior performance. such as: lightweight design, which reduces runtime overhead efficient garbage collection impressive concurrency model with Goroutines strong typing and static analysis These are just a few examples of the reasons why Go is so well-performing. All of these features enable the creation of highly optimized applications that can efficiently handle large workloads and easily scale even further. It's possible that one might argue that performance isn't crucial in many applications. In such cases, you could simply select Python, take code from a previous FastAPI project, base it around asyncio, and it should perform well enough. This argument may be valid. However, performance isn't just about speed; it also relates to resources. You might not notice speed differences when comparing an app written in Golang to one in Python - the differences are in nanoseconds. However, you will definitely see the difference in resource usage. The app developed in Go will use significantly fewer resources. This will be evident through lower costs. I recommend watching where he explains how rewriting a SAAS project in Go resulted in reduced costs. Ben Davis' video When it comes to performance, Python doesn't have the best reputation. This is largely due to the GIL, dynamic typing, and awkward concurrency model. While there are optimizations available, such problems are less frequent in Go. Therefore, when prioritizing performance, Go may be the better option. Summary Based on my experience with various technologies, I have some personal favorites. However, when building software, it is important to consider multiple factors before choosing the best tech stack. For backend development, Go stands out as an excellent option, especially when performance and cost are crucial. Nevertheless, it's not without any flaws. I believe that in modern Python, FastAPI and Pydantic are its strongest features for web development. They offer features like dependency injection, Swagger documentation, exceptional validation, and error handling right out of the box. I also find pytest to be the most well-designed test writing tool I've encountered. In summary, Go lacks some of the drawbacks that Python has, but Python has a wealth of features and libraries for app development. Ultimately, it depends on your specific needs and trade-offs. Also published here.

Gopher Pythonista #1: Transitioning from Python to Go

The Lion

The God of Tarzan

Tarzan's First Love

Inside Musk's Social Circle: Yes-men, Media Personalities, and (a lot of) right-wing VCs

Dentro del círculo social de Musk: sí-hombres, personalidades de los medios y (muchos) capitalistas de riesgo de derecha

À l'intérieur du cercle social de Musk : les hommes du oui, les personnalités des médias et (beaucoup) les VC de droite

मस्क के सोशल सर्कल के अंदर: हाँ-पुरुष, मीडिया हस्तियां, और (बहुत सारे) दक्षिणपंथी वीसी

Musk のソーシャル サークルの内部: イエスメン、メディアパーソナリティ、および (多くの) 右翼 VC

Dentro do Círculo Social de Musk: sim-men, personalidades da mídia e (muitos) VCs de direita

Bên trong vòng kết nối xã hội của Musk: Yes-men, Media Personalies, và (rất nhiều) nhà đầu tư mạo hiểm cánh hữu

马斯克的社交圈内幕：应声虫、媒体名人和（很多）右翼风投

Preface I was learning Golang recently. What interests me in Golang is the concurrency model. In Golang, we use goroutines to execute asynchronous tasks while these goroutines communicate with each other via the data structure called . goroutine + channel Channel As a frontend developer, I found that the goroutine + channel model in Golang is much like the event loop in Javascript. Javascript V8 engine uses the main thread to execute synchronous code while Golang uses the main goroutine. Javascript V8 engine maintains a thread pool to execute asynchronous code while Golang uses multiple goroutines. Javascript threads communicate with the main thread via a constantly polling event loop while Golang goroutines communicate with each other via . Channel So I am wondering if there is a way that we could control asynchronous tasks in Golang via pattern just like what we do in Javascript ES7. The answer is yes! Async/Await For more details → https://github.com/sun0day/async Async/Await Before we continue, let’s give a flashback about the common asynchronous model. There are two kinds of common asynchronous models: the blocking asynchronous models & non-blocking asynchronous models. The difference between a blocking asynchronous model and a non-blocking asynchronous model is whether the synchronous code execution should wait for the asynchronous task response to continue. In Javascript ES7, we use reserved word to tag a as an asynchronous task. Once you call a , it will be sent to another sub-thread to execute. Then the rest synchronous code will be executed immediately no matter whether the is done. The calling flow above is non-blocking asynchronous. We can also transform the calling flow to be blocking asynchronous using reserved word. async function async function async function async funciton async function await async function nonBlock() {...}

async function block() {...}

async function main() {
  nonBlock() // will not block main 

  await block() // will block main
} Golang provides a more granular and flexible way to control asynchronous task execution. But flexibility also means we have to write more code to control asynchronous task execution. And what’s worse is that we have to deal with exceptions such as goroutine leak, deadlock, panic, etc. Introducing Async/Await pattern in golang can help Goer to write more neat, readable, and robust code. Async Implementation in Golang In Golang we can convert a synchronous to be asynchronous using a named . The type definition is: func func Async Async func Async[V any](f func () V) func() *AsyncTask[V] {...} accepts a as a parameter and returns a new which will create a named . You should tell what the type of is via the generic parameter . represents the asynchronous task created by . It has two properties: & . Async func func struct AsyncTask Async return f V AsyncTask Async data err type AsyncTask[V any] struct {
  data chan V
  err chan any
} We can observe result through and the possible error through . Then let's take a look at the internals of . f() AsyncTask.data AsyncTask.err Async func Async[V any](f func () V) func() *AsyncTask[V] {
  exec := func() *AsyncTask[V] {
    data, err := handle[V](f)
    return &AsyncTask[V]{data: data, err: err}
  }

  return exec
} A named will create a pointer, inside , handles the synchronous result and possible error asynchronously. Both and capacity are set to 1 to prevent goroutine leaks. Once is done, the sub goroutine created by can exit immediately  regardless of whether there are other goroutines receiving and . func exec AsyncTask exec handle f data Channel err Channel f() handle data err func handle[V any](f func () V) (chan V, chan any) {
  data := make(chan V, 1)
  err := make(chan any, 1)

  go func() {
    var result V
    defer func() {
      if e:= recover(); e == nil {
        data <- result
      } else {
        err <- e
      }
      close(data)
      close(err)
    }()
    
    result = f()
  }()

  return data, err
} The example below shows how encapsulates and how works asynchronously. Async f f func main() {
  a := 1
  b := 2
  af := Async[int](func() int {
    c := a + b
    fmt.Println("f() result is", c)
    return c
  })

  fmt.Println("sync start, goroutine=", runtime.NumGoroutine())
  af()
  fmt.Println("sync end, goroutine=", runtime.NumGoroutine())
  
  time.Sleep(1 * time.Second)
  fmt.Println("async end, goroutine=", runtime.NumGoroutine())
}
/* stdout:
sync start, goroutine=1
sync end, goroutine=2
f() result: 3
async end, goroutine=1
*/ Await Implementation in Golang implementation in Golang is simple. What does is waiting for or sending data. Await Await AsyncTask.data AsyncTask.err func Await[V any](t *AsyncTask[V]) (V, any) {
  var data V
  var err any

  select {
  case err := <-t.err:
    return data, err
  case data := <-t.data:
    return data, err
  }
} The example below shows how waits for response. Await Async func main() {
  a := 1
  b := 2
  af1 := Async[int](func() int {
    c := a + b
    return c
  })
  af2 := Async[int](func() int {
    panic("f() error")
  })
  
  fmt.Printf("sync start, goroutine=%d\n", runtime.NumGoroutine())
  data, _ := Await[int](af1())
  _, err := Await[int](af2())
  fmt.Printf("sync end, goroutine=%d, af1() result=%d, af2() result='%s'\n", runtime.NumGoroutine(), data, err)
}

/* stdout
sync start, goroutine=1
sync end, goroutine=1, af1() result=3, af2() result='f() error'
*/ Conclusion This article shows how to implement ES7-style Async/Await API in Golang via goroutine and . In addition to the above definition, we still need to consider its execution result and state. Going a step further, we can also implement other asynchronous streaming APIs based on and , such as , , etc. Channel AsyncTask Async Await all race The full source code can be found . here

ES7-style Async/Await Implementation in Golang

Data has become the core of the modern business, and it is expanding more than ever. But these large datasets are not of much use unless organizations can make sense of it. That’s where Business Intelligence comes into play. If you are thinking about utilizing a platform that helps you pluck helpful, actionable insights from your data, then you are at the right place. In this blog, we are describing end-to-end Power BI implementation. What is Power BI? Power BI is the collective name for an assortment of cloud-based apps and services that help organizations collate, manage and analyze data from various sources through a user-friendly interface. It enables you to create a narrative around your data and tell a compelling visual story with actionable insights. Below are the Steps for Power BI Implementation 1. Get Data We are using widely used and publicly available data for visualization – “TLC Trip Record Data,” popularly known as “NYC Taxi data.” We can find it over the website ( ), which provides taxi trip records from 2009 to July 2021. https://www1.nyc.gov/ Visit the URL mentioned above, which has links to all trip records from January 2020 to December 2020, zone lookup tables, and data dictionary - https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page Now download “Yellow Taxi Trip Records” for January, “Taxi Zone Lookup Table” along with “Yellow Trips Data Dictionary.” 2. Extract Data As you can see, the data is in CSV format. We need to import this data to Power BI as we need to confirm whether there is any requirement for transformations. The file contained 64+ lakh records with 18 columns, and we easily imported it to Power BI. Also, we imported the data from the “Taxi Zone Lookup Table.” 3. Transform Data At this stage, we checked the data superficially and found that some columns had few nulls. Thus, they require transformation operations. We used the following Power BI transformation features – – To replace nulls with some acceptable values Replace Values – To add additional columns of “Vendor Name” & “Rate Type” based on their numerical values. (“Yellow Trips Data Dictionary” provides these details) Conditional Column – To add additional date columns to have only date values from trip pickup DateTime & trip drop DateTime columns. Add column from Examples Also, Power BI has a feature to create a date table. We created a date table having date hierarchy columns like the year, quarter, month, week, and day. Timeline-based reports have drill-down features which use the columns mentioned above. We can drill through a date hierarchy report from “Year” to “Day.” 4. Data Modeling Data modeling is used to connect multiple data sources in Power BI using a relationship. A relationship defines how data sources are related, and based on these; you can create exciting visualizations on various data sources. We have the following data sources ready with us now. Yellow taxi trip data Taxi zone lookup table Date Thus, we created a relationship between the LocationID of the zone lookup table and pick and drop LocationIDs of Yellow taxi trip data. We also created the relationship between the Date column of the date table consisting of pickup and drop off date columns from yellow taxi trip data. 5. Data Visualization Power BI has got several visuals such as stacked, line, pie, donut, map, and scatter charts. Using the easy-to-use interface and visual, we prepared . data visualizations Power BI has got several amazing features, and one of them is the feature. Suppose you have multiple visuals over a single interface. In that case, we can click over any visualization element, highlighting all other visualizations on the page corresponding to the part clicked. Check the pictorial representation- “Cross Highlighting” All Elements Highlighted in the Below Diagram A Single Element is Highlighted from the First Visual in the Above Dashboard (All other visuals got impacted corresponding to the highlighted element) Slicers In addition to graph plotting visuals, it also provides slicers to create data filters. Page Navigation Power BI has also got some cool features like bookmarks, buttons & selections. We can create page navigation similar to the web interface. The Path Ahead Power BI has several other features like Direct Query, DAX functions, themes, tooltips, custom backgrounds, and conditional drills. They help make more meaningful reports and dashboards that bring value to businesses. We will be covering them as well; till then, stay tuned with us.

A Step By Step Guide To Data Visualization With Power BI

Product Manager responsibilities can vary depending on the company size, industry, and location.  On 10/16, our NYC chapter launch focused on exploring the unique qualities that define Product Management in Silicon Valley and New York.  Despite heavy rain and the accompanying subway delays, we had a good showing with highly engaged attendees for our discussion on Contrasting PM Disciplines Between Coasts with a distinguished panel of successful women in product who were selected for their experience in operating across both coasts.  The panelists were: ●     Charlie Javice, Founder & CEO of FRANK., who had raised $15.5 million Series A round from venture capital firms across coasts and internationally; ●    Erica Windisch, Founder & CTO of IO|pipe, who has worked internationally building and selling products to technology teams across coasts; ●     Veronica Pinchin, Director of Product Management at Sidewalk Labs, who has extensive product management experience working for technology companies on both coasts. The panel was moderated by Crystal Huang, who is a Principal at New Enterprise Associates. The topic generated lively discussions about FRANK., IO|pipe, and Sidewalk Labs’ philosophy with regards to product management in addition to discussion about the different aspects of product development, contrasting the approach to culture, hiring, and style between the East Coast and West Coast.  Here are some of the key takeaways: Philosophy on Product and Its Role in the Company ●     East Coast – technology is a business enabler for accomplishing their mission and vision.  Companies such as FRANK (Fintech) and Sidewalk Labs (Urban Planning) are great examples where technology supports their business model, but is not the fundamental business driver. To accomplish their mission, they create products that enable them to better provide those services. ●     West Coast – there are more businesses where the product is the business. Facebook and Google are examples of companies where the product is the business. This can lead to different philosophies on product manager responsibilities and sometimes a conflict between business and product strategy. In companies where technology is the core business, these two strategies are one in the same. In New York, there tend to be more General Manager hires who lead business strategy while Product Managers are responsible for creating the technology to fulfill this vision. Focus on Revenue Generation ●      Venture capital firms based on the East Coast emphasize having a fast cycle to revenue generation and profitability and can skew towards being more risk-averse. ●      On the West Coast, VC’s prioritize company vision and engagement metrics over path to profitability. New York is a growing investment hub for venture capital and as more West Coast firms expand out east, we’ll continue to observe if this mindset begins to shift Philosophy on Product Management Hires ●     Associate Product Management programs abound in Silicon Valley, which results in many PMs who entered Product Management early in their careers ●      Alternatively, the East Coast features more professionals who transition into Product Management after gaining experience in banking, consulting and other industries. This can present a challenge for tech companies looking to hire for junior roles. ●     These backgrounds lead to PMs who heavily utilize frameworks and have a metrics-driven approach With the diversity in experience in East Coast Product Management, tech-enabled businesses can skew towards looking for PMs with domain expertise. That being said, there is an incredible startup ecosystem in NYC and also tech titans such as Google, Adobe and Amazon that are eager to hire product generalists. Business / Product Team Culture ●     There is more cultural diversity in the make-up of the company/product team based in New York City (47% of tech employees are from another country!).  It is not uncommon to have people from Germany, France, India, China, Israel, etc. working on the same team. [1] ●     It’s been well documented that diversity increases innovation and drives financial performance . With geographical diversity, East Coast companies incorporate a variety of perspectives and can address global opportunities that further expand their market reach. [2] Based on our observations of differences in product management philosophies and the East Coast’s emphasis on profitability, aspiring product managers on the East Coast should leverage their domain knowledge as unique skill set when making the transition into product management. There are fewer opportunities to be trained as an Associate Product Manager in New York, and we encourage those interested in transitioning to find a community that you can learn and grow with, such as AWIP, and to seek out sponsors who can guide you to achieve key milestones. References: [1] https://comptroller.nyc.gov/reports/our-immigrant-population-helps-power-nyc-economy/ [2] https://hbr.org/2018/01/how-and-where-diversity-drives-financial-performance

Product Management Between Coasts: Adapting to NYC and Silicon Valley Philosophies

A using image data from the New York City Department of Transportation showed that during a 10 day period in December 2017, on one street in Harlem, New York City, the machine learning algorithm bus stop was blocked 57% of the time (55% weekdays 7am to 7pm) bike lane was blocked 40% of the time (57% weekdays 7am to 7pm) Left: Computer Identified Vehicles Right: Computer Identified Violators The press is really excited about self driving cars and the automation of driving jobs. Less attention is paid to how the same advancements in computer vision will be superior to humans at jobs such as parking enforcement where the basic requirement is detecting a vehicle’s location, determining if the location is legal, and taking appropriate action. Autonomous Vehicle View Source:Bosch There are ; here is how video cameras and computers can do the same job with higher enforcement levels, less cost and greater equality. three thousand traffic enforcement officers in NYC Prototype NYC DOT Traffic Camera Source: invisibleboxes.info The New York City Department of Transportation (NYC DOT) maintains several hundred cameras throughout the city that post images in realtime to . To test the possible benefits of robo traffic cops I built a system which highlights parking violations at the corner of . http://dotsignals.org/ Saint Nicholas Ave and 145 street For a quick prototype its results are pretty interesting. green = bike lanes, blue = bus stop, red = cars on sidewalk The avenue has unprotected bikes lane in both directions highlighted in green. Throughout the city, unprotected bike lanes are often blocked by cars, trucks and especially . On the right of the image in blue is a bus stop but instead functions as a parking spot. . illegally UPS TRUCKS NYC public buses crawl at dismal speeds because of blocked bus lanes and stops The url for the camera at 145th and Saint Nicholas Avenue is . The machine learning algorithm code is available . In short, a pre-trained model was re-trained on . Then the model was used to classify the vehicles on the road and where they are stopped. http://dotsignals.org/google_popup.php?cid=532 here Tensorflow ~2000 pre-classified images The Results During a 10 day period in December 2017, bike lane was blocked 40% of the time (57% weekdays 7am to 7pm) bus stop was blocked 57% of the time (55% weekdays 7am to 7pm) Keep in mind this is just one average block. This means if you are riding in a bike lane you are swerving blind in and out of the bike lane every other block. And if you are on a bus your commute just got longer. Current Solution: Traffic Cops In NYC there are , who make from (let’s estimate $48k-$72k ) and gave (filter by issuer T) which is in 2014. ~3000 traffic police officers $30k-$45k per year including benefits, health, retirement seven million tickets ~1 ticket per agent per working day hour If the block at 145th and Saint Nicholas is indicative of other blocks in the city than we can assume that the number of tickets being given represents less than .0001% of infractions. So ~150 million in pay & benefits for traffic cops yielded NYC .0001% of enforcement. Future Solution:Camera Traffic Cops Higher Enforcement Levels If NYC DOT put four cameras at each intersection there would be near complete and total coverage throughout the city 24x7. There is simply no way that deploying police officers could match this level of enforcement. Less Cost According to the DOT there are If we assume that we need 4 cameras per intersection let’s round up and say we need 50,000 traffic cameras. The hypothetical robo traffic cop could be setup for either onboard processing or central processing. For onboard processing the cameras would include a computer vision system and a cellular connection to report video proof of infractions. For central processing the cameras would be connected by fiber back to a central server for group processing. Onboard processing would have higher per camera cost but less installation cost with central processing the opposite tradeoff. 12,460 traffic intersections in NYC. No matter which system, in order to be less expensive than current traffic enforcement each camera could cost $3,000 per year and NYC would get complete coverage of every street! $48k to $72k per officer * 3,000 officers / 50,000 cameras = $2.8k to $4.3k This article does not even attempt to address the huge issues of US labor productivity stagnation, unequal unemployment, public agency labor relations, and income inequality. All the camera systems discussed above would be much cheaper than the human officers who cover a tiny fraction of the city. Equality All laws are not equally enforced. Currently police officers decide on whom to enforce the law leading to inequality in enforcement often based on race and income. That said, . They are not perfect and can potentially be more biased than people. For instance a computer vision traffic enforcement system might erroneously single out certain groups for extra ticketing based on some visual characteristic of the vehicles. But as long as the data and algorithms are open for scrutiny we can work to eliminate bias to an extent not possible with police officer enforcement. algorithms can be just as biased as police Suggestions NYC DOT should evaluate installing four cameras at each intersection in NYC. The camera feeds should be higher quality but . The feeds should be open to all. The NYC DOT should not require NY State approval to install the cameras. The NYPD should contract out to an external company to build and maintain an open source system to process the video feeds using machine learning to identify all moving and parking violations. The system should present each violation to NYPD officers behind a computer screen who could review the footage and issue the ticket. This system, if implemented reasonable well, would be more effective, lower cost, and more equal than the current system. blur faces onboard the camera before transmitting to the internet Dear NYC, I am interested in working on this. DOT, NYPD, NYCGov how can I help you evaluate and create this system? Appendix What would Richard Stallman say? What if I already have a big brother? What is the price of privacy? Richard Stallman ( ) would probably say cameras on each NYC block . Sometimes I agree that increased surveillance is bad for society and will make it easier for our government and companies to prey upon society. But I also think that the current methods of enforcement (i.e. police) already prey on society, it’s just the poor and people of color who are most harmed. They are harmed by uneven enforcement of some laws and a lack of enforcement of others. I hope a system of video surveillance might improve this. At this point I think any change from the status quo would get us better results or at the least highlight the problems in the current system. I vote bring the cameras and surveillance to the streets but make the video feeds and let’s see what happens. rms imperils democracy open to all Tactically, I suggest a few things which should alleviate some privacy concerns and potentially introduce other issues. Firstly, the camera systems should blur all faces onboard the camera before transmitting to the internet. Secondly, the feeds should be open to all. Open to all? But then terrorists, stalkers, spousal abusers etc will have access to the cameras. Yes, but these people could already go down and view the street. By making the camera system open it also means that researchers, investigative journalists and lawyers could evaluate and monitor the performance of the data collection and algorithms to search and report on bias and inequality in enforcement. This should be a bigger conversation, not enough people are discussing the pros and cons. Questions People hate speed cameras/red light cameras, this is just bad. People who speed and go through red lights hate those cameras. Bicyclists and buses suck Ok How would you get license plates? The current DOT cameras are connected via Stealth Communication Inc which feeds fiber optics to each camera. The cameras installed are already capable of sufficient video resolution to detect license plates, the DOT has just decided not to broadcast higher resolution video. In the video why aren’t the detection boxes around all the cars? Why didn’t cars on the sidewalk get boxes? The algorithm isn’t perfect and was a quick first attempt. So there is a lot of improvement to further identify other types of vehicles. The training data was only vehicles front and backs, so the algorithms isn’t trained to identify the sides of vehicles. In the video there is a scene when rain covers the lens and the algorithm thinks there are cars all over the road See above, the algorithm is only a first implementation. But this is a real world concern and something to think about for a solution at scale. If the camera can’t see than the system doesn’t work. For the data represented here the time when the image was covered in rain represented a small portion of the total time and doesn’t significantly affect the results. Doesn’t NYC need NY State to install traffic cameras? If so, is this whole idea dead in the water? NY State sets out the traffic law for the state. So yes, in order to give automated tickets New York State has to approve certain installations. However, the DOT can install as many cameras as it wants. And I believe that the NYPD can, within the current law, issue tickets to motorists for violations they witness through cameras. So I think the system can be architected so as to not require NY State approval. Why not just build protected bike lanes and protected bus lanes and stops? Yes. We should do that. But that infrastructure might be even more expensive. And cars still find ways to block protected bike lanes and block protected bus stops. Why is the data for only 10 days? Why is this only one camera? For the 10 days of video my computer recorded over 800,000 images. It took Amazon AWS p2.xlarge instances running the model take .3 seconds per image. This means it took days and approximately $65 to process the 10 days worth of images. However, the cost is not linear so a greater number of cameras would be cheaper per camera to process. In short, while it is much cheaper than employing thousands of police officers, it still costs some money to run the algorithms and collect the data. Don’t these cameras already exist? There are already some . There are also some red light and speed cameras. They are few and far between. They are also entirely closed. I.e. we the people have no way of determining their number, their algorithms, their efficacy, their equality etc. bus lane cameras If these cameras are on 145th and Saint Nicholas then they are not working. That much should be obvious. What other violations could this setup catch? What other things could you do? Count traffic, vehicle speed, pedestrians, bicyclists etc. Trucks over 53’ https://twitter.com/illegal53NYC Are building owners or store owners shoveling the sidewalks? In time? Should we send out snow trucks to every street? Or deploy them to streets that video can verify have snow blocking them. Same for street cleaning. Should street cleaning happen even if the streets are clean? Trash pickup? Congestion pricing requires determining where vehicles are in the city and at what time in order to apply surcharges. TLC violations. The taxi and limousine commission has a great program, , in which you can submit a picture of a offending vehicle, say a taxi in a bike lane, and the TLC will after a process fine the driver. Why shouldn’t this exist citywide for all vehicles? http://www.nyc.gov/html/tlc/html/passenger/sub_consumer_compl.shtml Technical The github has all the code. page Some general things learned. The cameras are low resolution 352x240 CIF CTV and update at a framerate of ~1 fps. My Intel Core i7–4770K took about 24 hours to retrain the last layer of the model. I thought it would take longer and I would need to use cloud GPU’s. was really easy to get started using. I had taken some Machine Learning classes 8 years ago and things are getting easier and easier to get started. Tensorflow I had 800,000 images to process with the model to look for vehicle positions. Each 352x240 image took 5 seconds to analyze on my Intel Core i7–4770K CPU only. I thought it would be faster. I ended up using Amazon AWS p2.xlarge with 1 Nvidia K80 and it took .3 seconds per image. I also thought that would be faster. Side note: I tried to use google cloud but their response time in authorizing my account for ML usage was 2 days. I gave up after 12 hour and switched to Amazon who approved my GPU usage after 10 minutes. As you can see in the code the determining of whether or not the vehicle is in breach of the traffic law was not done with machine learning. Instead I just drew a simple polygon representing bus and bike lanes and counted a vehicle as blocking if it was the correct type of vehicle, i.e. not a bus, and whether the center of the vehicle was in the lane. I considered training the algorithm to try to spot bike lanes and bus stops and do its own determination. But I was concerned that with the street markings barely visible it would be very difficult. So instead, I just drew the lanes because I could visit the street and determine the actual markings. If this was to be used at scale there might have to be a manual portion of drawing or determining the lanes/stops and mapping that to pixels. I want to take a look at using YOLO for object detection. Specific semantics for certain vehicles- I tried annotating the images with identifiers for further subsets of vehicles such as taxis, ups trucks, or police cars with less impressive results. I will have to use embeddings to further classify the types of vehicles. I purchased a Movidius Tensor Compute stick and want to experiment running the detection in realtime on board a low power system suitable for mounting to a street pole and running off solar.

Stories, News & Mentions about NYMGO

Nymgo’s Stories on HackerNoon