Slices in go are not the same as slices in another programming language i.e Python. Assigning one slice to another only makes a shallow copy of the slice and should be used cautiously if you want to create a new slice from the existing slice.
Introduction
Go lang is undoubtedly one of the popular languages today. It is easier to understand and get started with but however, but it has its intricacies. I was working on a project and came across this behavior.
When working with programming languages like python the manipulation of slices is simple and expectable as it’s a high-level language. You create an array, slice them, re-slice them, concatenate them and the expected answer in your head matches the answer in the terminal output.
Take an example of the below python code:
#!/usr/bin/env python3
a = [0,1,2,3,4]
print(a)
array1 = a[:1]
print(array1)
array2 = a[2:]
print(array2)
array1 = array1 + array2
array1 = array1 + array2
print(array1)
print(a)
If you run the above code, you will get output like this:
➜ python3 main.py
[0, 1, 2, 3]
[0]
[2, 3]
[0, 2, 3, 2, 3]
[0, 1, 2, 3]
Let me explain what is happening above code and its output if you are new to python
- We created an array
a = [0,1,2,3,4] - Then we slice ( take out ) value first value from the original array
aand store it in another array calledarray1. After this array1 has a value of[0] - Then we again slice ( take out ) the second to last value from the original array
aand store it in another array calledarray2. After this array1 has a value of[2,3] - We then concatenate
array1andarray2and store again intoarray1resulting in our value ofarray1to be[0,2,3] - We again concatenate
array1andarray2and store again intoarray1. Since value ofarray1has been changed from the last operation which is[0,2,3],. new value ofarray1becomes[0,2,3,2,3]which seems logical as we concatenated[0,2,3]and[2,3] - Then we print the final value of
array1which is[0,2,3,2,3]and also print the original arrayavalue which is[0, 1, 2, 3]
The thing that happened above seems logical and understandable.
Now I had a similar requirement where I had to manipulate slices and create new ones using Go. Thinking that it would be similar to other languages, I followed a similar approach.
package main
import (
"fmt"
)
func main(){
slice := []int{0,1,2,3}
fmt.Println(slice)
newslice1 := slice[:1]
fmt.Println(newslice1)
newslice2 := slice[2:]
fmt.Println(newslice2)
newslice1 = append(newslice1, newslice2...)
fmt.Println(newslice1)
newslice1 = append(newslice1, newslice2...)
fmt.Println(newslice1)
fmt.Println(slice)
}
Looking at the code you can see that, it’s just a simple array manipulation using slices.
Same as the above python, there is an array, you slice it, re-slice it and concatenate by appending it to the original slice.
Now Take a moment and think what would be the output of the code. You might think its the same as the above python code, which is:
➜ go run main.go
[0, 1, 2, 3]
[0]
[2, 3]
[0, 2, 3, 2, 3]
[0, 1, 2, 3]
But wait, there is a gotcha. The output is completely different from what we expect and the actual output when you run the above code, it looks like this.
➜ go run main.go
[0 1 2 3]
[0]
[2 3]
[0 2 3]
[0 2 3 3 3]
[0 2 3 3]
If you noticed the output, it’s different from what we expect in our head and different from the same implementation in python.
The puzzling bit is even our original slice has been changed when in fact, there was no operation done on the original slice.
Problem identification
After going through a few rabbit holes, I found the answer. Turns out the slice is going is not the same as the slice in python.
When you create a slice, it’s not actually storing any data, but instead, it is a descriptor of an underlying array. Slice in go could be better called slice header or slice variable. slice variable is a data structure describing a continuous section of an array stored separately from the slice variables themselves? The slice that we created above is not an actual slice, but rather a description of an array that could be represented as ( theoretically )
type slice struct {
Length int
Capacity int
firstElement *int ( or pointer to underlying array )
}
We know that each slice has three attributes, length, capacity, and pointer to the data.
When we create a new slice of the original using newSlice1 := slice[:1], it creates a new data structure for which points to the original slice could be represented as ( theoretically )
type newSlice1 struct {
Length 1
Capacity 4
firstElement &underlyingArray[0]
}
And again When we create a new slice of the original using newSlice2 := slice[2:], it creates a new data structure for which points to the original slice could be represented as ( theoretically ):
type newSlice2 struct {
Length 2
Capacity 4
firstElement &underlyingArray[2]
}
In both of the case, new slice created points to the same original slice and shares values with the original slice, so when newSlice1 is created its value is [0] and newSlice2 value is [2,3]. For both arrays, length is changing as we sliced the slice, but capacity remains the same i.e 4. For newSlice1 one slot is used and 3 remain where are for newSlice2 two slots are used and 2 remain.
After the first append is called, the value of newSlice1 is changed to [0,2,3] with one free slot and length 3. Its data structure could be represented as:
type newSlice1 struct {
Length 3
Capacity 4
firstElement &array[0]
}
Since newSlice1 and original array slice share the same elements of the array i.e. values as both are pointing to the underlying array, the value of original slice is changed to [0,2,3,3] because we had overridden values from underlying arrays from the index 0 to 3 which ultimately changed the original slice.
This is why after the append to newSlice1 value is changed to [0 2 3 3 3] instead of [0, 2, 3, 2, 3].
Before appending:
After appending:
Here you can see the value of the underlying array of the slice gets changed after the append:
This can also be verified by checking the memory address of the underlying array. Since both of the slice point to a same underlying array memory address, any change in one affects the other as well.
➜ go run main.go
Memory location of first element of slice 0xc00001c0a0 <== Same
Memory location of first element of newslice1 0xc00001c0a0 <== Same
Memory location of first element of newslice2 0xc00001c0b0
Memory location of first element of newslice1 0xc00001c0a0
Memory location of first element of newslice1 0xc000018100
Memory location of first element of slice 0xc00001c0a0
In short, go append changes the underlying array of the slice when new slices are created and modified from it.
Solution
The solution to this would be to only use append when you actually want to append value to the already existing slice and not create a new one.
newArray1 = append(newArray1, "items")
If we want to create a new slice by re-slicing the original array or slice then we should use copy function as shown below. copy performs a deep copy of the slice.
package main
import (
"fmt"
)
func main(){
slice := []int{0,1,2,3}
fmt.Println(slice)
newslice1 := make([]int, len(slice))
copy(newslice1, slice)
newslice1 = newslice1[:1]
fmt.Println(newslice1)
newslice2:= make([]int, len(slice))
copy(newslice2, slice)
newslice2 = newslice2[2:]
fmt.Println(newslice2)
newslice1 = append(newslice1, newslice2...)
newslice1 = append(newslice1, newslice2...)
fmt.Println(newslice1)
fmt.Println(slice)
}
Now when you run the following code you will see the output as expected and the same as the output from the python code.
➜ go run main.go
[0 1 2 3]
[0]
[2 3]
[0 2 3 2 3]
[0 1 2 3]
This can again be verified by checking the memory address of the underlying array:
➜ go run main.go
Memory location of first element of slice 0xc00001c0a0 <== Not same
Memory location of first element of newslice1 0xc00001c0c0 <== Not same
Memory location of first element of newslice2 0xc00001c0f0
Memory location of first element of newslice1 0xc000018100
Memory location of first element of slice 0xc00001c0a0
Conclusion
Such intricacies are difficult to detect and even difficult to debug with working with go in production. I came across this when I am working on a volunteering project for a company. Hope this article was able to clear out some caveats with slices in Go.
Please feel free to reach out to me if you have any questions.
Also published here.