How to Append to a Slice in Golang

Slices in go are not the same as slices in another programming language i.e Python. Assigning one slice to another only makes a shallow copy of the slice and should be used cautiously if you want to create a new slice from the existing slice.

Introduction

Go lang is undoubtedly one of the popular languages today. It is easier to understand and get started with but however, but it has its intricacies. I was working on a project and came across this behavior.

When working with programming languages like python the manipulation of slices is simple and expectable as it’s a high-level language. You create an array, slice them, re-slice them, concatenate them and the expected answer in your head matches the answer in the terminal output.

Take an example of the below python code:

#!/usr/bin/env python3

a = [0,1,2,3,4]
print(a)

array1 = a[:1]
print(array1)


array2 = a[2:]
print(array2)

array1 = array1 + array2
array1 = array1 + array2

print(array1)
print(a)

If you run the above code, you will get output like this:

➜  python3 main.py
[0, 1, 2, 3]
[0]
[2, 3]
[0, 2, 3, 2, 3]
[0, 1, 2, 3]

Let me explain what is happening above code and its output if you are new to python

1. We created an array a = [0,1,2,3,4]
2. Then we slice ( take out ) value first value from the original array a and store it in another array called array1. After this array1 has a value of [0]
3. Then we again slice ( take out ) the second to last value from the original array a and store it in another array called array2. After this array1 has a value of [2,3]
4. We then concatenate array1 and array2 and store again into array1 resulting in our value of array1 to be [0,2,3]
5. We again concatenate array1 and array2 and store again into array1. Since value of array1 has been changed from the last operation which is [0,2,3],. new value of array1 becomes [0,2,3,2,3] which seems logical as we concatenated [0,2,3] and [2,3]
6. Then we print the final value of array1 which is [0,2,3,2,3] and also print the original array a value which is [0, 1, 2, 3]

The thing that happened above seems logical and understandable.

Now I had a similar requirement where I had to manipulate slices and create new ones using Go. Thinking that it would be similar to other languages, I followed a similar approach.

package main

import (
	"fmt"
)

func main(){
	slice := []int{0,1,2,3}
	fmt.Println(slice)

	newslice1 := slice[:1]
	fmt.Println(newslice1)

	newslice2 := slice[2:]
	fmt.Println(newslice2)

	newslice1 = append(newslice1, newslice2...)
	fmt.Println(newslice1)

	newslice1 = append(newslice1, newslice2...)
	fmt.Println(newslice1)

	fmt.Println(slice)

}

Looking at the code you can see that, it’s just a simple array manipulation using slices.

Same as the above python, there is an array, you slice it, re-slice it and concatenate by appending it to the original slice.

Now Take a moment and think what would be the output of the code. You might think its the same as the above python code, which is:

➜ go run main.go
[0, 1, 2, 3]
[0]
[2, 3]
[0, 2, 3, 2, 3]
[0, 1, 2, 3]

But wait, there is a gotcha. The output is completely different from what we expect and the actual output when you run the above code, it looks like this.

➜ go run main.go
[0 1 2 3]
[0]
[2 3]
[0 2 3]
[0 2 3 3 3]
[0 2 3 3]

If you noticed the output, it’s different from what we expect in our head and different from the same implementation in python.

The puzzling bit is even our original slice has been changed when in fact, there was no operation done on the original slice.

Problem identification

After going through a few rabbit holes, I found the answer. Turns out the slice is going is not the same as the slice in python.

When you create a slice, it’s not actually storing any data, but instead, it is a descriptor of an underlying array. Slice in go could be better called slice header or slice variable. slice variable is a data structure describing a continuous section of an array stored separately from the slice variables themselves? The slice that we created above is not an actual slice, but rather a description of an array that could be represented as ( theoretically )

type slice struct {
  Length  int
  Capacity int
  firstElement *int ( or pointer to underlying array )
}

We know that each slice has three attributes, length, capacity, and pointer to the data.

When we create a new slice of the original using newSlice1 := slice[:1], it creates a new data structure for which points to the original slice could be represented as ( theoretically )

type newSlice1 struct {
  Length  1
  Capacity 4
  firstElement &underlyingArray[0]
}

And again When we create a new slice of the original using newSlice2 := slice[2:], it creates a new data structure for which points to the original slice could be represented as ( theoretically ):

type newSlice2 struct {
  Length  2
  Capacity 4
  firstElement &underlyingArray[2]
}

In both of the case, new slice created points to the same original slice and shares values with the original slice, so when newSlice1 is created its value is [0] and newSlice2 value is [2,3]. For both arrays, length is changing as we sliced the slice, but capacity remains the same i.e 4. For newSlice1 one slot is used and 3 remain where are for newSlice2 two slots are used and 2 remain.

After the first append is called, the value of newSlice1 is changed to [0,2,3] with one free slot and length 3. Its data structure could be represented as:

type newSlice1 struct {
  Length  3
  Capacity 4
  firstElement &array[0]
}

Since newSlice1 and original array slice share the same elements of the array i.e. values as both are pointing to the underlying array, the value of original slice is changed to [0,2,3,3] because we had overridden values from underlying arrays from the index 0 to 3 which ultimately changed the original slice.

This is why after the append to newSlice1 value is changed to [0 2 3 3 3] instead of [0, 2, 3, 2, 3].

Before appending:

After appending:

Here you can see the value of the underlying array of the slice gets changed after the append:

This can also be verified by checking the memory address of the underlying array. Since both of the slice point to a same underlying array memory address, any change in one affects the other as well.

➜  go run main.go
Memory location of first element of slice 0xc00001c0a0 <== Same
Memory location of first element of newslice1 0xc00001c0a0 <== Same
Memory location of first element of newslice2 0xc00001c0b0
Memory location of first element of newslice1 0xc00001c0a0
Memory location of first element of newslice1 0xc000018100
Memory location of first element of slice 0xc00001c0a0

In short, go append changes the underlying array of the slice when new slices are created and modified from it.

Solution

The solution to this would be to only use append when you actually want to append value to the already existing slice and not create a new one.

	newArray1 = append(newArray1, "items")

If we want to create a new slice by re-slicing the original array or slice then we should use copy function as shown below. copy performs a deep copy of the slice.

package main

import (
	"fmt"
)

func main(){
	slice := []int{0,1,2,3}
	fmt.Println(slice)

	newslice1 := make([]int, len(slice))
	copy(newslice1, slice)

	newslice1 = newslice1[:1]
	fmt.Println(newslice1)

	newslice2:= make([]int, len(slice))
	copy(newslice2, slice)

	newslice2 = newslice2[2:]
	fmt.Println(newslice2)

	newslice1 = append(newslice1, newslice2...)
	newslice1 = append(newslice1, newslice2...)

	fmt.Println(newslice1)
	fmt.Println(slice)

}

Now when you run the following code you will see the output as expected and the same as the output from the python code.

➜ go run main.go
[0 1 2 3]
[0]
[2 3]
[0 2 3 2 3]
[0 1 2 3]

This can again be verified by checking the memory address of the underlying array:

➜ go run main.go
Memory location of first element of slice 0xc00001c0a0 <== Not same
Memory location of first element of newslice1 0xc00001c0c0 <== Not same
Memory location of first element of newslice2 0xc00001c0f0
Memory location of first element of newslice1 0xc000018100
Memory location of first element of slice 0xc00001c0a0

Conclusion

Such intricacies are difficult to detect and even difficult to debug with working with go in production. I came across this when I am working on a volunteering project for a company. Hope this article was able to clear out some caveats with slices in Go.

Please feel free to reach out to me if you have any questions.

Also published here.