Writing Bytes Without Tracking Offset In Go

2026-10-29

Using bytes.Buffer lets us write bytes without manually tracking offsets. Keeping writes and their position updates together helps prevent bugs and makes the code easier to understand.

Context

I'm building a small Raft implementation in Go. I'm defining a simple message format for communication between nodes. I ran into a couple of bugs caused by manually tracking byte offsets.

Managing Offsets Manually

Here is the Go struct for an AppendEntries request:

type LogEntry struct {
	Index   uint64
	Term    uint64
	Payload []byte
}

type AppendEntriesRequest struct {
	Term         uint64
	LeaderID     uint64
	PrevLogIndex uint64
	PrevLogTerm  uint64
	LeaderCommit uint64
	Entries      []LogEntry
}

Here is the code that encodes the message into bytes. It has two bugs. Can you spot them?

func (m *AppendEntriesRequest) EncodePayload() ([]byte, error) {
	entriesSize := 0
	appendEntryMetaSize := 8 * 5 // Term(8 bytes)|LeaderID(8 bytes)|PrevLogIndex(8 bytes)|PrevLogTerm(8 bytes)|LeaderCommit(8 bytes)
	logEntryMetaSize := 8 * 2 + 4 // Index(8 bytes)|Term(8 bytes)|PayloadSize(4 bytes)
	for i := 0; i < len(m.Entries); i++ {
		entriesSize += logEntryMetaSize + len(m.Entries[i].Payload)
	}
	payload := make([]byte, appendEntryMetaSize + entriesSize)
	binary.BigEndian.PutUint64(payload[:8], m.Term)
	binary.BigEndian.PutUint64(payload[8:16], m.LeaderID)
	binary.BigEndian.PutUint64(payload[16:24], m.PrevLogIndex)
	binary.BigEndian.PutUint64(payload[24:32], m.PrevLogTerm)
	binary.BigEndian.PutUint64(payload[32:40], m.LeaderCommit)
	currentOffset := appendEntryMetaSize
	for i := 0; i < len(m.Entries); i++ {
		entry := m.Entries[i]
		binary.BigEndian.PutUint64(payload[currentOffset:currentOffset+8], entry.Index)
		currentOffset += 8
		binary.BigEndian.PutUint64(payload[currentOffset:currentOffset+8], entry.Term)
		currentOffset += 8
		entryPayloadSize := len(entry.Payload)
		binary.BigEndian.PutUint32(payload[currentOffset:currentOffset+4], uint32(entryPayloadSize))
		currentOffset += 8
		copy(payload[currentOffset:currentOffset+entryPayloadSize], entry.Payload)
	}
	return payload, nil
}

The first bug is that the offset is increased by 8 instead of 4 after writing the entry's payload length.

binary.BigEndian.PutUint32(payload[currentOffset:currentOffset+4], uint32(entryPayloadSize)) // The payload size is encoded as a uint32 (4 bytes).
currentOffset += 8 // But the offset is increased by 8!

The second bug is that the offset is not updated after copying each entry's payload.

	for i := 0; i < len(m.Entries); i++ {
		// ...
		copy(payload[currentOffset:currentOffset+entryPayloadSize], entry.Payload) // entry.Payload is copied.
		// But the offset is not updated!
	}

As a result, the next entry overwrites the previous entry's payload.

Using bytes.Buffer

Instead of tracking the byte offsets manually, we can let bytes.Buffer handle it for us. Here is the updated implementation:

func (m *AppendEntriesRequest) EncodePayload() ([]byte, error) {
	var buf bytes.Buffer
	fields := [...]uint64{
    m.Term,
    m.LeaderID,
    m.PrevLogIndex,
    m.PrevLogTerm,
    m.LeaderCommit,
	}
	if err := binary.Write(&buf, binary.BigEndian, fields); err != nil {
		return nil, err
	}
	for _, entry := range m.Entries {
		fields := [...]uint64{
			entry.Index, entry.Term,
		}
		if err := binary.Write(&buf, binary.BigEndian, fields); err != nil {
			return nil, err
		}
		size := uint32(len(entry.Payload))
		if err := binary.Write(&buf, binary.BigEndian, size); err != nil {
			return nil, err
		}
		buf.Write(entry.Payload)
	}
	return buf.Bytes(), nil
}

Each write appends data to the buffer, so I no longer need to update an offset manually. This removes the risk of incrementing the offset by the wrong amount or forgetting to update it altogether. The code is also shorter and easier to read. Without manual offset calculations and magic numbers, it's easier to see which fields are being written and in what order.

Takeaways

Using bytes.Buffer lets us write bytes without manually tracking offsets. Keeping writes and their position updates together helps prevent bugs and makes the code easier to understand.