panic while merging in unit test

[mschoch]

 panic: runtime error: slice bounds out of range [:1153] with capacity 1152

goroutine 95 [running]:
github.com/blugelabs/ice/v2.(*Segment).copyStoredDocs(0xc000b40780, 0x200, 0xc000fc0000, 0x7d0, 0x7d0, 0xc000fad3f0, 0x0, 0x1)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:787 +0x745
github.com/blugelabs/ice/v2.mergeStoredAndRemap(0xc000621f40, 0x3, 0x3, 0xc000621f20, 0x3, 0x3, 0xc0006b88d0, 0xc0006b88a0, 0x3, 0x3, ...)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:659 +0x853
github.com/blugelabs/ice/v2.mergeToWriter(0xc000621f40, 0x3, 0x3, 0xc000621f20, 0x3, 0x3, 0x401, 0xc000621f60, 0xc000798fc0, 0xc000621f40, ...)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:130 +0x21c
github.com/blugelabs/ice/v2.mergeSegmentBasesWriter(0xc000621f40, 0x3, 0x3, 0xc000621f20, 0x3, 0x3, 0x9c2580, 0xc0009fe300, 0x401, 0xc000798fc0, ...)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:96 +0x15a
github.com/blugelabs/ice/v2.merge(0xc0006b8870, 0x3, 0x3, 0xc000621f20, 0x3, 0x3, 0x9c2580, 0xc0009fe300, 0xc000798fc0, 0x0, ...)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:85 +0x1d4
github.com/blugelabs/ice/v2.(*Merger).WriteTo(0xc000cd4910, 0x9c2cc0, 0xc0009d8110, 0xc000798fc0, 0x9c61e0, 0xc000044950, 0x0)
	C:/Users/runneradmin/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:48 +0x191
github.com/blugelabs/bluge/index.(*FileSystemDirectory).Persist(0xc0000ba780, 0x943751, 0x4, 0xd, 0x1d124eabf58, 0xc000cd4910, 0xc000798fc0, 0x9c4e60, 0xc000cd4910)
	D:/a/bluge/bluge/index/directory_fs.go:125 +0x2d5
github.com/blugelabs/bluge/index.(*Writer).merge(0xc0002f4480, 0xc0006b8870, 0x3, 0x3, 0xc000621f20, 0x3, 0x3, 0xd, 0x3, 0xc0009fe100, ...)
	D:/a/bluge/bluge/index/merge.go:368 +0x22b
github.com/blugelabs/bluge/index.(*Writer).executeMergeTask(0xc0002f4480, 0xc0007990e0, 0xc000621f00, 0xc000cd48c0, 0xc000621ea0)
	D:/a/bluge/bluge/index/merge.go:144 +0x88b
github.com/blugelabs/bluge/index.(*Writer).planMergeAtSnapshot(0xc0002f4480, 0xc0007990e0, 0xc0000fa980, 0xa, 0x4c4b40, 0x4024000000000000, 0xa, 0x7d0, 0x4000000000000000, 0x0, ...)
	D:/a/bluge/bluge/index/merge.go:118 +0x47c
github.com/blugelabs/bluge/index.(*Writer).mergerLoop(0xc0002f4480, 0xc0007990e0, 0xc000895380)
	D:/a/bluge/bluge/index/merge.go:56 +0x4e5
created by github.com/blugelabs/bluge/index.OpenWriter
	D:/a/bluge/bluge/index/writer.go:131 +0xf8e
FAIL	github.com/blugelabs/bluge	1.570s

[mschoch]

This is not confined to Windows, it just happened on the CI with Ubuntu:

panic: runtime error: slice bounds out of range [:1153] with capacity 1152

goroutine 99 [running]:
github.com/blugelabs/ice/v2.(*Segment).copyStoredDocs(0xc000a7a3c0, 0x200, 0xc000326000, 0x7d0, 0x7d0, 0xc000a75400, 0x0, 0x1)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:787 +0x74d
github.com/blugelabs/ice/v2.mergeStoredAndRemap(0xc000db9c50, 0x6, 0x6, 0xc000db9c20, 0x6, 0x6, 0xc000db9cb0, 0xc000db9c80, 0x3, 0x3, ...)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:659 +0x833
github.com/blugelabs/ice/v2.mergeToWriter(0xc000db9c50, 0x6, 0x6, 0xc000db9c20, 0x6, 0x6, 0x401, 0xc0006515c0, 0xc0002c0120, 0x30, ...)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:130 +0x205
github.com/blugelabs/ice/v2.mergeSegmentBasesWriter(0xc000db9c50, 0x6, 0x6, 0xc000db9c20, 0x6, 0x6, 0x963aa8, 0xc000013540, 0x401, 0xc0002c0120, ...)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:96 +0x158
github.com/blugelabs/ice/v2.merge(0xc0003015c0, 0x6, 0x6, 0xc000db9c20, 0x6, 0x6, 0x963aa8, 0xc000013540, 0xc0002c0120, 0xc000302188, ...)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:85 +0x1c5
github.com/blugelabs/ice/v2.(*Merger).WriteTo(0xc0011f9450, 0x9641e8, 0xc000302188, 0xc0002c0120, 0x966ea8, 0xc000308130, 0x0)
	/home/runner/go/pkg/mod/github.com/blugelabs/ice/v2@v2.0.0/merge.go:48 +0x191
github.com/blugelabs/bluge/index.(*FileSystemDirectory).Persist(0xc00019ce00, 0x8e81a6, 0x4, 0xe, 0x7f6ef888a338, 0xc0011f9450, 0xc0002c0120, 0x965e08, 0xc0011f9450)
	/home/runner/work/bluge/bluge/index/directory_fs.go:125 +0x2da
github.com/blugelabs/bluge/index.(*Writer).merge(0xc000272000, 0xc0003015c0, 0x6, 0x6, 0xc000db9c20, 0x6, 0x6, 0xe, 0x6, 0x4848a5, ...)
	/home/runner/work/bluge/bluge/index/merge.go:368 +0x23b
github.com/blugelabs/bluge/index.(*Writer).executeMergeTask(0xc000272000, 0xc0002c0240, 0xc000683dd0, 0xc0011f9400, 0xc000683db8)
	/home/runner/work/bluge/bluge/index/merge.go:144 +0x86b
github.com/blugelabs/bluge/index.(*Writer).planMergeAtSnapshot(0xc000272000, 0xc0002c0240, 0xc0002c3000, 0xa, 0x4c4b40, 0x4024000000000000, 0xa, 0x7d0, 0x4000000000000000, 0x0, ...)
	/home/runner/work/bluge/bluge/index/merge.go:118 +0x44f
github.com/blugelabs/bluge/index.(*Writer).mergerLoop(0xc000272000, 0xc0002c0240, 0xc000063380)
	/home/runner/work/bluge/bluge/index/merge.go:56 +0x4a5
created by github.com/blugelabs/bluge/index.OpenWriter
	/home/runner/work/bluge/bluge/index/writer.go:131 +0xf72
FAIL	github.com/blugelabs/bluge	1.693s

[mschoch]

I was able to reproduce this locally, the key is include the race detector:

go test -race -v -count=20

Or more specifically:

go test -race -v -count=20 -run=^TestIndexCountMatchSearch$

[mschoch]

So, it looks to me like this is just a general issue with the way we coded access to these slices.

In some cases we're at some random spot in the slice, and are expecting a varint. So we try to access from [pos:pos+binary.MaxVarintLen64] and this is just generally unsafe if you might be getting to the end of the buffer (actually varint took 1 byte, so slice ends before accessed range)

[mschoch]

It also looks like this issue is the same type of problem, but on a different path: blugelabs/ice#6

[mschoch]

fixed by #120