Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Reasonable memory restrictions when running local minio s3 plugin test
Set 2g memory limit to local Minio docker. Set s3 configuration chunksize and concurrency. S3 plugin upload and download chunksize defaults to 500MB and concurrent requests defaults to 6. This means that each s3 plugin upload call will reserve a buffer size of 500MB * 6 = 3G. This is observable during testing in the s3 plugin logs. ``` Uploading file [backup_file] with chunksize 524288000 and concurrency 6 ``` On a demo cluster of 3 segments this means that during backup or restore our plugin will try to reserve a total of 3G * 3 segments = 9G. This way out of line for local minio testing which only tries to upload a few bytes. If a system doesn't have this kind of free memory on hand it will error out with the following panic. ``` fatal error: runtime: out of memory runtime stack: ... goroutine 1 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00013f158 sp=0xc00013f148 pc=0x46a788 runtime.(*mheap).alloc(0x0?, 0x3?, 0xb8?) /usr/local/go/src/runtime/mheap.go:962 +0x5b fp=0xc00013f1a0 sp=0xc00013f158 pc=0x42cb3b runtime.(*mcache).allocLarge(0xc000024540?, 0xbb800000, 0x58?) /usr/local/go/src/runtime/mcache.go:234 +0x85 fp=0xc00013f1e8 sp=0xc00013f1a0 pc=0x41ab85 runtime.mallocgc(0xbb800000, 0xa39600, 0x1) /usr/local/go/src/runtime/malloc.go:1123 +0x4f6 fp=0xc00013f250 sp=0xc00013f1e8 pc=0x411d96 runtime.makeslice(0xb5aa93?, 0x36?, 0xc00013f3a0?) /usr/local/go/src/runtime/slice.go:103 +0x49 fp=0xc00013f278 sp=0xc00013f250 pc=0x451d69 bufio.NewReaderSize(...) /usr/local/go/src/bufio/bufio.go:57 github.com/greenplum-db/gpbackup-s3-plugin/s3plugin.uploadFile(0xc0000305d0?, 0xc0002f1c20, {0xc0002e6460, 0x4a}, 0xc00003c148) ... ```
- Loading branch information