Skip to content

Commit

Permalink
Preinitialize entire channel map before using it in parallel restore.
Browse files Browse the repository at this point in the history
Trying to assign channels to the map, and spin up a go routine that uses
the map in the same loop can result in the following panic at run time.

`fatal error: concurrent map read and map write`

This can also be detected using golang's race condition detector

```
go run --race gprestore.go --timestamp 20240201180314 --create-db
--redirect-db test2 --jobs 8 --verbose
...
==================
WARNING: DATA RACE
Read at 0x00c0003a6300 by goroutine 77:
  runtime.mapaccess1_fast64()
      /usr/local/go/src/runtime/map_fast64.go:13 +0x0
  github.com/greenplum-db/gpbackup/restore.ExecuteStatements.func1()
      /home/kyeap/workspace/gpbackup/restore/parallel.go:158 +0x1c4
  github.com/greenplum-db/gpbackup/restore.ExecuteStatements.func2()
      /home/kyeap/workspace/gpbackup/restore/parallel.go:159 +0x41

Previous write at 0x00c0003a6300 by main goroutine:
  runtime.mapassign_fast64()
      /usr/local/go/src/runtime/map_fast64.go:93 +0x0
  github.com/greenplum-db/gpbackup/restore.ExecuteStatements()
      /home/kyeap/workspace/gpbackup/restore/parallel.go:144 +0x805
  github.com/greenplum-db/gpbackup/restore.ExecuteRestoreMetadataStatements()
      /home/kyeap/workspace/gpbackup/restore/wrappers.go:321 +0xa4
  github.com/greenplum-db/gpbackup/restore.restorePredata()
      /home/kyeap/workspace/gpbackup/restore/restore.go:334 +0x9d1
  github.com/greenplum-db/gpbackup/restore.DoRestore()
      /home/kyeap/workspace/gpbackup/restore/restore.go:163 +0x184
 ```
  • Loading branch information
kyeap-vmware committed Feb 2, 2024
1 parent 80c4931 commit d3c0473
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions restore/parallel.go
Original file line number Diff line number Diff line change
Expand Up @@ -139,12 +139,15 @@ func ExecuteStatements(statements []toc.StatementWithType, progressBar utils.Pro
panicChan := make(chan error)
splitStatements := scheduleStatementsOnWorkers(statements, connectionPool.NumConns)
chanMap := make(map[int]chan toc.StatementWithType, connectionPool.NumConns)
// preinitialize channels to prevent concurrent read and write
for i := 0; i < connectionPool.NumConns; i++ {
chanMap[i] = make(chan toc.StatementWithType, len(splitStatements[i]))
for _, statement := range splitStatements[i] {
chanMap[i] <- statement
}
close(chanMap[i])
}
for i := 0; i < connectionPool.NumConns; i++ {
workerPool.Add(1)
go func(connNum int) {
defer func() {
Expand Down

0 comments on commit d3c0473

Please sign in to comment.