20201105-1333-golang-race-detector.md

title	date
golang-race-detector	2020-11-05 13:32

Golange race detector runtime/race

How does the ThreadSanitizer library work?

Here's an explainer of the algorithm.

The slides in the appendix have some good visualizations and the associated talk, Finding races and memory errors with compiler instrumentation

How is the ThreadSanitizer library integrated into golang's race detection

prereq

We'll be referencing the llvm and go github repos while exploring a bit of the internals of the go race detector.

dive in

• the go race detector is based on ThreadSanitizer (aka TSan) in llvm (originally by google)

• TSan is built with this script into an .syso file. These .syso objects are actually bundled in the golang/go main repo at src/runtime/race/race_linux_amd64.syso

• TSan API includes methods like the following, doing nm src/runtime/race/race_linux_amd64.syso reveals

   		........
   			0000000000024740 T __tsan_go_atomic64_exchange
   			0000000000025cb0 T __tsan_go_atomic64_fetch_add
   			0000000000028740 T __tsan_go_atomic64_load
   			0000000000027220 T __tsan_go_atomic64_store
   			0000000000016de0 T __tsan_go_end
   			0000000000006220 T __tsan_go_ignore_sync_begin
   			0000000000006270 T __tsan_go_ignore_sync_end
   			0000000000014f80 T __tsan_go_start
   			00000000000166c0 T __tsan_init
   			0000000000017140 T __tsan_malloc
   			0000000000013240 T __tsan_map_shadow
   			0000000000022f40 T __tsan_mutex_after_lock
   			000000000001a330 T __tsan_mutex_before_lock
   			0000000000023ab0 T __tsan_mutex_before_unlock
   		........
  

• While we're in the LLVM repo, notice the test.c which emulates 2 goroutines racing using the TSan API. Running this test program will output a DATA RACE detected message.

   			==================
   			WARNING: DATA RACE
   			Read at 0x00c011110000 by goroutine 2:
   				<null>()
   						<null>:0 +0x0
  
   			Previous write at 0x00c011110000 by main goroutine:
   				<null>()
   						<null>:0 +0x0
   				<null>()
   						<null>:0 +0x0
  
   			Goroutine 2 (running) created at:
   				<null>()
   						<null>:0 +0x0
   			==================
   			Found 1 data race(s)
  

  When using the -race flag in go, the TSan API is automatically called by the go runtime.

  In the test above, such a flow is artificially constructed. Notice the __tsan_func_enter, __tsan_write, ... calls issued.

• Go calls TSan API methods via optimized ASM wrappers in src/runtime/race*.s

What happens when we use the -race flag when building

These files are included in the build.

			src/internal/race/race.go:5:5:// +build race                                              
			src/runtime/race.go:5:5:// +build race                                                    
			src/runtime/race_arm64.s:5:5:// +build race                                               
			src/runtime/race_ppc64le.s:5:5:// +build race                                             
			src/runtime/race/race_test.go:5:5:// +build race                                          
			src/runtime/race/output_test.go:5:5:// +build race                                        
			src/runtime/race/timer_test.go:5:5:// +build race                                         
			src/runtime/race/race_unix_test.go:5:5:// +build race                                     
			src/runtime/race/race.go:5:5:// +build race,linux,amd64 race,freebsd,amd64 race,netbsd,amd
			src/runtime/race/sched_test.go:5:5:// +build race                                         
			src/runtime/race_amd64.s:5:5:// +build race                                               
			src/sync/atomic/race.s:5:5:// +build race                                                 

By including them, the runtime package is augmented with a number of new methods, which don't exist in the non -race builds.

			// Public race detection API, present iff build with -race
			func RaceRead(addr unsafe.Pointer)                        
			func RaceWrite(addr unsafe.Pointer)                       
			func RaceReadRange(addr unsafe.Pointer, len int)          
			func RaceWriteRange(addr unsafe.Pointer, len int)         
			.......

			// And a bit lower in the same (runtime/race.go) file
			// Internal API, in the runtime package
			func racefuncenter(callpc uintptr)             
			func racefuncenterfp(fp uintptr)               
			func racefuncexit()                            
			func raceread(addr uintptr)                    
			func racewrite(addr uintptr)                   
			func racereadrange(addr, size uintptr)         
			func racewriterange(addr, size uintptr)        
			func racereadrangepc1(addr, size, pc uintptr)  
			func racewriterangepc1(addr, size, pc uintptr) 
			func racecallbackthunk(uintptr)                

How does the compiler wire up the ThreadSanitizer API calls to the go code it's building?

The compiler instruments function calls in the gc/racewalk.go code.

			// The racewalk pass is currently handled in three parts.
			//
			// First, for flag_race, it inserts calls to racefuncenter and
			// racefuncexit at the start and end (respectively) of each
			// function. This is handled below.
			//
			// Second, during buildssa, it inserts appropriate instrumentation
			// calls immediately before each memory load or store. This is handled
			// by the (*state).instrument method in ssa.go, so here we just set
			// the Func.InstrumentBody flag as needed. For background on why this
			// is done during SSA construction rather than a separate SSA pass,
			// see issue #19054.
			//
			// Third we remove calls to racefuncenter and racefuncexit, for leaf
			// functions without instrumented operations. This is done as part of
			// ssa opt pass via special rule.

1. The code wraps the AST Function Nodes in calls to racefuncenterfp, racefuncexit, which we've mentioned above, those optimized assembly calls to ThreadSanitizer.

2. In the buildSSA processs, memory access is instrumented by inserting calls to runtime.{raceread,racewrite,racereadrange,...}.

Conclusion

Not much, just a fun exploration of how stuff works.