Skip to content

Instantly share code, notes, and snippets.

@AntoineAugusti
Last active April 8, 2024 08:33
Show Gist options
  • Save AntoineAugusti/80e99edfe205baf7a094 to your computer and use it in GitHub Desktop.
Save AntoineAugusti/80e99edfe205baf7a094 to your computer and use it in GitHub Desktop.
Limit the maximum number of goroutines running at the same time
package main
import (
"flag"
"fmt"
"time"
)
// Fake a long and difficult work.
func DoWork() {
time.Sleep(500 * time.Millisecond)
}
func main() {
maxNbConcurrentGoroutines := flag.Int("maxNbConcurrentGoroutines", 5, "the number of goroutines that are allowed to run concurrently")
nbJobs := flag.Int("nbJobs", 100, "the number of jobs that we need to do")
flag.Parse()
// Dummy channel to coordinate the number of concurrent goroutines.
// This channel should be buffered otherwise we will be immediately blocked
// when trying to fill it.
concurrentGoroutines := make(chan struct{}, *maxNbConcurrentGoroutines)
// Fill the dummy channel with maxNbConcurrentGoroutines empty struct.
for i := 0; i < *maxNbConcurrentGoroutines; i++ {
concurrentGoroutines <- struct{}{}
}
// The done channel indicates when a single goroutine has
// finished its job.
done := make(chan bool)
// The waitForAllJobs channel allows the main program
// to wait until we have indeed done all the jobs.
waitForAllJobs := make(chan bool)
// Collect all the jobs, and since the job is finished, we can
// release another spot for a goroutine.
go func() {
for i := 0; i < *nbJobs; i++ {
<-done
// Say that another goroutine can now start.
concurrentGoroutines <- struct{}{}
}
// We have collected all the jobs, the program
// can now terminate
waitForAllJobs <- true
}()
// Try to start nbJobs jobs
for i := 1; i <= *nbJobs; i++ {
fmt.Printf("ID: %v: waiting to launch!\n", i)
// Try to receive from the concurrentGoroutines channel. When we have something,
// it means we can start a new goroutine because another one finished.
// Otherwise, it will block the execution until an execution
// spot is available.
<-concurrentGoroutines
fmt.Printf("ID: %v: it's my turn!\n", i)
go func(id int) {
DoWork()
fmt.Printf("ID: %v: all done!\n", id)
done <- true
}(i)
}
// Wait for all jobs to finish
<-waitForAllJobs
}
@ajinabraham
Copy link

ajinabraham commented Aug 7, 2022

I did some benchmarking if anyone is interested, parsing diff patch and doing some regex check on files from around 2000 git commits from a local repo. exec.Command() is the bottleneck as it panics with open /dev/null: too many open files due to soft ulimit when the no of goroutines are uncontrolled.

Experiment Execution Time
Without Goroutine 3m33.714605387s
With Goroutine, following @crepehat snippet, with maxNbConcurrentGoroutines set to 50 35.2966391
With Goroutine, checking runtime.NumGoroutine() > 80 and adding 1s sleep 35.2392869
With Goroutine and semaphore NewWeighted(12) 28.851024029s
With limiter, NewConcurrencyLimiter set to 50 21.278964931s
go 1.17
MacBook Pro (16-inch, 2019)
2.6 GHz 6-Core Intel i7 | 32 GB RAM

@ajinabraham
Copy link

A bit more benchmarking done on level grounds.

Function execution times measured from Go with different goroutine concurrency limits (Average of 3 runs)

Sample Limit 12 Limit 50 Limit 80
Atomic 23.86s 23.40s 27.88s
Semaphore 29.54s 29.17s 35.38s
WaitGroup 28.62s 31.15s 38.26s
WaitGroup with Sleep 2m 56.85s 47.67s 37.93s
Without Goroutine 3m 33.71s NA NA

Go program’s total execution time measured using zsh’s time with different concurrency limits.(Average of 3 runs)

Sample Limit 12 Limit 50 Limit 80
Atomic 29.900 34.190 42.535
Semaphore 30.275 30.406 36.771
WaitGroup 29.464 32.382 39.660
WaitGroup with Sleep 2m 56.8521729s 48.905 39.100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment