the higher the concurrency, the slower the performance of a single coroutine？ #911
Description
my server and client node resource is: 56C 512G memory nvme ssd 10000Mbps bandwidth
rustfs version is latest
os is：ubuntu 24.04.1 LTS (Noble Numbat)
problem:
i used golang code to read a 32MiB object from rustfs server, i found If concurrency is set to 1, then a single coroutine will take 59ms， If concurrency is set to 2, then a single coroutine will take 110ms， If concurrency is set to 4, then a single coroutine will take 200ms，so it's increasing exponentially.
reproduce step:
- create object
1. install rustfs
2. mc mb rustfs/test
3. dd if=/dev/random of=bxx bs=1M count=32
4. mc cp bxx rustfs/test
- client code
package main
import (
"context"
"fmt"
"io"
"log"
"sync"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
endpoint := "10.50.5.28:9000"
accessKeyID := "rustfsadmin"
secretAccessKey := "rustfsadmin"
useSSL := false
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
Secure: useSSL,
})
if err != nil {
log.Fatalln("Error creating MinIO client:", err)
}
bucketName := "test"
concurrency := 2 //could be 1,2,3,4,5....
var wg sync.WaitGroup
for i := 0; i < concurrency; i++ {
wg.Add(1)
go func(id int) {
objectPath := "bxx"
defer wg.Done()
ctx := context.Background()
start := time.Now()
obj, err := minioClient.GetObject(ctx, bucketName, objectPath, minio.GetObjectOptions{})
if err != nil {
log.Printf("[Worker %d] Error getting object: %v at %v\n", id, err, time.Now())
return
}
defer obj.Close()
data, err := io.ReadAll(obj)
if err != nil {
log.Printf("[Worker %d] Error reading object: %v at %v\n", id, err, time.Now())
return
}
accessTime := time.Now()
log.Printf("[Worker %d] Read object %s, Size: %d bytes at %v (started at %v, duration: %v)\n",
id, objectPath, len(data), accessTime, start, accessTime.Sub(start))
}(i + 1)
}
wg.Wait()
fmt.Println("All reads completed.")
}
- result like:
goroutine=1
[Worker 1] Finished reading 67108864 bytes, duration: 59.333091ms
goroutine=2
[Worker 2] Finished reading 67108864 bytes, duration: 113.422243ms
[Worker 1] Finished reading 67108864 bytes, duration: 116.75297ms
I am confused about why the higher the concurrency, the longer the time taken by a single coroutine.
what's wrong with me, or how can i do to solve it, thanks in advantage