-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Closed
Labels
Description
Expected Behavior
Actual Behavior
Steps to Reproduce the Problem
start temporal worker servcie, log is:
TEMPORAL_ADDRESS is not set, setting it to 9.139.60.15:7233
2025/07/14 12:28:05 Loading config; env=docker,zone=,configDir=config
2025/07/14 12:28:05 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"Build info.","git-time":"2025-03-27T22:45:52.000Z","git-revision":"a31fd7a9e5db4c28627eb18676558e974e95e1ba","git-modified":false,"go-arch":"amd64","go-os":"linux","go-version":"go1.23.2","cgo-enabled":false,"server-version":"1.27.2","debug-mode":false,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:175"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: worker.historyscannerverifyretention oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: worker.historyscannerdataminage oldValue: nil newValue: { constraints: {} value: 604800000000000 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: limit.historycount.error oldValue: nil newValue: { constraints: {} value: 102400 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: history.cachettl oldValue: nil newValue: { constraints: {} value: 600000000000 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: worker.historyscannerenabled oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: frontend.namespacecount oldValue: nil newValue: { constraints: {} value: 200000000 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.568Z","msg":"dynamic config changed for the key: worker.executionsscannerenabled oldValue: nil newValue: { constraints: {} value: true }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.569Z","msg":"dynamic config changed for the key: limit.historysize.error oldValue: nil newValue: { constraints: {} value: 209715200 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.569Z","msg":"dynamic config changed for the key: history.cacheinitialsize oldValue: nil newValue: { constraints: {} value: 32 }","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:361"}
{"level":"info","ts":"2025-07-14T12:28:05.569Z","msg":"Updated dynamic config","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:207"}
{"level":"warn","ts":"2025-07-14T12:28:05.569Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:205"}
{"level":"warn","ts":"2025-07-14T12:28:05.585Z","msg":"Supplied configuration key/value mismatches persisted cluster metadata. Continuing with the persisted value as this value cannot be changed once initialized.","component":"metadata-initializer","key":"persistence.numHistoryShards","ignored-value":32,"value":512,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:822"}
{"level":"info","ts":"2025-07-14T12:28:05.586Z","msg":"Use rpc address 127.0.0.1:7233 for cluster active.","component":"metadata-initializer","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/cluster_metadata_loader.go:92"}
{"level":"info","ts":"2025-07-14T12:28:05.587Z","msg":"Service is not requested, skipping initialization.","service":"history","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:490"}
{"level":"info","ts":"2025-07-14T12:28:05.587Z","msg":"Service is not requested, skipping initialization.","service":"matching","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:510"}
{"level":"info","ts":"2025-07-14T12:28:05.587Z","msg":"Service is not requested, skipping initialization.","service":"frontend","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:539"}
{"level":"info","ts":"2025-07-14T12:28:05.587Z","msg":"Service is not requested, skipping initialization.","service":"internal-frontend","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:539"}
{"level":"info","ts":"2025-07-14T12:28:05.594Z","msg":"Quota changed","service":"worker","component":"persistence","scope":"host","current-quota":null,"new-quota":500,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/quotas/calculator/logged_calculator.go:124"}
{"level":"info","ts":"2025-07-14T12:28:05.600Z","msg":"Initialized service resolver for persistence rate limiting","service":"worker","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/fx.go:91"}
{"level":"info","ts":"2025-07-14T12:28:05.600Z","msg":"historyClient: ownership caching disabled","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/client/history/client.go:86"}
{"level":"info","ts":"2025-07-14T12:28:05.606Z","msg":"creating new visibility manager","visibility_plugin_name":"mysql8","visibility_index_name":"temporal_visibility","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/visibility/factory.go:155"}
{"level":"info","ts":"2025-07-14T12:28:05.606Z","msg":"Starting server for services","value":{"worker":{}},"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/server_impl.go:107"}
{"level":"info","ts":"2025-07-14T12:28:05.607Z","msg":"PProf listen on ","host":"localhost","port":7936,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/pprof/pprof.go:82"}
{"level":"info","ts":"2025-07-14T12:28:05.619Z","msg":"RuntimeMetricsReporter started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/runtime.go:141"}
{"level":"info","ts":"2025-07-14T12:28:05.620Z","msg":"worker starting","service":"worker","component":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/service.go:242"}
{"level":"info","ts":"2025-07-14T12:28:05.622Z","msg":"Membership heartbeat upserted successfully","address":"9.139.60.15","port":6939,"hostId":"fe07872e-60ad-11f0-aada-525400ad8fb7","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/monitor.go:340"}
{"level":"info","ts":"2025-07-14T12:28:05.623Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"9.139.62.137:6934,9.139.60.191:6933,9.139.62.44:6935,9.139.63.38:6935,9.139.60.167:6939,9.139.60.15:6939,9.139.62.114:6934,9.139.60.9:6934,9.139.61.143:6933,9.139.62.241:6935,9.139.60.113:6933,9.139.63.153:6933,9.139.62.183:6939,9.139.63.242:6934,9.139.61.241:6939,9.139.62.154:6934,9.139.62.83:6933,9.139.63.55:6935,9.139.60.6:6935,9.139.63.75:6939","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/monitor.go:382"}
{"level":"info","ts":"2025-07-14T12:28:05.636Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["9.139.60.113:7233","9.139.60.191:7233","9.139.61.143:7233","9.139.62.83:7233","9.139.63.153:7233"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:05.636Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["9.139.60.9:7234","9.139.62.114:7234","9.139.62.137:7234","9.139.62.154:7234","9.139.63.242:7234"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:05.637Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["9.139.60.6:7235","9.139.62.241:7235","9.139.62.44:7235","9.139.63.38:7235","9.139.63.55:7235"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:05.638Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["9.139.60.15:0","9.139.60.167:0","9.139.61.241:0","9.139.62.183:0","9.139.63.75:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:05.656Z","msg":"temporal-sys-tq-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:295"}
{"level":"info","ts":"2025-07-14T12:28:05.657Z","msg":"temporal-sys-executions-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:295"}
{"level":"info","ts":"2025-07-14T12:28:05.657Z","msg":"temporal-sys-history-scanner-workflow workflow successfully started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:295"}
{"level":"info","ts":"2025-07-14T12:28:05.872Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-executions-scanner-taskqueue-0","WorkerID":"[email protected]:0","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.876Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-tq-scanner-taskqueue-0","WorkerID":"[email protected]:0","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.879Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-history-scanner-taskqueue-0","WorkerID":"[email protected]:0","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.887Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-processor-parent-close-policy","WorkerID":"[email protected]:0","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.891Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"default-worker-tq","WorkerID":"temporal-system@temporal-worker-768cdf67cc-s7bc5","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.893Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-migration-activity-tq","WorkerID":"temporal-system@temporal-worker-768cdf67cc-s7bc5","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.895Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-delete-namespace-activity-tq","WorkerID":"temporal-system@temporal-worker-768cdf67cc-s7bc5","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.896Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-add-search-attributes-activity-tq","WorkerID":"temporal-system@temporal-worker-768cdf67cc-s7bc5","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.898Z","msg":"Started Worker","service":"worker","Namespace":"temporal-system","TaskQueue":"temporal-sys-dlq-activity-tq","WorkerID":"temporal-system@temporal-worker-768cdf67cc-s7bc5","logging-call-at":"/opt/hostedtoolcache/go/1.23.2/x64/src/sync/oncefunc.go:57"}
{"level":"info","ts":"2025-07-14T12:28:05.898Z","msg":"none","component":"worker-manager","lifecycle":"Started","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/worker.go:124"}
{"level":"info","ts":"2025-07-14T12:28:05.898Z","msg":"none","component":"perns-worker-manager","lifecycle":"Starting","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/pernamespaceworker.go:176"}
{"level":"info","ts":"2025-07-14T12:28:05.898Z","msg":"none","component":"perns-worker-manager","lifecycle":"Started","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/pernamespaceworker.go:188"}
{"level":"info","ts":"2025-07-14T12:28:05.898Z","msg":"worker service started","service":"worker","component":"worker","address":"9.139.60.15:0","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/service.go:271"}
{"level":"info","ts":"2025-07-14T12:28:06.445Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["9.139.60.15:0","9.139.60.167:0","9.139.61.241:0","9.139.62.170:0","9.139.62.183:0","9.139.63.75:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:07.624Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["9.139.60.15:0","9.139.60.167:0","9.139.61.241:0","9.139.62.170:0","9.139.62.183:0","9.139.63.56:0","9.139.63.75:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:11.760Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["9.139.60.15:0","9.139.60.167:0","9.139.62.170:0","9.139.62.183:0","9.139.63.56:0","9.139.63.75:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:28:14.770Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["9.139.60.15:0","9.139.60.167:0","9.139.62.170:0","9.139.63.56:0","9.139.63.75:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:329"}
{"level":"info","ts":"2025-07-14T12:30:42.485Z","msg":"Executions scavenger starting","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/scavenger.go:135"}
{"level":"info","ts":"2025-07-14T12:30:42.485Z","msg":"Executions scavenger started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/scavenger.go:140"}
{"level":"error","ts":"2025-07-14T12:31:03.960Z","msg":"Operation failed with internal error.","error":"ListConcreteExecutions is not implemented","error-type":"serviceerror.Unimplemented","operation":"ListConcreteExecutions","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1319","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1319\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1291\ngo.temporal.io/server/common/persistence.(*executionPersistenceClient).ListConcreteExecutions.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:352\ngo.temporal.io/server/common/persistence.(*executionPersistenceClient).ListConcreteExecutions\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:354\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).ListConcreteExecutions.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_retryable_clients.go:350\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).ListConcreteExecutions\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_retryable_clients.go:354\ngo.temporal.io/server/service/worker/scanner/executions.(*task).Run.(*task).getPaginationFn.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:218\ngo.temporal.io/server/common/collection.(*PagingIteratorImpl[...]).getNextPage\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/collection/paging_iterator.go:116\ngo.temporal.io/server/common/collection.NewPagingIteratorWithToken[...]\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/collection/paging_iterator.go:68\ngo.temporal.io/server/service/worker/scanner/executions.(*task).Run\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:116\ngo.temporal.io/server/service/worker/scanner/executor.(*fixedPoolExecutor).worker\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executor/executor.go:144"}
{"level":"error","ts":"2025-07-14T12:31:03.960Z","msg":"unable to paginate concrete execution","service":"worker","shard-id":6,"error":"ListConcreteExecutions is not implemented","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:124","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/service/worker/scanner/executions.(*task).Run\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:124\ngo.temporal.io/server/service/worker/scanner/executor.(*fixedPoolExecutor).worker\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executor/executor.go:144"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x271c865]
goroutine 25316 [running]:
go.temporal.io/server/common/persistence/versionhistory.GetVersionHistory(...)
/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/versionhistory/version_histories.go:61
go.temporal.io/server/common/persistence/versionhistory.GetCurrentVersionHistory(...)
/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/versionhistory/version_histories.go:193
go.temporal.io/server/service/worker/scanner/executions.(*mutableStateValidator).Validate(0xc000fb9c00, {0x317a396?, 0x18?}, 0xc000fb9f08)
/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/mutable_state_validator.go:80 +0x65
go.temporal.io/server/service/worker/scanner/executions.(*task).validate(0xc0008d2e40, 0xc000fb9f08)
/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:171 +0x59b
go.temporal.io/server/service/worker/scanner/executions.(*task).Run(0xc0008d2e40)
/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executions/task.go:129 +0x493
go.temporal.io/server/service/worker/scanner/executor.(*fixedPoolExecutor).worker(0xc000e49450)
/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executor/executor.go:144 +0x9b
created by go.temporal.io/server/service/worker/scanner/executor.(*fixedPoolExecutor).Start in goroutine 25311
/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/executor/executor.go:106 +0x45
Specifications

Reactions are currently unavailable