Skip to content

sql: v22.2.0: cannot perform TRUNCATE due to concurrent unknown mutation #93443

Closed
#98537
Closed
sql: v22.2.0: cannot perform TRUNCATE due to concurrent unknown mutation#93443
#98537
Assignees
chengxiong-ruan
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)branch-masterFailures and bugs on the master branch.branch-release-23.1Used to mark GA and release blockers, technical advisories, and bugs for 23.1
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Dec 12, 2022

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3803757482/?referrer=webhooks_plugin

Panic message:

truncate.go:368: cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
(1)
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.checkTableForDisallowedMutationsWithTruncate
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:368
| github.com/cockroachdb/cockroach/pkg/sql.(*planner).truncateTable
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:173
| github.com/cockroachdb/cockroach/pkg/sql.(*truncateNode).startExec
| github.com/cockroachdb/cockroach/pkg/sql/truncate.go:128
| github.com/cockroachdb/cockroach/pkg/sql.startExec.func2
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:518
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:112
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:297
| github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:79
| github.com/cockroachdb/cockroach/pkg/sql.walkPlan
| github.com/cockroachdb/cockroach/pkg/sql/walk.go:43
| github.com/cockroachdb/cockroach/pkg/sql.startExec
| github.com/cockroachdb/cockroach/pkg/sql/plan.go:521
| github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
| github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:147
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:178
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:90
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:247
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
| github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:92
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:246
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:291
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:695
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1611
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1334
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1541
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1177
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:687
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2382
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:128
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:218
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1998
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2000
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1846
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:828
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
Wraps: (4) cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
Error types: (1) *colexecerror.notInternalError (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: cannot perform TRUNCATE due to concurrent unknown mutation of type *tabledesc.mutation for mutation 1 in tabledesc.Mutable: {ID: 1993, Version: 23, ModificationTime: "1670869560.388020034,0", ParentID: 1771, ParentSchemaID: 2010, State: PUBLIC, NextColumnID: 8, Columns: [{ID: 1, TypeID: 2950, Null: false, HasDefault: true}, {ID: 2, TypeID: 20, Null: false}, {ID: 3, TypeID: 20, Null: false}, {ID: 4, TypeID: 20, Null: false}, {ID: 5, TypeID: 1184, Null: true}, {ID: 6, TypeID: 1184, Null: true}], NextFamilyID: 1, Families: [{ID: 0, Columns: [1, 2, 3, 4, 5, 6, 7]}], MutationJobs: [{MutationID: 3, JobID: 821818688580190246}], Mutations: [{MutationID: 3, Direction: ADD, State: WRITE_ONLY, Column: {ID: 7, TypeID: 1184, Null: false, Hidden: true, HasDefault: true, State: ADD, MutationID: 3}}, {MutationID: 3, Direction: ADD, State: WRITE_ONLY}], PrimaryIndex: 10, NextIndexID: 13, Indexes: [{ID: 10, Unique: true, KeyColumns: [{ID: 1, Dir: ASC}], StoreColumns: [2, 3, 4, 5, 6, 7]}, {ID: 11, Unique: false, KeyColumns: [{ID: 5, Dir: DESC}], KeySuffixColumns: [1]}]}
truncate.go:368: *withstack.withStack (top exception)
*assert.withAssertionFailure
*colexecerror.notInternalError

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/truncate.go

Lines 367 to 369 in 77667a1

} else {
return errors.AssertionFailedf("cannot perform TRUNCATE due to "+
"concurrent unknown mutation of type %T for mutation %d in %v", m, i, desc)
in pkg/sql.checkTableForDisallowedMutationsWithTruncate

cockroach/pkg/sql/truncate.go

Lines 172 to 174 in 77667a1

if err := checkTableForDisallowedMutationsWithTruncate(tableDesc); err != nil {
return err
in pkg/sql.(*planner).truncateTable

cockroach/pkg/sql/truncate.go

Lines 127 to 129 in 77667a1

for id, name := range toTruncate {
if err := p.truncateTable(ctx, id, tree.AsStringWithFQNames(t.n, params.Ann())); err != nil {
return err
in pkg/sql.(*truncateNode).startExec

cockroach/pkg/sql/plan.go

Lines 517 to 519 in 77667a1

}
return n.startExec(params)
},
in pkg/sql.startExec.func2

cockroach/pkg/sql/walk.go

Lines 111 to 113 in 77667a1

}
v.err = v.observer.leaveNode(name, plan)
}()
in pkg/sql.(*planVisitor).visitInternal.func1

cockroach/pkg/sql/walk.go

Lines 296 to 298 in 77667a1

}
}
in pkg/sql.(*planVisitor).visitInternal

cockroach/pkg/sql/walk.go

Lines 78 to 80 in 77667a1

}
v.visitInternal(plan, name)
return plan
in pkg/sql.(*planVisitor).visit

cockroach/pkg/sql/walk.go

Lines 42 to 44 in 77667a1

v := makePlanVisitor(ctx, observer)
v.visit(plan)
return v.err
in pkg/sql.walkPlan

cockroach/pkg/sql/plan.go

Lines 520 to 522 in 77667a1

}
return walkPlan(params.ctx, plan, o)
}
in pkg/sql.startExec

cockroach/pkg/sql/plan_node_to_row_source.go

Lines 146 to 148 in 77667a1

// This starts all of the nodes below this node.
if err := startExec(p.params, p.node); err != nil {
p.MoveToDraining(err)
in pkg/sql.(*planNodeToRowSource).Start

cockroach/pkg/sql/colexec/columnarizer.go

Lines 177 to 179 in 77667a1

ctx = c.StartInternalNoSpan(ctx)
c.input.Start(ctx)
if execStatsHijacker, ok := c.input.(execinfra.ExecStatsForTraceHijacker); ok {
in pkg/sql/colexec.(*Columnarizer).Init

cockroach/pkg/sql/colflow/stats.go

Lines 89 to 91 in 77667a1

func (bic *batchInfoCollector) Init(ctx context.Context) {
bic.Operator.Init(ctx)
bic.mu.Lock()
in pkg/sql/colflow.(*batchInfoCollector).Init

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 246 to 248 in 77667a1

return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
})
in pkg/sql/colflow.(*BatchFlowCoordinator).init.func1

cockroach/pkg/sql/colexecerror/error.go

Lines 91 to 93 in 77667a1

}()
operation()
return retErr
in pkg/sql/colexecerror.CatchVectorizedRuntimeError

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 245 to 247 in 77667a1

func (f *BatchFlowCoordinator) init(ctx context.Context) error {
return colexecerror.CatchVectorizedRuntimeError(func() {
f.input.Root.Init(ctx)
in pkg/sql/colflow.(*BatchFlowCoordinator).init

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 290 to 292 in 77667a1

if err := f.init(ctx); err != nil {
f.pushError(err)
in pkg/sql/colflow.(*BatchFlowCoordinator).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 319 to 321 in 77667a1

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}
in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 694 to 696 in 77667a1

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1610 to 1612 in 77667a1

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/distsql_running.go

Lines 1333 to 1335 in 77667a1

// the planner whether or not to plan remote table readers.
cleanup := dsp.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*DistSQLPlanner).PlanAndRunAll

cockroach/pkg/sql/conn_executor_exec.go

Lines 1540 to 1542 in 77667a1

}
err := ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory)
return *recv.stats, err
in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1176 to 1178 in 77667a1

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err = ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 686 to 688 in 77667a1

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 128 to 130 in 77667a1

err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execStmt.func1

cockroach/pkg/sql/conn_executor_exec.go

Lines 2381 to 2383 in 77667a1

} else {
err = op(ctx)
}
in pkg/sql.(*connExecutor).execWithProfiling

cockroach/pkg/sql/conn_executor_exec.go

Lines 127 to 129 in 77667a1

case stateOpen:
err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor_exec.go

Lines 217 to 219 in 77667a1

}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal

cockroach/pkg/sql/conn_executor.go

Lines 1997 to 1999 in 77667a1

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execCmd.func2

cockroach/pkg/sql/conn_executor.go

Lines 1999 to 2001 in 77667a1

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1845 to 1847 in 77667a1

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 827 to 829 in 77667a1

}(ctx, h)
return h.ex.run(ctx, s.pool, reserved, cancel)
}
in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in 77667a1

reservedOwned = false // We're about to pass ownership away.
retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn)
}()
in pkg/sql/pgwire.(*conn).processCommandsAsync.func1

pkg/sql/truncate.go in pkg/sql.checkTableForDisallowedMutationsWithTruncate at line 368
pkg/sql/truncate.go in pkg/sql.(*planner).truncateTable at line 173
pkg/sql/truncate.go in pkg/sql.(*truncateNode).startExec at line 128
pkg/sql/plan.go in pkg/sql.startExec.func2 at line 518
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal.func1 at line 112
pkg/sql/walk.go in pkg/sql.(*planVisitor).visitInternal at line 297
pkg/sql/walk.go in pkg/sql.(*planVisitor).visit at line 79
pkg/sql/walk.go in pkg/sql.walkPlan at line 43
pkg/sql/plan.go in pkg/sql.startExec at line 521
pkg/sql/plan_node_to_row_source.go in pkg/sql.(*planNodeToRowSource).Start at line 147
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Init at line 178
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Init at line 90
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init.func1 at line 247
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 92
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).init at line 246
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 291
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 320
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 695
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1611
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1334
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1541
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1177
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 687
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 129
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2382
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 128
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 218
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1998
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2000
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1846
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 828
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
Tag Value
Cockroach Release v22.2.0
Cockroach SHA: 77667a1
Platform linux amd64
Distribution CCL
Environment v22.2.0
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-22335

Metadata

Metadata

Assignees

Labels

C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-sql-foundationsSQL Foundations Team (formerly SQL Schema + SQL Sessions)branch-masterFailures and bugs on the master branch.branch-release-23.1Used to mark GA and release blockers, technical advisories, and bugs for 23.1

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions