python CopyFrom segmentation fault on version 4.24.0

What version of protobuf and what language are you using?
Version: v4.24.0
Language: Python upb

What operating system (Linux, Windows, ...) and version?
Repros on Linux and MacOS

What runtime / compiler are you using (e.g., python version or gcc version)
python 3.8
Installed Apple clang version 14.0.3 (clang-1403.0.22.14.1)

What did you do?
We have a proto message that fundamentally looks like this:

syntax = "proto3";

package message;

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
    map<int32, double> map_field = 3;
}

And we have a piece of python that tries to copy an empty version of one of these messages into the other:

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

This is resulting on a segmentation fault on exactly the CopyFrom line.

This failure started with us picking up version 4.24.0 with yesterday's release. Reverting back to any version prior fixes the issue.

Unfortunately building the above example proto as an MVP did not repro the segmentation fault for me. If there are any diagnostics I can provide that would make root causing this easier, happy to help.

What did you expect to see
No seg fault, behavior as before.

What did you see instead?
Segmentation fault failure on CopyFrom

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment

I've seen this issue with python 3.9 and 3.11 as well. It's blocking our project's CI runs.

Restricts protobuf dependency to < 4.24 due to protocolbuffers/protob…

Not sure why but I can NOT reproduce in google3 internal CL with

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
    map<int32, double> map_field = 3;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

Nor opensource on my linux:

$ pip show protobuf
Name: protobuf
Version: 4.24.0
$ python
Python 3.11.4 (main, Jun  7 2023, 10:13:09) [GCC 12.2.0] on linux
>>> from google.protobuf.internal import api_implementation
>>> print(api_implementation.Type())
upb
>>> from google.protobuf import timestamp_pb2
>>> msg = timestamp_pb2.Timestamp()
>>> msg2 = timestamp_pb2.Timestamp()
>>> msg2.CopyFrom(msg)
>>> from google.protobuf import struct_pb2
>>> msg = struct_pb2.Struct()
>>> msg2 =  struct_pb2.Struct()
>>> msg2.CopyFrom(msg)

Just a thought, could it be that the protos that we're intending to use are "compiled" with a previous protoc version, and the issue is that this new version throws in that case? In our project, it was failing when trying to use some protos from a TensorFlow dependency (which was released before this new protobuf release from yesterday).

python CopyFrom segmentation fault on version 4.24.0

Can you help to verify:
1, Does the segment fault still raise if not depend on TensorFlow.
2, Does the segment fault still raise for the message that do not have map:

message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)

3, Does the segment fault still raise for struct_pb2.Struct()

from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

Just a thought, could it be that the protos that we're intending to use are "compiled" with a previous protoc version, and the issue is that this new version throws in that case? In our project, it was failing when trying to use some protos from a TensorFlow dependency (which was released before this new protobuf release from yesterday).

This might actually be possibly how it's happening, as I could not produce a simple repro outside of our internal product.

Can you help to verify: 1, Does the segment fault still raise if not depend on TensorFlow. 2, Does the segment fault still raise for the message that do not have map:
message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)
3, Does the segment fault still raise for struct_pb2.Struct()
from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

I am working to answer your other questions, but I can confirm (1) that we do not depend on TensorFlow.

Can you help to verify: 1, Does the segment fault still raise if not depend on TensorFlow. 2, Does the segment fault still raise for the message that do not have map:
message TestMessage {
    optional int64 field_one = 1;
    optional int64 field_two = 2;
}

message = message_pb2.TestMessage()
message2 = message_pb2.TestMessage()
message2.CopyFrom(message)
3, Does the segment fault still raise for struct_pb2.Struct()
from google.protobuf import struct_pb2
msg = struct_pb2.Struct()
msg2 =  struct_pb2.Struct()
msg2.CopyFrom(msg)

Confirming for (3) that this does not fail with struct_pb2.Struct() and removing for (2), the segmentation fault still occurs with no map.

Thank you for these info. As both of you mentioned that the code is depending on old code gen, can you try if upgrade these gen files with newest protoc to see if it can resolve the issue?

upb CopyFrom the default empty message should just clear instead of d…

Test part: upb CopyFrom the default empty message should just clear i…

The issue should been fixed by #1514
25.0 will back to memory copy and no crash for copy the unsigned empty sub message

@anandolee

@anandolee Thank you, this is great to hear!

Relaxes restriction on protobuf dep since issue was fixed. (#6660)

R-tensorflow: Autospec creation for update from version 2.14.0 to ver…

Navigation Menu

Repository navigation

python CopyFrom segmentation fault on version 4.24.0 #13485

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

python CopyFrom segmentation fault on version 4.24.0 #13485

Description

Activity

arcra commented on Aug 9, 2023

anandolee commented on Aug 9, 2023

arcra commented on Aug 9, 2023

anandolee commented on Aug 9, 2023

sfc-gh-mussakli commented on Aug 10, 2023

sfc-gh-mussakli commented on Aug 10, 2023

anandolee commented on Aug 10, 2023

22 remaining items

anandolee commented on Aug 25, 2023

sfc-gh-mussakli commented on Aug 25, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions