-
Notifications
You must be signed in to change notification settings - Fork 16.1k
Import Error in Python on M1 Mac #10417
Description
What version of protobuf and what language are you using?
Version: 3.20.1
Language: Python
What operating system (Linux, Windows, ...) and version?
MacOS Monterey 12.5, M1 Version
What runtime / compiler are you using (e.g., python version or gcc version)
Python 3.10.4
What did you do?
Steps to reproduce the behavior:
In "sdks/python" directory of Apache Beam repo (https://github.com/apache/beam/), in a pyenv-virtualenv using Python 3.10.4:
pip install --upgrade pip setuptools
pip install -r build-requirements.txt
pip install -e .[gcp,test]
python -m pytest apache_beam/io/textio_test.py::TextSourceTest
This passes on my Linux Cloudtop, but fails on the Mac with the following:
ImportError while loading conftest '/Users/bjornpedersen/Documents/beam/sdks/python/conftest.py'. conftest.py:22: in <module> from apache_beam.options import pipeline_options apache_beam/__init__.py:92: in <module> from apache_beam import coders apache_beam/coders/__init__.py:17: in <module> from apache_beam.coders.coders import * apache_beam/coders/coders.py:58: in <module> from apache_beam.coders import coder_impl apache_beam/coders/coder_impl.py:62: in <module> from apache_beam.typehints.schemas import named_tuple_from_schema apache_beam/typehints/schemas.py:72: in <module> from apache_beam.portability import common_urns apache_beam/portability/common_urns.py:22: in <module> from .api import beam_runner_api_pb2_urns E ImportError: cannot import name 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api' (unknown location)
When I run python gen_protos.py to force the urns to be generated, I get the following:
org/apache/beam/model/interactive/v1/beam_interactive_api.proto:36:1: warning: Import google/protobuf/timestamp.proto is unused. Writing mypy to org/apache/beam/model/pipeline/v1/external_transforms_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/schema_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/metrics_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/endpoints_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/beam_runner_api_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/standard_window_fns_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_artifact_api_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_expansion_api_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_job_api_pb2.pyi Writing mypy to org/apache/beam/model/fn_execution/v1/beam_fn_api_pb2.pyi Writing mypy to org/apache/beam/model/fn_execution/v1/beam_provision_api_pb2.pyi Writing mypy to org/apache/beam/model/interactive/v1/beam_interactive_api_pb2.pyi Traceback (most recent call last): File "/Users/bjornpedersen/Documents/beam/sdks/python/gen_protos.py", line 561, in <module> generate_proto_files(force=True) File "/Users/bjornpedersen/Documents/beam/sdks/python/gen_protos.py", line 555, in generate_proto_files generate_urn_files(proto_package, PYTHON_OUTPUT_PATH) File "/Users/bjornpedersen/Documents/beam/sdks/python/gen_protos.py", line 126, in generate_urn_files import google.protobuf.pyext._message as pyext_message ImportError: dlopen(/Users/bjornpedersen/.pyenv/versions/env10.4/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): tried: '/Users/bjornpedersen/.pyenv/versions/env10.4/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))), '/Users/bjornpedersen/.pyenv/versions/3.10.4/envs/env10.4/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e)))
which seems to indicate the chip is the issue.
The wheel I'm using is protobuf-3.20.1-cp310-cp310-macosx_10_9_universal2.whl.
Seems potentially related to previous issue #8820