Airflow using mysql backend, ModuleNotFoundError

[whydo]

Hi All,

I'm trying to use a mysql backend for airflow but failing miserably .

My connection line is this:

sql_alchemy_conn = mysql+pymysql://AFUSER:AFPAS5y@host0001477.acme.com/airflow_db

I'm getting the error:

Traceback (most recent call last):

  File "/opt/mapr/airflow/airflow-2.7.1/bin/airflow", line 4, in <module>

    from airflow.__main__ import main

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/airflow/__init__.py", line 68, in <module>

    settings.initialize()

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/airflow/settings.py", line 526, in initialize

    configure_orm()

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/airflow/settings.py", line 222, in configure_orm

    engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True)

  File "<string>", line 2, in create_engine

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned

    return fn(*args, **kwargs)

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine

    dbapi = dialect_cls.dbapi(**dbapi_args)

  File "/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/pymysql.py", line 80, in dbapi

    return __import__("pymysql")

ModuleNotFoundError: No module named 'pymysql'

I see that ezmeral has the module included:

# locate -i pymysql

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql/__init__.py

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql/__pycache__

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql/patch.py

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql/__pycache__/__init__.cpython-39.pyc

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/aws_xray_sdk/ext/pymysql/__pycache__/patch.cpython-39.pyc

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/__init__.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/charset.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/connections.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/converters.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/cursors.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/err.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/times.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/util.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/CLIENT.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/COMMAND.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/ER.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/FIELD_TYPE.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/FLAG.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/SERVER_STATUS.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/jedi/third_party/typeshed/third_party/2and3/pymysql/constants/__init__.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/METADATA.toml

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/__init__.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/charset.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/connections.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/converters.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/cursors.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/err.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/times.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/util.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/CLIENT.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/COMMAND.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/CR.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/ER.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/FIELD_TYPE.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/FLAG.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/SERVER_STATUS.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/pymysql-stubs/constants/__init__.pyi

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/pymysql.py

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/__pycache__/pymysql.cpython-39.pyc

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/INSTALLER

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/METADATA

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/RECORD

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/REQUESTED

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/WHEEL

/opt/mapr/airflow/airflow-2.7.1/build/env/lib/python3.9/site-packages/types_PyMySQL-1.1.0.1.dist-info/top_level.txt

It also is installed on the base system.

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info

/usr/lib/python3.6/site-packages/pymysql

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info/PKG-INFO

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info/SOURCES.txt

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info/dependency_links.txt

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info/requires.txt

/usr/lib/python3.6/site-packages/PyMySQL-0.10.1-py3.6.egg-info/top_level.txt

/usr/lib/python3.6/site-packages/pymysql/__init__.py

/usr/lib/python3.6/site-packages/pymysql/__pycache__

/usr/lib/python3.6/site-packages/pymysql/_auth.py

/usr/lib/python3.6/site-packages/pymysql/_compat.py

/usr/lib/python3.6/site-packages/pymysql/_socketio.py

/usr/lib/python3.6/site-packages/pymysql/charset.py

/usr/lib/python3.6/site-packages/pymysql/connections.py

/usr/lib/python3.6/site-packages/pymysql/constants

/usr/lib/python3.6/site-packages/pymysql/converters.py

/usr/lib/python3.6/site-packages/pymysql/cursors.py

/usr/lib/python3.6/site-packages/pymysql/err.py

/usr/lib/python3.6/site-packages/pymysql/optionfile.py

/usr/lib/python3.6/site-packages/pymysql/protocol.py

/usr/lib/python3.6/site-packages/pymysql/times.py

/usr/lib/python3.6/site-packages/pymysql/util.py

/usr/lib/python3.6/site-packages/pymysql/__pycache__/__init__.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/__init__.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_auth.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_auth.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_compat.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_compat.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_socketio.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/_socketio.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/charset.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/charset.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/connections.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/connections.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/converters.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/converters.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/cursors.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/cursors.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/err.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/err.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/optionfile.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/optionfile.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/protocol.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/protocol.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/times.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/times.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/util.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/__pycache__/util.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/CLIENT.py

/usr/lib/python3.6/site-packages/pymysql/constants/COMMAND.py

/usr/lib/python3.6/site-packages/pymysql/constants/CR.py

/usr/lib/python3.6/site-packages/pymysql/constants/ER.py

/usr/lib/python3.6/site-packages/pymysql/constants/FIELD_TYPE.py

/usr/lib/python3.6/site-packages/pymysql/constants/FLAG.py

/usr/lib/python3.6/site-packages/pymysql/constants/SERVER_STATUS.py

/usr/lib/python3.6/site-packages/pymysql/constants/__init__.py

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/CLIENT.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/CLIENT.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/COMMAND.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/COMMAND.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/CR.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/CR.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/ER.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/ER.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/FIELD_TYPE.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/FIELD_TYPE.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/FLAG.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/FLAG.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/SERVER_STATUS.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/SERVER_STATUS.cpython-36.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/__init__.cpython-36.opt-1.pyc

/usr/lib/python3.6/site-packages/pymysql/constants/__pycache__/__init__.cpython-36.pyc

What am i missing?

Thanks in advance.

[Dave Olker]

I don't know what OS you're running, but have you checked whether the python*-mysql packages are installed?

On my RHEL 8.8 box:

# yum list python2-pymysql
Installed Packages
python2-PyMySQL.noarch 0.8.0-10.module+el8.1.0+3111+de3f2d8e @rhel-8-for-x86_64-appstream-rpms
Available Packages
python2-PyMySQL.noarch 0.8.0-10.module+el8.9.0+19487+7dc18407 rhel-8-for-x86_64-appstream-rpms

# yum list python3-pymysql
Installed Packages
python3-PyMySQL.noarch 0.10.1-2.module+el8.4.0+9657+a4b6a102 @rhel-8-for-x86_64-appstream-rpms

[whydo]

@Dave Olker 

Hi Dave,

We are running rhel 8.8 and yes we have it installed.

# rpm -qa | grep -y mysql

python2-PyMySQL-0.8.0-10.module+el8.1.0+3111+de3f2d8e.noarch

python3-PyMySQL-0.10.1-2.module+el8.4.0+9657+a4b6a102.noarch