Install `pycurl` on MacOS Big Sur 11.0.1

pycurl.md

Introduction

Installing pycurl on MacOS with Python 3.6+ and newer proved to be tricky, especially since a lot of the available resources seem to be outdated (1, 2, 3).

This gist was last updated at 2020-12-31.

Steps:

1. Remove previously installed Homebrew versions of curl:

    brew uninstall curl
    brew uninstall openssl
    brew uninstall curl-openssl

2. Install openssl and a curl version that uses openssl rather than SecureTransport, which MacOS curl uses by default. Note that Homebrew no longer allows options while installing, so it's not possible to do brew install curl --with-openssl. This was replaced with a different brew formula, curl-openssl, and again replaced on 2020-01-20 with the current version of just curl.:

    brew install openssl
    brew install curl  # this is the latest version with openssl

3. Now we need to use the new curl when installing pycurl. We can also export the path to the new curl to make it used by default:

    echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc  # or ~/.bash_profile
    echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc
    source ~/.zshrc
    source ~/.virtualenvs/foo_venv/bin/activate  # or whichever venv you're using
    export PYCURL_SSL_LIBRARY=openssl
    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"
    pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/openssl@1.1" pycurl

4. Test with:

    python
    >>> import pycurl

There should now be two different versions of curl:

1. The original OS X one in /usr/bin/curl:

    $ /usr/bin/curl --version
curl 7.64.1 (x86_64-apple-darwin20.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

2. The Homebrew one in /usr/local/opt/curl/bin/curl:

    $ /usr/local/opt/curl/bin/curl --version
curl 7.74.0 (x86_64-apple-darwin20.2.0) libcurl/7.74.0 (SecureTransport) OpenSSL/1.1.1i zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.42.0 librtmp/2.3
Release-Date: 2020-12-09
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

And our pycurl should use the second one, both at linker level and at compile level.

As of 2020-01-20, brew install openssl installs openssl@1.1. In the pip install command, the install-option for the openssl-dir should be updated to:
--install-option="--openssl-dir=/usr/local/opt/openssl@1.1"

I follow the example and got ld: library not found for -lcares, anyone knows how to add the missing cares?

I follow the example and got ld: library not found for -lcares, anyone knows how to add the missing cares?

found that it's called c-ares

in my case there still quite some missing library, I need to manually do
LDFLAGS="-L/usr/local/opt/curl-openssl/lib -L/usr/local/opt/c-ares/lib"

usually the missing dynamic library could be found under /usr/local/opt with a similar name

In catalina, it wants to install in /System/Library/Framework/ and fails

I got this working by slightly tweaking your step 3 to change openssl to openssl@1.1 as already mentioned above but I also needed to change curl-openssl to curl based on the directories that exist in my /usr/local/opt. You'll see on https://formulae.brew.sh/formula/curl that the curl-openssl formulea is now known as curl.

echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc  # or ~/.bash_profile
    echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc
    source ~/.zshrc
    source ~/.virtualenvs/pm-sem/bin/activate  # or whichever venv you're using
    export PYCURL_SSL_LIBRARY=openssl
    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"
    pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/openssl@1.1" pycurl

@idavidmcdonald @8bitter I updated the gist with your comments and confirm they work :) thanks for the effort!

@vidakDK I keep hitting the issue, even after following the instructions. Perhaps there has been an update to homebrew or BigSur?

For reference, my homebrew curl version

curl 7.75.0 (x86_64-apple-darwin20.2.0) libcurl/7.75.0 (SecureTransport) OpenSSL/1.1.1j zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2021-02-03
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

When trying to run my code, I still get this error

    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Has anyone hit this issue again?

EDIT: I am hitting this on Python 2, sorry if this comment is completely unrelated seeing as you have posted this gist for Python 3. I still think if anyone finds a fix to a similar problem then, this will help me.

I have recently upgrade my MacOS High serria to Catalina 10.15.7 and start facing this problem for virtual environment for pycurl. I did all your steps as it is to resolve my pycurl problem but am receving this error while executing this command
pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/openssl@1.1" pycurl

Error:
WARNING: Discarding https://files.pythonhosted.org/packages/fe/8f/711d436c76938093dd10dae465ccee316c6241cfdada06c5cc3b5c1f74b8/pycurl-7.18.2.tar.gz#sha256=cda1085b7911ca802746d816f37d541165ad1e20c241d22e4c304ddb2bc81ffe (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pycurl (from versions: 7.18.2, 7.19.0, 7.19.0.1, 7.19.0.2, 7.19.0.3, 7.19.3, 7.19.3.1, 7.19.5, 7.19.5.1, 7.19.5.2, 7.19.5.3, 7.21.5, 7.43.0, 7.43.0.1, 7.43.0.2, 7.43.0.3, 7.43.0.4, 7.43.0.5, 7.43.0.6, 7.44.0, 7.44.1)
ERROR: No matching distribution found for pycurl

I have a error when i import pycurl
pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.77.0)

I have the same error when I import pycurl

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

Same for me. Can't find solution.

Same:

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

I noticed:

    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"

are probably referencying my system curl.

brew list curl shows: usr/local/Cellar/curl/7.80.0/

So instead I do:

export LDFLAGS="-L/usr/local/Cellar/curl/7.80.0/lib"
export CPPFLAGS="-I/usr/local/Cellar/curl/7.80.0/include"

But fail:

(.venv)
🧢 pi@pPro18-4 ~/curlfail
> pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/openssl@1.1" pycurl
/Users/pi/curlfail/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py:229: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Collecting pycurl
  Downloading pycurl-7.44.1.tar.gz (227 kB)
     |████████████████████████████████| 227 kB 1.4 MB/s
Skipping wheel build for pycurl, due to binaries being disabled for it.
Installing collected packages: pycurl
    Running setup.py install for pycurl ... done
Successfully installed pycurl-7.44.1
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/pi/curlfail/.venv/bin/python -m pip install --upgrade pip' command.
(.venv)
🧢 pi@pPro18-4 ~/curlfail
> python
Python 3.10.0 (default, Oct 23 2021, 18:32:23) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

So I ran into new problems on my new laptop trying to get this to work. It might not be applicable for everyone else but thought I'd share just in case.

Context

pyenv --version

pyenv 2.0.7

brew list

...
openssl@1.1
openssl@3
...

curl --version

curl 7.80.0 ......

/usr/bin/curl --version                                                                                               
curl 7.64.1

where curl                                                                                                              

/usr/local/opt/curl/bin/curl
/usr/bin/curl

cat ~/.zshrc

...
export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"
export PATH="/usr/local/opt/curl/bin:$PATH"
...

I tried the following with a few different versions of python and a few different versions of pycurl

PYCURL_SSL_LIBRARY="openssl" \
LDFLAGS="-L/usr/local/opt/curl/lib" \
CPPFLAGS="-I/usr/local/opt/curl/include" \
pip install --ignore-installed pycurl==7.44.1 --global-option="--with-openssl" --global-option="--openssl-dir=/usr/local/opt/openssl@1.1"

but then always reached an error when doing

python -c "import pycurl; print(pycurl.version)"                                                                        

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

Thing that solved it for me

We spotted that on my team mates machines that worked they had:

gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

whereas I had

gcc --version                        [2:30:22]
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

By deleting the Xcode Apple app from my machine, this changed my gcc --version output to match and it fixed my problems.

@idavidmcdonald Thankyou for taking the trouble to research scientifically.
Presumably the macOS system holds both gcc versions.
I wonder if it is possible to force the pycurl installation to use the non-xcode gcc without deleting the XCode app...
What is the where command? I don't have it, it seems.

Answer has been provided to how to change InstalledDir here:
https://stackoverflow.com/questions/70119147/how-to-install-gcc-on-macos-such-that-gcc-version-reports-installeddir-li/70146246#70146246

sudo xcode-select -switch /Library/Developer/CommandLineTools

For this error:

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

2 Solutions:

1. Check your bash profile and see if you didn't mess up any python path.
2. Reinstall macOS (last Resort )
   I figured that I had messed up with the Python of my system, so I decided to erase my MacOS and reinstall it and it worked.
   It might not be the most optimum way but it fixed my issue.

This worked for me on a brand new M1 max MBP

brew install openssl curl
10127  pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

then you can test with

python -c "import pycurl"

None of the above solutions worked for me on Mac 12.1 and I was getting error ImportError: pycurl: libcurl link-time version (7.77.0) is older than compile-time version (7.81.0)

I managed to work around it building the exact curl binary I needed (7.77.0) from the libcurl repo and wiring that into pycurl using install_name_tool. Locations of these will depend on your system but these were my steps. I already had xcode cli tools installed.

Idea is:
install_name_tool -change old.dylib new.dylib pycurl.so

For example:
install_name_tool -change /usr/lib/libcurl.4.dylib /usr/local/opt/curl/lib/libcurl.4.dylib /Users/user/Library/Python/2.7/lib/python/site-packages/pycurl.so

Another example on a different mac with python 3.9 and pyenv I ran this:
install_name_tool -change /usr/local/Cellar/curl/7.81.0/lib/libcurl.4.dylib /opt/vagrant/embedded/lib/libcurl.4.dylib /Users/username/.local/share/virtualenvs/website-TeaFLnO_/lib/python3.9/site-packages/pycurl.cpython-39-darwin.so

1. brew uninstall curl (might not be necessary)
2. Use install_name_tool to link the dynamic library you need. Use find \ -name if to locate the dylib's if needed.
3. Build the exact version of curl you need. Run autoreconf -fi, ./configure --prefix=/usr/local/opt/curl --without-ssl, make
4. pip uninstall pycurl && pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

Sanity is restored and everything just works now. Hope this helps someone!

Faced the same problem with Mac 11.6.5 and python 3.10

~ > pip install pycurl
~ > python -c 'import pycurl'
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.77.0)

I didn't manage to change link-time version (7.64.1), so I download exact version of linked curl-7.64.1 and use it for pycurl compilation.

# Remove installed version
pip uninstall -y pycurl
brew uninstall curl

# Install openssl for pycurl compiling
brew install openssl || brew update openssl

# Downloading sources
wget https://curl.haxx.se/download/curl-7.64.1.tar.bz2
tar -xf curl-7.64.1.tar.bz2

# Setting up compiler flags and PATH
export PATH="$(pwd)/.cache/curl-${CURL_VERSION}/bin:$PATH"
export LDFLAGS="-L$(pwd)/curl-7.64.1/lib -L/usr/local/opt/openssl@3/lib"
export CPPFLAGS="-I$(pwd)/curl-7.64.1/include -I/usr/local/opt/openssl@3/include"

# Installing
pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

# Checking up 
python -c "import pycurl" && echo "Success!"

@eugen1j ,
This results to me in a:
ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Any suggestion?

@eugen1j , This results to me in a: ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Any suggestion?

@nffdiogosilva Seems like you have no ssl backend in link-time libcurl. Try to set export PYCURL_SSL_LIBRARY=openssl before running python -c "import pycurl" && echo "Success!"

Also, you can try to compile pycurl without --install-option="--with-openssl" flag

Thanks @eugen1j ! Based on your solution I was able to implement it for Mojave and Python2:

MacBook-Pro:v1.0 zollak$ source .env2.7/bin/activate
(.env2.7) MacBook-Pro:v1.0 zollak$ pip freeze
pycurl==7.43.0.5
(.env2.7) MacBook-Pro:v1.0 zollak$ python -c 'import pycurl'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.54.0) is older than compile-time version (7.64.1)

I didn't manage to change link-time version (7.54.0), so I download exact version of linked curl-7.54.0 and use it for pycurl compilation.

# Remove installed version
(.env2.7) MacBook-Pro:v1.0 zollak$ pip uninstall -y pycurl

# Downloading sources
(.env2.7) MacBook-Pro:v1.0 zollak$ mkdir curl
(.env2.7) MacBook-Pro:v1.0 zollak$ cd curl/
(.env2.7) MacBook-Pro:curl zollak$ wget https://curl.haxx.se/download/curl-7.54.0.tar.bz2
(.env2.7) MacBook-Pro:curl zollak$ tar -xf curl-7.54.0.tar.bz2

# Setting up compiler flags
(.env2.7) MacBook-Pro:curl zollak$ export LDFLAGS="-L$(pwd)/curl-7.54.0/lib -L/usr/local/opt/openssl@1.1/lib"
(.env2.7) MacBook-Pro:curl zollak$ export CPPFLAGS="-I$(pwd)/curl-7.54.0/include -I/usr/local/opt/openssl@1.1/include"

# Installing
(.env2.7) MacBook-Pro:curl zollak$ pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl
Successfully installed pycurl-7.43.0.5

# Checking up 
(.env2.7) MacBook-Pro:curl zollak$ python -c 'import pycurl' && echo "Success"
Success

I fell into this deep rabbit hole too many times, and finally this is probably the ultimate way of fixing this. Thanks @zollak. This works too for

• Python 3
• macOS Monterey 12.6.
• Python installed using Pyenv

--install-option is deprecated in the latest version of pip (e.g. pip 23.1.2).

Use --config-setting instead, e.g.

pip install --no-cache-dir --compile --ignore-installed  --config-setting="--build-option=--with-openssl" pycurl

Edit: Thanks to @jivens for pointing out, removed extra hyphen

The above command is absolutely wonderful except that it contains an extra hyphen, so use this:

pip install --no-cache-dir --compile --ignore-installed --config-setting="--build-option=--with-openssl" pycurl

Below command worked form me
MacOS: Apple M1 Pro (Ventura v13.4)

    pip uninstall pycurl
    PYCURL_SSL_LIBRARY=openssl LDFLAGS="-L/opt/homebrew/opt/openssl/lib" CPPFLAGS="-I/opt/homebrew/opt/openssl/include" pip install --no-cache-dir pycurl

Thanks @keshari-nandan , perfect job for me

@keshari-nandan Thanks