{"id":401,"date":"2023-04-20T17:23:52","date_gmt":"2023-04-20T17:23:52","guid":{"rendered":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/?p=401"},"modified":"2024-05-22T08:00:08","modified_gmt":"2024-05-22T08:00:08","slug":"python-development-on-m1-macs","status":"publish","type":"post","link":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/2023\/04\/20\/python-development-on-m1-macs\/","title":{"rendered":"Python Development on M1 Macs"},"content":{"rendered":"

Apple Silicon Python Setup<\/h1>\n

I recently received a new MacBook Pro for work. Great! The only catch is that Apple discontinued their Intel-based line of MacBook Pros in 2021<\/a> and their new line of laptops use the Apple silicon M-series coprocessors<\/a>. This might not seem like a problem at first, but the Apple silicon processors use ARM-architecture<\/a> instead of Intel’s x86 architecture<\/a>. For Python development, this is a potential problem because not all Python packages are installable for ARM architectures.<\/p>\n

<\/p>\n

There is a possibly simple solution, in May 2022 Anaconda released a\u00a0distribution supporting Apple silicon<\/a>. However, a few years ago I chose to move away from using Anaconda to manage my Python virtual environments in favour of a more lightweight solution that I have more control over.<\/p>\n

For years I have used the inbuilt venv module<\/a> for creating virtual environments in combination with pyenv<\/a> for installing multiple differnet versions of Python itself. With some help from this blog post<\/a> I was able to combine pyenv<\/code> with Rosetta2<\/a> (Apple’s x86 emulation tool that allows x86-compiled software to run on your Mac) to create a smooth setup to switch between different architecture versions of Python.<\/p>\n

The steps to set this up are below.<\/p>\n

Installing pyenv<\/h2>\n

Default ARM64 Installation<\/h3>\n

This section provides instructions on how to install Python using the default ARM64 architecture and pyenv<\/code>. This will cover most of the use cases for developing in Python on macOS. Hopefully, in a few years this will be all that is required.<\/p>\n

The best way to install pyenv<\/code>\u00a0is through Homebrew<\/a>. This is recommended by pyenv<\/code><\/a> and takes care of a lot of the additional shell environment setup<\/a>. Installed with:<\/p>\n

$ \/bin\/bash -c \"$(curl -fsSL https:\/\/raw.githubusercontent.com\/Homebrew\/install\/HEAD\/install.sh)\"<\/code><\/pre>\n
This defaults to install it in \/opt\/homebrew<\/code>. If you are prompted to install Xcode Commandline tools, say yes. You can now install many tools using the brew<\/code> command from Homebrew.<\/p>\n
Setup the recommended build environment<\/a> for pyenv<\/code>:<\/p>\n
$ brew install openssl readline sqlite3 xz zlib tcl-tk<\/code><\/pre>\n
Then install pyenv<\/code>:<\/p>\n
$ brew install pyenv<\/code><\/pre>\n
And finally, run to following to initialise your shell for pyenv<\/code> every time you open it:<\/p>\n
$ echo 'eval \"$(pyenv init -)\"' >> ~\/.zshrc<\/code><\/pre>\n
Restart your terminal to enable the new .zshrc<\/code> configuration.<\/p>\n
Note:\u00a0<\/strong>zsh has been the default login shell macOS since before the M1 chip was released, so I am assuming you are using zsh. If not, replace .zshrc<\/code> with .bashrc<\/code> whenever I mention it. Just be aware of potential differences in syntax<\/a>.<\/em><\/p>\n
You can then install a desired version of python (see list of possible versions with pyenv install --list<\/code>), and set it as the global (default) python version. I am installing 3.10.9, but you can install whichever version you want:<\/p>\n
\r\n$ pyenv install 3.10.9\r\n$ pyenv global 3.10.9\r\n<\/code><\/pre>\n
You can verify that your python version is the one you just set and the platform architecture is using the inbuilt ARM architecture with:<\/p>\n
\r\n$ python --version\r\nPython 3.10.9\r\n$ python -c \"import platform; print(platform.machine())\"\r\narm64\r\n<\/code><\/pre>\n
And you’re done! If all the packages you use are installable with arm64 architecture you can stop here, but if you run into this kind of error, read on…<\/p>\n
\r\n$ pip install <package>\r\nERROR: Could not find a version that satisfies the requirement <package>\r\nERROR: No matching distribution found for <package>\r\n<\/code><\/pre>\n
x86 Installation with Rosetta2<\/h3>\n
This section provides instructions on how to install Python using the x86 architecture emulated by Rosetta2.<\/p>\n
Commands can be run under emulated x86 architecture using Rosetta 2. Install it with:<\/p>\n
\r\n$ softwareupdate --install-rosetta\r\n<\/code><\/pre>\n
You can run shell commands under x86 using the arch -x86_64<\/code> command:<\/p>\n
\r\n$ uname -m\r\narm64\r\n$ arch -x86_64 uname -m\r\nx86_64\r\n<\/code><\/pre>\n
I will need to install an x86 version of Homebrew and pyenv<\/code>. I had this permissions issue<\/a> when installing this, so first fix this with (need sudo\/admin permissions):<\/p>\n
\r\n$ sudo chown -R $(whoami) \/usr\/local\/share\/zsh \/usr\/local\/share\/zsh\/site-functions\r\n<\/code><\/pre>\n
Then install the x86 version of Homebrew. It is the same install command as above, but by using the Rosetta2 emulation Homebrew knows to install the x86 version:<\/p>\n
\r\n$ arch -x86_64 \/bin\/bash -c \"$(curl -fsSL https:\/\/raw.githubusercontent.com\/Homebrew\/install\/HEAD\/install.sh)\"\r\n<\/code><\/pre>\n
This will install brew in \/usr\/local\/bin\/brew<\/code>, for which you will need to configure your shell environment. Notice how the location of the brew<\/code> executable changes from the default arm64 location, to the new x86 location:<\/p>\n
\r\n$ which brew\r\n\/opt\/homebrew\/bin\/brew\r\n$ eval \"$(arch --x86_64 \/usr\/local\/bin\/brew shellenv)\"\r\n$ which brew\r\n\/usr\/local\/bin\/brew\r\n<\/code><\/pre>\n
Note:<\/strong>\u00a0The command eval \"$(arch --x86_64 \/usr\/local\/bin\/brew shellenv)\"<\/code> will need to be run every time you need to use the x86 version of Homebrew to install. More on this later.<\/em><\/p>\n
Now, you can install pyenv<\/code> using the x86 version of Homebrew, don’t forget to still set up the recommended build environment<\/a> for pyenv<\/code>:<\/p>\n
\r\n$ arch --x86_64 brew install openssl readline sqlite3 xz zlib tcl-tk\r\n$ arch --x86_64 brew install pyenv\r\n<\/code><\/pre>\n
You now have all the tools to install multiple versions of Python in different architectures! However there are some teething problems and clunky user-experience that still needs to be set up before we do so.<\/p>\n
Set up a smooth user experience<\/h2>\n
Easily switch architectures<\/h3>\n
Set up some useful aliases by adding the following to your .zshrc<\/code>:<\/p>\n
\r\n# Alias for switching terminal architecture\r\nalias x86='arch -x86_64 zsh --login'\r\nalias arm64='arch -arm64 zsh --login'\r\n\r\n# Edit prompt name so you know the current architecture\r\nPROMPT=\"$(uname -m) $PROMPT\"\r\n<\/code><\/pre>\n
Restart your shell, you should now see arm64<\/code> at the front of your prompt, and it will change if you use the newly-created x86<\/code> alias to switch architecture:<\/p>\n
\r\narm64 $ uname -m\r\narm64\r\narm64 $ x86\r\nx86_64 $ uname -m\r\nx86_64\r\n<\/code><\/pre>\n
You can return to the ARM64 shell with exit<\/code> or with the arm64<\/code> alias.<\/p>\n
Python version name suffix<\/h3>\n
One limitation of using pyenv<\/code> to install different architecture versions of python is that it doesn’t allow you to give your different python versions custom names. So if I want to now install an x86 version of Python 3.10.9 (the same as above), I cannot do so without overwriting the ARM64 installation:<\/p>\n
\r\narm64 $ arch -x86_64 pyenv install 3.10.9\r\npyenv: \/Users\/adalessa\/.pyenv\/versions\/3.10.9 already exists\r\ncontinue with installation? (y\/N) N\r\n<\/code><\/pre>\n
To solve this issue, I adapted the broken pyenv-alias<\/code><\/a> plugin to create pyenv-suffix<\/code><\/a>, which allows you specify a suffix to append to the version name of a pyenv<\/code>-installed version of Python.<\/p>\n
Install it with:<\/p>\n
\r\n$ git clone https:\/\/github.com\/AdrianDAlessandro\/pyenv-suffix.git $(pyenv root)\/plugins\/pyenv-suffix\r\n<\/code><\/pre>\n
Now, if you set the PYENV_VERSION_SUFFIX<\/code> environment variable, it will be appended to any version you install with pyenv<\/code>.<\/p>\n
Shell configuration<\/h3>\n
The last thing that needs to be fixed is ensuring that the shell is configured correctly for the architecture you are using. Add the following to your .zshrc<\/code> and open a new terminal:<\/p>\n
\r\n# Alias for x86 version of Homebrew and pyenv\r\nif [ $(uname -m) = \"x86_64\" ]; then\r\neval \"$(arch --x86_64 \/usr\/local\/bin\/brew shellenv)\"\r\nalias pyenv='PYENV_VERSION_SUFFIX=\"x86\" \/usr\/local\/bin\/pyenv'\r\nfi\r\n<\/code><\/pre>\n
This will mean everytime you open a new x86 terminal, the Homebrew shell environment will be configured, and the PYENV_VERSION_SUFFIX<\/code> envirtonment variable will be set. This should mean seemless installation of anything with Homebrew and pyenv<\/code>.<\/p>\n
Install an x86 version of Python!<\/h3>\n
This is as simple as entering an x86 shell with the x86<\/code> alias and installing the desired Python version. First, check what versions of Python you have installed, it should look something like this:<\/p>\n
\r\narm64 $ pyenv versions\r\nsystem\r\n* 3.10.9 (set by \/Users\/<username>\/.pyenv\/version)\r\n<\/code><\/pre>\n
Then, enter the x86 shell and install the desired version:<\/p>\n
\r\narm64 $ x86\r\nx86_64 $ pyenv install 3.10.9\r\npyenv: \/Users\/adalessa\/.pyenv\/versions\/3.10.9 already exists\r\ncontinue with installation? (y\/N) y\r\nInstalling at \/Users\/adalessa\/.pyenv\/versions\/3.10.9x86\r\npython-build: use openssl from homebrew\r\npython-build: use readline from homebrew\r\nDownloading Python-3.10.9.tar.xz...\r\n-> https:\/\/www.python.org\/ftp\/python\/3.10.9\/Python-3.10.9.tar.xz\r\nInstalling Python-3.10.9...\r\npython-build: use readline from homebrew\r\npython-build: use zlib from xcode sdk\r\nInstalled Python-3.10.9 to \/Users\/adalessa\/.pyenv\/versions\/3.10.9x86\r\n<\/code><\/pre>\n
Now you should see the new version installed with the “x86” suffix appended (note I have exited the x86 shell, but it doesn’t matter):<\/p>\n
\r\narm64 $ pyenv versions\r\nsystem\r\n* 3.10.9 (set by \/Users\/<username>\/.pyenv\/version)\r\n3.10.9x86\r\n<\/code><\/pre>\n
How do you use it? There are three ways described in the pyenv<\/code> GitHub Page<\/a>. The simplest of which is the pyenv shell<\/code> command:<\/p>\n
\r\narm64 $ python -c \"import platform; print(platform.machine())\"\r\narm64\r\narm64 $ pyenv shell 3.10.9x86\r\narm64 $ python -c \"import platform; print(platform.machine())\"\r\nx86_64\r\n<\/code><\/pre>\n
Although my preferred method is to set the local version of python on a per-project basis. This is so that the python version automatically changes when you enter a project directory, and also because using pyenv shell<\/code> persists until you restart your shell.<\/p>\n
Open a new terminal. Enter your project directory and set the local version:<\/p>\n
\r\narm64 $ cd my_project\r\narm64 $ python -c \"import platform; print(platform.machine())\"\r\narm64\r\narm64 $ pyenv local 3.10.9x86\r\narm64 $ python -c \"import platform; print(platform.machine())\"\r\nx86_64\r\n<\/code><\/pre>\n
You will now see a .python-version<\/code> file added to your directory and that version of Python will be used whenever you are in that directory!<\/p>\n","protected":false},"excerpt":{"rendered":"
Apple Silicon Python Setup I recently received a new MacBook Pro for work. Great! The only catch is that Apple discontinued their Intel-based line of MacBook Pros in 2021 and their new line of laptops use the Apple silicon M-series coprocessors. This might not seem like a problem at first, but the Apple silicon processors […]<\/p>\n","protected":false},"author":1737,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[317604,317607,317608,317606,317605,268274,317609],"class_list":["post-401","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-apple","tag-arm","tag-development","tag-intel","tag-macos","tag-python","tag-software"],"_links":{"self":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/posts\/401","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/users\/1737"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/comments?post=401"}],"version-history":[{"count":30,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/posts\/401\/revisions"}],"predecessor-version":[{"id":440,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/posts\/401\/revisions\/440"}],"wp:attachment":[{"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/media?parent=401"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/categories?post=401"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs-staging.imperial.ac.uk\/research-software-engineering\/wp-json\/wp\/v2\/tags?post=401"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}