Compiling Django with Twitter support as a Mac OS X Universal Binary

Introduction

This post is a guide for building your own version of Apache’s mod_python as a Universal Binary in order to support a custom Django install containing the Twitter libraries. As you can probably gather, this information is likely only useful to advanced Mac users who are comfortable in Terminal with compiling and installing software from source. If you’re still interested, gird your loins, crack your knuckles, grab some Mountain Dew, and read on.

Mac OS X 10.5 “Leopard” is yet another step forward into the world of 64-bit. At the same time, Apple has to support both PowerPC and Intel architectures. This is no mean feat, and this is where “fat” or Universal binaries come in.  Apple also has an explanation of Universal binaries, although it’s heavy on PR. This is all well and good, but there is one problem: once you make this leap, all of your library dependencies must contain the architecture you’re running as. Much software is still built as 32-bit only; while it may be a “fat” binary, containing both Intel and PowerPC machine code, it only has the 32-bit versions thereof. For reference, the names of the various architecture flags:

  32-bit 64-bit
Intel i386 x86_64
PowerPC ppc7400 ppc64

Huzzah naming conventions! There’s a lot of history in those names. I’ve linked to the relevant Wikipedia articles if you’re curious; these flags will be coming up again later when configuring various builds. The main thing to note is that most build configurations default to i386 on Intel Macs (even though Core 2 and Xeon processors are natively 64-bit), probably because most software is developed for 32-bit versions of Windows and Linux. As you’ll see, we’ll be overriding that default in several places to get this whole mess working.

Unfortunately, Universality is a cancer, which in my case starts with the Apple-shipped version of the Apache web server in 10.5, a universal binary. Everything it touches needs to be Universal as well, so that Apache can run as a 64-bit process by default. I wanted to add Django support on my web server via mod_python, specifically to play with the Twitter API, which meant I also needed to build python-twitter and its dependencies, as well as a MySQL python module to allow Django to talk to my database. None of these are included in the default Leopard version of Python 2.5.1.

After getting all of this set up, and trying to start my test Django app, mod_python was giving me errors about architecture. As it turns out, the included version of Python is only a “fat” 32-bit binary, not a Universal binary… which means all of the new Python modules I just compiled to support Twitter and Django were only 32-bit, which in turn means that the included Universal version of Apache and mod_python couldn’t use them. Yay.

Below the cut you’ll find my complete instructions for compiling all of the relevant components and their dependencies. I also took the opportunity to update to the latest release version of Python 2.6 and MySQL 5.1, and as a side effect my database server is now running as a 64-bit process. Progress has been made here. Feel free to comment or contact me if you have questions.

Dependencies

Here’s a simple tree view of the package dependencies, followed by the order in which we’ll be compiling packages, with links to the relevant download sites. I’m assuming that all source tarballs have been expanded in /usr/local/src/, but obviously any prefix will do. I’m not listing any included packages; pretty much everything here depends on the custom build of the Python Framework. I also assume you have the Developer Tools isntalled from your Leopard DVDs.

  • mod_python
    • Python
    • Django
      • setuptools
      • MySQLdb
        • MySQL
    • python-twitter
      • simplejson

Click the package name to jump to the step in the instructions. Click the package version to download the source. (If you’re wondering why most Python packages are being built from source instead of eggs, it’s because there aren’t Python 2.6 eggs available for a lot of packages yet.)

  1. MySQL 5.1.30
  2. Python 2.6.1
  3. setuptools 0.6c9-py2.6
  4. MySQL-python 1.2.2
  5. Django 1.0.2-final
  6. simplejson 2.0.7
  7. python-twitter 0.5
  8. mod_python 3.3.1

Tips

In the sections below, you’ll see shell commands in <blockquote>s, one per line. If your browser and your shell cooperate, you should be able to copy-paste these without getting extraneous line breaks; just eyeball everything before you hit return. Triple-click to select the line should work.

If you’re unsure of whether or not you’re actually compiling as universal binaries, just use the file command to check what’s contained in a binary file. You should get five lines of output listing all four architectures, like this:

% file /usr/libexec/apache2/mod_python.so
/usr/libexec/apache2/mod_python.so: Mach-O universal binary with 4 architectures
/usr/libexec/apache2/mod_python.so (for architecture i386): Mach-O bundle i386
/usr/libexec/apache2/mod_python.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
/usr/libexec/apache2/mod_python.so (for architecture ppc): Mach-O bundle ppc
/usr/libexec/apache2/mod_python.so (for architecture ppc64): Mach-O 64-bit bundle ppc64

Compilation

MySQL

One huge caveat for this step: I compiled MySQL from source over a year ago, and I don’t remember exactly what I did. I just reused the same ./configure line that I had cached in an old config.log. The addition of multiple -arch flags initially caused a problem with running the preprocessor, but Marc Liyanage proved to once again be a great resource explaining the need for the --disable-dependency-tracking flag.

If you’re replacing an existing MySQL configuration, it wouldn’t hurt to have some sort of backup of your databases. I had no problems installing, and then killing mysqld. It restarted automagically as a 64-bit process.

cd /usr/local/src/mysql-5.1.30/

MACOSX_DEPLOYMENT_TARGET=10.5 CFLAGS='-arch i386 -arch x86_64 -arch ppc7400 -arch ppc64' LDFLAGS='-arch i386 -arch x86_64 -arch ppc7400 -arch ppc64' CXXFLAGS='-arch i386 -arch x86_64 -arch ppc7400 -arch ppc64' ./configure --prefix=/usr/local/mysql --with-extra-charsets=complex --enable-thread-safe-client --enable-local-infile --disable-shared --with-unix-socket-path=/usr/local/mysql/run/mysql_socket --with-mysqld-user=_mysql --disable-dependency-tracking

make all

make test

sudo make install

Note the four architecture flags set for all compiler and linker options; you’ll see those a lot throughout this post. I only allow local connections to MySQL, hence the socket path; you may use different depending on your configuration.

Python

While python.org provides a nice OS X Installer package for 2.6.1, it is compiled as a 32-bit “fat” binary, which doesn’t help us address the Universality problem. We’ll use their script for generating a package as well as installing the framework directly, after some minor changes, courtesy this mailing list post by Ned Deily.

If you’re feeling particularly trustworthy, you can download the disk image containing the Installer package that I built from my website. If not, you can make the changes yourself. Either way, Python.framework will be installed in /Library/Frameworks/.

In the Python 2.6.1 source directory, open the file Mac/BuildScript/build-installer.py in your favorite editor, and make the following changes on lines 1, 65, 68, 71, 633, and 1020:

1c1
< #!/usr/bin/python2.3
---
> #!/usr/bin/python
65c65
< DEPSRC = os.path.expanduser('~/Universal/other-sources')
---
> #DEPSRC = os.path.expanduser('~/Universal/other-sources')
68c68
< SDKPATH = "/Developer/SDKs/MacOSX10.4u.sdk"
---
> SDKPATH = "/Developer/SDKs/MacOSX10.5.sdk"
71c71
< ARCHLIST = ('i386', 'ppc',)
---
> ARCHLIST = ('i386', 'ppc', 'x86_64', 'ppc64')
633c633
< runCommand("%s -C --enable-framework --enable-universalsdk=%s LDFLAGS='-g -L%s/libraries/usr/local/lib' OPT='-g -O3 -I%s/libraries/usr/local/include' 2>&1"%(
---
> runCommand("%s -C --enable-framework --enable-universalsdk=%s --with-universal-archs=all LDFLAGS='-g -L%s/libraries/usr/local/lib' OPT='-g -O3 -I%s/libraries/usr/local/include' 2>&1"%(
1020c1020
< os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.3'
---
> os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.5'

You’ll note that some of the changes migrate this script to 10.5 (the hashbang path to Python, the deployment target and SDK version), and some of the changes add support for a Universal binary (adding to the architecture list. It’s possible that some of these changes will be fixed in a future source release of Python 2.6, but I’d guess that most efforts are focused on Python 3.0 at this point.

Once you’ve made the changes, just run the script. You need to be in the BuildScript subdirectory, unless you want to change some of the tmp paths the script uses. This script will do everything, build the framework, download and build any dependencies, install the framework, and generate a disk image. Just sit back and wait!

cd /usr/local/src/Python-2.6.1/Mac/BuildScript/

./build-installer.py

Depending on your shell configuration, you may need to make changes. In my $PATH /usr/local/bin/ trumps /usr/bin/, and I want to keep the Leopard system install of Python 2.5 in place in /System/Library/ and /usr/bin/. Here’s what I did:

cd /usr/local/bin/

for py in `find /Library/Frameworks/Python.framework/Versions/2.6/bin -name py*`; do ln -s $py; done;

This creates a symlink in /usr/local/bin/ for each of the various just-build Python binaries inside the framework (you’ll not that this is equivalent to the system install). If you run which python now, it should return /usr/local/bin/python. If not, adjust your $PATH as needed in your shell configuration file. This is beyond the scope of this post.

setuptools

This one is easy. Just execute the downloaded egg file as a shell script to add setuptools to your shiny new Python installation:

cd /usr/local/src/

sh setuptools-0.6c9-py2.6.egg

MySQL-python

MySQL-python installs using setuptools, as with most Python packages these days; however, we’ll need to make a few changes to get the C part of the extension to compile on OS X. I followed mickeyckm’s guide over at MangoOrange, which I’ve modified here to support all four architectures after looking at suggestions from Geert on the Python mailing list. Note that his manual compilation step isn’t necessary, from what I can tell.

First, you’ll need to modify site.cfg to point to the custom build of MySQL we created above; just uncomment and change the value of mysql_config on line 13 to match its actual install path as follows:

13c13
< #mysql_config = /usr/local/bin/mysql_config
---
> mysql_config = /usr/local/mysql/bin/mysql_config

The setup script for MySQL-python uses this tool to retrieve the compilation flags used with MySQL; since we built our own, this will automatically have all four architecture flags. You can run mysql_config directly and view the output if you don’t believe me.

Next, we need to make a small correction to _mysql.c in order to avoid a type conflict on OS X by deleting lines 37-39 and replacing uint with unsigned int on lines 484 and 485:

37,39d36
< #ifndef uint
< #define uint unsigned int
< #endif
484,485c481,482
< uint port = MYSQL_PORT;
< uint client_flag = 0;
---
> unsigned int port = MYSQL_PORT;
> unsigned int client_flag = 0;

Now we’re ready to build and install. Just run the standard setuptools procedure:

cd /usr/local/src/MySQL-python-1.2.2/

python setup.py build

sudo python setup.py install

Django

 

No special instructions here. Just run the standard setuptools procedure:

cd /usr/local/src/Django-1.0.2-final/

python setup.py build

sudo python setup.py install

simplejson

No special instructions here either. Just run the standard setuptools procedure:

cd /usr/local/src/simplejson-2.0.7/

python setup.py build

python setup.py test

sudo python setup.py install

python-twitter

Don’t you love it when how-tos get easier as you go along? Just run the standard setuptools procedure:

cd /usr/local/src/python-twitter-0.5/

python setup.py build

sudo python setup.py install

mod_python

Sadly, we end with some pain. Long story short, mod_python relies on apxs (the APache eXtenSion tool, best abbreviation ever) to configure its build, and does not obey any amount of environment variable manipulation to pass in multiple architectures. apxs retrieves useful compiler options, apparently based on the ones used to build your local copy apache… and herein lies the problem. apxs reads those flags from /usr/share/httpd/build/config_vars.mk, and if you inspect that file, you’ll see that there are no CFLAGS or LDFLAGS specified. This is particularly odd, considering that httpd is clearly running as a 64-bit process (according to Activity Monitor), and file reports that all of my other shared objects in /usr/libexec/apache2/ are Universal binaries.

This next bit is my idea, since I couldn’t find any suggestions by googling around. I know perl, so I just read the apxs script to figure out what it was doing, and how it was passing arguments to libtool.

This is probably a terrible idea.

Make a backup of config_vars.mk before adding the four architecture flags to CFLAGS and LDFLAGS on lines 59 and 62:

59c59
< CFLAGS = --- > CFLAGS = -arch i386 -arch x86_64 -arch ppc7400 -arch ppc64
62c62
< LDFLAGS = --- > LDFLAGS = -arch i386 -arch x86_64 -arch ppc7400 -arch ppc64

If you run apxs -q CFLAGS or apxs -q LDFLAGS, you’ll see these values, instead of empty strings.

Now just run the standard autoconf procedure to build and  install mod_python.so in /usr/libexec/apache2/:

cd /usr/local/src/mod_python-3.3.1/

./configure

make

cd test

python test.py

cd ..

make

sudo make install

Conclusion

At this point, you need to enable mod_python in your httpd.conf. If you passed the tests for it above, you shouldn’t need to worry about module issues, just double-check your configuration with sudo apachectl -t. When you’re ready to go, sudo apachectl restart, and welcome to the world of 64-bit.

If you have corrections to any of this, or need assistance, feel free to comment and I’ll try to get back to you with suggestions and/or edits.

Posted in How-Tos Tagged with: , , , , , , , ,

Nurd Up!

Nicolas Ward

Software engineer in Natural Language Processing research by day; gamer, reader, and aspiring UltraNurd by night. Husband to Andrle
Creative Commons License

Post History

January 2009
S M T W T F S
« Jun   Feb »
 123
45678910
11121314151617
18192021222324
25262728293031