Finding bottleneck in Python/Numpy

Callgraph of numpy array addtion See full image

For my GSoC project, I need to profile and find bottleneck in numpy code which is mostly written in c. The quick way is to see the callgraph and identify the path which consume most of the time. I am using Google profiling tool for profiling numpy's operations.

Setting up Gperftools

Following are the steps used to setup python c level profiler on Ubuntu 13.04. For any other system, options see [1]

1. Make sure to build it from source. Clone svn repository from http://gperftools.googlecode.com/svn/trunk/
2. In order to build gperftools checked out from subversion repository you need to have autoconf, automake and libtool installed.
3. First, run ./autogen.sh script which generate ./configure and other files. Then run ./configure
4. 'make check', to run any self-tests that come with the package. Check is optional but recommended to use
5. After all test gets passed, type 'sudo make install' to install the programs and any data files and documentation.

Running CPU profiler

Instead of writing profiler recording into code, I evoked profiler manually before running sample code. Consider following python code for profiling, in num.py file.

#!/usr/bin/python
import timeit
timeit.timeit('x+y',number=10000000,setup='import numpy as np;x = np.asarray(1.0);y = np.asarray(2.0);')

To start profiling use
$CPUPROFILE=num.py.prof LD_PRELOAD=/usr/lib/libprofiler.so python num.py

Alternatively, include profiler in code as follow

import ctypes
import timeit

profiler = ctypes.CDLL("libprofiler.so")
profiler.ProfilerStart("num.py.prof")
timeit.timeit('x+y',number=10000000,setup='import numpy as np;x = np.asarray(1.0);y = np.asarray(2.0);')
profiler.ProfilerStop()

To analysis stats use
$pprof --gv ./num.py num.py.prof 
Callgraph as shown in image above will appear. Each block represent method with local and cumulative percentage. 

Installing python with debug mode

When I used python available on Ubuntu repository, the callgraph was not created properly. There were lots of disconnected and isolated blocks. So I downloaded python2.7.5 source from python.org and then configure with --with-pydebug before make. For more information, please goto http://docs.python.org/devguide/

[1] http://gperftools.googlecode.com/svn/trunk/INSTALL