Running MISO on a cluster

I’m trying to get Yarden Katz’s program MISO running on the Harvard FAS computing cluster. It is not so easy to satisfy all the dependencies.

Some of the dependencies, including pygsl, scipy and numpy, are already available through add-on modules. I found that the problem is that they are not all the right versions and I get errors such as the following:
ValueError: can't handle version -1 of numpy.dtype pickle
ImportError: cannot import name defaultdict
ValueError: numpy.dtype does not appear to be the correct type object

I was able to solve these problems by installing my own copies of numpy and scipy of the exact same versions that Yarden has (I found these because I have access to the MIT cluster where he runs MISO). The numpy version is 1.4 (I used 1.4.1 and he has 1.4.0rc1, which I couldn’t find on the website) and the scipy version is 0.7.1. The weird thing about scipy is that that version is the one that was already installed on the Harvard cluster. However, I found that recompiling it solved the problem. This has been a big headache but hopefully worth it.

One more note is that the way to install your own version of a module to replace the system-installed version is to use the command python setup.py install --prefix=~. Here the tilde “~” just indicates my home directory, in which a lib/ subdirectory will be created and populated. You could put it somewhere else if you wanted. Then I add the following lines to the top of the main script (in my case this would be run_events_analysis.py):

import sys
sys.path.insert(0,"/n/home04/friedman/lib/python2.6/site-packages/")
import numpy
import scipy

Here the “/n/home04/friedman” is my home directory. You have to add “lib/python2.6/site-packages”.

In theory it should be good enough to export the environment variable PYTHONPATH. However, on our system this path, while respected, is further prepended with other directories containing the system-wide scipy and numpy modules, so my personal ones don’t get executed unlesss I cram them in the front within the script.

I still have a lot to learn about python.

0 comments ↓

There are no comments yet...Kick things off by filling out the form below.

You must log in to post a comment.