Search Mailing List Archives


Limit search to: Subject & Body Subject Author
Sort by: Reverse Sort
Limit to: All This Week Last Week This Month Last Month
Select Date Range     through    

[farmshare-discuss] Adding modules to Python?

Jason Bishop bishopj at stanford.edu
Wed Mar 13 12:32:24 PDT 2013



Hi Jeremy, I can think of 3 options off the top of my head.  If anybody has a clever idea to improve please let me know.

first, a little bit of background: Since ubuntu provides a good up-to-date python environment, this is the main python environment we provide on farmshare.  (This is not written in stone however, I have been considering looking to see if we can get any more speed out of numpy/scipy using ACML).


As such, python libraries can be managed in 3 ways:

1) The ubuntu provided package can be installed.  This is easy, just send us an email research-computing-support at stanford.edu

There are quite a few packages, so there is a good chance your package is there:

$ aptitude search python | wc -l
3992




2) If the ubuntu package does not exist or is the wrong version, you can install it in your home directory.

For example, there is no ubuntu package for globusonline-transfer-api-client, so you can use pip to install it with --user flag to have it installed in your home directory (in .local)


$ pip install --user globusonline-transfer-api-client
Downloading/unpacking globusonline-transfer-api-client
  Downloading globusonline-transfer-api-client-0.10.13.tar.gz
  Running setup.py egg_info for package globusonline-transfer-api-client
    
Installing collected packages: globusonline-transfer-api-client
  Running setup.py install for globusonline-transfer-api-client
    
Successfully installed globusonline-transfer-api-client
Cleaning up...

check to see if it is working (it is):
$ python -c "from globusonline.transfer import api_client; print api_client.__path__"
['/afs/ir/users/b/i/bishopj/.local/lib/python2.7/site-packages/globusonline/transfer/api_client']

Taking a look in .local, I can see all of the packages installed with easy_install or pip:

$ ls .local/lib/python2.7/site-packages/
boto		     globusonline					grapher.pyc		 lib
boto-2.8.0.egg-info  globusonline_transfer_api_client-0.10.13.egg-info	ioapps.so		 tai64n-0.3.3-py2.7.egg
easy-install.pth     grapher.py						ioprofiler-1.0.egg-info




3) If you have alot of packages which you want to keep as a self-contained group you can use virtualenv.  This sounds like the situation you describe where you have python eggs and scripts from the paper.  virtualenv also provides a nice way to collect and use the exact versions of python packages you want to use. 

Virtualenv sets up a new python area with its own packages, but shares all of the python interpreter and library environment from the original.

For example, we will clean the home directory install of the globusonline api we just did:

$ rm -r /afs/ir.stanford.edu/users/b/i/bishopj/.local/lib/python2.7/site-packages/globusonline*


And then setup a virtualenv environment called mysandbox

$ virtualenv --system-site-packages mysandbox
New python executable in mysandbox/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.


Then set your path to point to this environment so that python will execute from your mysandbox environment:

$ export PATH=$PWD/mysandbox/bin:$PATH

$ which python
/mnt/glusterfs/bishopj/vee/mysandbox/bin/python


Now pip install will work fine, installing everything into mysandbox:

$ pip install globusonline-transfer-api-client
Downloading/unpacking globusonline-transfer-api-client
  Downloading globusonline-transfer-api-client-0.10.13.tar.gz
  Running setup.py egg_info for package globusonline-transfer-api-client
    
Installing collected packages: globusonline-transfer-api-client
  Running setup.py install for globusonline-transfer-api-client
    
Successfully installed globusonline-transfer-api-client
Cleaning up...

$ python -c "from globusonline.transfer import api_client; print api_client.__path__"
['/mnt/glusterfs/bishopj/vee/mysandbox/local/lib/python2.7/site-packages/globusonline/transfer/api_client']



You can install eggs too:

$ easy_install DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg 
Processing DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg
creating /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg
Extracting DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg to /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages
Adding DDN-SFA-API 1.5.3.0.g-r12955 to easy-install.pth file

Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg
Processing dependencies for DDN-SFA-API==1.5.3.0.g-r12955
Searching for python-ntlm
Reading http://pypi.python.org/simple/python-ntlm/
Reading http://code.google.com/p/python-ntlm
Best match: python-ntlm 1.0.1
Downloading http://pypi.python.org/packages/source/p/python-ntlm/python-ntlm-1.0.1.tar.gz#md5=3b7ccdb2fd601596fea0f92d06f4fdaf
Processing python-ntlm-1.0.1.tar.gz
Running python-ntlm-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-V6gue6/python-ntlm-1.0.1/egg-dist-tmp-Q8g0mm
Adding python-ntlm 1.0.1 to easy-install.pth file
Installing ntlm_example_extended script to /mnt/glusterfs/bishopj/vee/mysandbox/bin
Installing ntlm_example_simple script to /mnt/glusterfs/bishopj/vee/mysandbox/bin

Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/python_ntlm-1.0.1-py2.7.egg
Searching for zope.interface>=3.8.0
Reading http://pypi.python.org/simple/zope.interface/
Best match: zope.interface 4.0.5
Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.5.zip#md5=caf26025ae1b02da124a58340e423dfe
Processing zope.interface-4.0.5.zip
Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-zVwcnX/zope.interface-4.0.5/egg-dist-tmp-r8gb1L
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Adding zope.interface 4.0.5 to easy-install.pth file

Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/zope.interface-4.0.5-py2.7-linux-x86_64.egg
Finished processing dependencies for DDN-SFA-API==1.5.3.0.g-r12955






----- Original Message -----
From: "Jeremy Hsu" <jlhsu at stanford.edu>
To: farmshare-discuss at lists.stanford.edu
Sent: Wednesday, March 13, 2013 12:03:23 AM
Subject: [farmshare-discuss] Adding modules to Python?



Hi all, 

I’m attempting to run some scripts from a paper, and am running into some issues. In short, after running the python scripts on corn, I keep getting “ File "<stdin>", line 1, in <module>, ImportError: no module named ____”. I managed to resolve most of these errors by temporarily changing my PYTHONPATH environment variable, but unfortunately the scripts look like they call some additional Python modules that aren’t contained within the scripts folder. I attempted adding the module but ran into permissions error, and it looks like changes to Python installed on the global bin are restricted to admins. 



Is there a way to have a module added to corn/barley’s Python, or is there another way I can include that module for these scripts? (Alternatively, I’ve thought about running a local version of Python and using that, instead of the global version, but that seems unnecessarily redundant.) 



Thanks! 



Jeremy 
_______________________________________________
farmshare-discuss mailing list
farmshare-discuss at lists.stanford.edu
https://mailman.stanford.edu/mailman/listinfo/farmshare-discuss



More information about the farmshare-discuss mailing list