Introduction
These are the steps to follow to get up and running the dev environment for Bunjil Forest Watch on a Windows 8 PC. It covers eclipse, powershell, python, python libraries, app-engine API, and earth-engine API.
While it is written for a particular app (bunjilfw.appspot.com) most of it would apply to setup to develop any app-engine python app that is connecting to earth-engine.
Keys & Accounts
To create an app engine app, you will need to create an account at https://appengine.google.com
You will also need a key for Google Maps API. This allows Google to assign traffic to maps to your application. You can create one yourself via the google maps site.
To talk to earth-engine you will need to have access to the earth-engine-api. This is currently only available to the trusted tester program as it is still changing rapidly. Contact earth-engine to apply for the trusted tester account. To request access try emailing the group: https://earthengine.google.org/#intro.
Once you are accepted, you will need to create a service account from the app engine console and send that the earth-engine to have it whitelisted.
Make sure your systems clock is synced with internet time (If not, this can cause issues with authentication.)
- Why Windows?
I am not advocating using Windows as a dev environment. You may be better of installing Virtual Box and creating an Ubuntu VM. It would be a bit more overhead to get started but would probably pay of in the flexibility and larger ecosystem of linux developers. But it you want to use Windows, these are the steps. They would be the same for Windows 7 or 8.1
PowerShell
You can use Powershell that comes with Windows, or CMD or even the bash shell installed by git. Cygwin is another choice. I have been using the windows Powershell. I find it necessary to customise the shell fonts sizes and to include a command history and to set up the path and initial directory.
http://technet.microsoft.com/en-us/library/ee692764.aspx
I mainly use the shell for git commands.
Eclipse
Check Java 7 is installed. It should be already.
Install Eclipse. I used Kepler Standard Java version 64 bit. This is the current release of Eclipse, an older release will be fine too.
PyDev
Install PyDev into Eclipse. This makes Eclipse Python friendly. follow the instruction at pydev.org
Python.
IMPORTANT Use 32 bit version, even if you run a 64 bit machine. The 64 bit version has issues. I loaded 2.7.6 Windows x86 MSI Installer from python.org. Installed to C:Python27. Add C:Python27 to Windows $PATH in system environment.
Create PYTHONPATH environment variable My Computer > Properties > Advanced System Settings > Environment Variables >
Then under system variables I create a new Variable called PythonPath
.
In this variable I have C:Python27Lib;C:Python27DLLs;C:Python27Liblib-tk;C:PythonScripts
Also add C:PythonScripts to my windows Path. This is where I will store pip and virtualenv.
Run the following command to get a list of your Python path:
>python -c “import sys; print(sys.path)”
PIP
It is easiest to install PIP before other Python libraries as this is tool that makes it easy to install other packages. This is the easiest way to install PIP on windows:
- goto: https://sites.google.com/site/pydatalog/python/pip-for-windows.
- download pip-Win_1.6.exe
- copy it to a sensible location like c:Python27Scripts
- add this path to your windows PATH environment.
- Make sure you run the upgrade after installing as the site explains.
- Test it. For example >pip install yolk
There are other ways to get PIP described on stack exchange. It may even be included in Python2.7 distribution one day.
Yolk
Yolk is a useful tool to see what python libraries are installed and active. >pip install yolk >yolk -l After following these instructions to the end, your luibraries should look like this
PS C:Python27> yolk -l earthengine-api - 0.1.9 - active pip 1.5.2 has no metadata pyDatalog - 0.14.0 - active pyOpenSSL - 0.13.1 - active pycrypto - 2.6 - active setuptools - 2.1 - active virtualenv 1.11.2 has no metadata yolk - 0.4.3 - active
Google App Engine
Install the Google App Engine SDK Python (first close Eclipse)
https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python
A good tutorial: Google App Engine Python Hello World Example Using Eclipse
GIT
Install GitHub for Windows from github (or just git ).
This will install a portable GIT somewhere.
I then added the line to include shell.ps1 file to the powershell profile. This puts git into my powershell path. Configure a local path for repos and email. (Resolve-Path “$env:LOCALAPPDATAGitHubshell.ps1”) I also had to muck around with the terminal fonts and sizes with powershell script in my profile. Add the path to PortableGIT to the powershell profile . (Resolve-Path “$env:LOCALAPPDATAGitHubshell.ps1”) Check that git and python commands execute from the shell.
Clone the repository
I used github for windows to clone intotecho/bunjilforestwatch to a local folder that the PyDEv project will import.
Create Eclipse Project
Create a PyDev Google App Engine project in Eclipse.
Then Click on the Project, select Import ->GIT->Project from GIT .
Either select the local GIT clone created above, or ask PyDev to clone it. I Use the local git repository.
At this stage, it should have loaded the source code files from the bunjil app but it may be necessary to import the following:
- The KEYS in settings.py – these are not in the public repository
- The oauth2client package
- The google app engine package
- PIL – image library (no sure if it is needed but stops a warning)
- earthengine-api package
Run Configurations Arguments
Set up a run configuration for running the project on a local dev server for debugging.
Eclipse->Run->Run Configurations.
Name launch-bunjilae It should be possible to load a shared launch file from the project in the Common tab, Save As select radio button for ‘Shared file‘. Set path to the env folder in the project bunjilaeenv It will then take the arguments from the file envlaunch-bunjilae.launch. This is shared in the repository.
Otherwise, set up locally with
- Project: bunjilae
- Main Module: C:Program Files (x86)Googlegoogle_appenginedev_appserver.py
- Working Directory: ${project_loc:/selected project name}
- Arguments: You may with to change the devserver arguments according to what you are doing. They can go on different lines in the run configuration.
"${project_loc:bunjilae}" --skip_sdk_update_check --port=8001 --logs_path=/bunjilsrc/logs/bunjilae.log --log_level=debug --clear_datastore=no
- Environment tab Click Select and add any needed environment variables from the OS.
Python Libraries
There are a number of libraries that the earth-engine requires that are not part of the code repository. So these must be installed separately.
After ‘installing’ these libraries, there is an instruction below on how to get the project to find them.
PyCrypto
PyCrpyto is needed to connect to earth engine with OAuth2. Here is a site of prebuilt binaries http://www.voidspace.org.uk/python/modules.shtml#pycrypto. This prebuilt binary hosted on a non-US site is a lot easier than installing Microsoft Visual C++ 2008 to build your own.
OpenSSL
PyOpenSSL is needed to connect to earth engine with OAuth2 https://pypi.python.org/pypi/pyOpenSSL/0.13.1 Run the installation package.
Oauth2client
Part of this package is included in the app engine package, but I import it from the source. I found it here: https://pypi.python.org/pypi/oauth2client
PIL
Unfortunately, >pip install PIL doesn’t work (does not find a distribution)
Fortunately, someone has built a windows installer here:
https://code.google.com/p/ocr-uvg/downloads/detail?name=PIL-1.1.7.win32-py2.7.exe
Download and run this.
Note: http://stackoverflow.com/questions/2485295/the-problem-with-installing-pil-using-virtualenv-or-buildout advises to use: Don't use pip install --no-index -f http://dist.plone.org/thirdparty/ -U PIL This also led to errors.
Some people argue to use the Pillow fork instead of PIL.
Importing Libraries
Installing the library just puts it into Python27Libsite-packages. It does not mean the project can use it.
I found it necessary to import the libraries into my project. But it should be possible to just install them into the environment.
Once you have created a project, you need the project to find the libraries.
From the eclipse project menu:
IMPORT->File System->
Select C:Program Files (x86)Googlegoogle_appenginelibgoogle-api-python-clientoauth2client –
Select All Files.
Into Folder: bunjilae
Tick create a top level folder.
This puts the oauth2client files into their own package folder called oauth2client, rather than moving the files into the bunjilae folder.
Repeat for PIL, PyCrypto and OpenSSL.
EARTH ENGINE API
With PIP and PIL fully installed, it would be easy to install earth engine using pip install earthengine-api. First, you may need PIP: http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows I used: https://sites.google.com/site/pydatalog/python/pip-for-windows Then install earth engine API from code.google.com/p/earthengine-api. pip install earthengine-api does not work if PIL is not installed. And PIL is hard to install. I get this error
Downloading/unpacking earthengine-api Downloading earthengine-api-0.1.16.tar.gz Running setup.py (path:c:userscgoodmanappdatalocaltemppip_build_cgoodmanearthengine-apisetup.py) egg_info for package earthengine-api WARNING: A Python library (PIL) used by the Earth Engine API mapclient module was not found. Information on PIL can be found at: http://pypi.python.org/pypi/PIL
Check the google page for updates. The earthengine-api can be downloaded manually without using pip, and then imported into your project. This is described in the Earth Engine API Installation Instructions
- Download the zip file, and extract the contents to say bunjilsrcpackages.
- This creates a new folder C:bunjilsrcpackagesearthengine-api-0.1.9
- cd to this folder and type > python setup.py install
- this will install the package, but may have issues with dependencies like openssl
-
Searching for pyOpenSSL>=0.11 Reading https://pypi.python.org/simple/pyOpenSSL/ Download error on https://pypi.python.org/simple/pyOpenSSL/: timed out -- Some packages may not be found!
If this succeeds to create the api in your python packages, you can import it into the project.
Import Earth Engine API (EE) into project
- In eclipse, right click on the bunjilae project
- Select Import
- Select from File System
- Navigate to the bunjilsrcpackagesearthengine-api-xyz folder
- Import to bunjilae
- Tick Create a top level folder.
- Click Import.
Keys
IMPORT your settings.py file. This is not in the repository as the keys are private credentials used to authenticate the bunjil app engine app to the earth engine. settings.py can be copied from settings.py.dist as a template, then changed to your keys.
For Windows, it seemed to be necessary to use the PEM format – but maybe this was due to a bad install of openssl.
Converted the key downloaded from app engine console to PEM format with
> openssl pkcs12 -in xxxxxxxxxxxxxxx-privatekey.p12 -nodes -nocerts > privatekey.pem
This command was actually embeded in a comment in crypt.py. Change the xxxx to your key’s filename,
However, the code still thought I had a pk12, and not PEM.
The error was in crypt.py was here
pkey = RSA.importKey(key)
else:
raise NotImplementedError(
‘PKCS12 format is not supported by the PyCrpto library. ‘
‘Try converting to a “PEM” ‘
‘(openssl pkcs12 -in xxxxx.p12 -nodes -nocerts > privatekey.pem) ‘
‘or using PyOpenSSL if native code is an option.’)
I looked at the key file and it had a few lines before the ‘—-BEGIN’ so the startswith() failed.
friendlyName: privatekey
localKeyID: xx 69 6D 65 20 31 33 36 xx xx 37 xx 39 xx 35 xx 30 xx
Key Attributes: <No Attributes>
—–BEGIN PRIVATE KEY—–
MIICdgIB…..
I deleted these lines from the PEM file and used this key.