Introduction

These are the steps to follow to get up and running the dev environment for Bunjil Forest Watch on a Windows 8 PC. It covers eclipse, powershell, python, python libraries, app-engine API, and earth-engine API.

While it is written for a particular app (bunjilfw.appspot.com) most of it would apply to setup to develop any app-engine python app that is connecting to earth-engine.

Keys & Accounts

To create an app engine app, you will need to create an account at https://appengine.google.com

You will also need a key for Google Maps API.  This allows Google to assign traffic to maps to your application. You can create one yourself via the google maps site.

To talk to earth-engine you will need to have access to the earth-engine-api. This is currently only available to the trusted tester program as it is still changing rapidly. Contact earth-engine to apply for the trusted tester account. To request access try emailing the group: https://earthengine.google.org/#intro.

Once you are accepted, you will need to create a service account from the app engine console and send that the earth-engine to have it whitelisted.

Make sure your systems clock is synced with internet time (If not, this can cause issues with authentication.)

• Why Windows?

I am not advocating using Windows as a dev environment. You may be better of installing Virtual Box  and creating an Ubuntu VM. It would be a bit more overhead to get started but would probably pay of in the flexibility and larger ecosystem of linux developers. But it you want to use Windows, these are the steps. They would be the same for Windows 7 or 8.1

PowerShell

You can use Powershell that comes with Windows, or CMD or even the bash shell installed by git. Cygwin is another choice. I have been using the windows Powershell. I find it necessary to customise the shell fonts sizes and to include a command history and to set up the path and initial directory.

http://technet.microsoft.com/en-us/library/ee692764.aspx

I mainly use the shell for git commands.

Eclipse

Check Java 7 is installed. It should be already.

Install Eclipse. I used Kepler Standard Java version 64 bit. This is the current release of Eclipse, an older release will be fine too.

PyDev

Install PyDev into Eclipse. This makes Eclipse Python friendly. follow the instruction at pydev.org

Python.

IMPORTANT Use 32 bit version, even if you run a 64 bit machine. The 64 bit version has issues. I loaded 2.7.6 Windows x86 MSI Installer from python.org. Installed to C:Python27. Add C:Python27 to Windows $PATH in system environment.

Create PYTHONPATH environment variable My Computer > Properties > Advanced System Settings > Environment Variables > Then under system variables I create a new Variable called PythonPath.

In this variable I have C:Python27Lib;C:Python27DLLs;C:Python27Liblib-tk;C:PythonScripts

Also add C:PythonScripts to my windows Path. This is where I will store pip and virtualenv.

Run the following command to get a list of your Python path:

>python -c “import sys; print(sys.path)”

PIP

It is easiest to install PIP before other Python libraries as this is tool that makes it easy to install other packages. This is the easiest way to install PIP on windows:

1. goto: https://sites.google.com/site/pydatalog/python/pip-for-windows.
2. download pip-Win_1.6.exe
3. copy it to a sensible location like  c:Python27Scripts
4. add this path to your windows PATH environment.
5. Make sure you run the upgrade after installing as the site explains.
6. Test it.  For example >pip install yolk

There are other ways to get PIP described on stack exchange. It may even be included in Python2.7 distribution one day.

Yolk

Yolk is a useful tool to see what python libraries are installed and active. >pip install yolk >yolk -l After following these instructions to the end, your luibraries should look like this

PS C:Python27> yolk -l
earthengine-api - 0.1.9 - active
pip 1.5.2 has no metadata
pyDatalog - 0.14.0 - active
pyOpenSSL - 0.13.1 - active
pycrypto - 2.6 - active
setuptools - 2.1 - active
virtualenv 1.11.2 has no metadata
yolk - 0.4.3 - active

Google App Engine

Install the Google App Engine SDK Python  (first close Eclipse)

https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python

A good tutorial: Google App Engine Python Hello World Example Using Eclipse

GIT

Install GitHub for Windows from github (or just git ).

This will install a portable GIT somewhere.

I then added the line to include shell.ps1 file to the powershell profile. This puts git into my powershell path. Configure a local path for repos and email. (Resolve-Path “$env:LOCALAPPDATAGitHubshell.ps1”) I also had to muck around with the terminal fonts and sizes with powershell script in my profile. Add the path to PortableGIT to the powershell profile . (Resolve-Path “$env:LOCALAPPDATAGitHubshell.ps1”) Check that git and python commands execute from the shell.

Clone the repository

I used github for windows to clone intotecho/bunjilforestwatch to a local folder that the PyDEv project will import.

Create Eclipse Project

Create a PyDev Google App Engine project in Eclipse.

Then Click on the Project, select Import ->GIT->Project from GIT .

Either select the local GIT clone created above, or ask PyDev to clone it. I Use the local git repository.

At this stage, it should have loaded the source code files from the bunjil app but it may be necessary to import the following:

• The KEYS in settings.py – these are not in the public repository
• The oauth2client package
• The google app engine package
• PIL – image library (no sure if it is needed but stops a warning)
• earthengine-api package

Run Configurations Arguments

Set up a run configuration for running the project on a local dev server for debugging.

Eclipse->Run->Run Configurations.

Name launch-bunjilae It should be possible to load a shared launch file from the project in the Common tab, Save As select radio button for ‘Shared file‘. Set path to the env folder in the project bunjilaeenv It will then take the arguments from the file envlaunch-bunjilae.launch. This is shared in the repository.

Otherwise, set up locally with

• Project: bunjilae
• Main Module: C:Program Files (x86)Googlegoogle_appenginedev_appserver.py
• Working Directory: ${project_loc:/selected project name}
• Arguments: You may with to change the devserver arguments according to what you are doing. They can go on different lines in the run configuration.

"${project_loc:bunjilae}"
--skip_sdk_update_check
--port=8001
--logs_path=/bunjilsrc/logs/bunjilae.log
--log_level=debug
--clear_datastore=no

• Environment tab Click Select and add any needed environment variables from the OS.

Python Libraries

There are a number of libraries that the earth-engine requires that are not part of the code repository. So these must be installed separately.

After ‘installing’ these libraries, there is an instruction below on how to get the project to find them.

PyCrypto

PyCrpyto is needed to connect to earth engine with OAuth2. Here is a site of prebuilt binaries http://www.voidspace.org.uk/python/modules.shtml#pycrypto. This prebuilt binary hosted on a non-US site is a lot easier than installing Microsoft Visual C++ 2008 to build your own.

OpenSSL

PyOpenSSL is needed to connect to earth engine with OAuth2 https://pypi.python.org/pypi/pyOpenSSL/0.13.1 Run the installation package.

Oauth2client

Part of  this package is included in the app engine package, but I import it from the source. I found it here: https://pypi.python.org/pypi/oauth2client

PIL

Unfortunately, >pip install PIL doesn’t work  (does not find a distribution)

Fortunately, someone has built a windows installer here:

https://code.google.com/p/ocr-uvg/downloads/detail?name=PIL-1.1.7.win32-py2.7.exe

Download and run this.

Note: http://stackoverflow.com/questions/2485295/the-problem-with-installing-pil-using-virtualenv-or-buildout advises to use: Don't use pip install --no-index -f http://dist.plone.org/thirdparty/ -U PIL This also led to errors. Some people argue to use the Pillow fork instead of PIL. 

Importing Libraries

Installing the library just puts it into Python27Libsite-packages. It does not mean the project can use it.

I found it necessary to import the libraries into my project. But it should be possible to just install them into the environment.

Once you have created a project, you need the project to find the libraries.

From the eclipse project menu:

IMPORT->File System->

Select C:Program Files (x86)Googlegoogle_appenginelibgoogle-api-python-clientoauth2client –

Select All Files.

Into Folder: bunjilae

Tick create a top level folder.

This puts the oauth2client files into their own package folder called oauth2client, rather than moving the files into the bunjilae folder.

Repeat for PIL, PyCrypto and OpenSSL.

EARTH ENGINE API

With PIP and PIL fully installed, it  would be easy to install earth engine using pip install earthengine-api. First, you may need PIP: http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows I used: https://sites.google.com/site/pydatalog/python/pip-for-windows Then install earth engine API from code.google.com/p/earthengine-api. pip install earthengine-api does not work if PIL is not installed. And PIL is hard to install. I get this error

Downloading/unpacking earthengine-api
 Downloading earthengine-api-0.1.16.tar.gz
 Running setup.py (path:c:userscgoodmanappdatalocaltemppip_build_cgoodmanearthengine-apisetup.py) egg_info for package earthengine-api
WARNING: A Python library (PIL) used by the Earth Engine API mapclient
 module was not found. Information on PIL can be found at:
 http://pypi.python.org/pypi/PIL

Check the google page for updates. The earthengine-api can be downloaded manually without using pip,  and then imported into your project. This is described in the  Earth Engine API Installation Instructions

1. Download the zip file, and extract the contents to say bunjilsrcpackages.
2. This creates a new folder C:bunjilsrcpackagesearthengine-api-0.1.9
3. cd to this folder and type  > python setup.py install
4. this will install the package, but may have issues with dependencies like openssl

5. Searching for pyOpenSSL>=0.11
   Reading https://pypi.python.org/simple/pyOpenSSL/
   Download error on https://pypi.python.org/simple/pyOpenSSL/: timed out -- Some packages may not be found!

If this succeeds to create the api in your python packages, you can import it into the project.

 Import Earth Engine API (EE) into project

1. In eclipse, right click on the bunjilae project
2. Select Import
3. Select from File System
4. Navigate to the bunjilsrcpackagesearthengine-api-xyz folder
5. Import to bunjilae
6. Tick Create a top level folder.
7. Click Import.

Keys

IMPORT your settings.py file. This is not in the repository as the keys are private credentials used to authenticate the bunjil app engine app to the earth engine. settings.py can be copied from settings.py.dist as a template, then changed to your keys.

For Windows, it seemed to be necessary to use the PEM format – but maybe this was due to a bad install of openssl.

Converted the key downloaded from app engine console to PEM format with
> openssl pkcs12 -in xxxxxxxxxxxxxxx-privatekey.p12 -nodes -nocerts > privatekey.pem

This command was actually embeded in a comment in crypt.py. Change the xxxx to your key’s filename,

However, the code still thought I had a pk12, and not PEM.
The error was in crypt.py was here

I looked at the key file and it had a few lines before the ‘—-BEGIN’ so the startswith() failed.

I deleted these lines from the PEM file and used this key.