Quickly expand your capacity: run Python script on Google Cloud
I am currently working on some simulation studies using agent-based models, which can quickly take a very long time to run. Rather than having my system slowed for hours/days, I wanted to find a way to quickly send code into the cloud. This seemed to be a very obvious use of cloud computing, but proved a little less straight-forward than expected. However, some existing solutions got me 80% there — but they still require a fair bit of manual tuning for every script, and as always, the last 20% ended up consuming a fair bit of time. That’s why I want to share a nearly out-of-the-box solution here.
Preview — or what pyscript2gce can do
Once the workflow is set up, it takes four steps to get your script to run on Google Cloud and to get your results:
- Paste your code into the template and add commands to save the outputs (and to send email updates, if you like)
- Push your code to GitHub (which can be one line if you set up a git lazy shortcut), then wait a moment for Docker image to get built
- Create the virtual machine that runs your code (with a simple command-line command), then wait for the email that tells you that it is done
- Download the output from Google Cloud Storage (again a one-line command)
How to get there
To get this workflow going, you need to fork the Github repo, sign-up to Google Cloud (if you haven’t done so; note that there are fairly generous free trials available), and then follow the step-by-step guide in the Github README.
NB: This is work-in-progress, and highly customisable. I hope it works for you - let me know if you have feedback.