Python at BGS // by Chris Williams, John Stevenson et al

This week BGS has been celebrating all things digital, as part of our Digital Week campaign. We've showcased the work of our BGS Digital team in GIS, programming, big data, data availability and digital models. Now, Chris Williams and John Stevenson - with contributions from Kat Daniels, Ekbal Hussain and Andy Hulbert - have taken the time to explain how we, as BGS, use Python...

There are so many different programming languages out there and each has its own pros, cons and associated learning curve. Here at the British Geological Survey, one language that is being increasingly used by a number of us for a variety of tasks is Python. This reflects the global increase in the use of Python as a language, which you can read about in this article from stackoverflow.

Python's growth relative to other languages based on Stack Overflow question views

 Ranging from those who are old hands at Python through to those who are relatively new, the BGS Python community is active, collaborative and growing! So let's have a quick run through why we are using Python, how we are applying it and how the BGS user community has developed over the past year.

Why Python?

• Python is extremely readable which makes it easy to share code with limited prior understanding – this means you can get the basics down fast, improving your productivity straight away whilst continuing to learn and advance
• Numerous packages exist providing tools that can help with a range of computational tasks, whatever your application e.g. NumPy, pandas

• Python is used in a range of areas of BGS work e.g. data analysis, making plots, machine learning, GIS, process automation, API development, database input/output – ultimately, those with a Python skillset could contribute to any of these areas

Current applications

There are a number of existing projects at BGS already being developed around a Python workflow, some of which have developed following these early Python user group meetings. Examples of existing projects include:

• The creation of the BGS_ETL library for easy access to Oracle and other databases, used for API development and data transfer
• Twitter data harvesting as part of the GeoSocial project
• Terrain analysis modelling including the handling of input data uncertainty
•  The development of a range of BGS products through ArcGIS including components of GeoSure
• Creating workflows for the handling of satellite time series data

(a) shows a satellite data-derived map of the average ground motions around Bandung city in Indonesia. Each of the pixels in this map has an associated time series, one of which is shown in (b)

(b) the blue bar on the time series shows how a Python developed automated change detector accurately determines the timing of the gradient change in the ground motion

• Implementing parallelisation of processing work flows over the BGS's High Performance Computer
• Volcanic ash dispersion model output plotting
• The development of an automated workflow for handling and plotting laboratory data focus on water movement through clay samples

As part of the workflow used to assist with laboratory experiments analysing the swelling pressure and permeability of a clay, data is taken from a custom-built constant volume steel pressure vessel containing the clay (a). Observations e.g. the absolute mechanical pressure over time (b) are then made available within a Python environment for further analysis

• Interrogation of meteorological data for input into assessing potential changes from climate-geological interactions

Seasonal rainfall records for various regions across Great Britain

The BGS Python user community 

To bring all Python users together at BGS, we have started a number of Python working groups meeting roughly every two weeks at both our Edinburgh and Keyworth sites. Meetings are open to anyone working at BGS and the content varies from week to week.

The aim of these clinics is to:

• Help new users install Python and configure their laptops to use it
• Expose scientists to software development methods and tools to will help them write better code
• Offer an environment for developers and analysts to share existing and in-development workflows, including problems
• Make everyone aware of who's who at BGS when it comes to Python programming, ultimately spurring on potential collaboration 

The meetings that have taken place so far have also helped us identify people’s varying skills and potential training requirements. As a result, we will soon be running general Beginner Python training courses both at both Edinburgh and Keyworth to introduce those new to Python (or even coding in general) for the first time and to provide a grounding in the core concepts.

For more information on Python, or our digital or data work, please visit the BGS Data website.