Cosmos-UK Soil Moisture (UKCEH)#

Agriculture Exploration Standard Python

license render review

binder binder

RoHub doi

Context#

Purpose#

To load and visualise daily hydrometeorological and soil data from the 2013-2019 public COSMOS-UK dataset.

Sensor description#

Since 2013 the UK Centre for Ecology & Hydrology (UKCEH) has established the world’s most spatially dense national network of cosmic-ray neutron sensors (CRNSs) to monitor soil moisture across the UK. The Cosmic-ray Soil Moisture Observing System for the UK (COSMOS-UK) delivers field-scale soil water volumetric content (VWC) measurements for around 50 sites in near-real time. In addition to measuring field-scale (or local) soil moisture, the network collects a large number of hydrometeorological and soil data variables, including VWC measured by point-scale (or site) soil moisture sensors.

This notebook explores a subset of 4 out of 51 stations available in the public COSMOS-UK dataset. These stations represent the first sites to prototype COSMOS sensors in the UK, see further details in Evans et al. (2016) and they are situated in human-intervened areas (grassland and cropland), except for one in a woodland land cover site.

The media below, available in the UKCEH YouTube channel, summarises the concept of cosmic-ray neutron sensors and how they provide non-invasive soil moisture measurments at field scale.

Highlights#

  • Fetch COSMOS-UK dataset files through intake.

  • Inspect the available metadata with information about the sites, their locations and other site-specific attributes.

  • Explore relationships between daily mean soil moisture and potential evapotranspiration derived from the meteorological measurements at the site.

  • Analyse yearly change of daily mean soil moisture observations.

  • Compare local and site soil moisture measurements at daily resolution.

Contributions#

Notebook#

  • Alejandro Coca-Castro (author), The Alan Turing Institute, @acocac

  • Doran Khamis (reviewer), UK Centre for Ecology & Hydrology, @dorankhamis

  • Matt Fry (reviewer), UK Centre for Ecology & Hydrology, @mattfry-ceh

Dataset originator/creator#

  • UK Centre for Ecology & Hydrology (creator)

  • Natural Environment Research Council (support)

Dataset reference and documentation#

  • S. Stanley, V. Antoniou, A. Askquith-Ellis, L.A. Ball, E.S. Bennett, J.R. Blake, D.B. Boorman, M. Brooks, M. Clarke, H.M. Cooper, N. Cowan, A. Cumming, J.G. Evans, P. Farrand, M. Fry, O.E. Hitt, W.D. Lord, R. Morrison, G.V. Nash, D. Rylett, P.M. Scarlett, O.D. Swain, M. Szczykulska, J.L. Thornton, E.J. Trill, A.C. Warwick, and B. Winterbourn. Daily and sub-daily hydrometeorological and soil data (2013-2019) [cosmos-uk]. 2021. URL: https://doi.org/10.5285/b5c190e4-e35d-40ea-8fbe-598da03a1185, doi:10.5285/b5c190e4-e35d-40ea-8fbe-598da03a1185.

Further references#

  • Jonathan G. Evans, H. C. Ward, J. R. Blake, E. J. Hewitt, R. Morrison, M. Fry, L. A. Ball, L. C. Doughty, J. W. Libre, O. E. Hitt, D. Rylett, R. J. Ellis, A. C. Warwick, M. Brooks, M. A. Parkes, G. M.H. Wright, A. C. Singer, D. B. Boorman, and A. Jenkins. Soil water content in southern england derived from a cosmic-ray soil moisture observing system – cosmos-uk. Hydrological Processes, 30:4987–4999, 12 2016. doi:10.1002/hyp.10929.

  • M. Zreda, W. J. Shuttleworth, X. Zeng, C. Zweck, D. Desilets, T. Franz, and R. Rosolem. Cosmos: the cosmic-ray soil moisture observing system. Hydrology and Earth System Sciences, 16(11):4079–4099, 2012. URL: https://hess.copernicus.org/articles/16/4079/2012/, doi:10.5194/hess-16-4079-2012.

Note

Data from COSMOS-UK up to the end of 2019 are available for download from the UKCEH Environmental Information Data Centre (EIDC). The data are accompanied by documentation that describes the site-specific instrumentation, data and processing including quality control. The dataset is available for download under the terms of the Open Government Licence.

COSMOS-UK work was supported by the Natural Environment Research Council award number NE/R016429/1 as part of the UK-SCAPE programme delivering National Capability.

Load libraries#

Hide code cell source 
import os
import pandas as pd
import intake
import holoviews as hv
import panel as pn
import matplotlib.pyplot as plt
from bokeh.models.formatters import DatetimeTickFormatter
from datetime import datetime

import hvplot.pandas
import hvplot.xarray  # noqa

import pooch

import warnings
warnings.filterwarnings(action='ignore')

pd.options.display.max_columns = 10
hv.extension('bokeh')
pn.extension()

Set project structure#

notebook_folder = './notebook'
if not os.path.exists(notebook_folder):
    os.makedirs(notebook_folder)

Fetch and load data#

Let’s download the sample data. We use pooch to fetch and unzip them directly from a Zenodo repository.

pooch.retrieve(
    url="doi:10.5281/zenodo.6567018/subset_COSMOS-UK_HydroSoil_Daily_2013-2019.zip",
    known_hash="md5:3755cb069bc48c5efc081905110e169b",
    processor=pooch.Unzip(extract_dir=os.path.join(notebook_folder,'data')),
    path=f".",
)

Downloading data from 'doi:10.5281/zenodo.6567018/subset_COSMOS-UK_HydroSoil_Daily_2013-2019.zip' to file '/home/jovyan/97469708ef44493ff5e8878f93e00890-subset_COSMOS-UK_HydroSoil_Daily_2013-2019.zip'.
Unzipping contents of '/home/jovyan/97469708ef44493ff5e8878f93e00890-subset_COSMOS-UK_HydroSoil_Daily_2013-2019.zip' to '/home/jovyan/./notebook/data'

['/home/jovyan/./notebook/data/COSMOS-UK_SiteMetadata_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_SH_2013-2019_Metadata.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_Daily_2013-2019_Metadata.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_SH_2013-2019/COSMOS-UK_CHIMN_HydroSoil_SH_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_SH_2013-2019/COSMOS-UK_WYTH1_HydroSoil_SH_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_SH_2013-2019/COSMOS-UK_SHEEP_HydroSoil_SH_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_SH_2013-2019/COSMOS-UK_WADDN_HydroSoil_SH_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_Daily_2013-2019/COSMOS-UK_WADDN_HydroSoil_Daily_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_Daily_2013-2019/COSMOS-UK_SHEEP_HydroSoil_Daily_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_Daily_2013-2019/COSMOS-UK_WYTH1_HydroSoil_Daily_2013-2019.csv',
 '/home/jovyan/./notebook/data/COSMOS-UK_HydroSoil_Daily_2013-2019/COSMOS-UK_CHIMN_HydroSoil_Daily_2013-2019.csv']

Load an intake catalog for the downloaded data

Hide code cell source 
# set catalogue location
catalog_file = os.path.join(notebook_folder, 'catalog.yaml')

with open(catalog_file, 'w') as f:
    f.write('''
sources:
  data_siteid:
    driver: intake.source.csv.CSVSource
    parameters:
      stationid:
        description: five letter code for the COSMOS-UK site
        type: str
        default: CHIMN
      resolution:
        description: temporal resolution
        type: str
        default: Daily
        allowed:
          - Daily
          - Hourly
          - SH
    args:
      urlpath: "{{ CATALOG_DIR }}/data/COSMOS-UK_HydroSoil_{{resolution}}_2013-2019/COSMOS-UK_{{stationid}}_HydroSoil_{{resolution}}_2013-2019.csv"
      csv_kwargs:
        na_values: [-9999]
        parse_dates: ['DATE_TIME']

  data_all:
    driver: intake.source.csv.CSVSource
    parameters:
      resolution:
        description: temporal resolution
        type: str
        default: Daily
        allowed:
          - Daily
          - Hourly
          - SH
    args:
      urlpath: "{{ CATALOG_DIR }}/data/COSMOS-UK_HydroSoil_{{resolution}}_2013-2019/COSMOS-UK_*.csv"
      csv_kwargs:
        na_values: [-9999]
        parse_dates: ['DATE_TIME']

  metadata_sites:
    driver: intake.source.csv.CSVSource
    args:
      urlpath: "{{ CATALOG_DIR }}/data/COSMOS-UK_SiteMetadata_2013-2019.csv"
      csv_kwargs:
        header: 0
        parse_dates: [ 'START_DATE','END_DATE']

  metadata_measurements:
    driver: intake.source.csv.CSVSource
    parameters:
      resolution:
        description: temporal resolution
        type: str
        default: Daily
        allowed:
          - Daily
          - Hourly
          - SH
    args:
      urlpath: "{{ CATALOG_DIR }}/data/COSMOS-UK_HydroSoil_{{resolution}}_2013-2019_Metadata.csv"
      csv_kwargs:
        header: 0

  location:
    driver: intake_xarray.image.ImageSource
    parameters:
      stationid:
        description: five letter code for the COSMOS-UK site
        type: str
        default: CHIMN
    args:
      urlpath: "https://eip.ceh.ac.uk/hydrodata/cosmos-uk/maps/airphoto/1000px/{{stationid}}.jpg"
    storage_options: {'anon': True}
''')

cat = intake.open_catalog(catalog_file)

Load metadata#

Here we load COSMOS-UK metadata into memory. The metadata contains multiple columns about the sites, their locations and other site-specific attributes.

metadata = cat.metadata_sites().read()
print(metadata.columns.tolist())

['SITE_NAME', 'SITE_ID', 'START_DATE', 'END_DATE', 'EASTING', 'NORTHING', 'EAST_NORTH_EPSG', 'LATITUDE', 'LONGITUDE', 'LAT_LONG_ESPG', 'ALTITUDE', 'SOIL_TYPE', 'LAND_COVER', 'BULK_DENSITY', 'BULK_DENSITY_SD', 'SOIL_ORGANIC_CARBON', 'SOIL_ORGANIC_CARBON_SD', 'LATTICE_WATER', 'LATTICE_WATER_SD']

metadata
SITE_NAME SITE_ID START_DATE END_DATE EASTING ... BULK_DENSITY_SD SOIL_ORGANIC_CARBON SOIL_ORGANIC_CARBON_SD LATTICE_WATER LATTICE_WATER_SD
0 Chimney Meadows CHIMN 2013-10-02 NaT 436113 ... 0.095 0.027 0.0016 0.011 0.0018
1 Sheepdrove SHEEP 2013-10-24 NaT 436039 ... 0.140 0.059 0.0086 0.027 0.0060
2 Waddesdon WADDN 2013-11-04 NaT 472548 ... 0.160 0.034 0.0023 0.021 0.0021
3 Wytham Woods WYTH1 2013-11-26 2016-10-01 445738 ... 0.190 0.028 0.0044 0.017 0.0047

4 rows × 19 columns

For this example, we will explore a subset of four stations, all of them with start date in 2013. Only the Wytham Woods station ceased on 10th January 2016. This station is situated in a Broadleaf woodland land cover which also hosts Environmental Change Network (ECN) and FLUXNET monitoring sites (see further details here). The dataframe contains each site name, id and corresponding land cover. CHIMN and WADDN are located situated in improved grassland, and SHEEP is in arable and horticulture.

metadata[['SITE_NAME','SITE_ID','LAND_COVER']]
SITE_NAME SITE_ID LAND_COVER
0 Chimney Meadows CHIMN Improved grassland
1 Sheepdrove SHEEP Arable and horticulture (previously improved g...
2 Waddesdon WADDN Improved grassland
3 Wytham Woods WYTH1 Broadleaf woodland

A key feature of COSMOS-UK stations is their capability of monitoring field-scale soil moisture. The CRNSs VWC value is an average soil moisture measurement (%) across an estimated, variable footprint of radius up to 200 m and estimated variable measurement depth of between approximately 0.1 and 0.8 m. It is worth mentioning the measurement depth depends on the soil moisture content as well as lattice water and soil organic matter water equivalent (see Cooper et al. 2021). The greater the actual soil water content, the shallower the penetrative depth. Let’s explore the notional footprint of the analysed stations from the CEH COSMOS-UK website.

Hide code cell source 
# set sliders
station_list = list(metadata.SITE_ID.tolist())

target_station = pn.widgets.Select(name = 'Station', options = station_list)

@pn.depends(target_station.param.value)
def plot_footprint(station):
    location_da = cat.location(stationid=station).to_dask()
    p = location_da.hvplot.rgb(x='x', y='y', bands='channel', data_aspect=1, flip_yaxis=True, xaxis=False, yaxis=None, hover=False)
    return p

plot_stations = pn.Row(
    plot_footprint,
    pn.Column(pn.Spacer(height=5), target_station, background='#f0f0f0', sizing_mode="fixed"),
    width_policy='max', height_policy='max',
)

plot_stations.embed()