Collaborative development of a Lidar Processing Pipeline

Abstract

Aerosols, clouds and their interactions are the source of the largest uncertainties in current climate change estimates. More frequent and higher quality measurement of aerosol, clouds, and the physical processes governing their link with climate are needed to reduce these uncertainties, and lidars are a powerful instrument to accomplish this task. However, lidars are generally developed by individual groups for particular applications, and hence typically have very different characteristics. In this sense, lidar networks play a key role as they coordinate the efforts of different groups, providing the guidelines for quality-assured routine measurements on a regional scale. Moreover, a coordinated effort is of utmost importance to homogenize the physical retrievals from the highly non-uniform instruments in a network. This is only possible with the development of a unified processing pipeline that accounts for the hardware heterogeneity in the pool of instruments. Here, we describe an ongoing effort to collaboratively develop a Lidar Processing Pipeline. The LPP is a collection of tools developed in C++ and Python that are planned to handle all the steps of lidar analysis. A first tool converts the raw data files into a single NetCDF file, including detailed information about the instrument and acquisition setup (Level 0 dataset). . Then, this is processed by another tool which applies the necessary corrections and computes the cloud-mask (Level 1 dataset). The final step is the elastic and/or inelastic retrieval of aerosol properties (Level 2 dataset). The development of LPP has been based on the existing analysis routines developed by individual groups, and hence takes advantage of previous efforts for algorithm comparison in the scope of Lalinet Network. A general concept and first steps of the ongoing project are reported, as well as the next steps towards the release of the first operational version of the processing pipeline.