Critical Assessment and Standardized Reporting of Vertical Filtering and Error Propagation in the Data Processing Algorithms of the NDACC Lidars

NEWS UPDATES:
Posted June, 6, 2015: FULL REPORT NOW PUBLICLY AVAILABLE (Download Here)

Posted Sept. 30, 2016: PEER REVIEWED PUBLICATIONS::
Leblanc et al., Atmos. Meas. Tech., 9, 4029-4049, 10.5194/amt-9-4029-2016, 2016
Leblanc et al., Atmos. Meas. Tech., 9, 4051-4078, 10.5194/amt-9-4051-2016, 2016
Leblanc et al., Atmos. Meas. Tech., 9, 4079-4101, 10.5194/amt-9-4079-2016, 2016

Project Summary
The international Network for the Detection of Atmospheric Composition Change (NDACC) is a global network of high-quality, remote-sensing research stations for observing and understanding the physical and chemical state of the Earth atmosphere. As part of NDACC, over 20 ground-based lidar instruments are dedicated to the long-term monitoring of atmospheric composition and to the validation of space-borne measurements of the Earth atmosphere from environmental satellites such as Aura (NASA) and ENVISAT (ESA). One caveat of large networks such as NDACC is the difficulty to archive measurement and analysis information consistently from one research group (or instrument) to another. Yet the need for consistent definitions has strengthened as datasets of various origin (e.g., satellite and ground-based) are increasingly used and ingested together in global assimilation systems.
Our Team of lidar experts was created to address existing issues in at least three critical aspects of the NDACC lidar data retrievals: 1) signal filtering and the vertical filtering of the retrieved profiles, 2) the quantification and propagation of the uncertainties, and 3) the consistent definition and reporting of filtering and uncertainties in the archived products. Additional experts from the satellite and global data standards communities will complement the group to address the issues specific to the latter aspect. Additional aspects of lidar data processing will be discussed if they are considered of critical relevance.

Project Outline
The first meeting will consist of a review and critical assessment of the methodologies used in the NDACC lidar analysis algorithms. Among them, the filtering schemes and the definition, quantification, and propagation of uncertainties will be reviewed and discussed. Based on the results of the first meeting, recommendations on the use of specific methodologies and/or approaches will be conveyed to the entire NDACC lidar community between the first and second meeting. Based on the conclusions of the first meeting and on the feedback received from the NDACC lidar community, the second meeting will consist of elaborating standardized definitions of the recommended methodologies/approaches, and implementing these methodologies so that critical information in connection with the use of these methods can be durably and consistently reported in the NDACC lidar data files. The last meeting will consist of the elaboration of one or several Technical Report(s) to be used as Reference Guide(s) by NDACC data users within both the NDACC and satellite communities.

Team Composition (First Meeting)
T. Leblanc (Team Leader and Coordinator), representing the NDACC lidars of JPL (USA)
F. Gabarrot, representing the NDACC lidars of La Reunion (France)
S. Godin-Beekmann, representing the NDACC lidars of CNRS (France)
A. Haefele, representing the Meteoswiss Lidar of Payerne(Switzerland)
F. Immler, representing GRUAN (DWD, GCOS)
J. Khanna, representing the NDACC Lidar of University of Western Ontario (Canada)
G. Liberti, representing the NDACC Lidar of Rome-Tor Vergata (Italy)
F. Madonna, representing the Potenza lidar (Italy), and European network EARLINET (EU)
T. McGee, representing two NDACC mobile lidars of NASA-GSFC (USA)
C. Retscher, representing NASA’s Aura Validation Data Center AVDC (USA)
B. Sica, representing the NDACC Lidar of University of Western Ontario (Canada)
B. Tatarov, representing the NDACC Lidar of NIES (Japan)
T. Trickl, representing the NDACC Lidar of IMK (Germany)
A. vanGijsel, representing the NDACC lidar of RIVM (The Netherlands)
C. Straub (Local Invitation), representing the NDACC microwave instrument of Univ. of Bern (Switzerland)

First Meeting
29 November – 3 December 2010
Meeting Agenda (Public Access)
Meeting Summary Notes
Meeting Agenda with Presentations (Restricted Access)

Second Meeting
14-17 June 2011
Present: T. Leblanc, F. Gabarrot, G. Payen, A. van Gijsel, S. Godin-Beekmann, J. Bandoro for B. Sica, M. Thetis for P. Keckhut
The main focus of the meeting was on the early Validation of the new NDACC Standardization Tools "NDACC_ResolDF" and "NDACC_ResolIR". The following ISSI Team Members attended the meeting: T. Leblanc, JPL (MLO and TMF lidars), S. Goding-Beekmann and Guillaume Payen, CNRS (OHP ozone lidar), Franck Gabarrot, LACy (Reunion Island Lidars), Anne van Gijsel, KNMI (Lauder Lidar), and Michelle Thetis, CNRS (OHP temperature lidar). Raw lidar signals with a varying degree of noise and interference were simulated (Leblanc), then analyzed by each lidar representative (Gabarrot, Leblanc, Payen, Thetis, van Gijsel). The results from the analysis were compared to the original temperature and ozone profiles used to simulate the signals. The Team's work during the meeting allowed to confirm the consistency of all the forward/reverse algorithms. The NDACC_ResolDF and NDACC_ResolIR standardization tools were then implemented inside the analysis softwares. A thorough validation of these new tools took place and the results are posted here.

Another focus of the meeting was on the initiation of guidelines for the progpagation of Uncertainties. A new document is being written that will describe the step-by-step NDACC-Standardized rules to propagate the uncertainties throughout the algorithms. In addition to the uncorrelated uncertainties associated with random noise from photon counting, the uncertainty sources to be considered include: background noise extraction, saturation correction, overlap correction, molecular extinction correction, ozone absorption correction (for temperature only), ozone cross-sections (for ozone DIAL only), a priori pressure and temperature uncertainties (for temperature only). The propagation rules for uncorrelated uncertainties will hold as often as possible. When uncertianties are found to be correlated, a special treatment using lidar simulated signals must be considered.

Post-Meeting Actions on Vertical Resolution
Validation of the new NDACC Standardization Tools (NDACC_ResolDF and NDACC_ResolIR)
Click Here

First Draft on the Treatment of Uncertainty
PDF Version: Click Here
WORD Version: Click Here

Third Meeting: September 10-14, 2012
NEW: Documentation in preparation of Meeting 3 (requires login info):
Overview on the analytical epxression of Uncertainty in the Ozone and Temperature algorithms (draft by T. Leblanc)
Overview on the statistical approach for the non-analytical expression of Uncertainty in the Ozone and Temperature algorithms (draft by T. Leblanc)
Preparatory document for the Evaluation of the Uncertainty associated with NO2 absorption (draft by G. Liberti)
Summary of Most Accurate Ozone Absoprtion Cross-sections (draft by T. Trickl and S. Godin-Beekmann)
Expressions of the source and propagation of Uncertainty in the temperature retrieval algorithm (draft by J. Bandoro and B. Sica)
Progress Update on Vertical Resolution (draft by T. Leblanc, F. Gabarrot, and G. Payen)

NEW JUNE 2015: FULL ISSI TEAM REPORT READY FOR DOWNLOAD:
CLICK HERE

Full Proposal
Click Here

List of Publications:
Click Here