NDACC Standardization Tools for Vertical Resolution

(T. Leblanc, J. Bandoro, G. Payen, F. Gabarrot, and A. vanGijsel)

1. Purpose
Currently, NDACC data originators do not use a consistent and homogeneous definition of vertical resolution. The purpose of producing tools for the standardization of vertical resolution is to insure that the Vertical Resolution values reported in the data files archived at the NDACC Data Handling Facility consistently use the same definition from one NDACC lidar data file to another.

2. Concept
The Tools currently developped are simple, easy-to-use, subroutines that can be directly inserted inside the NDACC PIs' lidar analysis algorithms and which purpose is to compute the Vertical Resolution based on an agreed standardized definition. As of today, 2 standardized definitions were selected by the ISSI Team. One definition is based on the cutoff frequency of a Digital Filter (DF). The type of filter is defined by the coefficients used to filter the data. The subroutine name for this definition is "NDACC_ResolDF". The second definition is based on the Full-Width at Half Maximum of the response to an impulse-type perturbation (a Dirac-Delta Function for smoothing filters, and a Heavyside-Step Function for differentiation filters). The subroutine name for this definition is "NDACC_ResolIR". In both "DF" and "IR" cases, the subroutines were written homogeneously in three different programming languages: IDL, FORTRAN, and MATLAB. It is planned to develop a PYTHON version as well.

3. Tools Validation
After being written, the tools needed a thorough validation to insure the consistency within all existing NDACC data. This validation was made using simulated lidar signals. Using a "reverse-type" model, synthetic lidar signals were created from known ozone and temperature profiles. The signals were simulated using various noise configurations, and with or without a number of common corrections usually applied to real lidar data. This flexibility insured to verify the full self-consistency between the reverse and forward models before the standardization tools themselves were tested. The results below therefore cover several steps of the validation, ranging from the self-consistency of the simulated and retrieved profiles (section 3.3) to the full and final validation of the tools (sections 3.4 and 3.5). Note that the validation results below are not meant to validate the ozone and temperature algoithms, but to validate the implementaiton of the Standadization Tools into these algorithms. That means focus must be made on the degree of agreement between the experimental and theoretical transfer functions (DF defintion), and the degree of agreemeent between the experimental and theroetical impulse response widths (IR definition). The main validation results and conclusions are summarized before all the plots are shown (top of sections 3.4 and 3.5). They apply similarly to all NDACC products listed in section 3.1. A few exceptions are covered individually below each relevant plot.

3.1 Contributing Investigators (instrument abbreviations are used to identify the plots below)
The validation results are presented below by alphabetical order of the instrument abbreviations, ozone first, temperature second. The following investigators participated to the validation work, covering eight existing or future NDACC products:
- Thierry Leblanc, for stratospheric ozone and temperature at Mauna Loa, Hawaii (MLO)
- Justin Bandoro, for temperature at Univ. of Western Ontario (PCT for Rayleigh and PCW for Raman)
- Guillaume Payen, for stratospheric ozone at Observatoire de Haute Provence (SHO)
- Anne vanGijsel, for stratospheric ozone and temperature at Lauder (SWL)
- Franck Gabarrot, for tropospheric ozone at Reunion Island (TRO)

3.2 Summary of the Simulations (by date) and their Purpose
Each date corresponds to a specific simulation configuration. The choice of a specific date has no geophysical meaning. A large range of dates was chosen only to avoid confusion between the simulation configurations. As a rule-of-thumb (01/14 excluded), even dates apply to the validation of the tools for ozone retrievals, while odd dates apply to the validation of the tools for temperature retrievals. Though we have listed 01/14, 01/20, and 01/21 below, no result for these dates are shown because they are only intermediate stages of the Tools validation process.

Simulated date: 2012/01/14
A near-real experiment. The simulated signals are representative of a real measurement, i.e., they contain background noise, saturation, partial overlap, they are absorbed by ozone and attenuated by molecular extinction. The purpose of these datasets was to insure that simulated signals could be ingested similarly to real signals without an unexpected crash of the data processing softwares. These signals were used for both ozone and temperature retrievals. This simulation run was a pre-requisite to all validation tests posted thereafter, but is not part of the Tools' validation in itself, and therefore not shown below.

Simulated date: 2012/01/18
Disturbance-free signals with the purpose of testing the consistency of the ozone forward and reverse models. The simulated signals contain no correction susceptible to introduce biases between the original and retrieved ozone profiles. They are therefore random noise-free, saturation-free, there is complete overlap, and no correction is necessary for molecular extinction. The ozone absorption cross-sections were taken as known constants to avoid possible discrepancies associated with their temperature dependence. This simulation run was a pre-requisite (for ozone) to all validation tests posted thereafter.

Simulated date: 2012/01/19
Disturbance-free signals with the purpose of testing the consistency of the temperature forward and reverse models. Like 01/18, the simulated signals contain no correction susceptible to introduce biases between the original and retrieved profiles. They are therefore random noise-free, saturation-free, there is complete overlap, and no correction is necessary for molecular extinction or ozone absorption. This simulation run was a pre-requisite (for temperature) to all validation tests posted thereafter.

Simulated date: 2012/01/20
Signals specifically dedicated to the validation of the Vertical Resolution Tool NDACC_ResolDF for ozone. The signals are just like those of 01/18, except that random (white) noise was added to the ozone profiles in order to quantify the effect of filtering and report this effect following the NDACC-Standardized DF cutoff frequency definition.

Simulated date: 2012/01/21
Signals specifically dedicated to the validation of the Vertical Resolution Tool NDACC_ResolDF for temperature. The signals are just like those of 01/19, except that random (white) noise was added to the temperature profiles in order to quantify the effect of filtering and report this effect following the NDACC-Standardized DF cutoff frequency definition.

Simulated dates: March and April 2013, even dates
Same as 2012/01/20. These are randomized repeats of 01/20, which are then averaged in order to minimize the noise associated with the experimental determination of the transfer function (TF) for ozone retrievals. The TFs are calculated for each of the 30 experiments, then averaged together. The resulting averaged TF is compared to the theoretical TF. Though a single experiment like 01/20 is sufficient when dealing with filters with a small number of coefficients, the averaging process is necessary to deal with filters having a large number of coefficients with respect to the total sampling window considered (for example 17 coefficients out of a 200-pts total window, or 133 coefficients out of 860-pts total window).

Simulated dates: March and April 2013, odd dates
Same as 2012/01/21. These are randomized repeats of 01/21, which are then averaged in order to minimize the noise associated with the experimental determination of the transfer function (TF) for temperature retrievals. The TFs are calculated for each of the 30 experiments, then averaged together. The resulting averaged TF is compared to the theoretical TF. Though a single experiment like 01/21 is sufficient when dealing with filters with a small number of coefficients, the averaging process is necessary to deal with filters having a large number of coefficients with respect to the total sampling window considered (for example 17 coefficients out of 200-pts total window, or 133 coefficients out of 860-pts total window).

Simulated date: 2012/01/22
Signals specifically dedicated to the validation of the Vertical Resolution Tool NDACC_ResolIR for ozone. The signals are just like those of 01/18, except that a Dirac Delta Function was added to the simulated ozone profile in order to quantify the effect of filtering and report this effect following the NDACC-Standardized FWHM Impulse Response definition.

Simulated date: 2012/01/23
Signals specifically dedicated to the validation of the Vertical Resolution Tool NDACC_ResolIR for temperature. The signals are just like those of 01/19, except that a Dirac Delta Function was added to the simulated temperature profile in order to quantify the effect of filtering and report this effect following the NDACC-Standardized FWHM Impulse Response definition

3.3 Checking the Consistency of the Reverse and Forward Models (01/18 and 01/19)

The work described in this section is only a pre-requisite to the validation results presented in section 3.4. If you are only interested in the validation of the vertical resolution standardization tools, you can skip this section and go to section 3.4 now.

The purpose of this check is to insure that, in the absence of noise, the retrieved profiles are fully consistent with the simulated profiles. If they were not, systematic differences between retrieved and simulated profiles would introduce an undesired "noise" component susceptible to bias the validation of the vertical resolution standardization tools. All eight plots in this section show basically the same behavior: For ozone, the retrieved and simulated profiles agree well, with only residual numerical noise owed to the discretization of the simulated, photon-counting-noise-free signals. This numerical noise may have a different magnitude due to the varying magnitude of the signals specifically simulated for each NDACC instrument. Numerical noise may also show at different altitudes due to the combination of high-intensity and low-intensity ranges for some instruments. For temperature, the main source of difference is at the top, and is due to the temperature tie-on to the a priori profile, which can be slightly different from the simulated profile. As integration occurs downward, this difference decreases exponentially. Like for ozone, residual noise can again be found due tothe numerical discretization of the photon counts. The small differences observed between the retrieved and simulated profiles illustrate the good self-consistency between the forward and reverse models. These results allow us to move on with confidence to the validation of the Vertical Resolution Standardization Tools since any difference between simulated and retireved profiles will be owed to data filtering rather than systematic errors.

MLO ozone

MLO Temperature

PCL Rayleigh Temperature

.

PCL Raman Temperature

SHO Ozone

Numerical noise above 30 km is higher for SHO above (and TRO below) than for the other ozone products (MLO and SWL) because of the number of significant digits used in the raw data format. This numerical noise is centred to zero and is not an issue for the validation of the ISSI tools. In the other simulation experiments presented below, the simulated signals contain photon-counting (random) noise and the data processing algorithms use filtering, so this numerical discretization problem does not exist (as can be seen in sections 3.4 and 3.5).

SWL Ozone

SWL Temperature

TRO Tropospheric Ozone

Numerical noise above 5 km is higher for TRO above (and SHO) than for the other ozone products (MLO and SWL) because of the number of significant digits used in the raw data format. This numerical noise is centred to zero and is not an issue for the validation of the ISSI tools. In the other simulation experiments presented below, the simulated signals contain photon-counting (random) noise and the data processing algorithms use filtering, so this numerical discretization problem does not exist (as can be seen in sections 3.4 and 3.5).

3.4 Validation of NDACC_ResolDF: Consistency of the Actual (observed) and Theoretical (NDACC_ResolDF) Transfer Functions (simulation dates: 30 days from March to April 2012)

For both temperature and ozone products, a series of 30 simulated experiments were performed to validate the vertical resolution standardization tool NDACC_ResolDF. For ozone, each of the 30 simulated profiles contained random (white) noise of mean magnitude 20% of the mean profile value. For temperature, each of the 30 simulated profiles contained random (white) noise of mean magnitude 3% of the mean value (approx. 6 K). One-by-one, the experiments were analyzed using several arbitrary filters. The type of filter is unchanged from that used in the normal NDACC routine analysis, but the number of coefficients is set to a constant with height in order to characterize the filter. The transfer function of the filter used for each of the 30 retrieved profiles is estimated by calculating the ratio of the Fast Fourier Transforms of the output product (the analyzed profile) and the input product (the simulated profile). The 30 Transfer Functions thus obtained were averaged, and the average was compared to the theoretical Transfer Function calculated directly from the coefficients of the filter used in the analysis (i..e, calculated by the new Standardization Tool NDACC_ResolDF). Theoretically the comparison could be done for just one simulated experiment, but the truncated sampling windows of the simulated and analyzed profiles introduce numerical noise into the calculated transfer function (as opposed to an idealized, infinite-length window). Despite the averaging, a slight difference remains for filters with a large number of coefficients, i.e., a number of coefficients of the order of 1/10th of the total sampling window length. This difference can be seen on several of the plots below, but does not have any incidence on the validation results, i.e., the good consistency between observed (blue) and theoretical (red) TF.

One interesting advantage of the new standardization tool is the abiltiy to be used multiple times within the same data processing chain, if necessary. When smoothing/filtering occurs at more than one occasion through the data processing chain, the subroutine NDACC_ResolDF can be inserted each time such a occurence is found. The output TF of the first call is used in input of the second call, and so on until all smoothing occurrences have been covered. The final TF is the product of the TF calculated separately for each smoothing filter applied.

The consistency of the observed and theoretical TFs observed in all the plots below validates the NDACC_ResolDF tool. The output value of the tool is referred to as "dz_cutoff" in the plots below. It is the reciprocal of the cutoff wavenumber (wavenumber where the TF equals 0.5). dz_cutoff can then be multiplied by the sampling width (bin size) to obtain the NDACC-Standardized Vertical Resolution (follwoing the DF definition). The tool can now be used for all NDACC Temperature and Ozone PIs/instruments to report Vertical Resolution based on the new standardized definition (defined uniquely from the number and value of the filter coefficients used). The new tool can also serve as a conversion device between the vertical resolution reported until today by the NDACC PIs and the vertical resolution meant to be reported from now on following the ISSI Team recommendations.

MLO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 7-pts

.

MLO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 13-pts

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 7-pts

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 15-pts

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 33-pts

SWL Ozone, Least-Square Polynomial Degree 1, Derivative filter, 5-pts

SWL Ozone, Least-Square Polynomial Degree 1, Derivative filter, 17-pts

TRO Tropospheric ozone, Least-Square Polynomial Degree 2, Derivative filter, 15-pts

TRO Tropospheric ozone, Least-Square Polynomial Degree 2, Derivative filter, 67-pts

MLO Temperature, Hann window, 9-pts

MLO Temperature, Hann window, 17-pts

PCL Rayleigh Temperature, "Smoothing by 3s and 5s" (7-pts)

PCL Raman Temperature, "Smoothing by 3s and 5s" (7-pts)

SWL Temperature, 2 filters: First, Least-Square Polynomial Degree 1, 3-pts, then Least-Square Polynomial Degree 1, 5-pts

The case of SWL-temperature (above and below)) is still under investigation. The PI has reported the use of two successive filters, namely, a Least Square polynomial fit of degree 1 over 3 points, followed by a Least-Square polynomial fit of degree 1 over 5 points for the case presented above (or 17 points in the case presented 2 plots below). However the current comparisons with the theoretical solution seems to point towards no effect of the 3-point smoothing filter. Much better agremeent with the theoretical solution is found if we assume non-existence of the first filter (plot below) than if we assume the existence of the first filter (plot above).

SWL Temperature, Least-Square Polynomial Degree 1, 5-pt

SWL Temperature, Least-Square Polynomial Degree 1, 17-pts

Though we are not showing the case of 2 successive, 3-point, then 17-point filters, the same conclusion as that for the 5-point filter seems to hold for the 17-point filter. The difference between the 2-filter case and 1-filter case is not as obvious as for the 5-point filter because the width of the second filter is much larger than that of the first filter, therefore shadowing the effect of the first, narrow width filter. For this reason we are only showing here the case of one 17-point filter.

3.5 Validation of NDACC_ResolIR: Consistency of the Actual (observed) and Theoretical (NDACC_ResolIR) FWHM Impulse Responses (simulation dates 01/22 and 01/23)

As for the validation of NDACC_ResolDF, simulated synthetic lidar signals were used for the validation of NDACC_ResolIR. An impulse perturbation (a Dirac Delta function with a magnitude of +100%) was introduced in the noise-free, simulated ozone (at 30 km) and temperature (at 40 km) profiles. The response to this perturbation is then examined and compared against the theoretical response caclulated by NDACC_ResolIR. The maximum magnitude of the response (i..e., the amplitude of the Gaussian-shaped response) is of minor importance here. The width of this response is the focus of the validation work. Though the theoretical and observed responses might not have the exact same maximum amplitude, they must have the same width at half-maximum (FWHM). In the plots below, the FWHM calculated by NDACC_ResolIR is shown by a black vertical line looking like an error bar. The proximity of each end of the vertical bar to the cyan curve shows how close the theoretical solution is from the observed solution. The (small) differences between the red curves and the blue curves at larger distances from the impulse (i..e, +/- npts/2) on the ozone plots are simply due to the effect of filtering on the overall shape of the ozone profile. The differences between the red curves and the blue curves at larger distances from the impulse (i..e, +/- npts/2) on the temperature plots are due to the effect of the asymmetric nature of the downward integration of atmospheric density. These differences do not affect the validation results (again, focus is on the shape of the response, not on its actual magnitude. For the Purple Crow Lidar temperature (PCT and PCW) , the match between observed and theoretical FWHM is not as straightforward because the verical resolution used (1000 m) is a non-negligible fraction of the total sampling window (approx. 60 km). The width of the observed response (cyan curve) is narrower than that theoretically predicted (red curve and black vertical line).

As for NDACC_ResolDF, the subroutine NDACC_ResolIR can be inserted and called multiple times in the data processing chain. The output response of the first call is used in input of the second call instead of the initial impulse. The final FWHM is the width of the final response obtained after the last call.

The excellent agreement observed on most of the plots below validate the NDACC_ResolIR tool. The output value of the tool is referred to as "dz_fwhm" in the plots below. It can then be multiplied by the sampling width (bin size) to obtain the NDACC-Standardized Vertical Resolution (following the IR definition). Though a few issues need to be resolved for a couple of cases presented below, the tool can now be used safely for all NDACC Temperature and Ozone PIs/instruments to report Vertical Resolution based on the new standardized definition (defined uniquely from the theoretical width of the impulse response). The new tool can also serve as a conversion device between the vertical resolution reported until today by the NDACC PIs and the vertical resolution meant to be reported from now on following the ISSI Team recommendations.

MLO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 13-pts

MLO Temperature, Hann Window, 9-pts

PCL Rayleigh Temperature, "smoothing by 3s and 5s" (7 pts)

Reaching full agreement between the expected and observed IR solutions for the Purple Crow Lidar case (Rayleigh "PCT" above and Raman "PCW" below) is problematic because the initial 24-m resolution data are binned together at a resolution of about 1000-m before being smoothed. The smoothing process is applied to this 1000-m resolution data. The aymmetric effect (with respect to the altitude of the impulse) of density integration over these thicker layers increases the differences between the simulated and retrieved temperatures above and below the impulse, and therefore tends to "stretch" the shape of the response farther away from the ideal, symmetrical shape. Expectedly, the observed Impulse Response does not fully match the theoretical Impulse Response. The difference is about one bin for a window width of 5 bins (20%). Possible actions to mitigate this problem are currently under consideration. One likely action will be to submit the PCL temperature product to the database using raw data binned to vertical resolutions similar to the other NDACC lidar temperature products presented here (e.g., 300 m for MLO and SWL).

PCL Raman Temperature, "smoothing by 3s and 5s" (7 pts)

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 7-pts

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 7-pts

SHO Ozone, Least-Square Polynomial Degree 2, Derivative filter, 7-pts

SWL Ozone, Least-Square Polynomial Degree 1, Derivative filter, 5-pts

SWL Ozone, Least-Square Polynomial Degree 1, Derivative filter, 17-pts

SWL Temperature, 2 filters: First a Least-Square Polynomial Degree 1, 3-pts, then a Least-Square Polynomial Degree 1, 5-pts

The case of SWL-temperature (above and below)) is still under investigation. The PI has reported the use of two successive filters, namely, a Least Square polynomial fit of degree 1 over 3 points, followed by a Least-Square polynomial fit of degree 1 over 5 points for the case presented here (or 17 points in the case presented 2 plots below). However the current comparisons with the theoretical solution seems to point towards no effect of the 3-point smoothing filter. Much better agremeent with the theoretical solution is found if we assume no first filter (plot below) than if we assume the existence of the first filter (plot above).

SWL Temperature, Least-Square Polynomial Degree 1, 5-pts

SWL Temperature, Least-Square Polynomial Degree 1, 17-pts

Though we are not showing the case of 2 successive, 3-point, then 17-point filters, the same conclusion as that for the 5-point filter seems to hold for the 17-point filter. The difference between the 2-filter case and 1-filter case is not as obvious as for the 5-point filter because the width of the second filter is much larger than that of the first filter, therefore shadowing the effect of the first, narrow width filter. For this reason we are only showing here the case of one 17-point filter.

TRO Tropospheric Ozone, Least-Square Polynomial Degree 2, Derivative filter, 15-pts

TRO Tropospheric Ozone, Least-Square Polynomial Degree 2, Derivative filter, 33-pts

3.6 What's next with NDACC_ResolDF and NDACC_ResolIR ?

The above results show that the NDACC_ResolDF routine provides a good estimate of the frequency, and therofre a good estimate of vertical resolution based on the standardized defintion selected by the ISIS Team at their December 2010 meeting. Also routine NDACC_ResolIR provides a good estimate of vertical resolution following the standardized defintionbased on the FHWM of the response to an impulse such as a Dirac Delta or Heaviside step function.

Technically, both subroutines now can be "permanently" inserted in the NDACC PIs' data processing softwares. The first use of these routines will consist of establishing a detailed map of the existing individual products' vertical resolution and their NDACC-standardized equivalent. Once the mapping is in place, the existing Ames files will be replaced by new ones that will contain the mapped values of vertical resolution following the NDACC-standardized definition. The second use of the routines consists of producing the values of vertical resolution directly based on the NDACC-standardizes defintion, i.e., without going through a prior conversion process.

3.7 The Cherry on the Cake: Can this work be used for other NDACC lidar products ?

Preliminary validation tests were made on an NDACC water vapor lidar product. Two cases using the JPL Table Mountain water vapor Raman lidar (TMW) algorithm are shown below. The Standardization Tools work just as good as for ozone and temperature. Though more water vapor (and backscatter ratio) lidar algorithms need to be used for a thorough validation of the Tools, the results below suggest that their implementaiton for water vapor and backscatter ratio is similar to that for ozone and temperature.