Data Management and Processing

Generating accurate, accessible scientific data is a fundamental principle of WBEA’s monitoring programs. Agencies such as Alberta Environment and Parks (AEP) and the AEP AirData warehouse use WBEA data to track regional air quality. The public looks to the Air Quality Health Index(AQHI), calculated using WBEA data, for information to guide outdoor activities.

WBEA’s Data Management Program develops, operates and maintains a secure, robust, reliable and transparent Data Management System (DMS). Redundant and reliable data collection, paired with simple data access portals, enables WBEA to consistently provide high quality data to our stakeholders. Documentation of WBEA’s complex DMS has recently been achieved with a new manual describing structures and components.

Data Management System Enhancement

  • In-house Data Management allows WBEA to rapidly and reliably improve the system as a whole, based on the needs of its members and the general public. Our website is now fully hosted and maintained by WBEA.
  • Consolidating WBEA’s Four Data Management Servers into a single high security facility provides the advantages of redundant internet connectivity and power supply, high availability cooling and restricted biometric access. Inter-server communication is now 100% reliable.
  • WBEA’s Four High Powered Servers collect continuous air monitoring data from analyzers at each of our air monitoring stations. The servers are fully redundant to ensure no single point of failure. To maintain DMS accessibility and reliability the servers are running ESXi virtualization which distributes processor and network loads evenly between each component.

Data Collection for Air Quality Monitoring

Each of WBEA’s continuous Air Monitoring Stations generates thousands of data points daily from some 142 air analyzers, which operate 24 hours a day, 365 days a year. Data loggers collect and transmit this data for storage in WBEA’s central secure database. The database is also populated with the historical record of final 5 minute and 1 hour data from 1997 to date. Recently, a user-friendly graphing function was added to the WBEA members website. WBEA members can now instantly display, save and print air pollutant concentration data for multiple pollutants per station, or for several stations. The public can access a modified version of this data interface.

A web-based documentation system called Doc-It allows entry of information by operations personnel at monitoring stations (or any web-enabled location) and retrieval by anyone in the system for data quality review purposes and routine processing. A data monitoring utility called Watch-It monitors continuous data in the PI database in near real time for programmed conditions such as flat-lines in data or exceedances of air quality guidelines.

A computerized information system called Sample-It guides the integrated sampling process and sends information recorded at each step, along with exposed samples, for analysis to the appropriate lab. Sample-It also receives and stores data from laboratory analysis in a Microsoft SQL Server database. Sample-It is being populated with the historical record of integrated sampling results.

Time-Integrated Air Quality Data

For years, WBEA’s continuous hourly updated air quality data has been publicly available for all stations. The WBEA Data Management team has recently developed software and user interfaces to enable non-continuous air quality data to be as transparent and readily available as our continuous data. The latest time-integrated current and historical data for volatile organic carbons (VOCs), particulate matter (PM10 and PM2.5),  and polycyclic aromatic hydrocarbons (PAHs) is now available on the WBEA website.

Improved Hardware Systems

Improvements to WBEA’s data collection system hardware in monitoring stations include:

  • Installation of sensors to monitor the status of support systems for air quality and meteorological data.
  • Consideration of alternate communications solutions, including satellite links from remote locations, to increase reliability of data transmission.
  • Upgrading of data loggers to new Campbell Scientific CR3000 units.
  • New data logger programs to standardize algorithms across the network, delay daily zero and span tests if an air quality episode is in progress and rotate the hour of daily zero and span tests throughout the month.
  • Accessing the internal self-monitoring electronic systems inside air quality analyzers, to transmit information about the status of analyzers to the operations center for review by network operators.
  • Collection of digital data from the majority of analyzers in the network.
  • Investigation of alternate back-up power sources for remote locations.

These upgrades improve effectiveness and efficiency in network operations and management by allowing access to monitoring stations and systems remotely over the internet, thus making more information accessible and available for review and detection ofoperating issues.

Routine Data Processing

Routine continuous data processing includes daily review of recently collected data from air quality analyzers and meteorological sensors and daily review of air quality analyzer zero and span check results. Daily review ensures proper instrument operation and identifies potential malfunctions for investigation. Monthly processing includes data summary, validation and baseline correction of air quality measurements for analyzer drift. Data arefurther reviewed by WBEA as an acceptance test before generation of monthly compliance reports and distribution to various locations, including the WBEA PI database for continuous data, the AEP AirData warehouse and WBEA members.

Enhanced Data Processing

Data collected by continuous monitoring programs, which include a regulatory compliance purpose, are often inadequate for science-based environmental effects monitoring. WBEA’s routinely produced, validated and baseline-corrected data are being reprocessed with sophisticated statistical techniques to provide data that satisfies requirements of environmental effects research. These requirements include calculation of concentrations where measured concentrations are less than operational detection limits and greater than operating ranges of air quality analyzers. The enhanced WBEA data management system presents an opportunity to undertake and develop utilities to support and streamline the data processing and other data-centric routine tasks.

Data Management System Development

To proactively address the needs of our stakeholders, WBEA Data and Systems Specialists are working closely with Environment and Climate Change Canada (ECC) scientists to stream the ECC ambient air mercury data collected at WBEA’s Patricia McInnes AMS in Timberlea, as well as the volatile organic compound (VOC) data from WBEA’s Bertha Ganter-Fort McKay AMS to the WBEA website. The WBEA Data Management Program continues to adapt and evolve to serve WBEA members, stakeholders and the public.

Click to read:
Toward a Comprehensive Data Management System for WBEA Monitoring Programs (Extended Abstract 2010-A-1092-AWMA)