Metering Data Management System (MDMS)

Facilitating complex meter data management and processing tasks.

$ 0 M
< 0 K
Data points
Photo Credit: APA Group



APA had was implementing an Energy Components (EC) software system in 2011, which required the reliable delivery of validated operational data into the EC solution.

Technical Situation

APA had several different SCADA platforms across the country collecting transmission data, but no central data repository.  There was an existing OSI PI system which had been developed some ten years previously for a specific application, but it was considered risky to expand this system to include the new requirements.

Nevertheless, APA had confidence in PI, and users had knowledge of PI tools, so it was desirable to continue using the PI platform.

Although users had good knowledge of the operation and of the market rules, the only existing tools in place to capture and validate metering data were spreadsheets tailored to specific SCADA systems.

There was a BizTalk framework in place for data exchange with the Energy Component (EC) system and corporate strategy dictated that it should be used as the standard for exchanging messages between systems.

The business process and therefore solution requirements were complex – GTS was engaged to help APA develop the Metering Data Management (MDM) system with key requirements including:

  • Real-time data collection from multiple SCADA sources across the country
  • File-based data integration via email
  • Interpretation of different data profiles to determine ‘correct’ end of gas period values
  • Implementing market estimation ‘rules’
  • Auto-validation of data and exception flagging
  • Allowing users to review, revise and approve data
  • Enforcing consistent business process
  • Integration of data through existing middleware
  • Raising alerts on system KPIs


A new multi-server OSIsoft PI system was established with the view that although a specific application would be developed on the platform it would also be scalable to serve as an enterprise data infrastructure in the future.  A primary/secondary server pair was established for the PI archive, PI Advanced Calculation Engine (PI ACE) and PI Asset Framework (PI AF).

Dual PI Interface hosts were established at the SCADA sources for data collection from SCADA and a single PI Universal File Loader (PI UFL) instance was established to receive emailed data.

APA chose to implement a tag naming standard in PI rather than trying to change all sources.  This presented a configuration task to map source points to PI points, but enabled the use of templates in AF and provided a consistent view of data to consumers.

The source data from the SCADA systems was not ‘neatly’ time-stamped events but rather a variety of data profiles including:

  • Incrementing index values which would return to zero on a meter rollover
  • A repeating value throughout the day with a step change at some point near the end of gas day
  • A single value received at some point near the end of gas day

To comply with market rules, a value must be selected to represent the gas period (hour or day) and predefined substitution/estimation rules must be applied in the event that ‘actual’ data was not available; this processing must also be auditable.

The concept of an ‘auditable point set’ was developed with each auditable value having a ‘raw’ source point (i.e. the value received from SCADA), an auditable value point (the value selected as the end of gas period value or estimated according to the rules) and an auditable record point (the record of changes to the auditable value).

OSIsoft’s PI Advanced Calculation Engine (PI ACE) was chosen to implement data processing algorithms.  PI ACE is a powerful development tool in combination with the PI Software Development Kit (PI SDK) because it allows extraordinary flexibility in developing scripts for scheduled or triggered action whether to perform calculations or validation on data or to deliver it.

For MDM, data processing scripts were developed that tapped directly into the PI Event Pipe to process data as it arrived, apply the data selection and estimation processes, and populate the audited value set.

There was a separate script or ‘strategy’ developed to suit each of the different raw data profiles and also for each permitted estimation method.

In addition, the data needed to be pre-validated so that possible issues could be detected and flagged for human review.  A set of validation rules and associated parameters was defined by the business and implemented in PI ACE modules.  Incoming data passed through a validation process with a PI point used to record the history of test results for each site.

A Gas Distribution Network model was developed in OSIsoft’s PI Asset Framework (PI AF).  PI AF is a powerful data infrastructure tool because it allows a logical, hierarchical structure to be defined that combines data from any source (such as PI points interfaced to SCADA, databases or other systems) with calculations or static data.  This is a great way to abstract any complexity in deriving data from the users.  For example, a user looking for the gas volume through a gate station can simply navigate to the gate station in the model and look at its volume attribute.  Behind the scenes there may be multiple data points and complex calculations but all of that is hidden from the user.

The MDM PI AF model included all of the relevant SCADA-based PI points, as well as relatively static network data and processing instructions. These include real-time processing and validation methods to be applied to the data at each site in the network.  The PI AF model was therefore an elaborate configuration structure for the PI ACE modules which would parse the structure on start-up and then periodically refresh it.

A user interface was developed in ASP.NET with the essential functions of:

  • Checking PI for gas dates requiring validation (i.e. where either new or modified data was available)
  • Loading audited values and any validation issues for human review
  • Permitting users to manually alter audited values or override validation issues prior to publishing the data to the EC system
  • Requiring users to follow an approval process where changes were made and capturing an audit trail
  • Providing a means for users to flag the data for publication

To integrate the PI data with EC, an interface was developed allowing EC to publish a list of sites it required data for; this list was received and written to the PI AF back-end SQL database.  A PI ACE module was implemented to read the site list, check for any data flagged to be sent, construct, and finally deliver a message to BizTalk.

The delivery of correctly validated data to EC – and from EC through to the market – was a critical task with regulatory rules and penalties for non-compliance.  To ensure any issues were detected as soon as possible OSIsoft’s PI Notifications were used and configured to monitor (among other things) the I/O rate of the SCADA interfaces and raise an alert whenever it fell outside expected parameters.  Alerts were also configured for failure to deliver data to BizTalk or for a variety of internal PI ACE process failures where the process itself might appear to be running but without correct data processing.

Project Outcomes & Benefits

The resulting solution

  • Provided a reliable and scalable data infrastructure integrated into the existing enterprise systems,
  • Provided a powerful data processing features and a highly-configurable AF model with an intuitive user interface.
  • Established a national data infrastructure, providing for the current business needs, and also provided a platform on which future applications could be developed in years to come.
  • Delivered a National Meter Data Management System
  • Delivered efficiency and consistency to complex data processing
  • Complex reporting and integration tasks automated
Andrew Todd
Andrew Todd

Chief Executive Officer

Complicated data processing in spread sheets was replaced entirely by a single enterprise-wide system bringing consistency, efficiency, and automation.

Andrew Todd

Chief Executive Officer at GTS Group

Other case studies