APA GROUP – METERING DATA MANAGEMENT SYSTEM
APA is Australia’s largest owner of gas infrastructure and a top 30 company on the Australian Stock Exchange. APA owns and operates two thirds of Australia’s onshore pipelines totalling more than 14,000 kilometres of gas transmission assets valued in excess of $12 billion.
APA is broadly divided into a transmission business responsible for bulk movement of natural gas from processing or storage facilities to domestic markets around the country; and a networks business for the distribution of gas to consumers. Both transmission and network operation is strictly regulated by the Australian Energy Regulator.
In 2004 APA implemented a PI-based system for managing gas consumption data for South Australian high demand customers in order to meet regulatory requirements for Full Retail Contestability (FRC). GTS partners Gavin Man and Andrew Todd were part of the original FRC project team establishing the PI system and the now expanded GTS team has been supporting it ever since.
APA had was implementing an Energy Components (EC) software system in 2011 which required the reliable delivery of validated operational data into the EC solution.
APA had several different SCADA platforms across the country collecting transmission data but no central data repository. There was an existing OSI PI system which had been developed some ten years previously for a specific application but it was considered risky to expand this system to include the new requirements.
Nevertheless, APA had confidence in PI, and users had knowledge of PI tools, so it was desirable to continue using the PI platform.
Although users had good knowledge of the operation and of the market rules, the only existing tools in place to capture and validate metering data were spreadsheets tailored to specific SCADA systems.
There was a BizTalk framework in place for data exchange with the Energy Component (EC) system and corporate strategy dictated that it should be used as the standard for exchanging messages between systems.
The business process and therefore solution requirements were complex – GTS was engaged to help APA develop the Metering Data Management (MDM) system with key requirements including:
- Real-time data collection from multiple SCADA sources across the country
- File-based data integration via email
- Interpretation of different data profiles to determine ‘correct’ end of gas period values
- Implementing market estimation ‘rules’
- Auto-validation of data and exception flagging
- Allowing users to review, revise and approve data
- Enforcing consistent business process
- Integration of data through existing middleware
- Raising alerts on system KPIs
A new multi-server OSIsoft PI system was established with the view that although a specific application would be developed on the platform it would also be scalable to serve as an enterprise data infrastructure in the future. A primary/secondary server pair was established for the PI archive, PI Advanced Calculation Engine (ACE) and PI Asset Framework (AF).
Dual PI Interface hosts were established at the SCADA sources for data collection from SCADA and a single PI Universal File Loader (UFL) instance was established to receive emailed data.
APA chose to implement a tag naming standard in PI rather than trying to change all sources. This presented a configuration task to map source points to PI points, but enabled the use of templates in AF and provided a consistent view of data to consumers.
The source data from the SCADA systems was not ‘neatly’ time-stamped events but rather a variety of data profiles including:
- Incrementing index values which would return to zero on a meter rollover
- A repeating value throughout the day with a step change at some point near the end of gas day
- A single value received at some point near the end of gas day
To comply with market rules, a value must be selected to represent the gas period (hour or day) and predefined substitution/estimation rules must be applied in the event that ‘actual’ data was not available; this processing must also be auditable.
The concept of an ‘auditable point set’ was developed with each auditable value having a ‘raw’ source point (i.e. the value received from SCADA), an auditable value point (the value selected as the end of gas period value or estimated according to the rules) and an auditable record point (the record of changes to the auditable value).
OSIsoft’s Advanced Calculation Engine (ACE) was chosen to implement data processing algorithms. ACE is a powerful development tool in combination with the PI Software Development Kit (SDK) because it allows extraordinary flexibility in developing scripts for scheduled or triggered action whether to perform calculations or validation on data or to deliver it. For MDM, data processing scripts were developed that tapped directly into the PI Event Pipe to process data as it arrived, apply the data selection and estimation processes, and populate the audited value set. There was a separate script or ‘strategy’ developed to suit each of the different raw data profiles and also for each permitted estimation method.
In addition, the data needed to be pre-validated so that possible issues could be detected and flagged for human review. A set of validation rules and associated parameters was defined by the business and implemented in ACE modules. Incoming data passed through a validation process with a PI point used to record the history of test results for each site.
A Gas Distribution Network model was developed in OSIsoft’s Asset Framework (AF). AF is a powerful data infrastructure tool because it allows a logical, hierarchical structure to be defined that combines data from any source (such as PI points interfaced to SCADA, databases or other systems) with calculations or static data. This is a great way to abstract any complexity in deriving data from the users. For example a user looking for the gas volume through a gate station can simply navigate to the gate station in the model and look at its volume attribute. Behind the scenes there may be multiple data points and complex calculations but all of that is hidden from the user.
The MDM AF model included all of the relevant SCADA-based PI points, as well as relatively static network data and processing instructions. These include real-time processing and validation methods to be applied to the data at each site in the network. The AF model was therefore an elaborate configuration structure for the ACE modules which would parse the structure on start-up and then periodically refresh it.
A user interface was developed in ASP.NET with the essential functions of:
- Checking PI for gas dates requiring validation (i.e. where either new or modified data was available)
- Loading audited values and any validation issues for human review
- Permitting users to manually alter audited values or override validation issues prior to publishing the data to the EC system
- Requiring users to follow an approval process where changes were made and capturing an audit trail
- Providing a means for users to flag the data for publication
To integrate the PI data with EC, an interface was developed allowing EC to publish a list of sites it required data for; this list was received and written to the AF back-end SQL database. An ACE module was implemented to read the site list, check for any data flagged to be sent, construct, and finally deliver a message to BizTalk.
The delivery of correctly validated data to EC – and from EC through to the market – was a critical task with regulatory rules and penalties for non-compliance. To ensure any issues were detected as soon as possible OSIsoft’s Notifications were used and configured to monitor (among other things) the I/O rate of the SCADA interfaces and raise an alert whenever it fell outside expected parameters. Alerts were also configured for failure to deliver data to BizTalk or for a variety of internal ACE process failures where the process itself might appear to be running but without correct data processing.
The resulting solution was a reliable and scalable data infrastructure integrated into the existing enterprise systems, providing powerful data processing features, a highly-configurable AF model, and an intuitive user interface.
- A national data infrastructure was established, providing for the current business needs, and also providing a platform on which future applications could be developed in years to come.
- Complicated source data processing in spread sheets was replaced by a single enterprise system bringing consistency, efficiency, and automation to the process.
- A user interface provided a valuable and easy to use tool for operators while enforcing a consistent business process.
- Integration with the EC target system was automated and any data processing or delivery issues were detected as soon as possible minimizing the risk of non-compliance.