Melbourne Water has a number of different operational systems developed to suit different purposes from water treatment plant operation to energy data and water quality data.  Like other large organisations in similar positions they faced issues in accessing data from multiple sources for day-to-day business use. 

In 2013 Melbourne Water were seeking an enterprise data infrastructure to facilitate central data processing and reporting and to improve access to data throughout the business.

GTS in partnership with Schneider Electric were engaged to design and build an Operational Data Store (ODS) with the key objectives of:

• Establishing a scalable enterprise historian initially with about 120,000 points
• Interfacing with three key data sources including Hydstra and Stark
• Implementing a representation of data quality and validity suitable to all business users
• Providing dashboard displays for specific water quality metrics
• Supporting business data calculation requirements

Technical Situation

Although the source systems and a BizTalk middleware were well-established, there was no central historian of any kind.

The initial data requirements were identified as interfacing to the primary SCADA system and largest source (Mosaic), the energy usage and billing system (Stark) and the waterways hydrology system (Hydstra).  It was understood that an interface was available for Mosaic but Stark and Hydstra data would be delivered in file structures of varying complexity.  Additionally, the files may contain additions, modifications or deletions of data.

Although each system had a point naming standard, data structure and method of representing data quality each was different.

Key reports and dashboards were produced by the business, but these were not automated e.g. through SSRS but were labour-intensive requiring multiple people to fetch data so that an analyst could consolidate and process the data into a report.

The scope was reasonably straightforward but there were complexities in interfacing with the data sources, providing a common data structure and aggregating data for reporting purposes. The high-level requirements included:

• Installation and configuration of the historian platform software
• Establishing interfaces to the core data sources for live data updates
• Implement a common representation for data quality and validity across the different data sources
• Provide data analytics/aggregation to meet operational dashboard requirements
• Implement a web-based multi-tier dashboard

Solution

GTS developed a solution based on OSIsoft’s PI platform that provided a reliable and scalable solution suitable for the historical data from the three core systems in phase one as well as planned expansion to include several other data sources in the future.

The multi-server architecture offered substantial storage and processing performance with dedicated hosts for the data archive and data processing tasks; reliability with primary and secondary archive hosts; while the ability additional servers offered substantial scalability for future needs

Custom Hydstra Interface

One of the challenges was interfacing to the Hydstra system for live data updates.  There was no off-the-shelf direct interface option, but data could be dumped to logs on a scheduled basis.  However, the file structure was a complex one, with data changes represented in ‘blocks’ that signify additions, deletions or modifications to PI events.

It was therefore necessary to parse the file, insert any events that were not already in PI, modify any that were, and remove any that were in PI but not in the file.  Although PI provided a very efficient and highly configurable file parser in its Universal File Loader (PI UFL) it didn’t have the capability to process the files in their raw form.

To resolve this issue GTS developed a Hydstra processing interface to parse the ‘blocks’ of data provided in a Hydstra export file and process the data into PI.  The interface utilised the PI SDK to:

• Read, cache and periodically refresh a list of target PI points
• Open and process files having a configurable filename pattern
• Writing to separate members of the PI database collective
• Inherently provide buffering by leaving files in the upload folder in the event of a connection failure

Custom NEM Interface

The energy data presented a similar challenge with no out-of-the-box interface option, and data files in a standard National Electricity Market (NEM) file structure.

These are also a more complex file structure requiring logic to construct tag names and extract value and quality data that was not supported by the PI UFL. However, there wasn’t the same need to synchronise the data in the PI system as there was with the Hydstra files, so the PI UFL could still be used if the files could be converted into a suitable format.  So, for the NEM files, GTS developed a pre-parsing application designed to monitor a folder location for NEM file drops, process and output data in a format ‘digestible’ by the PI UFL.

A hierarchical data model was developed in the PI Asset Framework (PI AF) for each source system.  PI AF is a powerful data infrastructure tool because it allows a logical, hierarchical structure to be defined that combines data from any source (such as PI points interfaced to SCADA, databases or other systems) with calculations or static data.  This is a great way to abstract any complexity in deriving data from the consumers of the data and to implement a standard structure across different sources.

PI AF design provided a PI AF Element for each source point with the view that data consumers using the data would reference the AF model as the data source rather than the using raw PI Points.  This provided a meaningful, logical structure to data consumers while abstracting the complexity in calculations in particular providing a means to ‘translate’ different data quality representations from each source system into a standard set of values to suit all users.

PI ProcessBook

OSIsoft’s PI ProcessBook provided the platform for the development of the dashboard graphics. The built-in graphic objects are integrated with PI AF through an ‘Element Relative Display’ allowing the graphic to automatically update with data from the PI AF element selected by the user.  This allows users to view a dashboard for a selected site, or to navigate to a rolled-up view at the region, line of business or enterprise levels.  The PI ProcessBook displays were presented using SharePoint, providing access to users across the business.

GTS trained Melbourne Water ODS users to use PI Datalink Microsoft Excel Plug-in for ad hoc data analysis.  PI Datalink is a very useful tool for browsing PI and PI AF data sources, fetching raw data values or applying statistical functions to the data stream as it’s imported into Excel.

GTS also provided PI ProcessBook training, enabling users to develop their own graphics, relevant to their business processes, combining data from multiple sources and implementing analytical formulas where necessary.

The result was a powerful and scalable data infrastructure integrated into the existing Melbourne Water business with live data feeds to all data sources.

Valuable analytics were provided through PI AF and the GTS plug-in toolkit to present a logical navigable data model.  Dashboard reports that were previously a time-consuming manual process were instantly available on the corporate intranet, providing users across the business with easy access to operational data and empowering them to develop their own analysis, reporting and graphics extensions.

Project Outcomes & Benefits

The resulting solution provided:

• A single source of truth for operational data
• Analysis in the hands of users
• Complex reporting tasks automated
• A powerful and scalable data infrastructure integrated into the existing Melbourne Water business with live data feeds to all data sources.
• Dashboard reports that were previously a time-consuming manual process were now instantly available on the corporate intranet
• Users to develop their own analysis, reporting and graphics extensions.