MELBOURNE WATER – OPERATIONAL DATA STORE (ODS) PROJECT
Melbourne Water is owned by the Victorian Government and is responsible for the day-to-day management of water supply catchments, treatment and supply of drinking and recycled water, removal and treatment of Melbourne’s sewage, and management of waterways and major drainage systems in the Port Phillip and Westernport regions.
Water infrastructure includes 10 interconnected storage reservoirs totalling more than 1,800 billion titres, 14 water treatment plants, 38 service reservoirs located around suburban Melbourne ranging in capacity from 2 to 250 million litres and thousands of kilometres of pipes.
Melbourne Water has a number of different operational systems developed to suit different purposes from water treatment plant operation to energy data and water quality data. Like other large organisations in similar positions they faced issues in accessing data from multiple sources for daily business use. In 2013 Melbourne Water were seeking an enterprise data infrastructure to facilitate central data processing and reporting and to improve access to data throughout the business.
GTS in partnership with Schneider Electric were engaged ta design and build an Operational Data Store (ODS) with the key objectives of:
- Establishing a scalable enterprise historian initially with about 120,000 points;
- Interfacing with three key data sources including Hydstra and Stark;
- Implementing a representation of data quality and validity suitable to all business users;
- Providing dashboard displays for specific water quality metrics; and
- Supporting business data calculation requirements.
Although the source systems and a BizTalk middleware were well-established, there was no central historian oi any kind.
The initial data requirements were identified as interfacing to the primary SCADA system and largest source (Mosaic), the energy usage and billing system (Stark) and the waterways hydrology system (Hydstra) It was understood that an interface was available for Mosaic but Stark and Hydstra data would be delivered in file structures of varying complexity. Additionally the files may contain additions, modifications or deletions of data.
Though each system had a point naming standard, data structure and method of representing data quality each was different.
Key reports and dashboards were produced by the business, but these were not automated e.g. through SSRS but were labour intensive requiring multiple people to fetch data so that an analyst could consolidate and process the data in a report.
The scope was reasonably straightforward but there were complexities in interfacing with the data sources, providing a common data structure and aggregating data far reporting purposes. The high-level requirements included:
- Installation and configuration of the historian platform software;
- Establishing interfaces ta the core data sources for live data updates;
- Implement a common representation for data quality and validity across the different data sources;
- Provide data analytics/aggregation to meet operational dashboard requirements; and
- Implement a web-based multi-tier dashboard.
GTS developed a solution based on OSIsoft’s PI platform that provided a reliable and scalable solution suitable for the historical data from the three core systems in phase one as well as planned expansion to include several other data sources in the future. The multi-server architecture offered substantial storage and processing performance with dedicated hosts for the data archive and data processing tasks; reliability with primary and secondary archive hosts; while the additional servers offered substantial scalability far future needs.
Custom Hydstra Interface
One of the challenges was interfacing to the Hydstra system for live data updates. There was an off-the-shelf direct interface option, but data could be dumped to logs on a scheduled basis. However, the file structure was a complex one, with data changes represented in ‘blacks’ that signify’ additions, deletions or modifications to PI events. It was therefore necessary to parse the file, insert any events that were not already in PI, modify any that were, and remove any that were in PI but not in the file. Although PI provided a very efficient and highly configurable file parser in its Universal File Loader (U FL) it didn’t have the capability to process the files in their raw form
To resolve this issue GTS developed a Hydstra processing interface to parse the ‘blocks’ of data provided in a Hydstra export file and process the data into PI. The interface utilised the PI software Development Kit (SDK) to:
- Read, cache and periodically refresh a list of target PI points;
- Open and process files having a configurable filename pattern;
- Writing to separate members of the PI database collective; and
- Inherently provide buffering by leaving files in the upload folder in the event of a connection failure.
The result was a powerful and scalable data infrastructure integrated into the existing Melbourne Water business with live data feeds to all data sources.
Custom NEM Interface
The energy data presented a similar challenge with no out-of-the-box interface option, and data files in a standard National Electricity Market (NEM) file structure. These files utilised a more complex file structure requiring logic to construct tag names and extract value and quality data that was not supported by the PI U FL. However, there wasn’t the same need to synchronise the data in the PI system as there was with the Hydstra files, so the LIFL could still be used if the files could be converted into a suitable format. So for the NEM files, GTS developed a pre-parsing application designed to monitor a folder location for NEM file drops, process arid output data in a format ‘digestible’ by the UFL.
A hierarchical data model was developed in the PI Asset Framework (AF) for each source system. AF is a powerful data infrastructure tool because it allows a logical, hierarchical structure to be defined that combines data from any source (such as PI points interfaced to SCADA, databases or other systems) with calculatior1S or static data, this is a great way to abstract any complexity in deriving data from the consumers of the data and to implement a standard structure across different sources.
The AF design provided an AF Element for each source point with the view that data consumers using the data would reference the AF model as the data source rather than the using raw PI Points. This provided a meaningful, logical structure to data consumers while abstracting the complexity in calculations in particular providing a means to ‘translate’ different data quality representations from each source system into a standard set of values to suit all users.
Asset Framework (AF)
Analytical components in the AF model were provided by formula data references for the simpler calculations; and in the case of more complex algorithms, the GTS AF plug-in toolkit provided a number of useful custom data references.
The ability to extend the native functionality by using custom-designed data references is a powerful feature of AF and GTS has developed a useful tool kit to provide features including:
- Complex aggregation of Attribute values based on AF structure and attribute categorization;
- Converting an integer representation of bit states to textual descriptions of active bits using enumeration sets;
- Representing data from other database sources into a collection of PI events;
- Shifting time periods e.g. to overlay data from different times in a single chart; and
- Fetching the time components of an AF data request (e.g. the day, month, year etc.).
A dedicated AF model was also built to support the dashboard reporting requirements which involved an aggregation of data from a series of water quality measurement points over a period of time to show a traffic light status, depending on the severity and quantity of excursions tram predefined quality indicators.
OSIsoft’s ProcessBook provided the platform for the development of the dashboard graphics. The built-in graphic objects are integrated with AF through an ‘Element Relative Display’ allowing the graphic to automatically update with data from the AF element selected by the user. This allows users to view a dashboard for a selected site, or to navigate to a rolled-up view at the region, line of business or enterprise levels. The ProcessBook displays were presented using SharePoint, providing access to users across the business.
The result was a powerful and scalable data infrastructure integrated into the existing Melbourne Water business with live data feeds ta all data sources, valuable analytics were provided through AF and the GTS plug-in toolkit to present a logical navigable data model Dashboard reports that were previously a time-consuming manual process were instantly available on the corporate intranet, providing users across the business with easy access to operational data and empowering them to develop their own analysis, reporting and graphics extensions.