header_image

SCADA Alarm History - 1/2

Often production and reliability departments have the necessity to go back in time and analyse past process data. All SCADA packages provide trending and alarm management features. SCADA application,

September 2, 2017

Often production and reliability departments have the necessity to go back in time and analyse past process data. All SCADA packages provide trending and alarm management features. SCADA application, though, are tailored for short term purposes, so that a new class of applications appeared in the market to overcome this limit. They are generically called historians. The main feature of these packages is the ability to connect to field data sources, including SCADA, and maintain a long term database of significant process points or alarm events. This data is then available for post analysis or reporting purposes. In this and the next post I am going to present an application I developed to build such a long term database for Citect Alarms, without purchasing external packages but using only Citect, Windows and .NET development tools already available.

System

The system where the application runs is composed by several networked computers with the following roles:

  • Citect SCADA servers: 5 Citect 7.2 and 7.4 Citect servers, divided in 3 clusters with two Primary/Standby couples. These servers are the data source (DS).

  • SQL Server 2014 server. This computer is the data target (DT).

Application Scope

Collect in DT all the alarms generated by DS in real-time. The saved alarm record scheme needs to contains at least the following fields:

  • Alarm ID

  • Alarm Description

  • Alarm Location

  • Time ON

  • Time OFF

  • Duration

First trial - configuration only

As first tentative, the easiest way has been tested.

  • A new Database named CitectAlarms has been created in the DT, containing the desired tables.

  • In all Citect servers, a new SQL device has been created and linked to the database through an ODBC connection. This is the Citect standard way to transmit alarm events to SQL server.

This standard implementation proved to be quite problematic, especially due to the insufficient performance of the ODBC connection to a remote SQL server against the necessities of a live SCADA instance. The network environment is not the most straightforward, including VLAN routing and inter-domain connections. After a few weeks of operations, the solution has been decommissioned as the results were unsatisfactory.

Final solution - .NET business layer

To avoid the problems found with the standard implementation, I had clear that it was necessary to avoid charging Citect with an expensive remote ODBC interactivity. A SCADA application is usually quite heavy as it is normally required to process data in a real-time fashion, which doesn't line up well with remote database interactions which are normally accepting asynchronous patterns and discontinuous access.

A good choice in cases like this is binding the data source with an asynchronous queuing service, which offers high availability and manages the burden of the remote transactions as delegate. There are many good open source packages for this purpose, but in this case I had already available Windows machines, so the natural choice was MSMQ.

Citect offers a feature called AlarmQueue. Once this feature is enabled, anytime a new alarm event is detected by the alarm server, a new entry is generated to an internal queue. The queue can be read using Cicode in real-time. The job to do is then to send these entries to a MSMQ queue in DT which in turn transforms the data into a SQL server table record. The advantages over the standard way are:

Reduced Citect load

All the heavy work is delegated to external processes, so Citect remains fully responsive in all situations. When a new event is detected, the data is sent to the local MSMQ, which is much less expensive than performing an INSERT into a remote SQL server table through ODBC. When dozens of new events are generated in seconds, this really makes the difference: instead of a perceivable slowing-down SCADA response the situation goes completely unnoticed to the SCADA user.

Better Availability and Reliability

Citect interacts with the local MSMQ service, which in terms of availability and reliability is much better than an ODBC connection to a remote server. The delivery of the Citect events is completely detached from the availability of the target SQL server. We pass from a weak to a strong dependency for Citect. In detail:

  • Maintenance on the target SQL server: if SQL services on the DT server are down for maintenance purposes, all the messages are kept in the server MSMQ queue. As soon as the SQL services are online, the queue is emptied without data loss.

  • Target server failure: as the sent messages are flagged as persistent, they are saved on the disk by MSMQ when received. If the server fails when there are message in the queue, again there is no data loss. Once the server is restarted, the pending messages are recovered from the disk and delivered to the SQL server.

  • Network failure: if a network breakdown prevents the Citect server from connecting to the target SQL server, the messages are kept by MSMQ outgoing queue in the Citect server. As soon as the target gets back online, the messages are then delivered and processed, all managed by MSMQ, with no impact to Citect.

In all these scenarios, the normal Citect ODBC connection would both slow down the SCADA application (trying to connect to the remote SQL server) and lose data.

Solution Workflow

Here is the summary of all the steps and artifacts in the solution.

Enable Citect AlarmQueue

The Citect AlarmQueue feature needs to be configured. To do this the appropriate entries must be added to the citect.ini file in the Citect servers:

Read AlarmQueue entries with Cicode and call .NET NCF.Messenger.DLL

A never-ending Citect Task needs to be started, whose duty is to read the AlarmQueue and forward the entries to MSMQ. The Cicode looks like this:

To be noted:

  • This task is started at startup (Citect function TaskNew).

  • In a multiprocess Citect configuration, the task should run under the Alarm Server process.

  • The functions _NCF_DLL_Open() and _NCF_DLL_Call() are Cicode functions defined in a different Citect module. They serve to call a .NET DLL written in F#, named NCF.Messenger.DLL (later in this post) where MSMQ is engaged. There are possibly other ways to write to MSMQ using Citect Ports, but I couldn't find enough documentation and in any case using a DLL is straightforward. These functions are essentially wrappers for the Citect library functions DLLOpen and DLLCall.

  • The excerpt reported here shows only part of the real application code. In particular, the real application takes into account that at startup, the Citect alarm server initializes the alarm cache with entries that were saved previously, and also synchronizes with the possible standby server. So the real Cicode script sends this initialized cache before focusing on the new alarms sent to the AlarmQueue. The full script can be seen in GitHub (NCF_Ext, NCF_DLL)

  • In the script a non-blocking QueueRead call has been used, so the task can do other things (not shown here) while the queue is empty. In different circumstances we could use the blocking version, that would freeze the task until new entries are available, with maximum CPU use optimization.

NCF.Messenger.DLL - send messages to MSMQ

This .NET DLL is developed in F#, using Microsoft Visual Studio 2017 community edition. The most significant parts of this source code are:

where:

  • Sender.fs contains the .NET logic that connects to the remote MSMQ queue.

  • Exports.fs contains the native Win32 entry point for Citect. Prior to version 2015, Citect didn't allow direct calls to .NET, but only to native code. To create native entry points in a .NET DLL we can use Interop Services, but there are open source packages available in NuGet that make the job easier, hiding all the plumbing. In this case I used the DLLExport NuGet project.

  • If Citect is running as 32 bit, the DLL has to be built upon that platform configuration. This is configured in the project settings in Visual Studio.

  • For Citect to correctly load the DLL (and its dependencies), it must be located in a searchable directory or the proper path must be given when calling DLLOpen; the current project implementation copies files in the same directory as citect32.exe.

See the Visual Studio project on GitHub

MSMQ need to be activated

MSMQ is an optional Windows feature, and is normally part of all recent Windows versions. but is not activated by default. Activating MSMQ is straightforward trough the Control Panel. I won't spend more on this, just Google it. It must be activated in all the involved computers (both DS and DT).

Creating the MSMQ queue in the target server

The target server (DT) must contain the destination MSMQ queue to be set in the previous sources in place of the placeholder. This task is also straightforward and plenty of information is available in the web. The created queue must be a simple non transactional queue.

Conclusion

In this first post I show how it is possible to collect alarm events from a Citect installation, composed of one or more servers, and send them to a target server as MSMQ queue messages. This is done without third party packages, but only using the native features offered by Citect and Windows. Once the messages are in the target queue, they can be read using the normal MSMQ Windows libraries. What I did as part of this application is create a continuous SQL server job that runs a SQL.NET stored procedure, written in F#. This is the subject of the second post in this series.

Related articles