Communication requirements

 

The communication inside the system levels – “system”, “object”, ”process” – is a core system’s element. Therefore the design of the whole system always adjust to the existing artifacts like communication infrastructure, technology, transmission medium etc.

The Real Time work regime requires the following system communication specifics:

  • the data flow between objects and the different Control Centers on the various system levels is done in parallel and independently
  • the transfer of technological data is well defined, i.e. there always exists response time interval even for the worst case scenario, and overrun of that time is a clear sign of connection loss
  • the connections used in the system has to be static – permanent. “Dynamic connections”, where established on demand and then closes are undesirable
  • the time requirements about data transfer are guaranteed independently of:

                  –  the data transfer velocity

                  –  distance among objects and the Control Center

                  –  information volume of the object

                  –  amount of process changes etc.

  • data transfer is protected with Hamming distance = 4 for information, and Hamming distance = 6 for commands
  • the communication infrastructure for the sake of automation is separated from other corporate or public networks

One could accomplish the above strict regulations by:

  • using direct telemechanical channels between Control Center and its adjacent objects by using “point – point” or “line” topology based on copper, optical or wireless channels, and
  • using special telemechanical protocols, i.e. IEC 60870–5–101.

It is clear that these strict conditions for Real Time environment require direct physical/cable connection between the Control Center and the objects.

A few well established communication technologies, based on:

  • IEEE 802.3 Ethernet or 802.11 WLAN
  • Carrier Sense Multiple Access with Collision Detection (CSMA/CD) – algorithm for network access
  • TCP/IP (Transmission Control Protocol/Internet Protocol) – network protocol a.k.a. TCP/IP networks,

have replaced the strict conditions mentioned above, and nowadays telemechanical systems simply adjust to the existing and well adopted communication technologies.

In order to define the metrics and approaches for adjustment, it is required to define and advantages and disadvantages of these networks in the frame of Real Time work regime. Main problems rooted inside their basics and building concepts, i.e.:

  • the environment does not allow direct access among subscribers. The connection is build dynamically and is established on demand. The connection build time depends on the subscriber count, traffic and other factors and thus is undefined. The same holds true for the information transfer.
  • this kind of networks allow the appearance of conflicts. Indeed the CSMA/CD algorithm, provides conflict resolution, but in case of intensive information flow and high network access count, a condition could be reached which blocks subscribers to access the network.
  • to overcome greater distances information might pass through various networks and environments, i.e. the use of 3rd and 4th layer of ISO model – Network Layer is used for routing the packages. On this level, the route and consecutiveness of packages are undefined, as well as package loss is allowed by definition.
  • passing through various networks and environments means mixing system information flow with regular (public) traffic and lack of physical or at least logical separation of the information being transferred in Real Time regime from other information flows, whose volume is greatly larger.
  • time for information delivery to a given address is undefined and always depends on the current network traffic. Receiving of information is not validated. Thus time and other criteriа for connection presence or loss are unavailable, which might mislead the operator, by presenting old information.

 

Part of above mentioned disadvantages could be diminished by appropriate system measures, but could not be fully removed as they are part of the TCP/IP network skeleton.

By the above mentioned writing it is clear that the information transfer in Real Time work regime, i.e. the time determination requirement, could not be accomplished.

There exist other types of networks build upon other principles, like Token Ring – mechanism. Such examples are PROFIBUS networks (or other similar) whose building principles are very close to the requirements of the Real Time, but are not much widespread and therefore unusable.

The advantages of the TCP/IP networks are the following:

  • they are widely used
  • building such network is relatively fast and cheap
  • conflicts and problems occur mainly if there is high traffic and overloading, otherwise it allows high speed data transfer
  • It allows great variety of configuration and therefore adapts to certain requirements and conditions

 

System measures, which need to be taken in order to adapt to the current communication technology are:

  • installing of telemechanical protocol IEC 60870-5-104 on the 7th layer of the OSI model and fully using its capability, e.g. using its communication control, Data confirmation etc. Data confirmation allows to find out if any packages are missing on both directions of the connection. Furthermore the Client could control the communication process by timing supervision and thus forming objective criteria for:

              –  detecting of connection losses

              –  channel load supervising

              –  traffic quality controlling etc.

  • send technological information with time tags – time of event occurrence. Thus one overcomes the problem with time indefiniteness when transferring information.
  • separate information flows, carrying technological information, from all others. This could be accomplished by logical separation of the network, where the traffic inside a logical network is defined only by the dynamics of the transferred information.
  • organize the program structure of the communication software into many separate and parallel branches (processes), where each paired connection is wrapped by “Client – Server” software. Thus a “point – point” topology is developed.
  • create mechanisms which secure the actuality, validity and completeness of the information.

 

 

Communication and data validity

 

The system supports parallel and independent communication through various communication environments, technologies and protocols.

The RTU communication is performed on three levels:

  • “process” level, where the data transfer with Intelligent Electronic Devices – IED is found
  • “object” level – here is the data transfer with other RTUs – Data Concentrator
  • “system” level – here is the data transfer with system structures found on higher control levels, like local Control Stations, SCADA, Human Machine Interface – HMI, Historical Information Servers – HIS and various existed external systems is carry out.

The binding with the process is done by parallel or serial. Parallel inputs are the analog or digital I/O’s that come from relay repeaters, sensors etc. Serial inputs are used for IED data.

The RTU support the following physical interfaces:

  • RS 232
  • RS 485
  • RS 422
  • Ethernet and other standard interfaces

Thus it could used the following communication mediums:

  • copper cable
  • optical cable
  • TCP/IP networks
  • PLC channels
  • radio networks
  • GPRS networks
  • VLAN networks etc.

On “process” level the RTU supports the following local networks:

  • CAN-BUS
  • MODBUS
  • I2C-Bus
  • SPI-Bus
  • other Buses could be adjusted (e.g. PROFIBUS etc.).

For data transfer with IED one could use the following protocols:

  • IEC 61107, IEC 62056
  • IEC 60870-5-101/104
  • IEC 60870-5-103
  • IEC 61850 Client
  • MODBUS etc.

On a system level, including the Control Center, the following protocols are used:

  • IEC 60870-5-101
  • IEC 60870-5-104
  • private protocol etc.

The required count of communication modules could be installed, depending on a given configuration, location of the Control Station, RTU and other objects. Each communication module has its own stack, used to adapt to a given communication infrastructure, e.g.:

  • communication environment
  • communication technology
  • physical interface
  • protocol etc.

Furthermore the Control Station as well as the RTU could work in parallel and independently as a Server for arbitrary count of Clients and as Client for arbitrary count of Servers.

The parameters of the communication and protocols for information exchange are separately defined for each Server, respectively Client.

The same holds true for communication protocols used.

As a server the RTU supports various Data Images for each Client, where the client stores its information with predefined/static size.

 

 

Data consistency

 

In latest years, as the communication between Control Centers and its adjacent objects is based mainly on TCP/IP protocol, and the sources of technological information are mainly Intelligent Electronic Devices – IED, a problem with the data consistency occurs, i.e. to deliver valid and up-to-date information from the objects to the Control Centers becomes more important and complex problem to solve.

It appears when the connection is lost on various levels. The communication disturbances appear on connection loss or channel overload.

The TCP/IP does not support functions for message conformation or any kind of feedback. This is done by the 7th – the application layer of the OSI model, i.e. IEC 60870–5–104 protocol.

Messages send from Server to Client, might be partially or completely lost and it validates the count of the received messages. Undelivered messages are have to be identified and resend. The last could become a cyclic process in case of bad channel.

Meantime other events occur which are also send to the Control Center with higher priority than undelivered messages that wait for resend.

Another problem is when the communication between RTU and Control Center is fine, but between RTU and IED (or any adjacent RTU object) is broken. After connection recovery, some of the devices are able to resend the missing information with the time of arise – time tag, which is already out-of-date. But the other devices that could not resend any missing information create lapses inside the records of the Control Station.

Any data consistency violation leads to information loss inside the Control Center, which makes the visualization of system objects’ state out-of-date, and/or creates lapses inside the records.

The problem becomes even more complex as this reflects practically each Client connection to the RTU, thus each connection has to be made safe, evading any data consistency violation.

As can be seen this problem cannot be solved with simply solutions, like “circular buffer” and similar measures, although this is a often practiced. There a composite solution is needed which reflects issues on all levels.

 

In the proposed solution, the communication software at all levels has complex structure of “queues” and other system mechanisms to ensure data consistency, regardless of the number of established connections, the status of communication paths etc.

This means that, in the absence of communication, the RTU has structures of sufficient diversity and depth to store the missing data and transmit them upon restoration of the connection. This applies to each connection and subscriber separately.

The data capacity is directly connected to the existing memory inside the RTU and its configuration settings which are defined at installation time.

In case of established or restored connection the actual data is immediately send. Unconfirmed messages, resulted because of channel overload, are classified and stored in various types of queues, and resend to the Control Center with lower priority.

The queues’ content is being reordered and updated, i.e. all successfully delivered to the appropriate Control Center messages are removed and new messages for dispatch are added. The process is repeated until all queues are empty.

In case of connection loss all up-to-date, data is saved in the archive database and after recovery:

  • initialization is started
  • all new changes are send
  • then all unconfirmed lower priority data that is waiting because of connection overload is also send, and
  • at the end the archived data are send

Because of the different user relevance the mentioned above structures are established for various data types, i.e. signals, measures, value counters etc. and the above description is valid for each independent Client connection.

 

 

Parallel structures and communication configuration

 

Development of the communication infrastructure is a particular case of building decentralization and parallelized software- and hardware structures. This approach diminishes any device injury and leads to better dynamical qualities of the system.

A rule of a thumb is that the communication has to be based on parallel and independent processes.

The communication between RTU and Intelligent Electronic Devices – IED is accomplished through Ethernet – IEC 61850 or RS 232/485/422, and 20 mA current loop for all other protocols.

The communication specifics of RS 485/422 are that a subscriber disturbance does lead to break up of the whole line, therefore, the communication lines’ distribution of IED is done as:

  • Relay protections, which transfer commands for control of devices by protocol IEC 60870–5-103, has to use “point – point” topology, i.e. a line with only one subscriber
  • IEDs, like P, Q, U, I meters, energy meters and other measure devices, are grouped in not more than 4 devices per line. Thus the line count increases, but all parallel connections increase the stability of the system.

 

 

Reserve of important system structures

 

It is recommended that all communication (Front_End) Servers installed in the Control Centers, are doubled, work in parallel and have equal access rights.

In case of Server failure or connection lost, its adjacent subscribers are immediately pass to the other working machine.

Each communication Server has to support two channels with each adjacent object – main and reserve. The data exchange with both Servers has to be carried out continuously and in parallel. Thus each Server has valid and up-do-date information and could continue to work without subsequent initialization and other procedures in case of Server change.