Motorola ACE3600 RTU Troubleshooting Guide

Motorola ACE3600 RTU Troubleshooting Guide

ACE3600 RTU often plays a triple role of "on-site data acquisition+local logic judgment+remote communication" in SCADA scenarios such as pump stations, sewage treatment, substation monitoring, distribution automation, communication station monitoring, and remote meter reading. During field maintenance, many failures are not caused by single module damage, but by the superposition of multiple factors such as power margin, extended frame configuration, communication link parameters, I/O filtering and hold value settings. When replacing or expanding the ACE3600 RTU, it is recommended to treat it as a small control system for inventory purposes, rather than just replacing one board.

Perform system inventory before replacement and expansion

The ACE3600 RTU adopts a modular structure of rack, power supply, CPU, and I/O modules. The rack can choose 0, 3, 5, 7, or 8 I/O slots. The 0-slot rack is only equipped with power supply and CPU; The 8-slot rack can be wall mounted or installed in a 19 inch cabinet. When replacing a discontinued module on site, the first step should be to record the slot position, order number, I/O type, number of terminal blocks, and on-site wiring method of the original module. The I/O module of ACE3600 supports hot swapping, but hot swapping should only be applied to the module itself and should not be used to disassemble the backplane, expand power cables, or communication cables when they are live.

The second step is to verify the CPU capability. Both CPU 3610 and CPU 3640 use 32-bit processors and real-time multitasking firmware, but I/O expansion must use CPU 3640 with Eth1 Ethernet port. If the original host rack CPU3610 on site is planned to add expansion frames, simply replacing the expansion module will not achieve expansion, and the CPU must be upgraded and the STS site configuration rechecked. There are SI1 and SI2 serial ports as well as PI1 and PI2 plug-in ports on the CPU module. SI1 can be configured as RS232 or RS485, SI2 is RS232, and CPU3640 also comes with a 10/100BaseT Ethernet port. After replacing the CPU, the serial protocol, baud rate, IP address, link ID, and FEP/IP gateway address should be checked to avoid "hardware replacement but communication still disconnected".

The third step is to backup STS configuration and applications. The site configuration, I/O link table, KLV/PDV predefined values, communication ports, and IP conversion table of ACE3600 all rely on STS management. If only the hardware is restored without downloading the configuration after replacing the module, the CPU may not be able to recognize the actual I/O combination, and the expansion module may also restart repeatedly due to inconsistent firmware versions.

I/O expansion failure: first check frame number, LAN, and initialization

The I/O expansion of ACE3600 is based on 100Base-T full duplex Ethernet. A single extension frame can directly connect the Eth1 port of the extension module to the Eth1 port of the main CPU through a cross LAN line; For multi frame systems, an extended LAN switch needs to be installed in the main rack I/O slot. A single switch can connect up to 7 extended frames, and a dual switch can connect up to 13 extended frames, forming a configuration of approximately 110 I/O modules. The extended frame includes an extended power supply, an extended module, and an I/O module. The extended module is installed in the CPU slot of the extended frame and communicates with the main frame through a dedicated LAN.

The common issue of "extension frames not going online" on site can be investigated in the following order: first, check the rotation frame number switch on the extension module, ranging from 1 to 13, where A, B, C, and D represent 10, 11, 12, and 13, respectively; Changing the frame number during operation will not take immediate effect and requires restarting the extended frame. Secondly, check if shielded Cat 5E FTP cables are used for LAN cables. For single frame direct connections, crossover cables should be used, and for multi frame connections, available ports of the extended LAN switch should be connected; The Eth1/M port of the switch is reserved for the main CPU and cannot be connected to ordinary computers, routers, or other non expansion devices, otherwise it may cause communication delays and abnormal expansion systems. Thirdly, observe the status of the MCOM LED and STS hardware testing of the expansion module. The main CPU defaults to waiting for the initialization of extension frames for about 60 seconds. Extension frames that fail to complete the timeout will be ignored until the next main CPU restart. Fourth, check the firewall address list. If the main frame has enabled a firewall and defined an address range, it may block communication from extended frames to the main frame; Either manually add the extended frame IP, or keep the address list empty to allow communication.

After the expansion module is powered on, it will load the expansion firmware image from the main CPU, then load user files, register the actual I/O module, and establish EMI communication. If the communication interruption between the extended frame and the main CPU exceeds the configurable time (about 1 minute by default), the extended module will restart and reinitialize. Therefore, when observing periodic disconnection of expansion frames on site, do not only replace the expansion module, but also check the LAN line shielding grounding, switch ports, main CPU load, and expansion power supply voltage drop.

Power supply and backup power: Don't just focus on "being able to light up"

The ACE3600 power module has multiple inputs: default 10.8 to 16V DC low-end power supply, 18 to 72V DC power supply, 100 to 240V AC power supply, and version with battery charger. The version with charger can be equipped with 6.5Ah or 10Ah sealed lead-acid backup power, supporting automatic switching of main power loss, temperature compensation charging, overcharge protection, and battery testing. During on-site maintenance, if the RTU frequently restarts or the CPU resets during wireless radio transmission, priority should be given to measuring the power module output and battery status, rather than repeatedly replacing the CPU.

When cascading power supply for extended frames, both power consumption and voltage drop must be calculated simultaneously. The extended power module can extend the main frame power supply to the extended frame, or cascade from the previous extended frame to the next extended frame, but the total output of the main power supply cannot be overloaded, and the input voltage of each stage of the extended power supply cannot be too low. In engineering, it is recommended that the input of each extended power supply after cascading should not be less than about 11V DC; The output voltage of the main power supply is about 13V DC when there is no battery, and it is determined by the battery voltage after power loss when there is a battery. If the voltage drop of the cable is too high, the DC cable should be shortened, the number of cascaded frames should be reduced, or an AC/DC power module should be used instead in the extended frames. The expansion power module is equipped with 4A and 8A slow melting fuses, which protect the expansion frame and cascade circuit respectively. The same specifications must be used when replacing the fuses on site.

In terms of battery maintenance, STS hardware testing or user program can be used to perform battery testing, including turning off the charger, discharging, and measuring capacity. If the battery fails, the charger will not continue charging and report the failure status to the CPU. For unmanned stations, it is recommended to regularly check battery terminal corrosion, charger alarms, and battery temperature to avoid "everything is normal when the mains power is normal, and the RTU immediately disconnects after power loss".

Communication troubleshooting: MDLC, IP conversion table, and ports

The main communication protocol of ACE3600 is MDLC, based on the OSI seven layer model, supporting conventional radio, cluster radio, serial port LAN、 Multiple media such as dial-up, with storage and forwarding capabilities, can achieve RTU to RTU, RTU to SCADA center, and multi hop communication. The central side is usually converted from MDLC to the protocol used by SCADA Manager by FEP; FEP maintains the local database when using OPC or Modbus, and acts as the protocol gateway when using TCP/IP gateway. When troubleshooting communication faults on site, the four layers of "physical link link link ID IP conversion table FEP/SCADA" should be confirmed first.

For MDLC over IP, data is encapsulated in UDP/IP and the default UDP port is 2002. Each IP port has an independent link ID, and the IP conversion table maps' site ID+link ID 'to an IP address or hostname. In dynamic IP scenarios, RTU should periodically send messages to FEP to enable FEP to learn its latest address; If RTU does not send for a long time, FEP may retain the old address. When using a host name, static LAN ports require manual configuration of DNS servers, while DHCP or PPP ports can automatically learn DNS. NTP servers can be used for clock synchronization, with an accuracy of approximately 1 millisecond within LAN and up to 100 milliseconds over wireless media.

After the firewall is enabled, ports other than DHCP, DNS, MDLC 2002, NTP, Modbus 502, and extended TCP/UDP ports may be blocked. If there is a situation where "ping is possible but MDLC is not working" on site, the UDP 2002 and IP conversion tables should be checked; If STS can connect but SCADA cannot, check the FEP link ID and SCADA driver configuration. When using an extended LAN switch, it is strictly prohibited to connect office network equipment to the extended LAN to avoid broadcasting or routing traffic interference with extended communication.

The wireless system also needs to verify the Channel Monitor Resolution and First Warm up Delay. All RTUs/FEPs of the same frequency and system should use consistent parameters; When mixing different models of radio stations, take the value that requires the longest time slot and the longest preheating time. Otherwise, typical faults such as "normal single station testing, system polling packet loss" may occur.

I/O module diagnosis: LED, hardware testing and holding values

The I/O modules of ACE3600 are equipped with ERR LED and channel status LED. The DI module has status lights for each channel, the AI module has overflow OF and underamplitude UF lights for each channel, and the AO module has Vout, Iout, and CAL lights for each channel. If both OF and UF lights up in the same channel of AI, it usually indicates that the channel is not calibrated. The DO relay module supports hardware readback, and will report to the CPU when the actual position of the relay does not match the logical requirements.

STS hardware testing is the core tool for on-site troubleshooting. Hardware testing can verify whether the module is online, display configuration and actual values, and temporarily modify DI filtering, AI filtering, and AO output. Freeze Mode allows user programs to continue running while freezing I/O at test values, making it easier to check channels without shutting down. It should be noted that AO calibration and forced output typically require the user program to stop or the module to freeze.

The DI module can be configured with fast capture, event timing, hardware filtering, and counter filtering; The AI module can be configured with 50/60Hz filtering, smooth sampling, input range, and COS variation; DO and AO can be configured with KLV or PDV. KLV represents maintaining the last value in case of module failure or communication failure, while PDV represents outputting predefined values. If there is a situation where the valve remains in place after a CPU failure or the output jumps after a communication interruption on site, the KLV/PDV and Mask settings in the I/O link table should be checked, rather than just suspecting the module hardware.

The DO relay module also supports disabling relay output through the 12V DO control line of the power module. When the 12V DO terminal is short circuited, the relay can operate, and when disconnected, the EE relay returns to the non excited state and the ML relay remains in the state. If on-site output needs to be prohibited during maintenance, this function should be utilized and the user program should be confirmed to be able to monitor the prohibited state. ML relays can also be configured to reset all magnetic holding relay positions during startup, and any modifications should be verified in accordance with process safety requirements.

Clock, encryption, and maintenance points

The internal system clock of ACE3600 is microsecond level, and RTC provides second level backup. The rechargeable lithium battery on the CPU can maintain SRAM and RTC for about 60 days when there is no power. When NTP/GPS is not connected, the clock drift is typically about 30ppm, about 2.6 seconds per day; It is recommended to have a reliable clock source RTU synchronize the entire network daily, or use NTP/GPS. In the expansion system, the clock of the expansion module is synchronized by the main CPU and processes time stamps and timing events in an orderly manner.

MDLC encryption uses a 128 bit key, and each RTU can store 9 replaceable keys. Only encrypted RTUs/IP gateways using the same key can exchange data. Key switching requires synchronization across the entire network and a transition window where both old and new keys are valid. When replacing the encryption system on site, it is necessary to first confirm the compatibility of STS, IP gateway, and RTU firmware versions before performing key download and activation.

In terms of installation and safety, ACE3600 should be installed by qualified personnel in accordance with national electrical regulations; High voltage I/O (>60V DC or>42V peak) should be considered as hazardous voltage and access should be restricted. SELV circuits should be clearly labeled. Pay attention to the heat dissipation gap and shock resistance when mounting the rack, and ensure sealing and waterproof cable entry when installing the NEMA 4/IP65 enclosure. Module replacement uses terminal block coding and TB holders to avoid wiring errors.