Unit is a Frame 9E, gas only, DLN-1 installed around 1997. It has a (factory installed) expansion card TBCB in <C> core and uses the <I> computer for communications (no HMI).
After I added a new input, TBCB card functioned normally for one to two days then stop updating inputs. No alarms, diagnostic or otherwise. I defaulted the card in the IO_Configurator and re-entered all the signals, generated a new IOCFG_C.DAT file and downloaded (and re-booted). Again, all worked well for a couple of days. Not sure if this is a new issue to this unit as the other inputs to this card tended to be very static and were for indication only.
I connected this new point into C_C_MAI26 and configuring it in the IO_Configurator as MA input on TCCB, Signal 12 (26 - 14 = 12 for the expansion card.) Signal worked well with correct configuration and was reading correctly up to the time it froze.
The only other change made to the <I> is that a Modbus connection was added, but I don't see how that could effect things. (?) There was no Modbus connection before.
Does anyone have experience with this card update problem? Suggestions are welcome.
Thanks in advance.
Interesting that there are two (2) extremely similar threads opened on the same day that are extremely similar. The other thread is:
I don't have access to my Mark V Application Manual (GEH-6195), but I think you are trying to say that something is amiss with the TCCB card; generally, TB in the first two characters of a Mark V card designation refers to a terminal board (where field wires, or wires connected to field devices, terminate). Generally, TC in the first two characters of a Mark V card designation refer to an I/O card (an 8-1/2-inch by 11 inch card in one of the cores (<C>, <R>, <S>, <T>, <P>, <PD>, <CD>, <QD1> and <QD2> if so equipped--and possibly a steam turbine PLU module (again, I don't have access to my App Manual).
Both threads refer to newly assigned mA inputs working for some short period of time and then not working, requiring a re-boot of <C> processor to get them working again.
Both threads refer to recent activation of a MODBUS link on an <I> (not a GE Mark V HMI running MS-Windows and CIMPLICITY--but a <I> running IDOS, the first operator interface used with Mark V turbine control systems).
It's not clear if these two threads refer to the same site (both with GE-design Frame 9E heavy duty gas turbines...), but if it is while the fact that on the face of it it shouldn't seem like MODBUS is causing a problem it very could be that MODBUS is using one or all of the newly assigned mA input signals--requesting their status at a fast rate. And it's possible that because of the way <C> multiplexes mA inputs it's getting hung up trying to send the data as quickly as it's being requested. (Multiplexing inputs refers to the way that inputs are read. Inputs to <C> are not deemed (by GE and the card/software designers) to be critical--so the analog inputs to <C> are generally not all read at the same instant in time. Rather they are read one at a time, serially, and so there is a time delay in the update rate in the CDB (Control signal DataBase). And since the <I> may be requesting the data from the newly assigned mA inputs at a rate much faster than once-per-second, <C> might be having a hard time responding, hence the issue with the inputs "freezing.")
This is pure speculation on my part--based solely on similar (but not exactly the same) issues I have experienced before with <C> analog inputs being requested by MODBUS from <C>.
It's also very possible that the Mark V is one of the early Mark V versions (there are Mark V "A" panels; "B" panels; "Hybrid A" panels; and "Hybrid B" panels); sometimes if you run CARD_ID.EXE from a command prompt it will tell you which version of Mark V panel is being used. And the software on the PROMs of the DCC/SDCC and/or LCC/SLCC and TCCB cards might have issues that were solved by newer revisions of PROM software. (The last two characters of the PROM P/N (Part Number) indicate the revision of the PROM software, so CB would be Major Revision 3, Minor Revision 2 (for the third and second characters of the alphabet, respectively.)
Failing all of the above, I would have to say there might be a problem with the TCCB card, or a problem with the MODBUS configuration. (I believe it's possible to ask for different signals via MODBUS at different rates (not Baud rates--the number of times per second), so you might try just stopping MODBUS and then waiting a couple/three days to see if the problem re-occurs. If it does not re-occur, then you've isolated the problem to something related to MODBUS, either the signal names were not entered correctly (possibly the wrong scale type, or mis-typed signals, or the wrong register type, etc.), or the timing of the requests from the <I> is too fast for the way the information is being updated. You could then try changing the timing of the requests for those signals in MODBUS and seeing if that resolves the problem.
If it's not a MODBUS or timing issue, then it's likely a TCCB card issue (try swapping in a new or refurbished TCCA/TCCB card), or a PROM issue (requiring updated PROMs).
I did think of one other possibility--the jumpers for signal isolation on the TBQx card where the new mA signals were terminated. I believe the Mark V does NOT provide power for <C> mA inputs but there IS a hardware ("Berg") jumper for grounding the input loop. Some other control systems have a different grounding system, and putting the hardware jumper in will cause problems for either the Mark V, or the other control system providing the signal. (That's something I didn't suggest to be checked--to see if the input signal is frozen and not updating, which would suggest it's not a Mark V problem at all!) Sometimes, it's necessary to use a signal isolator to prevent ground loops from hampering the signal and the control system(s). Many signal isolators are self-powered.
Finally, it could be that the input signal loading of the <C> mA inputs causes the sending control system to not be able to power the loop--especially if the Mark V was "added" to an existing loop. Most loops can only handle a total of 500 ohms; without being able to look at the Signal Flow Diagrams in Appendix D of GEH-6195 I can't say for certain what the dropping resistor is in the <C> mA input loops.
That's about all I can think of. Please write back to let us know what you find!
Sorry to be so long replying. There's been a lot of hypothesizing about this issue and I also managed to squeeze in a short vacation. :-)
You are correct that the two posts put in about the same time do refer to the same unit and issue. I was not aware that the other gentleman was going to open a post here. Sorry for any confusion.
So some more info on this problem:
- The panel is a "B" panel (based on the CARD_ID.EXE result).
- Additionally the TCCB (used in the TBCB - Loc 7 / TCCB - Loc 3 for optional MA inputs) card is detailed on the build sheets kept on the inside of the <C> core front door. This tells me the optional ma inputs came with the original Mk-V.
- The CARD_ID.EXE result shows the TCCB at version 4.1 We know there is an upgrade available but I do not know what the new version number is or what issues it solves. We would like to make that upgrade as soon as the unit comes down.
- The problem that occurs is that after a few days operation, all the ma inputs on the TCCB card stop updating. On this unit this problem never happened before we added modbus and one new 4 to 20 ma analog input.
- On another Mk-V unit on this site, for their steam turbine, they had a similar situation when they added a "SCADA look a like. In that HMI is also a historian included, read; a significant increase of data requested from HMI to C core. After taking this into service, after a few days the extended card freezes just like on the 301." (comment from site)
- The TCCB card has been swapped with a new card from spares several times with the same result (card input freeze).
- Our Modbus file contains:
55 analogs read from the Mk-V
32 analogs written to the Mk-V
32 digitals in two packed registers read from the Mk-V
We are starting to look at the modbus server to determine the request rate and see if we can reasonable change it.
One issue is that now the TCCB card inputs are not updating and will not reset until the <C> core is rebooted. The customer is not comfortable doing that until the unit is down, so it may be a few weeks when the PROMs get upgraded.
We are thinking of trying to initially run the unit with the modbus disconnected from <C> and our one new analog input still connected to the expansion ma inputs. This would be to see if the problem re-occurs without the modbus. If it does not, then the issue is either with the modbus or has been resolved with the new PROMs.
We would then reconnect the modbus to determine which one of those it is.
Question for anyone more knowledgeable about modbus than myself. I am assuming that with the <I> computer running iDOS, the ARCNET connection from <C> to the <I>, and the modbus serial connection to the <I>, that the data flow would be from the <C> to the <I>. Then when the modbus makes demands for data it would be getting it back from the <I>. With that understanding, how could the modbus crash part of <C>? It would seem to be isolated from <C> by the <I> computer. A problem might stop the modbus from working or maybe lock up the <I>, but I don’t understand how it could cause problems with the <C> core. Love to either hear an answer or maybe a point to where I could find the answer. Thanks in advance.
We'll keep you posted as things progress.
> With that understanding, how could the modbus crash
> part of <C>? It would seem to be isolated from <C> by the
> <I> computer. A problem might stop the modbus from working
> or maybe lock up the <I>, but I don't understand how it
> could cause problems with the <C> core. Love to either hear
> an answer or maybe a point to where I could find the answer.
Any modbus request to the <I> or <HMI> causes the <I> or <HMI> to request that data from the <C>. It is absolutely possible to cause memory issues in <C> with excessive modbus data requests. The issue may not just be the number of points, it could be excessive frequency of the read/write requests.
The number of points you indicate that are on the modbus list are absolutely not a problem. But, I have seen DCS systems setup to read/write to a HMI modbus slave at 200 milli-second intervals and it did cause <C> memory and timeout diagnostic alarms periodically.
Another thing you should look at, if the unit has an OSM you should try disconnecting the OSM for a time. In the 2014 time frame GE seems to have "dumbed down" the process by which the OSM point list is built. They collect all points that appear in the CSP, whether they need them or not and that is a large number of points and many are totally unnecessary. For example, on a 7FA they collected the digital and analog points for all 8 flame scanners, even though the 7FA only actually has 4 scanners. There are countless other examples of points GE collects into the OSM that are totally unnecessary and cause unnecessary loading of the MK-5 <C> core.
You/The site seem to be discounting the possibility that the problem could be related to wiring and grounding. If you would fully describe the new signal which was added to the TCCB card, where the signal was terminated in the Mark V, how it is powered (by the Mark V or by some other device/source), we may be able to offer some more suggestions.
All 4-20mA signals are not created equally. And, the new signal might just be causing the problem(s).
Tell the Customer to grow a pair and re-boot the <C>. If it trips (it should not if the Mark V is configured properly), consider it an opportunity to find and fix the cause so that it can be re-booted without problems in the future.
Regarding the added input to the <C> core, it is a 4-20 ma signal from a PLC and powered by that PLC.
The signal comes in to the <C> core on the TBCB card (location 7) as a two wire connection, landed on terminals 35 & 36 (POS & NEG) and configured as C_C_MAI26 in the IO.CFG file and defined in the IO_Configurator as Signal 12 on the TCCB Card Definition. Hardware jumper BJC-12 is NOT installed.
I am hoping to convince the customer to move this point to the <Q> core input Q_R_VDC2 with the ma signal daisy chained through the three cores with only jumper BJ-11 installed. This will give me a voltage signal into the three cores of <Q>, and get this new signal off of <C>. The added benefit of redundancy is also nice. This will at least get this added input away from the TBCB / TCCB issues.
BTW: Thanks mhwest for clarifying the Modbus question for me. People keep saying Modbus is easy, which it is if it is working. But if it's not working it can be very frustrating.
On your comment on an OSM. Not just sure what you mean by OSM. Is that an Historian? If so, this site does not have one. They do have two <G> computers. One sitting in the same box as the <I> and arcnet connected to <C>, and the second in a stand alone box and arcnet connected to <D>. They have redundant links to their DCS through the two <G>s.
Not sure if I could be successful convincing this customer to re-boot <C>. I agree it should do nothing, assuming all the <C> inputs have been kept to non-critical. However, this customer serves a big chemical plant with no sense of humor.
Thanks for all the help,
Thanks for the feedback. If the Customer won't go for moving the signal to <Q> I would suggest trying to install a mA signal isolator between the mA output of the PLC and the mA input of <C>. And, then I would try inserting the BJC12 jumper. (I would actually try inserting the BJC12 jumper anyway.) Signal isolators are relatively inexpensive and readily available in most parts of the world, and are miracle devices for problem applications like this.
I think I've mentioned it before, but I believe the <C> mA inputs are multiplexed--meaning that the Mark V switches from one input to the next at regular intervals, and doesn't read all the inputs simultaneously. This happens with RTD inputs to <C>, and it's the reason that digital RTD simulators can't be used with the Mark V--digital simulators want a constant load, and the way the Mark V <C> multiplexes the RTD inputs there's NOT a constant load; there's only a load when that signal is being read. It's entirely possible that the PLC doesn't deal well with the multiplexing well.
It's also possible that the PLC doesn't like the 100 ohm dropping resistor on the TBCB card; I've seen that happen, too. It might work for a while, but then-intermittently--it just doesn't.
It could be a combination of all of the above that's causing the mA inputs to the TCCB (via the TBCB--isn't this fun!!!) to intermttently fail. A mA signal isolator would probably solve this problem, but moving it to <Q> will also probably solve the problem--BUT then it might not, either. Since the signal is powered by an external source (the PLC), there could be other problems with reference grounding, and a mA signal isolator would be the best solution in this case also. (The Mark V wasn't exactly designed to play well with others (control systems, that is).) GE has learned a LOT since then, and has put what they've learned into their newer turbine (and DCS/BOP) control system designs.
Please write back to let us know how this all works out!
> On your comment on an OSM. Not just sure what you mean by
> OSM. Is that an Historian?
OSM=> On site Monitor, a PC/server GE puts at the site and connects to the arcnet to feed data to the GE monitoring and diagnostic center when either a unit is still within the warrantee period or there is a long term service contact of some type.
Thanks for the explanation. This site does not have an OSM. On the ArcNet they only have one <I> in the PEECC connected to <C> core; a second <I> in their control room connected to <C> through a fiber link; and two <G>'s in the PEECC connected, one each, to <C> and <D> cores.
>Thanks for the explanation. This site does not have an OSM.
>On the ArcNet they only have one <I> in the PEECC connected
>to <C> core; a second <I> in their control room connected to
><C> through a fiber link; and two <G>'s in the PEECC
>connected, one each, to <C> and <D> cores.
So some new information on our issue.
Over the weekend the <C> core automatically re-booted. All by itself. The inputs on the TCCB card are now working again. Our status on the unit is the following:
1) The new signal wired in to the <C> core expansion inputs is currently un-wired.
2) The other inputs moved from <R> to <C> expansion inputs during this outage are connected and are now updating normally
3) The modbus is connected and working OK
4) Both the <G1> and <G2> computers have their arcnet connection going to <D>
Our position right now is to watch these inputs on the TCCB card to see if they remain working for a few more days. (They have been crashing after 24 to 48 hours).
We are working to procure a MOORE Isolator for the new (currently disconnected) input. We plan to leave that one disconnected until the isolator is installed.
Assuming all stays good, we are thinking our next step would be to bring the arcnet connection for the first <G> core that was moved from <C> to <D> back to <C>. That would put those <G> arcnet connections back to original. Then watch for a while to see if any problems crop up or if the TCCB card freezes. If not, re-connect the last remaining ma input through a MOORE Isolator and see how that goes.
There may be a question regarding the PROM upgrade on the TCCB card due to $$$$. People may be wiling to see how the isolator goes. :-(
Thanks for the feedback.
Are both <G>s configured to get MODBUS data?
Moving the <G>s to <D> likely wouldn't resolve the "loading" issue of the MODBUS requests from <C>. <D> still has to get the information from <C>, and the TCCB.
Please write back to let us know the progress.
To answer your questions, one <G> is connected via arcnet to the <C> core, and the second <G> is connected to the <D> core, also via arcnet. Each <G> has a connection to a WDPF DCS with an Ethernet connection. The connection is designated as "ETHERNET coax cable 10 Base T" and goes into the "IIU" input of two separate DPUs.
Both links have the exact same content regarding the amount and type of data points. Each <G> link to the DCS contains 538 data points.
HELP Another question has come up as the <C> core has done a second independent reboot within a one week period. What is it that triggers an independent reboot? It does sound like we have something really wrong.