Software Monitoring of ALMA Correlator Using Quadrant Control Cards
CDL Outline Functionality of Quadrant Control Cards (QCC) in the system, QCC CAN functions still needing software support. Importance of software support for QCC, ALMA Correlator Workshop in May 2016. CDL QCC system block diagram presentation. Photos of QCC standard control card and power control card. Mechanism for dealing with QCC warnings and errors highlighted. Proposed new method for dealing with warnings using a CAN-based GUI. Detailed readout of warning count block and various warnings present.
Uploaded on Feb 26, 2025 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Software Monitoring of the ALMA Correlator Using the Quadrant Control Cards Joe Greenberg Atacama Large Millimeter/submillimeter Array Karl G. Jansky Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array
CDL Outline Functionality of the Quadrant Control Cards (QCC) in the system QCC CAN Functions that still have to have software support written Why it would be useful to implement software support for the QCC 2 2 ALMA Correlator Workshop, May 2016
CDL QCC SYSTEM BLOCK DIAGRAM ALMA Correlator Workshop, May 2016 3
CDL Photo of QCC Standard Control Card Microprocessor Single Xilinx Uses separate 28 Volt Supply with a UPS Backup. Allows QCC can be ON while the rest of the Quadrant is OFF Allows a record of the shutdown to be kept ALMA Correlator Workshop, May 2016 4
CDL Photo of the Power Control Card (PCC) Turns Quadrant on or off by controlling Solid State Relays which switch the AC Power to the 48 Volt Power Supplies. Contains KILL logic to shut down all four Quadrants ALMA Correlator Workshop, May 2016 5
CDL Current Mechanism for Dealing with QCC Warnings and Errors Warning- Condition exists where the correlator is not functioning properly Error- Condition exists where the correlator may be damaged ALMA Correlator Workshop, May 2016 6
CDL Proposed New Method of Dealing with Warnings A CAN based GUI can alert issue an alert that a Warning is present. Smart software could automatically flag astronomical data from the compromised portion of the Correlator. ALMA Correlator Workshop, May 2016 7
Warning Count Readout Readback of WARN_COUNT Block: RCA 0x1000, FC 4 DESCRIPTION LIMIT 0 Temperature Warnings. > 45.0 C. 1 9UBPS 3.3V Current Lower Limit Warnings. < 1.0 A. 2 1.8V Voltage Lower Limit Warning. < 1.6 V. 3 3.3V Voltage Lower Limit Warning. < 3.0 V. 4 5.0V Voltage Lower Limit Warning. < 4.5 V. 5 |-48V| Under Voltage Warning. < 40.0 V. 6 Rectifier AC Failure Warnings. 7 Rectifier Communications Lost (Rectifier disappeared). 8 DLL out of lock Warnings. 9 Timing Event Warnings. 10 Isolated Shutdown Command Warnings. 11 Rectifier Minor Fault. 12 Rectifier Fan Failure. 13 QPCC Offset Voltage Out of Range (Used in -48V Current Measurement.) 3.0 to 7.0 V. 14 Master Clock Board PLL Out of Lock. 15 A Slave Clock Board PLL Out of Lock. 16 A Station Rack Fan has stopped. 17 A Station Rack Fan will not servo. 18 A TFB Airflow Temperature Sensor is reading too high. 19 A TFB Airflow Temperature Sensor is reading too low. CDL For Each Row reads back: Number of warnings Time and Date of most recent warning Could corrupt Correlator data Could lead to Correlator Shutdown ALMA Correlator Workshop, May 2016 8
Get Shutdown Status CDL GET_SHUTDOWN_STATUS: RCA 0x20501 Description Get an integer value that tells the cause of the most recent rack shutdown. An integer is returned to be decoded as follows: Possibly caused by transient external event RCA 2 05 01 15- QCC Under Voltage V5 16- QCC Over Voltage V1.8 17- QCC Over Voltage V3.3 18- QCC Over Voltage V5.0 19- 6UBPS Over Current V3.3 20- No Cooling Air 21- Rectifier Major Error 22- QPCC Under Voltage V3.3 23- QPCC Over Voltage V3.3 24- QPCC Under Voltage V5.0 25- QPCC Over Voltage V5.0 26- Absolute Value of -48V Under Voltage 27- Absolute Value of -48V Over Voltage 28- TFB Airflow Sensor Over Temperature 29- QCC 125 MHz Clock has stopped 30- Probable Power outage. 48 Volts low plus other causes 31 Kill Signal from other Quadrants. 32 TFB 1.2V Supply Upper Current Limit Error >45.0 A 99- Multiple Causes Full Quadrant Correlator Shutdown Status Definition 0- No Known Cause 1- Over Temperature 2- 28V Voltage Lower Limit 3- 28V Voltage Upper Limit 4- Rectifier High Voltage 5- Rectifier Low Voltage 6- Over Voltage V1.8 7- Over Voltage V3.3 8- Over Voltage V5.0 9- Rectifier Over Temperature 10- 9UBPS, 6UBPS, or Mezzanine Card Over Current V1.8 11- 9UBPS Over Current V3.3 12- BPS Over Current V5 13- QCC Under Voltage V1.8 14- QCC Under Voltage V3.3 ALMA Correlator Workshop, May 2016 9
CDL CAN FUNCTIONS Maintenance Turns Correlator ON or OFF Remotely TURN_ RACKS_ON: RCA 0x20507 RCA 2 05 07 Description Racks . Turn On the Correlator Typical Interval Rare TURN_ RACKS_OFF: RCA 0x20508 RCA 2 05 08 Description Racks . Turn Off the Correlator Typical Interval Rare ALMA Correlator Workshop, May 2016 10
Readback of Error Count CDL Readback of ERROR_COUNT Block: RCA 0x1000, FC 5 COUNT DESCRIPTION LIMIT 0 No Known cause 1 Temperature Greater than Upper Limit Error. > 50.0 C. 2 28V Voltage Lower Limit Error. < 27.0 V. 3 28V Voltage Upper Limit Error. > 29.0 V. 4 Rectifier High Voltage Error. 5 Rectifier Low Voltage Error. 6 1.8V Voltage Upper Limit Error. > 2.2 V. 7 3.3V Voltage Upper Limit Error. > 3.6 V. 8 5.0V Voltage Upper Limit Error. > 5.5 V. 9 Rectifier Over Temperature Alert. 10 1.8V Current Upper Limit Error. > 60.0 A. 11 9UBPS 3.3V Current Upper Limit Error. > 50.0 A. 12 5.0V Current Upper Limit Error. > 3.0 A. 13 QCC V1.8 Lower Limit Error. < 1.6 V. 14 QCC V3.3 Lower Limit Error. < 3.0 V. 15 QCC V5.0 Lower Limit Error. < 4.5 V. 16 QCC V1.8 Upper Limit Error. > 2.2 V. 17 QCC V3.3 Upper Limit Error. > 3.6 V. ALMA Correlator Workshop, May 2016 11
Readback of Error Count (Continued) CDL Readback of ERROR_COUNT Block: RCA 0x1000, FC 5 (Continued) 18 QCC V5.0 Upper Limit Error. > 5.5 V. 19 6UBPS 3.3V Current Upper Limit Error. > 60.0 A. 20 Airflow Not On Error 21 Rectifier Major Error. 22 QPCC V3.3 Lower Limit Error. < 3.0 V. 23 QPCC V3.3 Upper Limit Error. > 3.6 V. 24 QPCC V5.0 Lower Limit Error. < 4.5 V. 25 QPCC V5.0 Upper Limit Error. > 5.5 V. 26 |V48| Under Voltage Error. < 40.0 V. 27 |V48| Over Voltage Error. > 56.0 V. 28 TFB Airflow Over Temperature Error > 50.0 C. 29 125 MHz Clock has stopped Error. 30 PCC|V48| Under Voltage Error plus other causes.< 40.0 V. 31 Kill Signal from other Quadrants. 32 TFB 1.2V Supply Upper Current Limit Error >45.0 A ALMA Correlator Workshop, May 2016 12
CDL Reset Racks RESET_RACKS: RCA 0x20504 RCA 2 05 04 Description Microprocessors in the quadrant. Send a reset pulse to all the Control Card Useful as a Maintenance trouble shooting tool to get back to initial conditions without having to cycle power. ALMA Correlator Workshop, May 2016 13
CDL READ BACK OF PARAMETERS Monitor Data Blocks: RCA 0x1000, FC 3 Block Read back of Voltages, Currents, and Temperatures from selected Correlator Card, 9UBPS, 6UBPS, PCC, or QCC GET_TE_DLL_ERROR_STATUS: RCA 0x20304 Read QCC TE or Delay Locked Loop (DLL) Error GET_PLL_STATUS: RCA 0x20503 Read the Clock Boards to see if their PLL s are locked ALMA Correlator Workshop, May 2016 14
CDL QCC Engineering Port Display of Parameters CORRELATOR AND STATION RACK 1 VALUES ============================================================================================================= 1.8V Currents 3.3V Currents 5V TEMP TOTAL 1.8V ( I0 + I1 + I2 ) = SUM 3.3V ( I0 + I1 ) = SUM 5V Current 48V DEG C POWER OUT -------------------------------------------------------------------------------------------------------------- QCC 1.81V 3.30V 5.01V 22.8C 9UBPS-1 1.81V 4.85A 5.13A 3.32V 17.36A 18.70A 36.06A 4.98V 0.42A -47.9V 19.4C 131W 9UBPS-2 1.81V 4.92A 5.06A 3.32V 17.29A 18.84A 36.13A 4.99V 0.42A -47.7V 17.8C 131W 9UBPS-3 1.81V 4.85A 4.99A 3.31V 16.24A 18.42A 34.66A 4.97V 0.35A -47.3V 15.3C 125W 9UBPS-4 1.81V 4.85A 4.92A 3.31V 18.00A 17.50A 35.50A 4.99V 0.35A -47.4V 13.0C 128W 6UBPS-1A 1.82V 33.88A 34.16A 3.32V 22.78A 22.92A 4.99V 0.56A -47.6V 21.5C 141W 6UBPS-1B 1.82V 34.66A 34.87A 3.32V 33.04A 33.18A 4.96V 0.07A -47.8V 18.7C 174W 6UBPS-2A 1.83V 34.38A 34.59A 3.31V 34.16A 34.30A 4.97V 0.42A -47.7V 19.9C 179W 6UBPS-2B 1.83V 33.81A 34.02A 3.32V 21.51A 21.65A 4.98V 0.07A -47.8V 16.7C 134W 6UBPS-3A 1.83V 34.45A 34.66A 3.32V 23.20A 23.34A 4.97V 0.42A -47.5V 18.1C 143W 6UBPS-3B 1.83V 34.09A 34.30A 3.32V 22.14A 22.21A 4.97V 0.07A -47.8V 14.0C 137W 6UBPS-4A 1.82V 23.90A 24.04A 3.32V 20.67A 20.74A 4.97V 0.49A -48.1V 11.7C 115W 6UBPS-4B 1.82V 23.97A 24.11A 3.31V 19.96A 20.10A 4.99V 0.21A -47.8V 10.0C 111W BIN1-CC0 1.79V 26.15A 27.35A 53.57A 3.30V 4.97V -48.5V 46.9C 96W BIN1-CC1 1.79V 23.62A 22.57A 46.26A 3.30V 4.96V -48.3V 40.0C 83W BIN1-CC2 1.80V 26.99A 27.91A 54.97A 3.30V 4.97V -48.1V 37.5C 99W BIN1-CC3 1.79V 22.71A 24.04A 46.82A 3.30V 4.97V -48.1V 33.8C 84W BIN1-CC4 1.79V 24.96A 27.28A 52.30A 3.29V 4.97V -48.1V 26.0C 94W BIN1-CC5 1.78V 23.06A 22.00A 45.13A 3.30V 4.96V -47.8V 27.6C 80W BIN1-CC6 1.80V 26.15A 27.63A 53.85A 3.30V 4.96V -47.7V 29.7C 97W BIN1-CC7 1.79V 22.78A 24.25A 47.10A 3.29V 4.95V -47.6V 29.3C 84W o o BIN4-CC0 1.79V 25.17A 27.13A 52.37A 3.29V 4.99V -47.9V 29.5C 94W BIN4-CC1 1.79V 22.07A 22.92A 45.06A 3.29V 5.00V -47.9V 31.1C 81W BIN4-CC2 1.79V 25.59A 27.13A 52.79A 3.29V 4.99V -47.7V 26.1C 95W BIN4-CC3 1.79V 22.21A 23.69A 45.97A 3.29V 4.99V -47.6V 20.8C 82W BIN4-CC4 1.79V 26.50A 25.52A 52.09A 3.29V 4.99V -47.4V 17.8C 93W BIN4-CC5 1.79V 23.20A 22.35A 45.62A 3.29V 4.99V -47.6V 19.0C 82W BIN4-CC6 1.79V 26.08A 27.13A 53.28A 3.29V 4.99V -47.4V 19.9C 95W BIN4-CC7 1.79V 23.97A 22.49A 46.54A 3.29V 4.99V -47.4V 20.3C 83W ALMA Correlator Workshop, May 2016 15
CDL QCC Engineering Port Display of Parameters (continued) TFB CURRENTS: BIN TFB-A0 TFB-A1 TFB-A2 TFB-A3 TFB-B0 TFB-B1 TFB-B2 TFB-B3 1 11.18A 13.15A 12.44A 14.41A 11.95A 12.37A 13.08A 13.08A 2 13.64A 14.48A 13.36A 13.08A 12.23A 12.37A 12.51A 12.30A 3 12.30A 13.50A 12.44A 13.29A 12.02A 11.81A 13.08A 13.00A 4 9.77A 9.98A 11.95A 13.15A 12.30A 10.97A 9.63A 9.70A Temperature Probe# Temperature Fan# CMD TACH Fan# CMD TACH. Target Temperature=25.0C 0 24.0C 0 22 28 1 22 28 1 23.9C 2 26 35 2 23.5C 3 26 36 3 23.1C 4 22 29 5 22 29 QPCC: V3.3=3.31V IA=238A IB=178A IA+IB=416A Voff=4.97V V48=-48.6V V5=5.01V V28=28.4V Temp=22C Rectifier status: Fans OK, AC OK, Voltage OK, Comm OK, Temp OK TOTAL POWER IN = 20198W TOTAL POWER OUT = 17277W EFFICIENCY = 85.5 percent DLL Status - LOCKED Timing Event Synchronized Type WIN S to stop this display. loop_count=4 scan_count=308682 The SYSTEM Switch is ON. System is ON. Automatic Power Down ENABLED. Use the WINDOW command to toggle this window on and off. Cooling air is On. PLL Status (L=locked, U=Unlocked) Master-U, 1-L, 2-L, 3-L, 4-L, 5-L, 6-L, 7-L, 8-L ALMA Correlator Workshop, May 2016 16
CDL Example of Smart Software Flagging Correlator Data as Compromised Warning Detected indicating a low 1.8 Volt supply Voltage Read in the parameters to determine which supply is affected and if it has shut down If it was a Station Bin Power Supply, determine which two antennas are associated with that half Station Bin Flag data for all baselines associated with those antennas as invalid Optionally let the operator know that data is being flagged invalid and why ALMA Correlator Workshop, May 2016 17
CDL Conclusions There is possibly quite a bit of software development remaining to be done for the QCC. Implementing the CAN support could: Result in more useful data being retrieved from the correlator since operations would be made aware that a part of the correlator is not functioning correctly. Result in less down time for the Correlator, since Warnings could cause corrective action to be taken before things get so bad that the Correlator shuts down. Similar software could be written for the other Control Cards as well. ALMA Correlator Workshop, May 2016 18
