Design of the Compact Processing Module for the ATLAS Tile Calorimeter

The upgraded TileCal modules will be segmented into identical sub-modules, called mini-drawers, to boost the availability of the readout electronics by the suppression of single-points of failure and the use of redundant power supplies. The Long Barrel modules will consist of four mini-drawers with 45 PMTs, while the Extended Barrel modules will be composed of three mini-drawers with 32 PMTs.

Each mini-drawer forms an independent readout subsystem capable to operate up to 12 PMT channels. A mini-drawer is composed of a mechanical aluminum structure which supports one Mainboard, one Daughterboard, and one high voltage regulation board to read out and operate up to 12 PMT blocks equipped with FENICS front-end boards. The PMT signals are shaped and amplified in two gains by the FENICS, and digitized in the Mainboard by 12-bit dual ADCs. The digitized pulses are transferred to the Daughterboard FPGAs which create and transmit a data packet with the samples to the off-detector electronics through high-speed optical modules.

Figure 3 presents one fully-assembled prototype of a mini-drawer populated with the upgraded on-detector readout electronics and the mechanical aluminum substructure for supporting the PMTs and the electronics.

The off-detector electronics of TileCal for the HL-LHC will be formed by 32 TilePPr modules and 32 TDAQi boards.

The TilePPr module is composed of an Advanced Telecommunications Computing Architecture (ATCA) [8] Carrier Blade Board (ACBB) and four FPGA-based Compact Processing Modules (CPM).

Figure 4 presents a picture of the ACBB. This board has been designed as an ATCA cutaway carrier board to allow the integration of large heat sinks for the CPMs. The ACBB interconnects the CPMs with the TDAQi system through the Zone 3 connector and provides slow control capabilities through the ATCA infrastructure. The ACBB can deliver up to 400 W at 12 V through the Zone 1 connector.

In order to enhance the maintainability and upgradability of the ACBB during the entire HL-LHC lifetime, the functionalities of the ACBB are implemented into three mezzanine boards:

• •

  The Tile Computer on Module (TileCoM): This board is a System-on-Chip-based board which permits remote configuration and monitoring of the on-detector and off-detector readout systems and interfaces with the Detector Control System (DCS).

• •

  The GbE Switch: This mezzanine provides GbE communication between the CPM, TDAQi and TileCoM boards, and the ATLAS TDAQ system.

• •

  The Intelligent Platform Management Controller (IPMC): The CERN IMPC [9] implements the IPMI (I for Interface) functionalities required for controlling and monitoring the cooling and power system of the ACBB.

The CPM has been designed as single full-size Advanced Mezzanine Card (AMC) hosting one Xilinx UltraScale FPGA (Main FPGA), one Xilinx Artix FPGA (Control FPGA) and eight Samtec FireFly optical modules. The prototypes of the CPM assembled with the 8 Samtec FireFly modules is presented in Figure 5.

Figure 6 shows a complete block diagram of the CPM and its interconnections. The core processing of the CPM is a Kintex UltraScale FPGA, called Main FPGA. The first CPM prototypes mount a Kintex KU085-1A1517C FPGAs with 48 GTH MultiGigabit Transceivers (MGT) capable of operating up to 12.5 Gbps data rates. The final version of the CPM to be installed in ATLAS will be equipped with a Kintex KU115-1A1517C FPGAs in order to provide a large margin of resources for the implementation of more complex energy reconstruction algorithms.

The Main FPGA provides a bidirectional optical communication path with the on-detector electronics via 32 MGTs connected to 8 FireFly modules, while 5 MGTs interface with the TDAQi and FELIX systems through the AMC connector. The remaining MGTs are used for the communication with the GbE Switch and to the other AMCs in the ACBB.

The Modular Management Controller (MMC) of the CPM also communicates through the AMC connector with the IPMC in the carrier to manage the hot swap power sequence and for transmitting health monitoring data.

The slow control and monitoring of the peripherals and power modules relies on a Xilinx XC7A35T-1CSG325C, called Control FPGA. The Control FPGA receives the detector monitoring data from the Main FPGA and transmit it to the ATLAS DCS system through the TileCoM board. Furthermore, the Control FPGA is intended to implement a phase monitoring circuit based on the Digital Dual Mixer Time Difference phase detector [11] for the monitoring of phase variations in the distributed clock with a precision of 30 $\text{ps}_{\text{RMS}}$.

The CPM is responsible for the distribution to the on-detector electronics of the recovered bunch-crossing clock from the FELIX input stream. The designed clock architecture in the CPM permits to extract the recovered clock directly from the MGTs using the dedicated output clock buffers OBUFDS_GTE3 and to clean it with a Si5345 jitter cleaner before routing it back onto the Main FPGA through the transceiver reference clock pins. This clock routing introduces lower levels of jitter than buffering the clock through regular general purpose input/ouput (GPIO) pins to the jitter cleaner [12].

The jitter cleaner outputs six copies of the clean version of the bunch-crossing clock to the MGT banks of the Main FPGA, so that all MGTs can operate using the same reference clock. Additional copies of the bunch-crossing clock are transmitted to the Control FPGA for phase monitoring purposes, and to MMCX connectors to measure the quality of the clock.

The clocking circuitry can be configured to provide three possible clock inputs to the Si5345 jitter cleaner:

• •

  Local programmable oscillator Si570: Used for the initialization of the transceivers and lab tests.

• •

  Recovered clock from FELIX: This clock is buffered through an On Semiconductor NB6LQ572 multiplexer clock. The clock multiplexer permits to choose between different copies of the clock recovered with the MGTs.

• •

  Clock from the ACBB: This clock is intended for lab tests and qualification purposes during the final production of the CPMs.

The PCB comprises 14 layers, where 6 are devoted to power distribution and 8 layers are dedicated for high-speed and control signals. The total PCB thickness is 1.6 mm $\pm$10%, compliant with the AMC standard. Isola FR408HR was selected as dielectric material due its cost-effective, low dielectric constant ($\epsilon_{r}\simeq$ 3.68) and low loss tangent ($\text{tan}\delta\simeq$ 0.01) compared with other standard FR4 materials.

During the PCB design, signal and power simulations were performed with ANSYS Electromagnetics Suite to verify the signal integrity of the high-speed lines and power planes. Several studies were carried to minimize the insertion loss and impedance discontinuities along the high-speed paths introduced by differential vias and DC-blocking capacitors.

Figure 7 presents the simulation results of the differential insertion loss (SDD21) for the interconnects of the Samtec FireFly module farthest from the Main FPGA. The simulated high-speed interconnects have a maximum length of 7.65 cm.

As can be observed in the results, the insertion loss at the Nyquist frequency (4.8 GHz) is between $-1.24$ dB and $-1.74$ dB, ensuring signal integrity on the high-speed interconnects between the FireFly modules and the Main FPGA.

The 32 optical links connected to the Main FPGA have been qualified with the Xilinx IBERT IP core [13] through Bit Error Rate (BER) tests running the links at 9.6 Gbps.

The setup consisted in two separated CPMs transmitting and receiving $2^{31}-1$ Pseudo-Random Bit Sequence (PRBS31) data patterns during seven days. No errors were found in any link providing a BER better than $5\cdot 10^{-16}$ per link with a Confidence Level (CL) of 95 %, and a total combined BER better than $1.6\cdot 10^{-17}$ with a CL of 95 %.

Furthermore, the power consumption during the test was about 32 W. The CPM consumption during operation is expected to increase by 12 W due to additional firmware. This estimation was obtained from the Xilinx Power Estimator tool and the expected resource use for the final firmware. These preliminary estimations predict that the power consumption of the CPM will be within the power consumption budget of 60 W per AMC specified by the ATCA standard [8].

As part of the validation tests of the CPMs, the eye diagrams were measured at the output of the Samtec FireFly modules using a DCA-X 86100D sampling oscilloscope from Keysight. The measurements were done transmitting a PRBS31 data pattern at data rates of 4.8 Gbps and 9.6 Gbps.

Figure 8 shows the results of the eye diagram measurements for 4.8 Gbps (a) and for 9.6 Gbps (b). In both cases, the eye diagrams are widely opened without signs of excessive jitter, presenting satisfactory noise margins.

Signal quality parameters were extracted from the eye diagram data as part of the validation tests. Table I summarizes the extracted jitter values, eye height and width.

The signal quality at the input of the CPM was also studied using the built-in capabilities of the Xilinx transceivers. For these measurements, the same setup presented in section IV-A is used. The receiver of the MGTs includes a second independent sampler after the equalizer which runs in parallel with the Clock and Data Recovery (CDR) data sampler. The results from both samplers are compared to perform the BER tests and to compose eye diagrams.

Figure 9 presents the eye diagram at the input of the FPGA for a BER better than $10^{-9}$ per sampling point and a data rate of 9.6 Gbps. The eye diagram is scanned across vertically and horizontally by controlling the clock phase of the sampler and the offset voltage.

An horizontal BER test scan was performed by shifting the phase clock of the second sampler horizontally across the eye, and setting the voltage offset to the center of the eye.

The resulting BER curve is a bathtub curve, so-called because of its characteristic shape, shown in Figure 10. The BER is below $10^{-12}$ at the center of the eye and increases when getting closer to the eye crossings.

The distance between the left and the right curves defines a noise margin of 0.62 Unit Interval ($\sim$64.58 ps) at the specified BER level of $10^{-12}$. All channels were verified and validated showing similar quantities.

In order to study the performance of the clock distribution system, phase noise measurements were carried with a MS2840A signal analyzer from Anritsu at room temperature. Figure 11 presents a block diagram of the setup for the clock qualification tests. The setup consisted in a CPM with a GBT link transmitting at 4.8 Gbps to a GBTx chip [14] in a test board via a SFP module.

The GBTx chip is a 130 nm CMOS rad-hard device intended for high-speed communications and slow control in High Energy Physics experiments. This chip is able to recover the bunch-crossing clock from the input serial link providing up to 8 independent output clocks with programmable phase and up to 40 clocks with a fixed phase. Both types of clocks can be configured to buffer 40, 80, 160 and 320 MHz clocks.

However, the phase-programmable clocks present higher levels of random jitter than the e-link clocks, showing a phase noise peak around 1 MHz introduced by the internal delay lines used for shifting the phase. For this reason, the e-link clock is preferred over the Phase-Shift clock (PS clock) as reference clock for driving the MGTs of the Daughterboard FPGAs. Despite its higher jitter, the PS clock is used as input clock to the Mainboard ADCs since the phase-tuning capability is desirable to minimize the error of the real time energy reconstruction algorithms and for pulse characterization purposes and calibration.

Figure 12 presents the measured phase noise plot of the e-link and PS clocks for different frequencies, as well as the corresponding phase noise mask provided by the FPGA manufacturer to ensure minimal jitter in the MGT outputs [15]. In this test, the clock source was the on-board Si5345 jitter cleaner driven by the on-board Si570 oscillator.

The e-links were configured to provide recovered clocks with frequencies of 80 MHz, 160 MHz and 320 MHz. The phase noise of the 40 MHz e-link clock is not shown for simplicity, since 40 MHz is outside the frequency range for the reference clock of the MGTs. Similarly, only the phase noise of the 40 MHz PS clock is presented because it is the only possible input clock frequency for the 12-bit ADCs on the Mainboard.

As observed in Figure 12, the phase noise values of all the e-link clocks exceeds the phase noise limits for the reference clocks addressed by the FPGA manufacturer. However, stable communication between the CPM and the Daughterboard v5 [16] is achieved using the GBT protocol at 4.8 Gbps for the downlink and 9.6 Gbps for the uplink.

Table II shows the integrated RMS jitter extracted from the phase noise measurements of the two types GBTx output clocks and for all frequencies. The RMS jitter was obtained integrating the phase noise response over the entire bandwidth (from 10 Hz to 10 MHz) and over the sensitive bandwidth of the MGT (from 10 kHz to 1 MHz).

The RMS jitter of the 80/160/320 MHz PS clocks for the entire bandwidth is about three times higher than in the e-link clocks. In the frequency range specified by the manufacturer, the RMS jitter is up to seven times higher in the PS clocks.

Finally, the phase noise at the outputs of the Si5345 jitter cleaner of the CPM was measured. Two different clock frequencies of 120 MHz and 240 MHz were considered as reference clocks for the implementation of the GBT protocol.

Figure 13 shows the phase noise measurements for the two reference clocks in the CPM and the corresponding phase noise masks.

As observed in Figure 13 and Table III, the phase noise for the 120 MHz clock is higher than for the 240 MHz clock in the range of frequencies from 10 kHz to 1 MHz, exceeding the manufacturer specifications. However, the tests reveled that the configuration of the Si5345 chip did not cause any relevant effect on the phase noise of the recovered clock indicating that the main jitter contribution is dominated by the CDR of the GBTx.