Large Scale Distributed Oscilloscope for FLARE

FLARE: Facility for Laboratory Reconnection Experiments required large scale, high speed DAQ:

• Number of Simultaneous Analog Inputs: 1200
• Sample Rate: 20MSPS
• Maximum Skew between channels : 20nsec

The skew requirement is extremely tight, and would be very difficult to achieve with a conventional copper clock distribution. However D‑TACQ proposed a system based on White Rabbit, following the Distributed Oscilloscope concept. White Rabbit (WR), WR is an extension of IEEE1588 Precision Time Protocol, provides extremely accurate timing on a standard fiber optic Ethernet link, with the possibility to work over long distances and large numbers of links. In this case, the physical distance is small since all the digitizer appliances are in the same cabinet, but WR is still ideal, because it guarantees that all boxes use the same clock, with the same phase, and they all see the same trigger, with simple one-fiber-per-box cabling solution. ACQ2106 Digitizer units already support WR as standard. D‑TACQ uses an automated tuning method to guarantee that the phase of each 40MHz sample clock is identical on all boxes. The tuning is stable for hours. IOC logic checks for loss of time-sync and forces re-tuning if there is a loss of timing - we find that never happens unless there is an outside fault like cable removal.

A key feature of the system is the use of a "DD" double depth box - the ACQ2106 1U box, normally 240mm deep, is doubled to 480mm, the appliance motherboard and analog modules occupy only the rear of the enclosure, leaving the front side free for custom signal conditioning. In this case, this is customer-developed-custom signal conditioning, where the customer plans variety of compatible signal conditioning modules. The DD box represents a great saving in both space and expensive cabling over a traditional separate-box approach.

System Featues:

• 26 ACQ2106 boxes, each fitted 6 ACQ481ELF analog modules, for a total 48 channels, 20MSPS/s, capture 1920MB/s to local memory.
• JOG, RUN, SPRINT feature, allows programmed sample step changes in-shot:
  "JOG" at 2.5MSPS to save memory, RUN at 20MSPS and "SPRINT" at 40MSPS for short duration for region of interest.
• Each ACQ2106 "DAQ Appliance", an embedded Linux computer with 1GB DRAM, has an EPICS IOC inside, providing extensive monitoring, supervisory and control features.
• The ACQ2106 boxes are Double Depth DD, with space inside to accomodate customer signal conditioning. The customer signal conditioning owns the front panel connectors, receiving clean power and a low-speed control interface from the system. Customers will configure the signal conditioning using their own-developed python/EPICS applications running on the embedded system.
• 2 White Rabbit Switches WRS, a COTS open hardware product available from several vendors, thanks to the CERN Open Hardware project.
• 2 Fiber Ethernet Switches FS, each with 10GB/s uplinks.
• System detects a physical Plant Trigger signal at T-1s and uses WRTD compatible signaling (a network multicast) to enable each unit to trigger at the same WR time, with 20nsec precision.
• The standard shot is 100msec, 192MB/box, achieving post shot offload at ~500MB/s thanks to link aggregation on the FS switches.
• Host side PC control using Python, with bulk data offload (one file per box), and data archive with MDSplus.

The system has been through extensive testing, both our standard module tests for analog quality and calibration on every channel, synchronization tests to show that the combined set of 26 boxes meet the skew requirement between every box on every channel, and endurance testing to 1000 cycles per run.