\section{Data acquisition} \label{rwmacleod:sec1} \subsection{Overview} The data acquisition goal for \hd\ is to accept the level 1 trigger rate without incurring any DAQ system deadtime. The high rate of 70-180\khz\ drives the design of the trigger, the front-end electronics, and the DAQ system. Both the \fadc\ and the \tdc\ digitizing modules must store data for several microsecond so that the level 1 trigger can be processed. When the level 1 trigger is satisfied these devices must be read-out without deadtime. When the level 1 trigger is asserted, a time slice of each ring buffer will be copied, compressed and stored. Events will be buffered into groups of at least 64 on each electronics board and then transferred first across a backplane to be built into crate event fragments and then to a computer farm to be built into complete events. The farm will perform a quick analysis to reduce the event rate by approximately a factor of ten before recording to magnetic media. This design allows \hd\ to start running with a modest tagged photon rate and then to scale-up by an order of magnitude. The DAQ system will be an important part of the whole \hd\ data analysis system. The rates and complexity of the DAQ system for \hd\ are challenging, nevertheless we will provide a flexible system that will allow different modes of read-out: \begin{itemize} \item production physics events, \item full \fadc\ and \tdc\ time window events, and \item several microseconds streams of raw \fadc\ and \tdc\ data for the whole detector. \end{itemize} Additionally, since demand for access to a DAQ system during accelerator down-time is very high, we propose that the system should allow independent trigger, control and read-out of sub-systems so that effective use of this time can be made. Each of the components of the data acquisition system will be discussed below. \subsection{Data flow} % \item Digitization & Trigger In \hd\ there are approximately 16,000 \fadc\ channels. If we assume a typical occupancy of 2\%, a 250 MHz, 8 bit \fadc, a time window of 100 nanoseconds, and that the full time window is read-out, then the total amount of \fadc\ data would be: 16000 channels * 0.02 (\% occupancy) * 25 bytes/channel $=$ 8Kbytes per event. Reduction of the \fadc\ data will be performed in real-time. The result will be an energy, a time, and a channel identifier. This would allow us to reduce the amount of data per hit to 10 bytes per \fadc\ channel thereby lowering the total \fadc\ data to 3.2 Kbytes per event. The computational resources required to achieve this reduced data rate are currently being studied. It may be possible to build this processing into the gate array, or to include digital signal processors on the \fadc\ boards. In \hd\ there are approximately 8,000 \tdc\ channels. The data volume for the \tdc s would be: 8000 channels * 0.02 (\% occupancy) * 4 bytes/channel $=$ 640 bytes per event. The total event size would be about 4 Kbytes; we assume 5 Kbytes per event for the DAQ design. Once the event fragments have been selected from a stream in each channel and reduced, they must be built into complete events and pass a filter that will reject accidentals and low energy events. Board level event fragments will be read out over a VME backplane and crate level event fragments will then be assembled and sent to the level 3 trigger farm. One important detail of the DAQ design is that the boards would have to buffer 64 (or more) events locally before transferring this group of event fragments to the crate computer. This is necessary because the tasks that the crate computer is required to do can't be done today at a 100\khz\ interrupt rate with a commercial real-time operating system. At 64 events per buffer, we will have to transfer about 23.5 KB in less than 640 microseconds. This transfer speed (36.7 MB/sec) is achievable with current VME technology. Depending on the CPU power available, the crate CPU may reorganize the crate event fragments so all data from one event is contiguous. Using current technology this data transfer rate could be handled using either gigabit ethernet or ATM OC-12 network links. Regardless of the network hardware, significant CPU resources will be required at the crate level to transmit the data. A fast switch with a backplane bandwidth greater than 672 MB/second and more than 40 ports will also be required. This switch technology is under development by many networking companies\footnote{One product from Cisco is described at http://www.cisco.com/warp/public/146/1971.html} and in a few years it should be common and less expensive. % FNAL model of event filter. The architecture of the CPU farm which performs event building, filtering and recording is presented below. % (design is stolen from FNAL) Event building and recording will be done in parallel on 10-20 computers. Each of these systems would send events to a sub-farm for filtering. The sub-farm nodes would send a pass/fail answer for each event received and the master node would record all events that passed the filter. It is possible that further information could be added to the event or that the events selected by the filter could even be reformatted before being sent to the recording computer(s). \subsection{Level 3 trigger} The goal of the level 3 trigger is to reduce the event rate given by the level 1 trigger to an acceptable rate to tape. In low intensity running ($10^7$ tagged photons/s) the level 1 trigger rate is expected to be 15\khz. Since the DAQ system is being designed to handle this rate to tape, the level 3 trigger farm will not have to cut any events, although it may be used to reduce the event rate somewhat. In high intensity mode where the level 1 rate may be as high as 180 \khz, the level 3 trigger must be able to reduce the event rate by a factor of ten. Most of these unwanted events result from an untagged (mostly lower energy) photon interacting in coincidence with a tagged photon. Rejecting these events means that level 3 must be able to calculate, with reasonable accuracy, the energy of the photon which produced the event. This involves accurately reconstructing tracks, matching them with the calorimeters, and adding additional energy deposited by neutral particles in the calorimeters. Because of the accuracy requirements and the demands of linking information from different detectors, we have decided to use a processor farm architecture for level 3 instead of building a dedicated hardware processor. All events passing the level 1 trigger will be read into the level 3 processor farm where they will be reconstructed; events passing the cuts applied will then be written to tape. This approach allows for algorithmic flexibility and improvements, and the ability to adjust cost-effectively to higher rates, but it does put pressure on the DAQ system. An estimate of the processing power required can be made as follows. The Hall-B full event analysis, with momentum resolution of less than 1\%, requires approximately 45 milliseconds on a 21 SPECint processor (Pentium III - 500). We believe that \hd\ will be similar and will require a similar amount of processing power. To perform the level 3 trigger analysis we think that 1/10 the number of instructions will be adequate because unlike the 1\% precision required in a full reconstruction we are aiming for only 2-5\% precision. In fact Hall-B has such a pattern recognition program in development; it is able to achieve 10\% momentum resolution using only 3\% as much processing power. Therefore we believe that approximately 5 milliseconds on a 21 SPECint processor will be required for the level 3 analysis of each event. This yields a figure of 0.105 SPECints/ev/s as a good estimate of the processing power required. This means an aggregate processing power of 18,900 SPECints is required for the level 3 analysis. Allowing for 50\% processor utilization (due to I/O overhead etc.) this means that a farm composed of 200 computers each capable of 200 SPECints could handle the I/O and perform the level 3 analysis for the high intensity configuration. Table \ref{tab:l2trig} shows the rates, sizes, and processing requirements for the level 3 trigger. \begin{table} \begin{center} \begin{tabular}{|l|c|c|} \hline \hline & Low Rate & High Rate \\ \hline \hline Event Size & 5 KB & 5 KB \\ \hline Event Rate to Farm & 10\khz & 180\khz \\ \hline Data Rate to Farm & 50 Mbytes/s & 900 Mbytes/s \\ \hline Num Links to Farm & 20 & 20 \\ \hline Data Rate per Link & 2.5 Mbytes/s & 45 Mbytes/s \\ \hline Link Technology & 100 Megabit Ethernet & Gigabit Ethernet \\ \hline Events/s per Link & 500 & 9000 \\ \hline SPECints/ev/s for L2 & 0.105 & 0.105 \\ \hline Num SPECints/link & 52.5 SPECints & 945 SPECints \\ \hline Num SPECints/link x 2 & 105 SPECints & 1890 SPECints \\ \hline Num 200 SPECint & 1 & 10 \\ processors/link & & \\ \hline Total Num 200 & 20 & 200 \\ SPECint processors & & \\ \hline \hline \end{tabular} \caption{Rates, sizes, and processing requirements for the level 3 trigger.} \label{tab:l2trig} \end{center} \end{table} \subsection{Monitoring and control} Monitoring and control tasks include hardware configuration and control (``slow controls''), bookkeeping, online event monitoring, alarm systems,and messaging systems. These are less demanding tasks than data acquisition in \hd, and should not present unusual challenges. We plan to follow some examples from Hall B, but to also make use of lessons learned there. In particular, we plan to integrate offline data analysis tools with the online software at the outset to reduce the total cost of software development. % % This not only avoids a % large duplication of effort, but it also makes it seamless to take software % developed on one data set and apply it directly to future runs. The number of events that can be processed online is largely determined by the number of processors available. In \hd\ a monitoring component can be added to the main event filtering code to gather statistics about both the event that is being written to tape and those being discarded by the level 3 farm. The choice of the type and number of processors can wait until soon before the system is installed, when prices are lower and performance is better known. The framework for slow controls should be uniform for all subsystems in \hd, but the choice is not obvious. VME-based {\sc epics} works in Hall B, but does not mesh well with the online requirements and has proven to be manpower intensive. In fact, a number of Hall B systems do not use {\sc epics} or VME, but instead resort to {\sc camac} or other options. Other software possibilities include message-based systems and hardware alternatives include PC or mixed hardware systems. We believe that an open, message-based system that implements a uniform interface to diverse hardware is probably best. Commercial products are currently available that implement this at moderate cost. Bookkeeping tasks include all recordable activities of the experiment other than raw and calibration data. We expect this will be done using relational and/or object-oriented databases. Current commercial and public domain database technology should be adequate. The alarm and messaging framework allows sub-systems to communicate their state to monitoring programs and operators. This system needs to be integrated across the entire online, DAQ, and database systems in a simple, uniform manner. The scale and performance requirements of this system are modest, and similar to other projects in progress or completed at Jefferson lab. %\subsection{Cost estimates} %\begin{table} % \begin{center} % \begin{tabular}{|l|r|r|r|} % \hline %Item & Unit cost & Quantity & Total \\ %\hline \hline %SBC CPU & 3 \ \ \ \ \ \ \ & 35 & 105\\ \hline %Main Switch & 20 \ \ \ \ \ \ \ & 2 & 40 \\ \hline %Sub-Farm Switch & 4 \ \ \ \ \ \ \ & 10 & 40 \\ \hline %DAQ Sun SMP & 20 \ \ \ \ \ \ \ & 10 & 200 \\ \hline %DAQ Field CPU & 2 \ \ \ \ \ \ \ & 200 & 400 \\ \hline %Misc PC/Sun & 3 \ \ \ \ \ \ \ & 10 & 30 \\ \hline %DAQ Raid Disk & 40 \ \ \ \ \ \ \ & 4 & 160 \\ \hline %CC Raid Disk & 40 \ \ \ \ \ \ \ & 4 & 160 \\ \hline %DAQ Tape Drive & 100 \ \ \ \ \ \ \ & 6 & 600 \\ \hline %CC computer & 20 \ \ \ \ \ \ \ & 2 & 40 \\ \hline %CC Tape Silo & 300 \ \ \ \ \ \ \ & 2? & 600 \\ \hline %Analysis Farm & 2 \ \ \ \ \ \ \ & 250 & 500 \\ \hline %Tape Media/Year & 0.1K/TB \ \ \ \ \ \ \ & 15000 & 1500 \\ \hline %Rack/Desks/Misc & 30 \ \ \ \ \ \ \ & 1 & 30 \\ %\hline \hline %{\bf Total} & & & 4405 \\ \hline %\end{tabular} %\caption[Estimated cost of the \hd\ Data Acquisition System.] %{Estimated cost of the \hd\ Data Acquisition System. Costs are listed %in thousands of 1999 US dollars. The abbreviations ``SBC'' and ``CC'' refer to %``single board computer'' and ``computer center'', respectively.} %\label{tab:daqcost} %\end{center} %\end{table} %A significant cost savings can be realized if we recycle tapes after some %period of time.