% Responsible : Ken Oyama, Yvonne Pachmayer, Oliver Busch

\chapter{Shift Operation}

This chapter explains to shift operators how to operate TRD in ALICE.

\subsection{Shifter's Responsibilities}

Shifters have responsibility of the following subjects:

\begin{itemize}
\item
    Operation of detector and its infrastructures according to ALICE run schedule, TRD local plan, and requirements from surrounding activities. 
\item
    Keeping reasonable quality of data and efficient running in terms of fractions of properly configured chambers, data size, not keep writing non-useful data to storage, and so on. 
\item
    Keep detectors safe by watching delicate part of detectors. Regularly watching high voltage current, voltage stability, DCS boards temperature, cooling status and so on are very important. 

In addition, the following things are strongly recommended:
\item
    Reporting run summary at the end of shift everyday to alice-trd mailing list. 
\item
    Reporting any kind of problem to experts by phone (for emergency) or e-mail (use alice-trd-p2crew mailing list for local discussion and alice-trd for TRD-wide discussion). 
\item
    Having enough communications with around, and especially with other shifters when shift will be changed (every day and at the end of shifts). 
\item
    Try to improve things. Even this operation manual can be improved by shifters. 
\end{itemize}

\subsection{Keep in Mind}

\begin{itemize}
 \item Watch HV from time to time 
 \item Check the trigger rates and data size
 \item Check the data with Online Monitor
 \item Update the wiki if any update
\end{itemize}

\newpage 

\section{Detector Controls}

This section describes the detector controls (DCS), that is operation and monitoring of the TRD, via the ALICE remote user interface (UI) 
based on the PVSSII software running on windows operating systems. A collection of passwords, a list of the subsystem experts 
and FAQ can be found in the TRD wiki pages ({\tt http://wiki.kip.uni-heidelberg.de/ti/TRD/index.php/TRD\_Operation\_Manual}). 

\subsection{Shift Prerequisites (DCS)}

For DCS operation you need access to the TRD operator node, a WINDOWS server running the ALICE TRD UI, which 
is granted by the ALICE DCS network administrator. Please obstain from contacting her/him directly, but send an e-mail to 
the TRD DCS coordinator specifying your NICE user name, about \textbf{2 working days before the start of your shift}. \par 
Please also make sure to be registered to use the CERN terminal services \\ 
({\tt cern.ch/nice} $\to$  Terminal Services $\to$ register). \par 
You can check your access grants from outside ACR connecting via remotedesktop to the DCS application gateway, alidcscom001.cern.ch. \par 

\subsection{Start/End of shift}

In the beginning of your shift, login the Operator Node and the TRD control console. Make sure with the previous shifter that you can take all locks. 
If a lock is taken by somebody else, the Windows tooltip text (yellow box which appears when you leave the mouse pointer a couple of 
seconds over the lock item) indicates the owner/system. In the end of your shift, release all locks and logout 
(not just disconnect) from the operator node.


\subsection{Access - Logging In} 
 
The shift operator controls TRD connecting to the TRD operator node (ON) from the ALICE control room (ACR). 
To start, connect to the ON via remote desktop. 

\begin{itemize}
\item log into the DAQ machines aldaqacrXXX (user: trd, password). 
\item {\tt rdesktop -g 1280x1024 -D -u username alitrdon001.cern.ch} \\ 
      (NICE credentials) 
\end{itemize}


\subsection{ALICE UI}

\begin{figure}[ht]
\centering
\includegraphics[width=0.8\textwidth]{figs/obusch/UI_topnodePanel.pdf}
  \caption{The ALICE UI displaying the TRD top node panel.}
  \label{fig:UI}
\end{figure}

On the operator node, the UI starts automatically, otherwise it can be launched via 
{\tt START $\to$ PROGRAMS $\to$ STARTUP $\to$ TRD CONTROL CONSOLE}. Here, to log in : 

\begin{itemize}
\item user: trdOp   ({\tt ENTER}) 
\item password   ({\tt ENTER})
\end{itemize}

The UI is shown in in Fig.~\ref{fig:UI}, along with the top node panel explained in Sec.~\ref{sec:topNodePanel}. 
The user panels for monitoring and operation can be opened from the FSM tree browser in 
the left. The FSM states of the different nodes are displayed in the FSM monitoring zone. To open the alarm screen, click the button 
with the red triangle. 

\subsubsection{Access Control} 

In the ALICE UI there exist 4 user groups with different privileges: Observers, Operators, Experts, Developers. 
Currently there are 3 standard users established: trd (observer), trdOp and trdDCS (developer). In addition, dedicated 
TRD expert groups exist for HV and cooling plant operation. The shapes (buttons etc.) in the monitoring panels are 
enabled/disabled according to the access rights granted to the logged user. 
To login as a differen user, open the login prompt by clicking on the key symbol in the top left. \par
 
\subsection{FSM Tree} 

The TRD control system is modeled as a detector oriented hierarchy of objects behaving as finite state machines (FSM), displayed 
in the FSM tree browser. The possible states of the detector are mapped to FSM states, operation proceeds via FSM commands, 
moving from state to state. The states are propagated up from the lowest level of the FSM tree (representing the devices, such as LV/HV channels or DCS boards). 
The commands are propatated from the highest level down. The TRD top node {\tt TRD\_DCS} controls the substyems {\tt TRD\_FED} (Front-End Devices), 
{\tt TRD\_SM\_LV} (Low Voltage for the FED), {\tt TRD\_SM\_HV} (High Voltage),{\tt TRD\_PT} (Pre-trigger), 
{\tt TRD\_GTU} (Global Tracking Unit), {\tt TRD\_INFRASTRUCTURE}, and {\tt TRD\_DCS\_RUN} (the Run Unit communicating the 
run readiness of the detector to ECS and executing scripts at start and end of run. \par 

The FSM states of the main nodes are displayed in the FSM monitoring zone. 
In addition, more detailed information about the detector state is displayed in panels for the individual subsystems. 
The FSM tree allows navigation between the different panels and executing FSM commands: 

\begin{itemize}
\item to display a user panel, select a node via left-click, then right-click $\to$ {\tt VIEW PANEL} (note: not for each FSM node there is also a panel)  
\item to view the FSM state and possible actions, open the FSM control panel for the selected node (red frame): 
  right-click and choose {\tt OPEN FSM CONTROL}, or simply press the FSM button on top left of the FSM tree view 
\end{itemize}


The FSM contol panel displays the FSM states as well as the inclusion state (lock symbol) of the selected node and its children. 
The state of the node is calculated as function of the states of all included children. The user can take control over the 
FSM node via left-click $\to$ {\tt TAKE} on the lock symbol. This action will include automatically all children. To exclude a child, one can 
use left-click $\to$ {\tt EXCLUDE AND LOCK OUT} (the 'lock out' option prevents automatic re-inclusion when the lock of the parent is released and taken 
again - recommended !). A coloured frame around the lock symbol indicates that there are excluded children. Note: a child can only be in-/excluded by the 
owner of the lock of the parent. To find out the owner, place the mouse pointer over the lock and wait 2 seconds - the name is shown in a yellow 'bubble'. 
Please note: any FSM node should only be excluded from the FSM tree if all devices/channels governed by that node are {\tt OFF} or {\tt STANDBY}. Otherwise, bringing the 
detector back e.g. to {\tt STANDBY} via FSM will leave these excluded channels on, but this is (by definition) not shown by the global detector FSM state. \par 
To execute FSM commands, click the state and choose from the list of all possible transitions. If the node is in an indefinite ({\tt MIXED}) state, you 
first have to bring it to a consistent state by acting on its subsystems.  



\subsubsection{Special FSM Objects}
 
There are some DCS specific types of FSM nodes which are not shown in the FSM tree view, but in the FSM control panel: 

\begin{itemize}

\item majority units \\
(e.g. {\tt trd\_LV\_FWMAJ}). Used to set a threshold on the number of children in a given state to trigger a change of the parent state. 
Example: change to {\tt ERROR} only if more than 2 children are on {\tt ERROR}. The thresholds are presently all set to 0 (as if there were no 
majority unit) but can be set by the DCS expert. Possible states: {\tt MAJORITY\_OK, MAJORITY\_ERROR}. 

\item configurators \\
(e.g. {\tt TRD\_SM00\_HV\_ConfDB}) are used to download and apply recipes (lists of settings for devices) via PVSS from a database during FSM transitions. 
Foreseen for HV. If you find them excluded from the FSM tree, leave them excluded. 

\end{itemize}

\subsubsection{How to kill Locks}
\textbf{Only if unavoidable}: open the FSM overview panel via the FSM tool button in the lower part of the UI (right of monitoring zone), 
as shown in Fig.~\ref{fig:FSMOverviewPanel}. All FSM nodes, their FSM states and the 
locks ownership are displayed. To unlock a node, click on the red cross. 


\begin{figure}[h]
 \centering
 \includegraphics[width=0.7\textwidth]{figs/obusch/UI_withFSMControlPanel.pdf}
 \caption{FSM control overview panel. Click the red cross to unlock a node.}
 \label{fig:FSMOverviewPanel}
\end{figure}


\subsection{Top Node FSM}

\begin{figure}[htb]
 \hspace{3.0cm} \includegraphics*[width=0.6\textwidth]{figs/obusch/ALICE_topNodeFSM_TRD.pdf} 
 \caption{ALICE FSM state diagram as implemented by TRD.}
 \label{fig:topNode}
\end{figure}

The state diagram of the ALICE FSM top node is shown in Fig.~\ref{fig:topNode}. From {\tt OFF} to {\tt STANDBY}, the DCS boards and the 
LV for the FED are switched on. Configuration of the MCMs proceeds from {\tt STANDBY} via 
{\tt STANDBY\_INIT} to {\tt STANDBY\_CONFIGURED}. If HV is on and the drift chambers at operational anode voltage 
for physics data taking and the electronics configured, the state is {\tt READY}. In case of beam injection or 
whenever a HV safe state is required, the anode voltage can be ramped down to intermediate values, corresponding to {\tt BEAM\_TUNING}. 
If the electronics is not fully configured, but HV up and hence TRD in an unsafe state, this is indicated by the prefix {\tt READY\_}.  
The detailed synchronisation table of the top node and the states of the subsystems is presented in Fig.~\ref{fig:synchTableTop}. 
In addition to these standard states, there are also the 'undefined' state {\tt MIXED}, indicating that the children of a given node are not 
in all in the same state, and the {\tt ERROR} state. Note: in the synchronization table, the infrastructure node {\tt TRD\_INFRA} is not shown. 
It should always be {\tt READY}, otherwise the top node will be {\tt OFF} or in {\tt ERROR}.  

\vspace{1.0cm}

\begin{figure} [htb]
\centering 
\includegraphics*[width=0.6\textwidth]{figs/obusch/TRDFSMSynch.pdf} 
\caption{TRD FSM synchronization table. Note: the node TRD\_INFRA (not shown here) should always be in {\tt READY} state.}
\label{fig:synchTableTop}
\end{figure}


\newpage

\subsection{Top Node Panel} \label{sec:topNodePanel}

The top node panel is shown in Fig.~\ref{fig:UI}. It shows the FED FSM states (colors) of the individual 
TRD sectors. From here, the user can open the panels to set the FED configuration tags for different run types (Sec.~\ref{sec:FED}), 
browse the FED log messages and launch, if needed, the execution of the SOR/EOR scripts 
(Sec.~\ref{sec:SOR}) by hand . \par 
Two buttons allow the TRD operator to ramp down/up the voltage by a step of $\Delta$U = 400~V from {\tt READY} (operational voltage, nominal gain) 
to {\tt BEAM\_TUNING} (injection safe state). The highest voltage measured in the system is displayed. The panel also implements the 
DCS emergency button. It shows the HV safety condition of TRD and allows the DCS operator 
to ramp down TRD HV and bring the detector to a injection safe state in case of an emergency. 


\subsection{Shift Overview Panel - Alarms} 

\begin{figure}[ht]
\centering 
\includegraphics*[width=0.6\textwidth]{figs/obusch/OverviewPanel.pdf} 
\caption{Shift overview panel. Can be opened from the FSM nodes {\tt TRD\_FED, TRD\_HV or TRD\_LV} In the bottom, the alarms are displayed.}
\label{fig:overviewPanel}
\end{figure}

The overview panel, shown in Fig.~\ref{fig:overviewPanel} can be opened from the FSM nodes {\tt TRD\_FED, TRD\_HV or TRD\_LV}. 
It gives an overview over the FSM states of all TRD chambers. Tabs allow to toggle between the views for different subsystems. \par 
In the bottom the most recent active alarms are displayed. The complete list of all active alarms can be viewed in the 
alert panel, opened from the UI via the red triangle in the top. An alarm is a notification to the user, issued when a measured value is 
outside a given bound (and does, strictly speaking, not trigger an automatic action, in contrast to an interlock). It requires two actions (in any order): 

\begin{itemize}
\item removal of the alarm condition 
\item acknowledgment of the alarm 
\end{itemize}
 
The acknowledge requirement assures that the alarm didn't go unnoticed, even if the alarm condition eventually vanished without intervention. 
The alarm panel displays the datapoint and value triggering the alarm, the time the alarm occurred, the severity level (warning/error/fatal), 
and a short description. A detailed \textit{alarm instruction panel}, with explanations, instructions, and the names of the experts 
to contact, can be opened via right click $\to$ {\tt open\_help}.  \par
Another class of alarms is related to FSM states: whenever an {\tt ERROR} state occurs, the operator is informed by an alarm, in addition 
to the display in the FSM monitoring zone. 


\subsection{FED} \label{sec:FED}

\begin{figure}[h]
\hspace{3.5cm} \includegraphics*[width=0.6\textwidth]{figs/obusch/trdFsmDiagramFed.pdf} 
\caption{The FED FSM diagram.}
\label{fig:FEDFSM}
\end{figure}


The FED FSM diagram is shown in Fig.~\ref{fig:FEDFSM}. Issuing from {\tt OFF} the command {\tt $\to$ GO\_STANDBY}, the LV of the DCS boards (one for each drift chamber) 
is switched on, the boards are booted and the FEE servers start to publish (via the DIM protocol) the states of the chambers electronics and environment 
temperatures measured at the boards. The state {\tt NO\_CONN} occuring temporarily at boot time indicates that the DCS board LV is on, 
but no FEE server is running - an expected 
condition during booting, however resulting in an {\tt ERROR} state during normal operation. Moving to {\tt INITIALIZED}, 
the on-detector electronics is switched on and partially configured. The full configuration, including e.g. settings for the zero-suppression and on-line tracking, 
is downloaded from a database during the transition to the {\tt CONFIGURED} state. \par 

\begin{figure}[h]
\hspace{2.5cm} 
\includegraphics*[width=0.7\textwidth]{figs/obusch/FEDMonitorPanel.pdf} 
\caption{FED monitor panel.}
\label{fig:FEDMonitor}
\end{figure}

States and temperatures can be monitored for each SM in the FED monitoring panel shown in Fig.~\ref{fig:FEDMonitor}. In addition, the panel allows to 

\begin{itemize} 

\item view the trend of the average, maximum and minimum temperature per SM (trending button on top) 
\item monitor the status of the PVSS DIM client and restart it 
\item set the interlock temperature (Experts!) - see below  
\item open a panel for each DCS board individually (button {\tt CONF}), with further trends and some experts options  
\end{itemize}

The LV distribution to the different DCS boards within one TRD supermodule (SM) is controlled via the Power Control Unit (PCU). 
Advanced users can operate the PCU via the corresponding panel (see Sec.~\ref{Sec:PCU}).  

\subsubsection{FED Software Interlock}
To protect the electronics from overtheating, a PVSS software interlock permanently monitors the DCS board temperatures and 
switches off individual DCS boards (via the PCU) if a critical temperature is reached. The status of the interlock (runnning) is 
displayed in the monitoring panel, as well as the interlock temperature. 

\subsubsection{FED Configuration} \label{FEDconfiguration}

FED configuration proceeds in two steps: 

\begin{itemize}
\item define configuration for each run type 
\item {\tt CONFIGURE} via FSM, specifying run type 
\end{itemize}

The first step needs \textbf{not} be repeated each time the electronics is configured. The configurations for the different run types change 
occasionally, e.g. when a new configuration is released by the experts. The shift operator should 
check the configurations from time to time. \par

\begin{figure}[thb]
\hspace{2.5cm}
\includegraphics*[width=0.7\textwidth]{figs/obusch/FEDConfigPanel.pdf} 
\caption{FED configurations panel: define and check the configuration for the possible run types.}
\label{fig:FEDConfig}
\end{figure}

The different FED configurations are labelled with numeric tags. The run type (e.g. {\tt PHYSICS}) is appointed by the 
run coordinator. To define or check the configuration tag for each run type, open the 
FED configurations panel from the top node panel, as shown in Fig.~\ref{fig:FEDConfig}. A 
list of all available configurations is shown. A tag can be chosen, sector by sector, and mapped to a run type using the buttons in 
the {\tt STORE CONFIGURATON} section. The stored tags can be displayed via the {\tt LOAD CONFIGURATION} buttons. \par 
FED configuration is issued from the FSM top node via the command {\tt CONFIGURE}, which is available from {\tt STBY\_INIT}, but 
also from {\tt STANDBY} (proceeding via {\tt STBY\_INIT} automatically) and from {\tt STBY\_CONFIGURED/READY} (between runs, to reconfigure 
and reset e.g. the internal event counters). A panel opens, asking for the run type. Specify the type and press enter 
(will probably be automatized by the time the cosmic run starts). 

\subsubsection{FED Message Browser} 
The FED log messages from the DCS board are stored in a MySQL database (DB). The FED message browser (opened from the top node panel) 
allows the user to compose and execute DB queries. 


\subsection{Pre-Trigger}

Should be instructions here. Expert: Stefan Schmiederer. 

\subsection{GTU}

Should be instructions here. Expert: Stefan Kirsch.  

\subsection{Low Voltage}

The Low Voltage (LV) for the on-detector electronics as well as the channels for the DCS boards are switched on during the FSM top node 
transition from {\tt OFF} to {\tt STANDBY}. \par 

\begin{figure}[htb]
\hspace{2.5cm}
\includegraphics*[width=0.7\textwidth]{figs/obusch/LVMonitorPanel.pdf} 
\caption{LV monitoring panel.}
\label{fig:LVMonitor}
\end{figure}

The LV monitoring panel is presented in Fig.~\ref{fig:LVMonitor}. It displays the measured voltage and current for the 
channels supplying the digital and analog parts of the MCMs 
as well as the channel supplying the DCS boards. Furthermore, the network connection status ('pingability') of 
the power supplies and the status of the WIENER OPC server is shown. 
Clicking the {\tt view} button, current/volage trends for individual channels can be displayed. 
Individual channels can be switched on or off by the operator, however, adjustment of the settings requires expert privileges. 


\subsubsection{Low Voltage Trip}

The LV power supplies have a threshold of the maximum tolerable current. A high frequency of pre-trigger signals can cause a trip of the power supply, 
if the current limit is reached. In this case, to avoid problems with electronics noise, the two LV channels supplying the same TRD layer pair are 
also switched off. At the same time, the electronics of the layer pair is switched off, moving to the FED FSM state {\tt STANDBY}. 

\subsubsection {Power Distribution Box} 

The Power Distribution Box (one per SM) distributes LV for the DCS boards, according to the settings at the power control unit (PCU). 
The LV channels supplying the DCS boards are governed by the FSM node {\tt TRD\_DCS\_PDB}. 

\subsubsection{LV software interlocks} 

In the LV system, there are two PVSS software interlocks running: 

\begin{itemize} 

\item LV power supply overheating protection: if the temperature of the power supply (measured at the DC side) exceeds a certain limit, all 
channels of the power supply are switched off. The temperature can be adjusted from the LV monitoring panel (experts). 

\item cooling loop state: triggered, if any of the cooling loops leaves the state ''ON''. 
      If the loop supplied cooling water to a TRD sector (and not to the TPC-TRD thermal screen), all LV channels 
      supplying the corresponing TRD sector are switched off. 
      
\end{itemize}

In any case, the shifter is notified by an alarm and in addition the experts are informed via e-mail/sms.  


\subsection{High Voltage}


At detector startup, the initial ramping of the High Voltage (HV) from 0 is performed by the HV expert. The operators on shift only 
have to the anode HV by a fixed step ($\Delta$U = 400~V) between the states {\tt READY} and {\tt BEAM\_TUNING} (reduced gain) when 
requested by the central DCS operator, e.g. prior to beam injection. The ramping can be performed via FSM commands ({\tt GO\_BEAM\_TUNING, GO\_READY)}) 
or from the TRD\_DCS top node panel (Sec.~\ref{sec:topNodePanel}). \par 

\begin{figure}[h]
\hspace{0.5cm}
\includegraphics*[width=0.8\textwidth]{figs/obusch/HVMonitorPanel.pdf} 
\caption{HV monitoring panel.}
\label{fig:HVMonitor}
\end{figure}

More details about the HV state are displayed in the HV monitoring panel for each sector, shown in Fig.~\ref{fig:HVMonitor}. 
The measured currents and voltages for each drift chamber are published 
to PVSS via OPC server and are displayed, seperately for anode and drift, along with the FSM state of the chambers. A panel with more details 
and the trends of current and voltage can be opened clicking on the corresponing rectangle. The state of the crate and of the OPC server are 
also indicated. All controls, expect the option to switch HV off, need expert privileges. \par 


\subsubsection{HV Crate State}

The HV crates should always be {\tt ON}, otherwise the crate state display will blink red, an alarm will be issued, and the {\tt INFRASTRUCTURE} FSM will 
go to {\tt ERROR}. If this should be the case, e.g. after an instability of the power network, the OPC items from the 
OPC server cache are likely not refreshed with the correct (that is, zero) values of the currents and 
voltages from the device. So, even if these 'measurements' look misleadingly normal, 
the HV crate is off and the experts should be notified. 

\subsection{Infrastructure} 

The {\tt TRD\_INFRA} FSM node governs the following nodes: 

\begin{itemize} 

\item {\tt TRD\_PCU}: power control unit, controls LV to DCS boards
 
\item {\tt TRD\_PT\_LV}: Pre-trigger LV 

\item {\tt TRD\_VME}: Pre-trigger VME crate 

\item {\tt TRD\_HVCRATE\_00, TRD\_HVCRATE\_08}: HV crates 

\item {\tt TRD\_COOLING}: cooling plant and loops 

\item {\tt TRD\_GAS}: gas system 

\item {\tt TRD\_GOOFIE}: drift gas monitor  

\end{itemize}

\subsubsection{PCU} \label{Sec:PCU}
(advanced users) \par
The DCS board power of TRD is controlled by 4 PCUs (2 primary systems: PCU00, PCU01 and 2 redundant backup systems: PCU02, PCU03). 
The PCU panel, shown in Fig.~\ref{fig:PCU}, displays the status of the PCU channels (1 channel = 1 SM). The information 
which DCS boards are on is shown for each sector after clicking the corresponding button. The staus of the 
LV channels supplying the PCU device itself is displayed on the right 
column of the PCU panel (operation of the LV is prohibited). \par 

\begin{figure}[h]
\centering
\includegraphics*[width=0.8\textwidth]{figs/obusch/PCUPanel.pdf} 
\caption{PCU control panel.}
\label{fig:PCU}
\end{figure}

On each PCU a process called {\tt watchdog timer} is running. It switches off the PCU (and hence the DCS boards and FEE of 9 supermodules) 
if the connection to PVSS (where the software interlock is running) and to a TRD DCS workernode (where a second, redundant, interlock is running) 
is lost. The status of the watchdog processes is displayed. \par 
Each PCU also implements a timeout mechanism to switch off the DCS boards if 
the connection to PVSS is lost, i.e. if within a certain time interval no update of the DIM services from PVSS is requested. 
If the timeout is not set, this is indicated by the FSM state {\tt NO\_TIMEOUT}. The time interval 
(say, 30s) can be set by FSM ({\tt SET\_TIMEOUT}) or via the corresponding text field in the PCU panel. 


\subsubsection{Cooling} 
 
The cooling panel shows the state (temperature, pressure etc.) of the cooling plant and cooling loops (1 loop = 1 sector). Further trends can be obtained 
via left-click on the corresponing buttons. 


\subsection{Run Unit}

The run unit the FSM interface beteen the Experiment Control System (ECS) and TRD DCS. It signals run readiness to ECS and receives 
the Start of Run (SOR) and End of Run (EOR) commands. \par 

\subsubsection{RUN\_OK and RUN\_INHIBIT}

Run Readiness is indicated by the FSM states {\tt RUN\_OK} or {\tt RUN\_INHIBIT}. The states are determined by a script crosschecking 
the GTU DDL (detector data link) list of DDLs for which GTU is configured and the ECS DDL list reflecting the DAQ configuration. Furthermore, 
it is checked wether all chambers included in the FSM tree are configured and if the configuration tag agrees with the run type expected by ECS. 
If any mismatch or inconsistency is found, the run unit goes to {\tt RUN\_INHIBIT} and a short notification about the reason is 
displayed in the run unit panel. If needed, the run unit can forced manually to move to {\tt RUN\_OK} executing the corresponding FSM command. 
\par 

\subsubsection{SOR/EOR Scripts} \label{sec:SOR}

The SOR or EOR signal trigger execution of a script to: 

\begin{itemize} 

\item start/stop the pre-trigger 

\item force archiving of all datapoints relevant for offline analysis/calibration into the DCS database 
      (to ensure, that in case of short runs this information is available at least once in the beginning and the end of the run) 

\item at EOR: start rstate logging, the readout of the FEE configuration, counters etc.  via the DCS boards 
 
\end{itemize}

The SOR/EOR scripts can also be executed manually, via the panel opening after clicking the corresponding button on the top node panel. 

%\section {FAQ}

%\begin{itemize}

%\item  I click on the "FSM" button, but the panel doesn't open. 

%\item[$\to$] maybe the small FSM control panel is already open, but hidden behind the large UI panel? 

%\item The UI is frozen 

%\item[$\to$] stop UI via WINDOWS task manager: right-click on grey bar in the bottom {\tt $\to$ Task Manager $\to$ Applications $\to$ Vision\_1:Top $\to$ End Task}. 
%Then restart via  {\tt START $\to$ PROGRAMS $\to$ STARTUP $\to$ TRD CONTROL CONSOLE}

%\item The UI doesn't start up. \\ 
%Error message \textit{D:\textbackslash\textbackslash PVSS\_Projects \textbackslash\textbackslash trd\_ON\_UI \textbackslash\textbackslash 
%config \textbackslash\textbackslash config : proj\_path nonexistent, inaccessible or not a directory: 
%\textbackslash\textbackslash alidcscom099 \textbackslash PVSS\_Projects \textbackslash trd\_fed \textbackslash PVSS00ui}. 

%\item[$\to$] too many users try to access the shared folder on WN003, probably because they didn't log off properly but only locked out. Troubleshooting: 
%kick out disconnected ''users'' with the WINDOWS task manager: right-click on grey bar in the bottom {\tt $\to$ Task Manager $\to$ Users} 
%$\to$ right-click user $\to$ {\tt log off} 

%\item The ON is very slow. 

%\item[$\to$] Maybe too many users have not logged out (but only locked off) and are still running their processes? Start the WINDOWS task manager and 
%kick out disconnected users as explained above. 

%\item I can't control the PCU LV. 

%\item[$\to$] You shouldn't - it should always stay on. The PCU power is plugged to a GTU power supply which can not be switched via PVSS controls. 
%If really needed, you can use the tool ''pcu-monitor'' under LINUX.

%\item The UI doesn't start up. Error message "can't connect - retry in 20 seconds". 

%\item[$\to$] The project on the ON is running as a service called PVSS00pmon. Try to restart/stop and restart the service 
%via {\tt Start $\to$ Programs $\to$ Admnistrative tools $\to$ Services} (restart takes some time, order 1 minute).

%\item (should be obsolete:) The PDB LV has tripped and I don't manage to recover. 

%\item[$\to$] Reset the PCU registers, otherwise the PCU tries to switch on all DCS boards at once when you restart the power supply, 
%making it trip again. From the PCU console, open the panel for the supermodule concerned, and click on "all DCS boards off". Then switch on the 
%PDB power supply and power the DCS boards in the standard way.  

%\end{itemize}



% EOF
