Pdev data taking software.
- Jeff's code:
- AO System Overview
- The Details
What jeff provided:
Jeff provided the following programs:
-> prun -> psrv -> disc
- pnet - A perl script to
drive the data taking. It talks to 1 or more prun tasks running on the
spectrometer boxes. It also can route messages through the prun task to
the psrv task on the file servers. It is driven by command line options
and several xxx.conf
- prun- c program running
on the spectrometer box.
- There is one copy for each band of a spectrometer (two per
- It controls the spectrometer hardware.
- It takes commands from pnet. It can forward commands from pnet
to psrv and pass back the replies.
- prun starts the data taking, gets the data from the
spectrometer, and passes it via a socket to the psrv program running on
the file server.
- When the integration is done, the prun process exits.
- psrv - C program running
on file server that reads data from prun and writes it to disc.
- There is a psrv task for each prun task (one for each subband
on a spectrometer box).
- prun starts psrv via xinetd by connecting to the well known
- psrv accepts commands from pnet via prun.
- When data taking starts, psrv reads the data stream over the
same socket that was used for the comands.
- Data from the input socket is buffered in a circular 20 mb
- By default the sequence numbers of each integration are checked.
- Data is written to disc as it comes in. Disc files can grow to
a user defined maximum size before switching to the next file.
- When the requested number of buffers have been input , psrv
stops writing to disc.
- Psrv will exit when the input socket is closed by prun.
- psrv will also exit if it has an error.
- Messages can be sent back msgs to pnet via prun using the input
How jeff's datataking
starts, runs, and stops
Starting/connecting: pnet -> prun -> psrv
The user starts the perl script pnet.
Pnet starts prun and then psrv:
- pnet connects to prun (starting it if not running) on each
- /etc/inetd.conf on the spectrometer starts prun
- /etc/services on the spectrometer box has port 1421 attached to
- pnet (in routine pnet_connect_prun) connects to the socket 1421
on the spectrometer box. This starts prun
- pnet connects to prun on file servers.
- pnet sends the A command
to each prun box. This message contains the file server name:port for
- prun uses gethostbyname() to find the address of the file
server. The port is extracted from the filename.
- prun connects to psrv on the file server.
- On the file server psrv is started by xinetd. The port number
is 1421 (in the services file).
- Now pnet is connected to each prun on each box. Each box has a
connection to their own file server. pnet can send commands to psrv via
prun. The X cmd allows for a generic message to be forwarded to
psrv via prun.
- The configuration files are read by pnet.
- The config info is sent to prun (pnet_setup)
- pnet_dump is called to start the datataking:
- The first available file sequence number is found by querying
all psrvs (O cmd).
- Filename with seq number is sent to all psrv's (P cmd).
This opens the files.
- pnet makes the headers and then sends the 4K block to each psrv
(Q cmd). The command includes the number of dumps that are to be taken.
At this point psrv is in data acquistion mode (just waiting for data
blocks from prun).
- dma is engine is started on prun (pnet_startdma sends L command
- pnet_sync() will wait for the next 1 PPS from the first
spectrometer box (assuming not start immediate). This uses the H cmd.
- pnet sends the M cmd to prun to take data forever.
- While taking data:
- psrv will send back a status message every 1 second (w
command). This is passed back thru prun.
- prun does a select on: read: psrv, fpga, and pnet. It does a
select on write for psrv.
- pnet is waiting on any messages from prun.
- psrv is counting the blocks of data it reads from prun. The
number it needs to complete the run was sent during the writeheaders
- When psrv reads the final block of data, it sends a message back
to pnet (x cmd) saying that it has completed the run. In the meantime
psrv will continue reading from prun but the data will be discarded.
- pnet receives the x (done) message from each psrv. When the
number of end messages equals the number of spectrometers running, pnet
- pnet exitting causes the socket to prun to close.
- When prun sees the socket from pnet returning an error, it exits
(after stopping the datataking).
- When psrv sees the socket from prun return an error, it exits.
changed in jeff's design
the design to: pnet -> prun ->
-> sharedMemoryBuffers --> bpInp --> shareDMemory -> bpOut
I made the following
modifications to psrv:
- The 20 mb input buffer was discarded.
- psrv requests a shared memory buffer from the shared memory
free list when it needs to store info.
- For headers or commands it asks for a small buffer (about 4k)
- For storing data it requests a large buffer (about 20Mb).
- Data buffers are always filled with an integral number of
integrations. An integration will never span a buffer.
- When a buffer is filled, all the sequence numbers in the buffer
are checked at once. This is easy since they are located at fixed
intervals. All the code to keep track of partial record headers is gone.
- When checking the sequence numbers, the 8 byte header is byte
swapped so it matches jeff's pdev documentation. The sequence number
and fft integration are in the first 32 bits, the overflow flags are in
the 2nd 32 bits.
- The filled buffer is then placed on a queue in shared memory
(bpInp) to be processed by the bpInp program.
- For now, some commands from pnet will be passed to the bpInp
program.. File names and max sizes so that the i/o programs will know
where to write the data. Eventually this will be replaced by commands
from the cima gui.
AO system overview:
The buffering system:
The datataking uses
a large block of shared memory to buffer the input data. Two sets of
fixed length buffers are allocated: large buffers of 20 Mb each and
small buffers of 4kb each. Queues (linked lists in shared memory) are
used to keep track of the free buffers, and various stages of
processing. Programs request a free buffer from the free list,
fill/process the buffer, and then place the buffer on the next queue
for processing. When done the with a buffer, it will be put back on the
Buffers can hold data (usually the large buffers) or
commands for the program (the small buffers). The data processing is
queued with a possibly large pipeline delay. Commands to a program go
thru the same queueing system as the data buffers to insure that the
commands are executed in the correct order.
The buffers system is centered around a Node. There
is 1 node for each buffer in the system. The first part of each node
contains pointers that can be used to link the node into a linked list.
Nodes also contain some space to specify what they contain (data,
header, command, etc..). The final elements are the pointers to the
buffer for this node. The node to buffer mapping is setup by bufpoolD
at boot time and does not change. The pointers/addressing are done
using offsets from the start of shared memory rather than absolute
addresses since different processes can map shared memory into their
local address space at different locations. The routines make the
addressing transparent to the user (when you want it, you get a ptr to
programs will be multi threaded. One thread will wait on the input
queue for that program. When a buffer becomes available, it will
process it and then pass it on to the next step. Once a second this
thread will wake up to see if there is anything else to do. A second
thread will wait on communications from the outside world (probably a
tcp socket?). When a request arrives (say from the gui) it will
process it and probably place it on the input queue for the first
program. This guarantees that the commands are executed in the correct
Some things that need to be worked out with the
queueing system are:
- When a command gets processed from a queue, there may need to be
a reply going back somewhere. This requires some type of address in the
queued command for the reply.
The current list of programs:
- bufpoolD: This is a daemon that gets started
at boot time. It allocates the shared memory block, fills in the data
structures in shared memory, and then sits around forever. It needs to
be run by root to be able to allocate more than 32k of locked shared
memory. Since it never exits, the shared memory will not be freed.
- psrv: A modified version
of jeff mocks psrv. It requests a free buffer , fills it with data from
the spectrometer, and then places this buffer on the bpInp list in
shared memory for further processing.
- bpInp: Takes data off of
the bpinp queue, bit flips the data, does any other type of processing,
and then places the data on the bpout queue.
- bpOut: Takes buffers off
of the bpout queue and then outputs them to wherever the user has
requested. If fits files are being written it will probably merge the
header info from the scramnet before writing here.
The Queues in shared memory:
- bpfrlL: Buffer Pool Free
List of Large buffers. These are 20 Mb each. The data gets put here.
- bpfrlSt: Buffer Pool
Free List of Small buffers. Headers and command can be placed in these
Buffer Pool INPut buffers. psrv fills buffers and places them here for
further processing. The program bpInp will take the buffers off of this
list and process them.
- bpout: Buffer Pool
OUTput buffers. The program bpInp will place the processed buffers
here. The program bpOut will take the buffers from here an output them.
- Inputs data from
- It reads data from prun via a socket. It places this data in
buffers requested from shared memory. When the buffer is full, psrv
places the buffer on the bpinplist
in shared memory. It then continues to read from the input socket.
- I've tried to minimize the changes in jeffs routine. Most of
the processing has been moved to the bpInp and bpOut programs.
- processes the data in the
- It waits for buffers on the bpInp
Once a second it wakes up to check if it has been signaled to exit.
- It will process the buffer and then place it on the bpOut queue so it can be written
- The processing depends on the type of buffer:
- Cmd buffer:
- If this is a comand buffer than it will do something with
the command. For now it will pass filenames on to the output program.
- Hdr buffer:
- When a header buffer arrives, it is stored in a data
structure for later use. There is room to store 2 separate headers (one
from each 170 Mhz band). The bit flip index array is recomputed if the
fftlen or the start/stop channels have changed.
- For now the header is also put on the output queue so it
can be written out.
- Data Buffer:
- For each integration in the data buffer, the spectra are
bit reversed to put them in the correct fft order. The start/stop
bins option means that this can not be done in place. At 65 mb/sec this
take about 30% of 1 cpu.
- Any other processing that needed to be done could be
inserted in this routine.
- The buffer is then placed on the output queue so it can be
- The fits generating routines could be here or in the bpOut
program. It may be more convenient to put them in the bpOut routine
since they could then write each record to disc as it was processed. If
it was done here, we'd have to merge the header and data in a shared
memory buffer, pass it to bpOut and then write it to the header and
stack locations of the data file.
- Take the
processed buffers and output them somewhere.
- It waits for buffers on the bpOut
queue. Once a second it will wake up to check if it should
- Cmd buffers. These might be:
- Filename and max file size. Store this for later use.
- enable/disable file formats. I can think of allowing a
.pdev file format and a .fits file format for now.
- enable/disable i/o destinations. This command can turn on/off
recording to a particular device (disc, socket, etc..). I'm not sure
whether this should be the same for all file formats? maybe you want to
do online monitoring with .pdev file formats while disc i/o with fits
- any other commands that we can think of.
- hdr buffers:
- For now store the hdr buffer in case we need it for later
- If fits file processing is enabled we may want to build up
the fits header info from the scramnet info when we get a new pdev hdr.
- If .pdev output enabled, output the header to all enabled
- Data buffers:
- If .pdev output is enabled, output to all enabled devices.
Keep track of max file sizes so we can switch when we hit the max size.
- If .fits file output is enabled, grab get the field info for
this rec and output it as well as the raw data.
- Some things to thing about:
- The data buffers can hold many integrations. For slow
integration times there may be a large lag between when the record was
taken and when we process it for output. This means that we will have
to buffer the scramnet info and have a timestamp for each integration
in the buffer (or at least 1 timestamp for the start of the buffer).
- For slow data taking it may be better to not use the entire
20 Mb buffer. Just put one integration (128Kb) in a 20mb buffer and
pass it on. Then the delay from data taking to output won't be so long.
At least there should be a command to limit how many integrations we
put in a single buffer.