Diamond FA Archiver

The FA Archiver captures full beam position orbit data at 10 kHz to a short term rolling archive, and republishes the live data stream interested client applications. The distribution here includes tools for reading from the archive and displaying live beam data.

The FA Archiver consists of three components which have to be separately installed and configured. The FA Archiver should be run on a dedicated machine with a fibre connection to a beam position communication network.

  • FPGA firmware. The FA Sniffer FPGA image runs on a Virtex-5 LXT ML555 FPGA Development Kit for PCI Express or Simple PCIe FMC carrier (SPEC) and transfers frames from the Diamond Communication Controller network to memory in the archiver machine.
  • FA Sniffer device driver. This is a Linux kernel device driver to provide an interface to the FA Sniffer card. This has been built and run on a variety of kernels.
  • FA Archiver and associated tools. The FA archiver proper consists of three main tools and a number of supporting tools. See notes below for some details.

FA Sniffer Firmware Downloads

ML555/v3.00.3 pcie_cc_top.bit for FPGA configuration on Xilinx ML555 board.
pcie_cc_top.mcs for PROM configuration on Xilinx ML555 board.
SPEC/v1.00.0 spec-fa-archiver.tar.gz includes firmware (*.bin) and sw-tools for the SPEC board.

Programming on-board PROM for ML555

You will need (1) to copy pcie_cc_top.mcs file using the link above, and (2) copy v5_ml555_loadPROM.cmd file in the same directory. Then run impact in command line mode as follows:

 $(XILINX)/bin/lin/impact -batch v5_ml555_loadPROM.cmd 

Programming on-board PROM for CERN SPEC

Extract spec-fa-archiver.tar.gz from the link above and run the command ./fpga_load.py as root.

FA Sniffer Device Driver Downloads

1.9 pcie_sniffer-1.9.tgz Fix regression and bug in 1.8 release.
1.8 (do not use) Adds support for CERN SPEC board. Do not use this release: it causes a kernel panic with the Xilinx card, and it may fail to correctly initialise the clock on the SPEC card.
1.7 pcie_sniffer-1.7.tgz Adds improved 64-bit compatibility and improved timestamp support. Also allow for more than 256 ids in a single transfer.
1.5 pcie_sniffer-1.5.tgz

Building and Installation

This driver should build without any complications, placing the resulting kernel module in

kbuild-$(uname -r)/fa_sniffer.ko

Alternatively, on RPM based systems with dkms run make rpm to build a dkms (architecture independent) kernel module and install the rpm.

After loading the module a device node /dev/fa_sniffer0 should be created, together with associated sysfs nodes. Data can be read from this node to verify operation of the sniffer.

GIT Repository

A git repository containing work in progress for the sniffer device driver can be cloned by running the command:

git clone http://controls.diamond.ac.uk/downloads/other/fa-archiver/pcie_sniffer.git

There is also a copy on github

FA Archiver

v1.22 fa-archiver-1.22.tgz Release April 2015, now running at Diamond. A handful of miscellaneous bug fixes, mostly.
v1.18 fa-archiver-1.18.tgz Release June 2014. The main change is in the way that the mapping from node name to id is handled.
v1.15 fa-archiver-1.15.tgz Release 4th April 2013. A variety of small changes and improvements including:
  • Preload memory mapped pages to reduce risk of data dropout
  • Provide timestamps for continuous decimated data
  • Implement fa_load in pure matlab
v1.12 fa-archiver-1.12.tgz Release 20th Sep 2012. Fixes a bug in release 1.11 where multiple concurrent readers could cause the archiver to hang.
v1.9+ fa-archiver-1.9+.tgz Release 17th Oct 2011, essentially identical to 1.9 release previously running at Diamond.

Building and Installation

The archiver should build, run make, without any complications, placing three tools, fa-prepare, fa-archiver and fa-capture, in the subdirectory build/$(uname -m). If you have Python docutils installed then make docs should place html and man page documentation in build/docs.

Installation of the archiver is optional, there are no library or other dependencies beyond standard system libraries, so each application will run standalone. The Python tools are currently harder to install as there are some unresolved internal Diamond dependencies; any feedback on automating this installation is welcome, and I am happy to answer any inquries.

Read the man pages for information about using the archiver, but a brief summary follows below.

  1. Define an area of disk to use for the archive. At Diamond we use Linux disk mapping to join a set of dedicated 3TB disks into a single 30TB volume: this gives us 16 days of archive for all our BPMs. For large archives a dedicated block device is highly recommended, but an ordinary file can be used. Let's call this $DISK.
  2. Determine which FA IDs will be archived. At Diamond we archive the following ids: 1-255. Let's call this $FAIDS.
  3. Initialise the archive using the fa-prepare command. If $DISK is a block device no size needs to be given and the following command will serve:

    fa-prepare $FAIDS $DISK

    otherwise -s must also be used to specify the file size.
  4. The archive is now ready for use. Run the archiver:

    fa-archiver -c filters/decimate.config -t $DISK

To retrieve data from the archiver use the fa-capture command, or in Matlab the fa_zoomer tool can be useful.

GIT Repository

A git repository containing work in progress for the FA archiver can be cloned by running the command:

git clone http://controls.diamond.ac.uk/downloads/other/fa-archiver/fa-archiver.git

There is also a copy on github

Documentation

FA archiver documentation Manual pages
Paper presented to ICALEPCS 2011 THCHMUST03.pdf
Slides from presentation fa-archiver.pdf