Project Dependencies

General Dependencies

The Cerberus program requires a vast amount of dependencies in order to run. These range from the operating system (CentOS) to smaller things like individual programs (Git, Tree) and smaller parts even still like configuration files.

Luckily for the user, all of this is handled ahead of time with provisioning. This is one of the largest selling points of the project; the user merely has to run the program at it performs any number of self healing tasks. This means that once the program and headnode appliance are installed and activated, the day to day user will have no dependencies to resolve.

UNIX: Tree

Tree is a simple utility that lists all of the files in a directory in a simple manner. This utility is provided under GNU GPL and is used by the Cerberus program to map and error check directories from before the processing is run and afterwards. If there are differences between the two, they are recorded and reconciled.

Tree is available from the standard YUM Repos and accessible through a basic Yum package manager install.

UNIX: Stress and Sysbench

Stress is a multi-purpose benchmark tool to test various aspects of a computer system. Sysbench is very similar in goal and helps test a variety of benchmarks as well.

Stress and Sysbench were used to test the maximum potential of the entire cluster. This was particularly valuable in testing whether or not a given location possessed a reasonable power supply to fuel the cluster. This set of testing tools is also responsible for a variety of power outages across campus as the result of several over-ambitious cluster stress tests.

Stress and Sysbench are both available from the standard YUM Repos and accessible through a basic Yum package manager install.

UNIX: NFS tools

A variety of NFS packages are also installed. This allows the system to connect to all of the FreeNAS fileshares which house project infrastructure and all relevant files. NFS is ideal for use in this project because it is the Linux standard in file sharing and is intensely reliable.

NFS tools are available from the standard YUM Repos and accessible through a basic Yum package manager install.

BBCP

BBCP is a state of the art network transfer tool. This is used to replace usilities like RSync and SCP. BBCP helps fragment, compress and copy files by saturating TCP lines between hosts without costly UDP error checking and endline authentication. This tool is used to copy any and all information between hosts in the cluster and has shown massive speed increases. Frames copy near 140% of the speed they took with SCP previously although exhaustive networking tests will be covered later in this paper.

BBCP is an open-source software package, although it is not a common part of the YUM repositories and has to be downloaded and compiled from source.

Image Processing: GraphicsMagick

Currently, image processing is handled by GraphicsMagick, an open source, C++ based software package designed to run natively on Linux operating systems. GraphicsMagick is an improved, highly optimized version of the immensely popular ImageMagick software package.

GraphicsMagic excels at introducing multithreaded CPU support which is the main draw in implementing this software package over ImageMagick or other UNIX tools. Introducing multi-thread support allows for massive optimization of the code. Testing time comparisons between ImageMagick and GraphicsMagick improved almost 60% when GM was implemented.

GraphicsMagick also allows for LUT operations to be handled in stream. Although covered later in detail, GraphicsMagick intelligently uses the HALD LUT system, which makes it an intuitive piece of the image processing pipeline.