CytoBuoy Data File Access

This repository contains a DLL that allows accessing the data generated by CytoBuoy instruments directly from your own (.net) software. The project was created to allow our customers to implement their own data processing pipelines for handling our files, independent of the analysis software that we provide.

Open sourcing this code assures them that access to their data is guaranteed now and in the future. The work involved in open sourcing this was paid for (in part) by the OBAMA-NEXT project.

Licensing

The licensing terms for the software are described in the file CyzFile-API_LICENSE.TXT, in laymen's terms, this is a modified MIT license where the rights for you to sell the software or use our name in publications without consent have been removed. For details consult the actual license text.

Repository

The repository consists of 3 projects (and a Visual Studio Solution):

1. CyzFile: The actual DLL that can be used to read the data files from our instruments.
2. TestCyzFile: A test project that tests some aspects of the DLL using existing data files.
3. CyzFileDump: This is an example command line tool that opens a data file, and dumps a selection of the data to the console.

This library was not written from scratch for this repository, instead it was extracted from existing software. And to ensure compatibility now, and in the future, it is still used by our software for reading and writing the data files. Parts of the software are more then 10 years old and e.g. deal with old instrument with rarely used options. Over the years dependencies have slipped into the code that, in hindsight, were not a very good idea. However for reasons of backwards compatibility to a large amount of existing data files, or because the amount of time involved would be prohibitive not al these dependencies were removed.

This means that even though the purpose of (providing) this library is to extract measurement data from the .cyz file, you may run into code related to other things (such as analyzing or plotting data, creating datafiles, etc.) This code will probably be incomplete, is not documented, and I do not expect you to be able to use it, and you should not :-).

CyzFileDump

The CyzFileDump was written as a (very quick) introduction to the library. It shows you how to open a data file and extract the some of the important data from it.

The data file contains several groups of information (and then some other data.):

1. Instrument Description: Which instrument was used, what features does it have.
2. Measurement Information:
   • Settings used for the measurement
   • Results: how many particles were measured, how much fluid was analyzed, etc.
3. Auxillary Sensor Data, such as temperatures, pressures, GPS data, etc.
4. Particle Data: The actual information for each measured particle, possibly including images for a subset of the particles.

We will have a quick look at getting each of the different kinds of information.

Opening a data file

Step one is opening the actual datafile, to open a data file, create a DataFileWrapper wrapper class, passing in the name of the file to open. The constructor will take care of loading the data file, and processing any differences between the different versions of the file format that exist. Depending on the actual file format, loading of the particle data may be postponed until the first particle is accessed.

This is demonstrated below:

using using CytoSense.Data;

DataFileWrapper dfw = new DataFileWrapper(filename);
dfw.CytoSettings.setChannelVisualisationMode(ChannelAccessMode.Optical_debugging);
ChannelData.VarLength = 13; // Calculate the alternate variable height length parameter at a height of 13%

NOTE: dfw is short for DataFileWrapper, you will see that often in this document and the sample code.

In the example we set the ChannelVisualizationMode to Optical_debugging. Doing this will allow access to the split Forward Scatter Left(FWS) and Forward Scatter Right channels. These can be used to check the alignment of the instrument. Normally only the combined Forward Scatter is available, but when you enable the optical debugging mode, then the channel list will contain both the combined FWS channel, and the split FWS Left and FWS Right channels. (For instruments that have separate FWS detectors, most have but a few very old ones do not.)

The next line controls the calculation for the Variable Height Length Parameter, please refer to the CytoClus documentation for details on this parameter. It is a variation of the normal length parameter, but calculated at a different height of the pulse shape.

Some older instruments had problems with concentration measurements. This results in data files where the measured pre-concentration is significantly different from the concentration during the measurement. In most cases the pre-concentration is the correct one, but not always. Simply opening these files and using the concentration data will result in an exception. There is a special check function that can be used to check the files before hand, and select explicitly which concentration to use. Depending on the data files, you could choose to always use this code, or to ignore it until you run into the exception.

The code to check is in the example program:

double concentration = 0.0;
double preconcentration = 0.0;
if (dfw.CheckConcentration(ref concentration, ref preconcentration))
{
    Console.WriteLine("CONCENTRATION MISMATCH!");
    Console.WriteLine($"Pre-Concentration: ${preconcentration}, Concentration: ${concentration}");
    Console.Write($"Would you like to use the PRE concentration instead of the concentration(Y/N)? :");
    char c =  Console.ReadKey().KeyChar;
    if (c =='y' || c=='Y')
        dfw.ConcentrationMode = ConcentrationModeEnum.Pre_measurement_FTDI;
    else
        dfw.ConcentrationMode = ConcentrationModeEnum.During_measurement_FTDI;
    Console.WriteLine();
}

Instrument Description

A description of the instrument and all its features and options is stored in every measurement file. A large part of that data is only interesting to us and our software for controlling the instrument (e.g. information on which option is connected to which IO pin on the micro-controller). Other information such as the name/serial number, the number (and type) of the detector channels, the camera installed in the system (if any), optical magnification, etc. are also interesting to the user. This information is available via the CytoSettings property in the DataFileWrapper class. An example of it's use is shown in the DumpGeneralInformation method in the example project:

private static void DumpGeneralInformation( DataFileWrapper dfw )
{
    // ... 
    Console.WriteLine($"Instrument Information:");
    Console.WriteLine($"    Name:               {dfw.CytoSettings.name}");
    Console.WriteLine($"    Serial Number:      {dfw.CytoSettings.SerialNumber}");
    Console.WriteLine($"    Number of Channels: {dfw.CytoSettings.ChannelList.Count}");
    int numChannels = dfw.CytoSettings.ChannelList.Count;
    for (int chIdx=0; chIdx<numChannels; chIdx++ ) 
        Console.WriteLine($"        Chan {chIdx}:         {dfw.CytoSettings.ChannelList[chIdx].Description}");

   // ...
}

Looking through the properties of the CytoSettings object will show you much more available properties.

Measurement Information

Configuration

There is quite a bit of configuration involved in running an actual measurement. All this information is again stored in the data file, so you can always check how a measurement was actually performed. The basic configuration is stored in an a member of the DataFileWrapper called: MeasurementSettings. It's use can again be seen in the DumpGeneralInformation method in the example project:

private static void DumpGeneralInformation( DataFileWrapper dfw )
{
    // ... 
    Console.WriteLine($"Measurement Settings:");
    Console.WriteLine($"    Name:              {dfw.MeasurementSettings.TabName}");
    Console.WriteLine($"    Duration:          {dfw.MeasurementSettings.StopafterTimertext} (seconds)");
    Console.WriteLine($"    Samplepump Speed:  {dfw.MeasurementSettings.ConfiguredSamplePompSpeed} (muL/s)");
    Console.WriteLine($"    Trigger Channel:   {dfw.MeasurementSettings.TriggerChannel}");
    Console.WriteLine($"    Trigger Level:     {dfw.MeasurementSettings.TriggerLevel1e}");
    Console.WriteLine($"    Use Smart Trigger: {dfw.MeasurementSettings.SmartTriggeringEnabled}");
    if (dfw.MeasurementSettings.SmartTriggeringEnabled)
        Console.WriteLine($"    Smart Trigger:     {dfw.MeasurementSettings.SmartTriggerSettingDescription}");
    Console.WriteLine($"    Take Images:       {dfw.MeasurementSettings.IIFCheck}");
   // ...
}

Please check the actual object for a list of all the properties stored.

Results

Overall results of the measurement are stored (for the most part) in the MeasurementInfo property. FOr a usage example please check the DumpGeneralInformation method in the example project:

private static void DumpGeneralInformation( DataFileWrapper dfw )
{
    // ... 
    Console.WriteLine($"Measurement Results:");
    Console.WriteLine($"    Start:                        {dfw.MeasurementInfo.MeasurementStart}");
    Console.WriteLine($"    Duration:                     {dfw.MeasurementInfo.ActualMeasureTime}");
    Console.WriteLine($"    Number of Counted Particles:  {dfw.MeasurementInfo.NumberofCountedParticles}");
    Console.WriteLine($"    Number of Particles in File:  {dfw.MeasurementInfo.NumberofSavedParticles}");
    Console.WriteLine($"    Number of Pictures:           {dfw.MeasurementInfo.NumberOfPictures}");
    Console.WriteLine($"    Pumped Volume:                {dfw.pumpedVolume} (muL)");
    Console.WriteLine($"    Analyzed Volume:              {dfw.analyzedVolume} (muL)");
    Console.WriteLine($"    Particle Concentration:       {dfw.Concentration} (n/muL)");
   // ...
}

As you can see, not all information is available via the MeasurementInfo, some of it, such as pumped volume or particle concentration si available directly from the DataFileWrapper.

Auxillary Sensor Data

This is a record of all other data that is recorded during a measurement, this includes the temperature in various parts of the system, the internal pressures and for some systems the external pressures as well. A log of GPS coordinates during the measurement, if a GPS was available, etc. An average temperature or pressure can be retrieved from the MeasurementInfo directly. This can be seen in the following example:

private static void DumpGeneralInformation( DataFileWrapper dfw )
{
    // ... 
            Console.WriteLine($"Auxiliary Sensor Data:");
            Console.WriteLine($"    Average System Temperature:    {dfw.MeasurementInfo.SystemTemp} (C)");
            Console.WriteLine($"    Average Sheath Temperature:    {dfw.MeasurementInfo.SheathTemp} (C)");
            Console.WriteLine($"    Average Absolute Pressure:     {dfw.MeasurementInfo.ABSPressure} (mbar)");
            Console.WriteLine($"    Average Differential Pressure: {dfw.MeasurementInfo.DiffPressure} (mbar)");

    // ...
}

If you are interested in all the actual measurements they are provided via the MeasurementInfo.sensorLogs, which contains for each sensor a list of all the measurements (usually once per second). The actual sensors available will depend on the instrument. So often only some of the lists will be populated with data, as the other sensors were simply not present in the instrument.

Particle Data

The data file wrapper contains an array of all the particles, which is called SplittedParticles (for historic reasons). To access the particles you simply loop over the array. Each parameter supports a number of channels, this number and the type of the channels can be found in the instrument description. And in the documentation for the specific instrument.

Then for each channel, several different parameters can be retrieved, such as the maximum value, or the total amount. These are the parameters that are normally used in dot plots to analyzed the data. The different parameters are documented in the help files for our CytoClus software.

An example of accessing this is given below, here we look over the (first 100) particles, and we output the number of samples a particle was long, for the FWS channel the calculated particle length, and for the FlRed channel the total amount. And finally a flag to indicate if we have an image for the particle.

For the number of samples we access p.ChannelData[0].length, this channel data is the actual recorded pulse for this particle, so via the ChannelData array you access to the pulse shapes for each of the detector channels.

private static void DumpParticleInformation( DataFileWrapper dfw )
{
    int numParticles = dfw.SplittedParticles.Length;
    for(int pIdx=0; pIdx<numParticles; ++pIdx)
    {
        if (pIdx >= 100)
            break;
        var p = dfw.SplittedParticles[pIdx];
        Console.WriteLine($"{pIdx,3}: #samples={p.ChannelData[0].Data.Length}, " + 
                            $"length={p.getChannelByType(ChannelTypesEnum.FWS).get_Parameter(ChannelData.ParameterSelector.Length)}, " +
                            $"total red={p.getChannelByType(ChannelTypesEnum.FLRed).get_Parameter(ChannelData.ParameterSelector.Total)}, " + 
                            $"image={p.hasImage}");
    }
}

To get a clearer overview of all the information available, the example program also contains the function DumpCompleteParticle which is called only for the first information:

private static void DumpCompleteParticle( Particle p)
{
    Console.WriteLine($"Particle ID: {p.ID}");

    Console.WriteLine($"Pulse Shapes:");
    foreach( ChannelData cd in p.ChannelData)
    {
        string[] values = cd.Data.Select((v) => v.ToString(System.Globalization.CultureInfo.InvariantCulture)).ToArray();
        Console.WriteLine($"    - {cd.Information.Description}: {String.Join(',', values)}");
    }

    Console.WriteLine($"Parameter Values:");
    foreach( ChannelData cd in p.ChannelData)
    {
        Console.WriteLine($"    - {cd.Information.Description}:");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Length]}: {cd.get_Parameter(ChannelData.ParameterSelector.Length)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Total]}: {cd.get_Parameter(ChannelData.ParameterSelector.Total)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Maximum]}: {cd.get_Parameter(ChannelData.ParameterSelector.Maximum)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Average]}: {cd.get_Parameter(ChannelData.ParameterSelector.Average)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Inertia]}: {cd.get_Parameter(ChannelData.ParameterSelector.Inertia)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.CentreOfGravity]}: {cd.get_Parameter(ChannelData.ParameterSelector.CentreOfGravity)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.FillFactor]}: {cd.get_Parameter(ChannelData.ParameterSelector.FillFactor)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Asymmetry ]}: {cd.get_Parameter(ChannelData.ParameterSelector.Asymmetry )}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.NumberOfCells ]}: {cd.get_Parameter(ChannelData.ParameterSelector.NumberOfCells )}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.SampleLength]}: {cd.get_Parameter(ChannelData.ParameterSelector.SampleLength)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.TimeOfArrival]}: {cd.get_Parameter(ChannelData.ParameterSelector.TimeOfArrival)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.First]}: {cd.get_Parameter(ChannelData.ParameterSelector.First)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Last]}: {cd.get_Parameter(ChannelData.ParameterSelector.Last)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.Minimum]}: {cd.get_Parameter(ChannelData.ParameterSelector.Minimum)}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.SWSCOV ]}: {cd.get_Parameter(ChannelData.ParameterSelector.SWSCOV )}");
        Console.WriteLine($"        + {ChannelData.ParameterNames[(int)ChannelData.ParameterSelector.VariableLength ]}: {cd.get_Parameter(ChannelData.ParameterSelector.VariableLength)}");
    }
}

It first writes the recorded pulse for each channel to the output, and then it calculates all the supported parameters for each of the channels.

Image Data

In the example above we checked if a particle has an image, when looking specifically for images, there is a different array: SplittedParticlesWithImages, this contains only the particles with images, so it makes processing just the images a little easier and faster.

The image part of the sample program looks like:

if (exportImages)
{ 
    Console.WriteLine($"Exporting background image");
    Cv2.ImWrite("background_image.jpg", dfw.CytoSettings.iif.OpenCvBackground);
    ExportImages(dfw);
    ExportCroppedImages(dfw);
}

Apart form images of particles we (almost) always take an image of the empty flow cell, this is called the background image and is used for our automatic image processing. If you want to implement your own image processing then you will need this image as well. (It is provided as an OpenCV Mat object, so we use OpenCV functions to save it to a file.)

The next step is accessing the complete images. These are stored in the file as a JPEG compressed image stream, so exporting them to file is simply copying the stream to a file, no processing is needed.

private static void ExportImages( DataFileWrapper dfw )
{
    int numImagedParticles = dfw.SplittedParticlesWithImages.Length;
    for( int imgIdx=0; imgIdx<numImagedParticles;++imgIdx)
    {
        if (imgIdx >= 100)
            break;

        var imgP = dfw.SplittedParticlesWithImages[imgIdx];
        string imgName = $"particle_{imgP.ID}.jpg";
        Console.WriteLine($"{imgIdx,2}: Writing {imgName}");
        using (FileStream fileStream = new FileStream(imgName, FileMode.Create, FileAccess.Write) )
        {
            imgP.ImageHandling.ImageStream.Position = 0;
            imgP.ImageHandling.ImageStream.CopyTo( fileStream );
        }
    }
}

Each imaged particle contains an ImageHandling property, that has an ImageStream, which contains the complete recorded image as a JPG file. In the sample above we just copy the image into a data file. Note that you should not forget to reset the image streams position to 0 before you start using it. Sometimes it is still set to the end because of earlier use of the property and then you would see an empty file.

And finally the ExportCroppedImages function will use the built in object detection to extract images of just the object.

private static void ExportCroppedImages( DataFileWrapper dfw )
{
    int numImagedParticles = dfw.SplittedParticlesWithImages.Length;
    for( int imgIdx=0; imgIdx<numImagedParticles;++imgIdx)
    {
        if (imgIdx >= 100)
            break;

        var imgP = dfw.SplittedParticlesWithImages[imgIdx];
        string imgName = $"particle_{imgP.ID}_cropped.jpg";

        var crpImg = imgP.ImageHandling.GetCroppedImage(25, 1.1, 7, 1);
        if (imgP.ImageHandling.CropResult == CytoImage.CropResultEnum.CropOK)
        { 
            Console.WriteLine($"{imgIdx,2}: Writing {imgName}");
            Cv2.ImWrite(imgName, crpImg);
        }
        else // there was a problem cropping the image, examine the result enum to see what the problem was.
        {
            Console.WriteLine($"{imgIdx,2}: Cropping failed ('{imgP.ImageHandling.CropResult}')");
        }
    }
}

The image handling part provides the function GetCroppedImage. The functions takes several parameters, these are all described in the CytoClus documentation. The returned image is an OpenCV Mat object, this can be saved to disk using OpenCV functions just as we did with the background image.