This page explains how Serial Studio builds a dashboard from incoming data. Once you're familiar with the user interface, the next step in mastering Serial Studio is gaining a high-level understanding of how it works. This knowledge will help you to adapt Serial Studio to suit your needs, no matter how complex your project is.

Who should read this: Anyone who wants to understand how Serial Studio processes data, especially if you're working with custom data formats or debugging parsing issues.

Prerequisites: Complete the Getting Started tutorial first.

What you'll learn:

• How data flows from device to dashboard
• What frame delimiters are and how to use them
• How the frame parser function works
• Data conversion methods (Plain Text, Hex, Base64, Binary Direct)

Quick Data Flow Pipeline:

Device → Circular Buffer → Frame Identification (delimiters) →
Data Conversion (if configured) → JavaScript Parser → Datasets → Dashboard Widgets

Related pages:

• JavaScript API Reference - Complete parser function documentation
• Project Editor - Where you configure parsing
• Troubleshooting - Common parsing problems

We'll begin with a simple diagram and gradually introduce more details.

Whether you're a new user or someone trying to convince others to use this tool, you might start with the following explanation:

"Serial Studio lets you create a dashboard for your embedded project and optionally export data as CSV."

Here’s a visual representation of this idea:

The left side represents your embedded device, which generates the data. Serial Studio processes that data and creates real-time visualizations. While the specifics of your device are beyond the scope of this document, Serial Studio provides a flexible framework to manage and consume data effectively.

The process that Serial Studio uses to generate dashboards can be divided into three steps:

1. Data Buffering: Incoming data is stored in a circular buffer.
2. Frame Identification: An algorithm reads the buffer to find useful frames of data.
3. Frame Parsing: Each frame is parsed into meaningful pieces of information.

This flow of data can be visualized as follows:

A circular buffer is a data structure that uses a fixed-size buffer to manage incoming data efficiently. Think of it as a queue where data wraps around to the beginning once the buffer is full. You don’t need to understand the technical implementation to use Serial Studio, but it’s important to know why we need it:

• Your device might send data with unpredictable delays or timings.
• The length and format of your data frames may vary.
• The buffer helps manage incoming data consistently before processing.
• The buffer implementation ensures that memory usage is not (too) excesive.

Since we don't know the format of your data, we need a way to identify where each frame of data begins and ends. This is done using frame delimiters — special characters or strings that mark frame boundaries.

Serial Studio provides three approaches to handle incoming data:

1. No delimiters: Treat all incoming bursts of data as frames.
2. End delimiter: Use a specific character or string (e.g., a newline \n) to separate frames.
3. Start and end delimiters: Use specific characters or strings to mark both the start and end of a frame.

If you've followed the Quick Plot Tutorial, you’ve seen the second approach in action, where a newline character serves as the frame end delimiter.

Consider the following data received from your embedded device:

$12:00 PM,45,3.5,90,15.2;$12:05 PM,48,4.0,87,16.1;$12:10 PM,50,3.8,88,15.9;

By using frame delimiters, we can split this into individual frames for further processing. In this case, the start delimiter can be set to $ and the end delimiter can be set to ;, producing the following frames:

12:00 PM,45,3.5,90,15.2
12:05 PM,48,4.0,87,16.1
12:10 PM,50,3.8,88,15.9

Once frames are identified, the next step is to extract meaningful data from within each frame. To do this, we interpret the content of each frame by splitting it into smaller items. These items can then be assigned specific properties and displayed in your dashboard.

Imagine you're tracking a potato launcher to analyze its performance. Your device logs the following data:

Each row is a frame, each cell is an item/value and each column corresponds to a dataset. A dataset doesn’t just contain the value and title of an item—it also specifies properties that define how it’s displayed. For example:

• Widget Type: Line plots, bar indicators, LEDs, gauges, etc.
• Analysis Tools: Options like FFT (Fast Fourier Transform) for frequency analysis.
• Metadata: Descriptions or units of measurement (e.g., "m/s" for speed).

The following diagram illustrates how Serial Studio processes frames and maps data to datasets:

The continuous mapping of frame data to datasets creates the foundation of your dashboard. Over time, Serial Studio dynamically updates the dashboard as new frames are received.

📝 Groups are simply a collection of datasets that are related in some way and can be used to build more complex widgets, such as attitude indicators, plots with multiple curves and GPS maps.

Serial Studio provides a flexible way to parse frames using the Frame Parser Function—a piece of JavaScript code embedded in your project file. This function processes incoming frames and splits them into arrays of data, which are then used to populate your dashboard.

By default, the parser splits frames using commas as delimiters:

function parse(frame) {
    return frame.split(',');
}

For example, if the incoming frame is:

12:05 PM,48,4.0,87,16.1

The parser transforms it into an array:

["12:05 PM", "48", "4.0", "87", "16.1"]

This array is then used by Serial Studio to populate the dashboard. Each element in the array corresponds to a dataset, and you can assign properties to each dataset via the Project Editor, enabling real-time visualization.

Serial Studio supports more complex formats, such as JSON, binary data, and key-value pairs. You can customize the Frame Parser Function to handle these formats. For instance, here’s how you could parse a JSON-based frame:

function parse(frame) {
    if (frame.length > 0) {
        let data = JSON.parse(frame);
        return [data.time, data.speed, data.angle, data.distance];
    }
    
    return [];
}

Given the following JSON frame:

{
    "time": "12:05 PM",
    "speed": 48,
    "angle": 4.0,
    "distance": 87
}

This parser would return:

["12:05 PM", 48, 4.0, 87]

This output array is then used by Serial Studio to dynamically update the dashboard.

Before the frame reaches the parsing stage, Serial Studio offers an optional data conversion step, which can be configured in the Project Editor. This feature allows you to convert raw binary data into a format suitable for parsing, such as a hex string or Base64-encoded string. This is especially useful when working with devices that send binary data, as the frame argument passed to the JavaScript parser is always a string.

Before version 3.1.1, all frame data was passed to the JavaScript parser as a string. This meant that binary data had to be converted ( typically to hexadecimal or Base64) to avoid corruption or parsing errors caused by invalid characters in the string.

The data conversion step ensures that binary payloads are safely transformed into a text-compatible format, which the parser can then decode.

Supported conversions include:

• Hexadecimal Strings: Converts binary data to a hex-encoded string (e.g., 0x12 → "12").
• Base64 Strings: Encodes binary data in Base64 format for compact and safe string representation.

As of Serial Studio v3.1.1, if you select Binary (Direct) mode, this conversion step is skipped entirely. The raw QByteArray is passed directly to the JavaScript parser as an array of bytes, allowing efficient and lossless binary parsing without string encoding.

As of version v3.1.1, Serial Studio introduces a new decoder mode: Binary (Direct). This mode allows the raw QByteArray data to be passed directly to the JavaScript parser as a JavaScript array of byte values (0–255), bypassing all intermediate string conversions.

This is ideal for high-performance binary protocol handling, where frame data should be interpreted at the byte level without UTF-8, hex, or Base64 encoding.

Example:

function parse(frame) {
    let values = [];
    for (let i = 0; i < frame.length; ++i)
        values.push(frame[i] * 5.0 / 255); // Scale byte to 0–5V
    return values;
}

The HexadecimalADC example has been updated to work with binary data conversion.

Note: To maintain compatibility with older project files and existing scripts, the following decoder modes still perform byte-to-string conversion before calling the JavaScript parser:

• Plain Text (UTF-8)
• Hexadecimal
• Base64

Only Binary (Direct) provides raw byte-level access in the JS parser.

Suppose your device sends the following binary frame:

[0x12, 0x34, 0x56, 0x78]

Using the hexadecimal data conversion option, the frame is converted to:

"12345678"

You can then decode this in your Frame Parser Function:

function parse(frame) {
    // Split the hex string into an array of byte values
    let bytes = [];
    for (let i = 0; i < frame.length; i += 2) {
        // Take two characters (one byte) at a time and convert to a number
        let byte = parseInt(frame.substring(i, i + 2), 16);
        bytes.push(byte);
    }

    return bytes;
}

This function would return:

[18, 52, 86, 120]

If your device sends the same binary frame and you choose Base64 encoding, the converted frame would look like this:

"EjRWeA=="

You can decode this in your Frame Parser Function by implementing a naive atob() function:

/**
 * Decodes a base64-encoded string into a binary string.
 * @param {string} input - The base64-encoded string.
 * @returns {string} - The decoded binary string.
 * @throws {Error} - If the input contains invalid base64 characters.
 */
function atob(input) {
    input = input.replace(/[\s=]+$/, '');
    const base64Chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
    let output = '';
    let buffer = 0, bits = 0;

    for (let i = 0; i < input.length; i++) {
        let value = base64Chars.indexOf(input[i]);
        if (value === -1) {
            throw new Error('Invalid character in base64 string');
        }

        buffer = (buffer << 6) | value;
        bits += 6;

        if (bits >= 8) {
            bits -= 8;
            output += String.fromCharCode((buffer >> bits) & 0xFF);
        }
    }

    return output;
}

/**
 * Parses a base64-encoded frame into an array of byte values.
 * @param {string} frame - The base64-encoded string to parse.
 * @returns {number[]} - An array of byte values (0-255), or an empty array if an error occurs.
 */
function parse(frame) {
    try {
        let binaryString = atob(frame);
        let bytes = Array.from(binaryString).map(char => char.charCodeAt(0));
        return bytes;
    }
    
    catch (e) {
        console.log(e.message);
        return [];
    }
}

This function would also return:

[18, 52, 86, 120]

By converting binary data to a string first, the integrity of the frame is preserved, allowing for seamless decoding and parsing in JavaScript.

Suppose your device sends data in a format like this:

temperature=22.5,humidity=60.3,pressure=1013

However, not all frames contain every key, or the order of keys might change. For example:

humidity=58.2,pressure=1010

To ensure the data is consistently mapped to the correct dataset indexes (e.g., temperature is always index 0, humidity is index 1, pressure is index 2), you can use a keymap and a regular expression. Here’s how you can implement this:

// Define a keymap to associate keys with specific indexes
const keyToIndexMap = {
    temperature: 0,
    humidity: 1,
    pressure: 2
};

// Initialize an array to hold parsed values with default values (e.g., 0)
const parsedValues = new Array(Object.keys(keyToIndexMap).length).fill(0);

function parse(frame) {
    // Regular expression to match "key=value" pairs
    const regex = /([\w]+)=([\d.]+)/g;
    let match;

    // Iterate over all matches in the frame
    while ((match = regex.exec(frame)) !== null) {
        const key = match[1]; // Extract the key (e.g., "temperature")
        const value = parseFloat(match[2]); // Extract the numeric value (e.g., 22.5)

        // Check if the key exists in the map
        if (keyToIndexMap.hasOwnProperty(key)) {
            const index = keyToIndexMap[key]; // Get the mapped index
            parsedValues[index] = value; // Update the corresponding value in the array
        }
    }

    return parsedValues;
}

Input 1:

temperature=22.5,humidity=60.3,pressure=1013

Output 1:

[22.5, 60.3, 1013] // All keys are present and correctly mapped

Input 2:

humidity=58.2,pressure=1010

Output 2:

[22.5, 58.2, 1010] // `temperature` retains its previous value

Input 3:

pressure=1012

Output 3:

[22.5, 58.2, 1012] // Only "pressure" is updated; others remain unchanged

1. Index Consistency: The keymap ensures each dataset is always mapped to a specific index, regardless of the key order or presence in the frame. For instance:

   • temperature → index 0 in the array, index 1 in the Project Editor.
   • humidity → index 1 in the array, index 2 in the Project Editor.
   • pressure → index 2 in the array, index 3 in the Project Editor.

2. Default Initialization: The parsedValues array starts with default values (e.g., 0) for all keys. Missing keys retain their previous values after the first frame update.

3. Extensibility: You can easily extend the keymap to include new keys or datasets as needed.

To test and refine your regular expressions, you can use online tools like Regex101. These tools allow you to experiment with your frame format, verify matches, and debug your regex patterns in an interactive environment. Simply copy your frame data and regex into the tool, and you’ll see instant feedback on how the pattern works.

1. Index Consistency: Regardless of the order or presence of keys in the frame, the data is always mapped to the correct indexes. For example, temperature is always index 0, even if it’s missing from a frame.
2. Default Values: Missing keys automatically use default value at start, and retain the last read value once updated.
3. Flexibility: You can easily extend or modify the keymap to support new keys or datasets.

This approach is powerful and flexible, ensuring your data parsing logic remains robust even as your frame format evolves.

Serial Studio transforms raw data into real-time visualizations by:

• Buffering Incoming Data: Stores incoming data in a circular buffer.
• Identifying Frames: Finding meaningful boundaries in the data stream.
• Parsing Frames: Converting frames into arrays of items.
• Mapping Datasets: Assigning properties to each item to build a dashboard.

With this understanding, you can track and visualize almost any kind of data—from industrial sensors to potato launch experiments. The flexibility of Serial Studio can help you create dashboards that meet the needs of your project.

Building Dashboards:

• Project Editor - Configure frame parsing and create custom dashboards
• Widget Reference - Choose the right widget for your data
• JavaScript API Reference - Complete parser function documentation

Understanding Serial Studio:

• Operation Modes - Quick Plot, Project Files, Device-Defined modes
• Communication Protocols - Protocol overview and comparison
• Data Sources - Configure your connection

Troubleshooting:

• Troubleshooting Guide - Solutions to parsing problems
• Common Mistakes - Fix parsing errors

Examples:

• Examples Repository - Real-world parsing examples
  • HexadecimalADC - Binary data parsing
  • LorenzAttractor - JSON parsing
  • MPU6050 - CSV with validation

Questions? Check the Troubleshooting Guide or ask on GitHub Discussions!