Functions perform work within a csvpath. Some focus on creating values. Others on deciding if a line matches. And a few provide a side-effect, rather than contributing values or matching.
Like a Python function, a CsvPath function is represented by a name followed by parentheses. They may take zero to an unlimited number of arguments within the parentheses, separated by commas.
Functions can contain:
- Terms
- Variables
- Headers
- Equality tests
- Variable assignment
- Other functions
They can not include when/do expressions. This means you cannot use -> within a function.
Many functions take qualifiers. With only a three exceptions, all functions can take the onmatch qualifier. An onmatch qualifier indicates that the function should be applied only when the whole path matches. See the individual function pages for what built-in qualifiers a function supports.
Some functions will optionally make use of an arbitrary name qualifier to better name a tracking variable.
Most of the work of matching is done in match component functions. There are over one hundred and fifty functions in several groups.
Creating custom functions straightforward. Once you create a function, you register it with the FunctionFactory class. You can register your functions either programmatically or by creating an import file listing your functions. Your import file must be referenced in your config.ini file at [functions][imports] like this:
[functions]
imports = my_project_assets/my_functions.imports
Each custom function has its own line in your imports file. The format is the same as you use for importing classes into Python files -- under the hood the mechanism is similar. E.g.
from my_function.for_stuff.me import Me as function_meIn this example your class is Me and the name of the function you use in your csvpath is function_me()
Use your functions in csvpaths by simply referring to them by name like you would any built-in function.
While it is quite possible to create your own function by copying one of the existing ones, it is probably a good idea to contact [email protected] to talk through what you're working on to minimize the chance of running into gotchas or reinventing the wheel.
not(count()==2)add( 5, 3, 1 )concat( end(), regex(/[0-5]+abc/, #0))
There are lots more simple examples on the individual function pages, linked below, and on csvpath.org. FlightPath Data has a rich set of examples that you can run right away. They are generated in and examples folder of each new project you create.
The functions are self-documenting. Each can programmatically explain how it works. You can see the output in the CLI, in the language help window in FlightPath Data, and in these online function docs.
When in doubt, the most up to date version of the function docs can be seen in the online function docs or in the most recent release's CLI. The individual function docs pages linked directly from this page are left available for their descriptions and examples; however, they are not the most up to date information.
Each function can display:
- A short description
- Any aliases it has
- Its allowable match component arguments
- The allowable data types that the match component arguments can evaluate to
- If it is a schema type (I.e. can it be used in a
line()to construct a schema similar to that of a SQL table?) - If it produces a value, determines a match, or is a side-effect
Functions can have multiple signatures. Meaning, the same function can be called with multiple types or numbers of arguments to get a different result.
Many functions have aliases that are intended to help make csvpaths more readable. For example the less-than test can be written:
lt(a, b)before(a, b)below(a, b)
While that may seem confusing to some people, others appreciate having options that can make their csvpaths more sentence-like.
This list is often not up to date. For complete documentation, coming from the function classes themselves, see the FlightPath Data language help window, the CsvPath Framework CLI, or the generated online function docs.
| Function | What it does |
|---|---|
| after(value, value) | Finds things after a date, number, string. Aliases: gt(), above(). |
| all(value, value, ...) | An existence test for all selected values or all headers, or all variables. |
| and(value, value,...) | Returns true when all match. |
| any(value, value, ...) | An existence test across a range of places. |
| before(value, value) | Finds things before a date, number, string. Aliases: lt(), below(). |
| between(value, value, value) | Returns true when a value is found between to others. Aliases: outside(). |
| empty(value) | Tests if the value is empty. |
| exists(value) | Tests if the value exists. |
| equals(value, value) | Tests equality. |
| in(value, list) | Match against a pipe-delimited list, values, or references. |
| missing(value, value, ...) | An existence test for all selected values, all headers, or all variables. |
| no() | Always false. Alias: false() |
| not(value) | Negates a value. |
| or(value, value,...) | Returns true when any match. |
| yes() | Always returns true. Alias: true() |
| Function | What it does |
|---|---|
| count() | Counts the number of matches. |
| count(value) | Count the matches of values. |
| count_lines() | count the lines of data to this point in the file. |
| count_scans() | count lines we checked for match |
| counter() | a streamlined way to increment a counter variable |
| count_bytes() | Returns the number of bytes written to `data.csv`. |
| count_dups(header, ...) | Returns the number of duplicate lines. |
| every(value, number) | Matches every Nth time a value is seen. |
| has_matches(header, ...) | Matches when any other match component matched anywhere in the file. |
| increment(value, n) | Increments a variable by n each time seen. |
| line_number() | Gives the physical line number. |
| tally(value, value, ...) | Counts times values are seen, including as a set. |
| total_lines() | Returns the number of rows in the file being scanned. |
| Function | What it does |
|---|---|
| now(format) | Returns a datetime, optionally formatted, for the current moment. |
| Function | What it does |
|---|---|
| append(value, value) | Appends a new header value at the end of every line. |
| collect(value, ...) | Identifies the header values to collect when a row matches. |
| count_headers() | Returns the number of headers expected. |
| count_headers_in_line() | Returns the number found in the line. |
| empty_stack(value, ...) | Returns a stack of the names of empty header values. |
| end(int) | Returns the value of the line's last header value. |
| header_name(value, value) | Returns the header name for an index. |
| header_index(value, value) | Returns the header index for a name. |
| headers(value) | Indicates to another function that it should look in the headers. |
| header_names_mismatch(value) | Checks the header names against a delimited list of expected headers. |
| insert() | Inserts a header and its values. |
| mismatch() | Returns the difference in number of value vs. number of headers. |
| replace(value, value) | Replaces a header value with another value. |
| reset_headers() | Sets the headers to the values of the current line. |
| Function | What it does |
|---|---|
| advance(int) | Skips the next n-rows |
| after_blank() | Matches when a line was preceded by a blank line |
| has_dups(header, ...) | Returns true if there are duplicate lines. |
| count_dups(header, ...) | Counts duplicate lines. |
| dup_lines(header, ...) | Returns the line numbers of duplicate lines. |
| firstline() | Matches on the 0th line, if scanned. |
| firstscan() | Matches on the 1st line scanned. |
| firstmatch() | Matches on the 1st line matched. |
| last() | Returns true on the last row that will be scanned. |
| first(value, value, ...) | Matches the first occurrence and captures the line number. |
| skip(value) | Skips to the next line scanned if a condition is met. |
| stop(value) | Stops scanning lines if a condition is met. |
| take(value) | Skips to the next line scanned if a condition is met. Matches, or "takes", the line it skips out of. |
| Function | What it does |
|---|---|
| add(value, value, ...) | Adds numbers. |
| divide(value, value, ...) | Divides numbers. |
| float(value) | Converts to a float. |
| int(value) | Converts to an int. |
| mod(value, value) | Returns the modulus of two numbers. |
| multiply(value, value, ...) | Multiplies numbers. |
| num(value, int, int, int, int) | A number defined as min/max before and after the decimal. |
| odd(value) and even(value) | Test a number to find if it is odd or even. |
| round(value) | Rounds a number. |
| subtotal(value) | Returns a running subtotal of a value subtotaled by another value. |
| subtract(value, value, ...) | Subtracts numbers or makes a number negative. Aliases: minus(). |
| sum(value) | Returns a running sum of the value. |
| Function | What it does |
|---|---|
| error() | Sends error messages to the error manager |
| line_fingerprint(), store_fingerprint(), file_fingerprint() | Functions for creating digests. |
| import() | Injects another csvpath into the current csvpath. |
| random(starting, ending) | Generates a random int from starting to ending. |
| shuffle(starting, ending) | Generates a random int from starting to ending with no replacement. |
| Function | What it does |
|---|---|
| header_table(value, value) | Prints a formatted table of the headers with header numbers. |
| jinja(value, value) | Applies a Jinja2 template. |
| print(str, value) | Prints the interpolated string. |
| print_line(value,value) | Prints the current line unchanged. |
| print_queue(value,value) | Returns the number of strings printed. |
| row_table(value, value) | Prints a formatted table of a row. |
| var_table(value, value) | Prints a formatted table of the variables. |
| Function | What it does |
|---|---|
| average(number, type) | Returns the average up to current "line", "scan", "match". |
| correlate(value, value) | Gives the running correlation between two values. |
| max(value, type) | Returns the largest value seen up to current "line", "scan", "match". |
| median(value, type) | Returns the median value up to current "line", "scan", "match". |
| min(value, type) | Returns the smallest value seen up to current "line", "scan", "match". |
| percent(type) | Returns the percent of total lines for "scan", "match", "line". |
| percent_unique(header) | Returns the percent of unique values found in the header values. |
| stdev(stack) and pstdev(stack) | Returns the standard deviation of numbers pushed on a stack. |
| Function | What it does |
|---|---|
| alter(value, value, value) | Changes a string by replacing substrings |
| caps(value, value) | Uppercases initial chars. |
| concat(value, value, ...) | Joins any number of values. |
| contains(value, value) | True if the second string is contained in the first. |
| ends_with(value, value) | Checks if the first value ends with the second. |
| exact(regex-string, value) | Exact match on a regular expression. |
| find(value, value) | Returns the index of the second string within the first. |
| length(value) | Returns the length of the value. |
| lower(value) | Makes a value lowercase. |
| max_length(value) | Returns the length of the value. |
| metaphone(value, value) | Returns the metaphone transformation of a string or does a reference look up. |
| min_length(value) | Returns the length of the value. |
| regex(regex-string, value) | Matches on a regular expression. |
| starts_with(value, value) | Checks if the first value starts with the second. |
| strip(value) | Trims off whitespace. |
| substring(value, int) | Returns the first n chars from the value. |
| upper(value) | Makes a value uppercase. |
| Function | What it does |
|---|---|
| blank(value) | Expects a specific header with any value. |
| boolean(value) | Declares a boolean value. |
| date(value, format) | Declares a date parsed according to a format string. |
| datetime(value, format) | Declares a datetime parsed according to a format string. |
| decimal(value, value, value) | Declares a decimal. |
| email(value) | Declares an integer value. |
| integer(value) | Declares a string as an email. |
| none() | Expects a None. |
| string(value, max, min) | Declares a string, optionally with max and min lengths. |
| url(value) | Declares a string as a standard url. |
| wildcard(value) | Expects one or more headers that are unspecified. |
| Function | What it does |
|---|---|
| fail() | Indicate that the CSV is invalid. |
| failed() | Check if the CSV is invalid. |
| fail_and_stop() | Stop the scan and declare the file invalid at the same time. |
| line(function, function, ...) | Declares a typed ordered structure for lines using core data type functions like string(), int(), etc. |
| valid() | Check if the CSV is valid or invalid. |
| Function | What it does |
|---|---|
| get(value, value) | Gets a variable value. |
| peek(name, int) | Accesses a value at an index in a stack. |
| peek_size(name) | Returns the size of a stack. |
| pop(name) | Pops a value off a stack. |
| push(name, value) | Pushes a value on a stack. |
| put(value, value) | Sets a variable value. |
| stack(name) | Returns a stack variable of pushed values. |
| track(value, value) | Tracks a value by name. |
| variables() | Indicates to another function that it should look in the variables. |