# CsvPath Framework

## CsvPath

- [CSV, Excel and JSONL Ingestion](https://www.csvpath.org/readme.md): Stop writing custom import and validation scripts. CsvPath Framework automates CSV and Excel ingestion and data quality checks. Open source. Python.
- [CSV Validation Automation](https://www.csvpath.org/csv-validation-automation.md): CSV File Validation Automation for Data Engineers
- [DATA PREBOARDING](https://www.csvpath.org/data-preboarding.md): Why we all need to care about data preboarding and the trusted publisher model
- [Quickstart](https://www.csvpath.org/getting-started/quickstart.md): Get started managing CSV/Excel file validation and ingestion with less manual effort, lower risk, and greater predictability.
- [Get the FlightPath Data app](https://www.csvpath.org/getting-started/get-the-flightpath-data-app.md): FlightPath is a purpose-built CSV and Excel preboarding frontend to CsvPath Framework
- [The FlightPath Data Examples](https://www.csvpath.org/getting-started/the-flightpath-data-examples.md): The built-in examples that ship with FlightPath Data to help you start processing your CSV/Excel files fast.
- [Counting](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting.md): CsvPath Validation Language examples of counting CSV data
- [Track values](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting/track-values.md)
- [Counters](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting/counters.md)
- [Tallying](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting/tallying.md)
- [Summing up](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting/summing-up.md)
- [Thresholds](https://www.csvpath.org/getting-started/the-flightpath-data-examples/counting/thresholds.md)
- [Duplicates](https://www.csvpath.org/getting-started/the-flightpath-data-examples/duplicates.md): CsvPath Validation Language examples of handling duplicate CSV/Excel data
- [Unique lines by headers](https://www.csvpath.org/getting-started/the-flightpath-data-examples/duplicates/unique-lines-by-headers.md)
- [Last duplicate seen](https://www.csvpath.org/getting-started/the-flightpath-data-examples/duplicates/last-duplicate-seen.md)
- [Percent overlapping](https://www.csvpath.org/getting-started/the-flightpath-data-examples/duplicates/percent-overlapping.md)
- [Duplicate lines](https://www.csvpath.org/getting-started/the-flightpath-data-examples/duplicates/duplicate-lines.md)
- [Headers](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers.md): CsvPath Validation Language examples for managing CSV/Excel file headers
- [Collect matching lines](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/collect-matching-lines.md)
- [Replace values](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/replace-values.md)
- [Insert and append](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/insert-and-append.md)
- [No blanks](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/no-blanks.md)
- [Reset headers](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/reset-headers.md)
- [Name check](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/name-check.md)
- [Position check](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/position-check.md)
- [More header checks](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/more-header-checks.md)
- [End](https://www.csvpath.org/getting-started/the-flightpath-data-examples/headers/end.md)
- [Lookups](https://www.csvpath.org/getting-started/the-flightpath-data-examples/lookups.md)
- [index\_zipcodes.csvpaths](https://www.csvpath.org/getting-started/the-flightpath-data-examples/lookups/index_zipcodes.csvpaths.md)
- [zipcode\_lookup.csvpaths](https://www.csvpath.org/getting-started/the-flightpath-data-examples/lookups/zipcode_lookup.csvpaths.md)
- [Math](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math.md): CsvPath Validation Language examples of using math functions in processing CSV/Excel files
- [Averaging across groups](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/averaging-across-groups.md)
- [Sum](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/sum.md)
- [Subtract](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/subtract.md)
- [Min and max](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/min-and-max.md)
- [Running median and average](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/running-median-and-average.md)
- [Percents](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/percents.md)
- [Above average](https://www.csvpath.org/getting-started/the-flightpath-data-examples/math/above-average.md)
- [Schemas](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas.md): Examples of using CsvPath Validation Language schemas to validate CSV and Excel data
- [Intersection](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas/intersection.md)
- [Three entity schema](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas/three-entity-schema.md)
- [Three entity schema v2](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas/three-entity-schema-v2.md)
- [Three entity schema v3](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas/three-entity-schema-v3.md)
- [Three entity schema v4](https://www.csvpath.org/getting-started/the-flightpath-data-examples/schemas/three-entity-schema-v4.md)
- [More CSV and Excel Validation](https://www.csvpath.org/getting-started/more-csv-and-excel-validation.md): Preboarding without validation is like pizza without cheese
- [Your First Validation, The Lazy Way](https://www.csvpath.org/getting-started/more-csv-and-excel-validation/your-first-validation-the-lazy-way.md): Start here if you want a code-free introduction!
- [Your First Validation, The Easy Way](https://www.csvpath.org/getting-started/more-csv-and-excel-validation/your-first-validation-the-easy-way.md): CsvPath is a very flexible language. There is often a simpler way than you first thought.
- [Your First Validation, The Hard Way](https://www.csvpath.org/getting-started/more-csv-and-excel-validation/your-first-validation.md): Super simple rules to demonstrate how validation works
- [DataOps Integrations](https://www.csvpath.org/getting-started/dataops-integrations.md): CsvPath has built-in integrations for storage, observability, formats, and more
- [OpenTelemetry](https://www.csvpath.org/getting-started/dataops-integrations/opentelemetry.md): Observability for the masses
- [OpenLineage](https://www.csvpath.org/getting-started/dataops-integrations/openlineage.md): Get started with Edge Data Governance the easy way. The instructions on this page should take you 15 to 45 minutes, depending on network speeds, docker startup times, etc.
- [A Helping Hand](https://www.csvpath.org/getting-started/a-helping-hand.md): We're here to help
- [How-tos](https://www.csvpath.org/topics/how-tos.md): Quick examples to help you get started using CsvPath Framework
- [Storage Backends](https://www.csvpath.org/topics/how-tos/storage-backends.md)
- [AWS S3](https://www.csvpath.org/topics/how-tos/storage-backends/aws-s3.md): CsvPath's three file stores can be local or on S3. Mix and match!
- [Azure](https://www.csvpath.org/topics/how-tos/storage-backends/azure.md): Trusted publishing using Azure blobs is really quite straightforward.
- [Google Cloud Storage](https://www.csvpath.org/topics/how-tos/storage-backends/google-cloud-storage.md): Trusted publishing using Azure blobs is really quite straightforward.
- [HTTPS](https://www.csvpath.org/topics/how-tos/storage-backends/https.md)
- [S3, SFTP, Google Cloud, or Azure](https://www.csvpath.org/topics/how-tos/storage-backends/s3-sftp-google-cloud-or-azure.md)
- [Parquet](https://www.csvpath.org/topics/how-tos/parquet.md): How to use CsvPath Framework to output Parquet files
- [Sqlite](https://www.csvpath.org/topics/how-tos/sqlite.md): Capturing run events to Sqlite can make searching for answers faster
- [Postgres and MySQL](https://www.csvpath.org/topics/how-tos/postgres-and-mysql.md): Some multi-user archives may benefit from capturing events to the database
- [Transfers](https://www.csvpath.org/topics/how-tos/transfers.md): Sending result data where you need it.
- [File System Transfers](https://www.csvpath.org/topics/how-tos/transfers/file-system-transfers.md): Exporting files from CsvPath Framework runs to other filesystem locations
- [AWS Lambda](https://www.csvpath.org/topics/how-tos/aws-lambda.md): Setup a Lambda to automatically process files arriving to an S3 bucket
- [Webhooks](https://www.csvpath.org/topics/how-tos/webhooks.md): Calling a webhook at the end of a named-paths group run is a straightforward way to integrate systems
- [Airflow](https://www.csvpath.org/topics/how-tos/airflow.md): An Easy Way To Add FlightPath Server API To Your Airflow Project
- [Slack](https://www.csvpath.org/topics/how-tos/slack.md): Send yourself notifications about CsvPath runs via Slack webhooks
- [Scripts](https://www.csvpath.org/topics/how-tos/scripts.md)
- [File Arrival Activation](https://www.csvpath.org/topics/how-tos/file-arrival-activation.md): Automatically run a named-paths group when a new file arrives
- [Rewind / Replay](https://www.csvpath.org/topics/how-tos/rewind-replay.md)
- [Replay Using References](https://www.csvpath.org/topics/how-tos/rewind-replay/replay-using-references.md): Options for when you need to rerun a named-path group
- [Doing rewind / replay, part 1](https://www.csvpath.org/topics/how-tos/rewind-replay/doing-rewind-replay-part-1.md): How to start a run from an intermediate point in a past run
- [Doing rewind / replay, part 2](https://www.csvpath.org/topics/how-tos/rewind-replay/doing-rewind-replay-part-2.md): Can we achieve the same goals using only the CLI? Yes!
- [Referring to named-file versions](https://www.csvpath.org/topics/how-tos/rewind-replay/referring-to-named-file-versions.md): It's common to need to rerun an older file version again -- and easy to do.
- [SFTP](https://www.csvpath.org/topics/how-tos/sftp.md): SFTP is a bedrock backend provider that also enables seamless operations
- [SFTP Export](https://www.csvpath.org/topics/how-tos/sftp/sftp-export.md): Forward your results from CsvPath's archive to an SFTP account
- [SFTPPlus](https://www.csvpath.org/topics/how-tos/sftp/sftpplus.md): Create an effective automated MFT solution in minutes
- [SFTPPlus Implementation Checklist](https://www.csvpath.org/topics/how-tos/sftp/sftpplus/sftpplus-implementation-checklist.md): A checklist for implementing SFTPPlus + CsvPath
- [Registering Files From SFTP](https://www.csvpath.org/topics/how-tos/sftp/registering-files-from-sftp.md): Registering files from an SFTP server into a named-file
- [Creating a Custom Listener](https://www.csvpath.org/topics/how-tos/creating-a-custom-listener.md): CsvPath Framework listeners are easy to create and powerful
- [CKAN](https://www.csvpath.org/topics/how-tos/ckan.md): How to use CsvPath to publish data products to CKAN with confidence
- [CKAN CsvPath Setup](https://www.csvpath.org/topics/how-tos/ckan/ckan-csvpath-setup.md): CsvPath is integrated with CKAN, the leading open source data portal.
- [Jenkins](https://www.csvpath.org/topics/how-tos/jenkins.md): An example of using a simple automation tool to feed data to CsvPath
- [config.ini](https://www.csvpath.org/topics/how-tos/config.ini.md): Configuring CsvPaths is easy. The first step is adding it to your project using your Python dependency management tool of choice. Then create a simple test harness file and run it.
- [Debugging](https://www.csvpath.org/topics/how-tos/debugging.md)
- [Creating a Derived File](https://www.csvpath.org/topics/how-tos/creating-a-derived-file.md)
- [A Longer Example](https://www.csvpath.org/topics/how-tos/a-longer-example.md)
- [Another Example, Part 1](https://www.csvpath.org/topics/how-tos/a-longer-example/another-example-part-1.md)
- [Another Example, Part 2](https://www.csvpath.org/topics/how-tos/a-longer-example/another-example-part-2.md)
- [How-to videos](https://www.csvpath.org/topics/how-tos/how-to-videos.md)
- [Practical Stuff](https://www.csvpath.org/topics/practical-stuff.md)
- [Language Basics](https://www.csvpath.org/topics/practical-stuff/language.md)
- [CsvPath Framework Cheatsheet](https://www.csvpath.org/topics/practical-stuff/csvpath-framework-cheatsheet.md): A super high-level cheatsheet for concepts and approaches
- [The CLI](https://www.csvpath.org/topics/practical-stuff/the-cli.md): CsvPath's command line interface is a productivity booster
- [Working With Error Messages](https://www.csvpath.org/topics/practical-stuff/working-with-error-messages.md)
- [The Modes](https://www.csvpath.org/topics/practical-stuff/the-modes.md)
- [The Reference Data Types](https://www.csvpath.org/topics/practical-stuff/the_reference_data_types.md)
- [Manifests and Metadata](https://www.csvpath.org/topics/practical-stuff/manifests-and-metadata.md): The CsvPath Library is all about the structure you need to trust the data you process
- [Serial Or Breadth-first Runs?](https://www.csvpath.org/topics/practical-stuff/serial-or-breadth-first-runs.md)
- [Namespacing With the Archive](https://www.csvpath.org/topics/practical-stuff/namespacing-with-the-archive.md)
- [Higher-level Topics](https://www.csvpath.org/topics/higher-level-topics.md): Topics and ideas. Watch this space!
- [Organizing Inbound Data](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data.md): Organizing your preboarding data layout upfront makes the onboarding process efficient
- [Dataflow Diagram](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/dataflow-diagram.md)
- [The Three Data Spaces](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/the-three-data-spaces.md)
- [Source Staging](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/the-three-data-spaces/source-staging.md)
- [Validation Assets](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/the-three-data-spaces/validation-assets.md)
- [Trusted Publishing](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/the-three-data-spaces/trusted-publishing.md)
- [How Data Progresses Through CsvPath Framework](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework.md)
- [Staging](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging.md)
- [Data Identity](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/data-identity.md)
- [Handling Variability](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/handling-variability.md)
- [Templates](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/handling-variability/templates.md)
- [Named-file Reference Queries](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/handling-variability/named-file-reference-queries.md)
- [Registration API and CLI](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/registration-api-and-cli.md)
- [Loading](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/registration-api-and-cli/loading.md)
- [Going CLI-only](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/staging/registration-api-and-cli/going-cli-only.md)
- [Validation and Upgrading](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/validation-and-upgrading.md)
- [Templates](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/validation-and-upgrading/templates.md)
- [Run Using the API](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/validation-and-upgrading/run-using-the-api.md)
- [Running In the CLI](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/validation-and-upgrading/running-in-the-cli.md)
- [Named-paths Reference Queries](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/validation-and-upgrading/named-paths-reference-queries.md)
- [Publishing](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/publishing.md)
- [Inspect Run Results](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/publishing/inspect-run-results.md)
- [Result API](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/publishing/inspect-run-results/result-api.md)
- [More Templates and References](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/publishing/inspect-run-results/more-templates-and-references.md)
- [Export Data and Metadata](https://www.csvpath.org/topics/higher-level-topics/organizing-inbound-data/how-data-progresses-through-csvpath-framework/publishing/export-data-and-metadata.md)
- [Why CsvPath?](https://www.csvpath.org/topics/higher-level-topics/why-csvpath.md): As has been said on other pages, in technology there are always multiple ways to get to a result. So why should you use CsvPath when there are other options?
- [CsvPath Use Cases](https://www.csvpath.org/topics/higher-level-topics/csvpath_use_cases.md)
- [Paths To Production](https://www.csvpath.org/topics/higher-level-topics/paths-to-production.md): There are many paths. What's the simplest thing that could possibly work?
- [Solution Storming](https://www.csvpath.org/topics/higher-level-topics/solution-storming.md): Let's figure out how CsvPath can make a difference for your use case.
- [Data, Validation Files, and Storage](https://www.csvpath.org/topics/higher-level-topics/data-validation-files-and-storage.md)
- [Named Files and Paths](https://www.csvpath.org/topics/higher-level-topics/data-validation-files-and-storage/named_files_and_paths.md)
- [Where Do I Find Results?](https://www.csvpath.org/topics/higher-level-topics/data-validation-files-and-storage/where-do-i-find-results.md): CsvPath serializes results for your future use
- [Storage Backends](https://www.csvpath.org/topics/higher-level-topics/data-validation-files-and-storage/storage-backends.md): You have a few simple choices for where to store your assets
- [File Management](https://www.csvpath.org/topics/higher-level-topics/data-validation-files-and-storage/file-management.md)
- [Validation](https://www.csvpath.org/topics/higher-level-topics/validation.md)
- [Schemas Or Rules?](https://www.csvpath.org/topics/higher-level-topics/validation/schemas-or-rules.md)
- [Validation Strategies](https://www.csvpath.org/topics/higher-level-topics/validation/validation_strategies.md)
- [Python](https://www.csvpath.org/topics/higher-level-topics/python.md)
- [Python vs. CsvPath](https://www.csvpath.org/topics/higher-level-topics/python/python-vs.-csvpath.md)
- [Python Starters](https://www.csvpath.org/topics/higher-level-topics/python/python-starters.md): A few really basic scripts to get you started
- [Glossary](https://www.csvpath.org/topics/glossary.md): CsvPath is a Data Preboarding Framework enabling Edge Governance and Trusted Publishing through the Architectural Pattern called Collect, Store, Validate, Publish. Phew — so many buzzwords!
- [Release Notes](https://www.csvpath.org/topics/release-notes.md): High-level descriptions of CsvPath Framework and FlightPath Data releases
- [Privacy Policy](https://www.csvpath.org/privacy-policy.md)