CODAP

Common Online Data Analysis Platform

A Bit of History

Development of this codebase began in 2010 as part of the work of KCP Technologies' NSF-supported Data Games project. In a data game, the data generated by playing the game flows into CODAP for analysis and visualization. In a well-designed data game, the player cannot progress very far without using the data to inform and refine a strategy.

Starting in 2012 CODAP became the data analysis environment for Concord Consortium's NSF-supported InquirySpace project. Developed to promote scientific inquiry in high school science classes, students work with simulations or gather data with probes. The data flow into CODAP for analysis, facilitating hypothesis generation and iteration of experiments.

In April, 2014, CODAP received NSF funding as a project in its own right with the goal of developing a platform that can be used by developers of online curriculum materials that engage learners with data. Initial collaborations are: continuing work with InquirySpace and new work with OceanTracks at EDC and Terra Populus at the Minnesota Population Center. CODAP's success depends on the extent to which a community of developers embraces it as a way to bring dynamic, interactive data science tools to classroom (and other) learning. This work resulted in CODAP Version 2 which, by 2022 was being used by ~40K unique "28-day-active-users" in over 70 countries (stats from Google Analytics).

July 2022 saw the beginning of the development of CODAP Version 3, a complete reimplementation of CODAP using modern web development tools such as React, TypeScript, MobX State Tree and D3. The goal of version 3 is that while the underlying software is completely new, the application features and user interface are largely unchanged so that users' transition to version 3 is as seamless as possible.

What CODAP Looks Like

What CODAP looks like will depend strongly on the context. But below is a screen shot of CODAP in use with an InquirySpace investigation. The Terminal Velocity component is a simulation that is streaming data to CODAP. Note the hierarchical structure of the data in which there are five runs, each containing a time series. Note also the linked selection in which the fourth run and its time series are selected.

![CODAP as it appears in InquirySpace]

The License

CODAP software is provided here under an open-source MIT license. For details on this license, please see the LICENSE file included with this repository or The MIT License (MIT). CODAP was initially released under the Apache 2.0 license, and so you may see references to that license in the sources as well.

---

Developer instructions

CODAP v3 is being developed alongside CODAP v2 in the same codap repository. During this period of active development on both versions, master continues to be the master/main branch for v2 development while main is the master/main branch for v3 development.

Initial steps

1. Clone this repo and cd into its v3 directory
2. Checkout the main branch
3. Run npm install to install dependencies
4. Run npm start to run webpack-dev-server in development mode with hot module replacement

Run using HTTPS

Additional steps are required to run using HTTPS.

1. install mkcert : brew install mkcert (install using Scoop or Chocolatey on Windows)
2. Create and install the trusted CA in keychain if it doesn't already exist: mkcert -install
3. Ensure you have a .localhost-ssl certificate directory in your home directory (create if needed, typically C:\Users\UserName on Windows) and cd into that directory
4. Make the cert files: mkcert -cert-file localhost.pem -key-file localhost.key localhost 127.0.0.1 ::1
5. Run npm run start:secure to run webpack-dev-server in development mode with hot module replacement

Alternately, you can run secure without certificates in Chrome:

1. Enter chrome://flags/#allow-insecure-localhost in Chrome URL bar
2. Change flag from disabled to enabled
3. Run npm run start:secure:no-certs to run webpack-dev-server in development mode with hot module replacement

Building

If you want to build a local version run npm run build, it will create the files in the dist folder. You do not need to build to deploy the code, that is automatic. See more info in the Deployment section below.

To run just the TypeScript compiler for a quick type check without a full webpack build, run npm run build:tsc.

Notes

1. Make sure if you are using Visual Studio Code that you use the workspace version of TypeScript. To ensure that you are open a TypeScript file in VSC and then click on the {} button next to TypeScript JSX in the status bar and select Use Workspace Version in the popup menu.

Development

V3 pull requests should target the main branch and should apply the v3 label. V2 pull requests should target the master branch and apply the v2 label.

Deployment

Production releases to S3 are based on the contents of the /dist folder and are built automatically by GitHub Actions for each branch pushed to GitHub and each merge into the main branch.

Production releases are deployed to the root at http://codap3.concord.org.

Other branches are deployed to https://codap3.concord.org/branch/{branch-name}/, e.g. https://codap3.concord.org/branch/main/ for the main branch. Note that the trailing slash is required. For arcane reasons leaving off the slash results in a redirect to an invalid url. ¯\(ツ)/¯

To deploy a new release:

The codap-v3-build skill can be invoked to initiate an interactive walkthrough of the process of building a release of CODAP v3. The process has 6 phases:

1. Prepare the Release - Set up Jira version and gather context
2. Prepare Release Notes - Interactive walkthrough to create CHANGELOG entry
3. Update Version Files - Update package.json, versions.md, CHANGELOG.md
4. Create Release PR - Build, capture asset sizes, create PR
5. Tag and Release - After PR merge, create git tag and GitHub release
6. Deploy - Stage, QA, deploy to production

The prior manual steps for building a release are listed below for posterity:

1. Create a new release version in Jira using the Manage Releases tab of the CODAPv3 Jira board. (The new version number is the number found in the top right corner of v3, incremented by one.) The status of the release can remain Unreleased for now.
2. Run git log --reverse <last-version>...HEAD and/or use the GitHub UI to see a list of PRs merged since the last release and identify their corresponding Jira stories. Tag the Jira stories based on this list as 3.0.0-pre.<new-version> in the Fix versions field.
3. Update the version number in package.json and package-lock.json.
   • npm version --no-git-tag-version 3.0.0-pre.<new-version> or
   • npm version --no-git-tag-version 3.0.0-beta.<new-version>
4. Create a new entry in versions.md with the new version and release date.
5. Create a new entry in CHANGELOG.md with a description of the new version.
6. Create v3-release-<version> branch and stage versions.md, package.json and package-lock.json.
7. Run npm run release-notes 3.0.0-pre.<new-version> in the dev-templates/scripts repo.
8. Run npm run build.
9. Copy asset size markdown table from previous release and change sizes to match new sizes in dist. In the Terminal from the v3 folder, the command is ls -la dist/assets.
10. Stage CHANGELOG.md, commit changes, push to GitHub, and create PR for the release. Add v3 and run regression labels so cypress tests are run. Merge PR once tests complete.
11. Checkout main and pull.
12. Make a new version tag using your local git client with the version number in the description (e.g. git tag -a 3.0.0-pre.<new-version> -m "version 3.0.0-pre.<new-version>") and push the tag to the remote origin (e.g., git push origin 3.0.0-pre.<new-version>).
13. Create a new GitHub release at https://github.com/concord-consortium/codap/releases corresponding to the new tag.
14. Watch the GitHub actions build to see that the S3 Deploy step finished and then QA this version using the versioned URL (e.g., https://codap3.concord.org/version/3.0.0-pre.<new-version>/) or alternatively QA the staged release in the next step.
15. Stage the release by running the Release v3 Staging Workflow and entering the version tag you just pushed.
16. Test the staged release at https://codap3.concord.org/index-staging.html.
17. Update production by running the Release v3 Production Workflow and entering the release version tag.
18. In the Manage Releases tab of the CODAPv3 project in Jira, mark the new release as Released. Clicking on the release shows a list of all stories tagged as fixed in this version.

Versions

Here is a link to the various versions of CODAP along with their release dates.

Testing

• run npm run lint to lint the source files
• run npm run lint:fix to lint the source files and fix correctable issues
• run npm run test to run the jest tests
• run npm run test -- a-particular.test to run a particular test
• run npm run test:coverage to run the jest tests and print out coverage statistics for the tested files
• run npm run test:coverage -- a-particular.test to run a particular test and print out coverage statistics just for the specified tests
• run npm run test:watch to run the jest tests in watch mode, re-running when files change
• run npm start in one terminal and npm run test:cypress to run the cypress tests locally against a headless browser
• run npm start in one terminal and npm run test:cypress -- --spec cypress/e2e/a-particular.spec.ts to run a particular cypress test locally
• run npm start in one terminal and npm run test:cypress:open to run the cypress tests against an actual browser with the ability to debug, etc.

Cypress Run Options

Inside of your package.json file:

1. --browser browser-name: define browser for running tests
2. --group group-name: assign a group name for tests running
3. --spec: define the spec files to run
4. --headed: show cypress test runner GUI while running test (will exit by default when done)
5. --no-exit: keep cypress test runner GUI open when done running
6. --record: decide whether or not tests will have video recordings
7. --key: specify your secret record key
8. --reporter: specify a mocha reporter

Cypress Run Examples

1. cypress run --browser chrome will run cypress in a chrome browser
2. cypress run --headed --no-exit will open cypress test runner when tests begin to run, and it will remain open when tests are finished running.
3. cypress run --spec 'cypress/integration/examples/smoke-test.js' will point to a smoke-test file rather than running all of the test files for a project.

Debugging

Various developer features can be enabled by adding a debug local storage key with one or more of the following flags separated by spaces. Local storage is specific to the domain that CODAP is running on. In Chrome local storage can be edited by opening the developer tools and going to the "Application" tab. Then find "Storage/Local storage" and the domain that CODAP is running on.

• caseIds displays the id of each case in the index column of the case table
• cfmEvents console log all events received from the CFM
• cfmLocalStorage enable the CFM local storage provider so documents can be saved and loaded from the browser's local storage
• cfmNoAutoSave disable CFM auto-save for any document loaded by the CFM
• document this will add the active document as window.currentDocument, you can use currentDocument.toJSON() to views the current documents content. You can also use currentDocument.treeManagerAPI.document.toJSON() to inspect the history of the document.
• eventModification print warnings with stack traces when code is modifying default browser event behavior. See debug-event-modification.ts for the list of methods that are monitored.
• formulas print info about recalculating formulas
• history this will: print some info to the console as the history system records changes, print the full history as JSON each time it is loaded from Firestore, and provide a window.historyDocument so you can inspect the document while navigating the history.
• logger console log all messages sent to the logging service
• map print info about interactions with the map component
• pixiPoints this adds a map of tileId keys to PixiPoint instances as window.pixiPointsMap. This is also always available in Cypress.
• plugins enable some extra plugins in the plugin menu and print information about interactions with plugins.
• saveAsV2 this will cause the app to save documents in V2 format.
• undo this will print information about each action that is added to the undo stack.

Why Did You Render?

To debug React rendering problems, the why-did-you-render utility is available. To use it:

• uncomment the import "../why-did-you-render.ts" line in index.tsx
• by default, all "pure" components are tracked
• this can be configured in the local why-did-you-render.ts file
• individual components can be tracked by setting MyComponent.whyDidYouRender = true

Remember not to commit any of these debug-only changes to the repository.

License

CODAP is Copyright 2025 (c) by the Concord Consortium and is distributed under the MIT license.

See license.md for the complete license text.