Skip to content

Comments

Protect against including null strings when reading PDAL vlr types#4566

Merged
hobu merged 1 commit intomasterfrom
null-string-pdal-vlr-types
Dec 4, 2024
Merged

Protect against including null strings when reading PDAL vlr types#4566
hobu merged 1 commit intomasterfrom
null-string-pdal-vlr-types

Conversation

@hobu
Copy link
Member

@hobu hobu commented Dec 4, 2024

We were including null characters when reading PDAL JSON VLR types. Because we don't reinterpret the JSON anywhere, these NULLs were being passed on through and causing invalid output.

@hobu hobu added this to the 2.8.3 milestone Dec 4, 2024
@abellgithub
Copy link
Contributor

abellgithub commented Dec 4, 2024

There is no null termination of JSON, as such.

JSON that contains NULL bytes is invalid.

I'm not sure what the issue is, but I don't think removing any NULL bytes seems right.

Perhaps we should just throw an exception because the data is bad?

@hobu
Copy link
Member Author

hobu commented Dec 4, 2024

JSON that contains NULL bytes is invalid.

Correct. The issue is the VLR is a null-terminated string and we were reading dataLength not dataLenght-1 and passing it into Metadata::addWithType (json), which simply uses the string as-is. It has nulls in it because we're reading the last character, and then we have a problem.

The reason why I chose to replace any \0's in the string instead of simply doing dataLength-1 is because VLR dataLength regularly lies. I wouldn't expect us to be writing bad lengths, but I also wouldn't have expected us to slurp up JSON as a string without interpreting and validating it in addWithType(json), which is what we do.

@hobu hobu merged commit 88531fc into master Dec 4, 2024
@hobu hobu deleted the null-string-pdal-vlr-types branch December 4, 2024 23:16
hobu added a commit that referenced this pull request Dec 4, 2024
@hobu
protect against including null strings when reading PDAL vlr types (#…
814f0c1 
…4566)
@hobu
Copy link
Member Author

hobu commented Dec 4, 2024

picked back to 2.8-maintenance at 814f0c1

hobu added a commit that referenced this pull request Dec 23, 2024
@hobu
Squashed commit of the following:
82324c4 
commit c720e6d
Author: Howard Butler <[email protected]>
Date:   Mon Dec 23 12:41:31 2024 -0600

    Local documentation build conveniences (#4592)

    * A local build Makefile

    * add more stuff for local build to environment.yml

    * Add hobu logo stuff

    * remove conf.py – this is now generated by jupyterbook

    * drop conda-provided sed/graphviz/doxygen

    * a little doc for building locally

    * update substitutions

commit 74382cb
Author: Howard Butler <[email protected]>
Date:   Mon Dec 23 09:52:32 2024 -0600

    adapt standard gitignore for CMake files (#4596)

commit 9bab3a3
Author: Guilhem Villemin <[email protected]>
Date:   Tue Dec 17 18:13:22 2024 +0100

    Enhance `filters.gpstimeconvert` to make it streamable (#4568)

    * make filter streamable

    * fix typo in disclaimer

    * update doc

    * thow when invalid conversion

    * revert spurious edit

    * add missing Streamable

    * adding run method

    * add wrapped tolerance arg

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit 991e67c
Author: Howard Butler <[email protected]>
Date:   Tue Dec 17 09:51:10 2024 -0600

    Fix #4578, doc nit in readers.rdb (#4590)

commit 998be56
Author: Howard Butler <[email protected]>
Date:   Mon Dec 16 10:30:14 2024 -0600

    update community page with link to Matrix (#4587)

commit 2d4a3ce
Author: Andrew Bell <[email protected]>
Date:   Mon Dec 16 09:47:35 2024 -0500

    Make sure points are added to a new point view if a filter returns no views and (#4583)

    where_merge == 'true'.
    Close #4572

commit cda0631
Author: Andrew Bell <[email protected]>
Date:   Mon Dec 16 09:39:22 2024 -0500

    Write binary in binary mode. (#4584)

    Some general cleanup.

commit 213a6a6
Author: chambbj <[email protected]>
Date:   Thu Dec 12 12:13:34 2024 -0600

    Isolate dependencies and tasks for doc generation and preview (#4579)

commit 2271196
Author: Howard Butler <[email protected]>
Date:   Thu Dec 12 11:59:06 2024 -0600

    adapt GDAL's backport bot (#4580)

commit 97522d1
Author: chambbj <[email protected]>
Date:   Wed Dec 11 11:07:00 2024 -0600

    Use tabs for the OS variants of using Conda (#4575)

commit ab8a5d1
Author: chambbj <[email protected]>
Date:   Wed Dec 11 10:45:39 2024 -0600

    Rewicker the pixi config (#4571)

    * Rewicker the pixi config

    * Enable NITF and also mark this as a win-64 platform task

    * Use target_compile_definitions for the _XKEYCHECK_H fix

    * Add task for linux-64 configure, move docs, build, and test tasks

commit 88531fc
Author: Howard Butler <[email protected]>
Date:   Wed Dec 4 17:16:29 2024 -0600

    protect against including null strings when reading PDAL vlr types (#4566)

commit 9800957
Author: Howard Butler <[email protected]>
Date:   Wed Dec 4 15:23:11 2024 -0600

    bump master CMake version to 2.9.0 to eliminate confusion

commit d172b32
Author: chambbj <[email protected]>
Date:   Wed Dec 4 15:11:26 2024 -0600

    Add Pixi install instructions (#4565)

    * Add instructions for setting up a Pixi environment

    * Fix nested code block

commit de41500
Author: Andrew Bell <[email protected]>
Date:   Tue Dec 3 15:58:31 2024 -0500

    `filters.expression` support for filtering NaN and LOWEST/HIGHEST (#4561)

    * Checkpoint.

    * testing WIP

    * Checkpoint.

    * Don't call checkEnd() in MathParser.
    Add failure tests.

    * Remove dead comment.

    * Provide nan support.

    * Remove junk.

    * Fix typos.

    * Test.

    * Test.

    * Test.

    * Test 2.

    * Test 3.

    * Test 4.

    * Test 5.

    * bump ci

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit cc3e18f
Author: Guilhem Villemin <[email protected]>
Date:   Tue Dec 3 17:28:43 2024 +0100

    Extend `filters.gpstimeconvert` to handle day seconds.

    * convert to day seconds

    * typo

    * finish day conversion

    * fixes

    * restore conversion arg, mark it as deprecated

    * declare m_converison

    * fix type declaration

commit cf6f6ef
Author: Howard Butler <[email protected]>
Date:   Sun Dec 1 10:37:20 2024 -0600

    Validate checksums and version number in automated release CI (#4564)

    * add release version checks

    * set exit

    * dummy 7.7.7 release version

    * explicit verify of checksums

    * cd into correct dir

    * need conda for testing

    * syntax

    * break apart steps a bit

    * set path to environment

    * bad path :(

    * bad path again :(

    * dont build tests

    * check installed version version

    * use pdal-config for clean version string

    * only release for x.x.x

commit 3334e6e
Author: chambbj <[email protected]>
Date:   Fri Nov 22 15:52:28 2024 -0600

    Detect path to conf.py and set JSON/CSV paths relative (#4560)

    * Detect path to conf.py and set JSON/CSV paths relative

    * Try running it as a separate pre_build step

    * fix typo

    * jeez

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit eab55c5
Author: Howard Butler <[email protected]>
Date:   Wed Nov 20 16:18:38 2024 -0600

    Automate release creation (#4559)

    * automate release artifact generation

    * on push

    * fix syntax

    * test builds for any tag

    * explicit source tree build job

    * working-directory

    * only draft release for x.x.x tags

    * fix syntax

    * update HOWTORELEASE
hobu added a commit that referenced this pull request Jan 10, 2025
@hobu @ibell13
pdal tindex tutorial (#4599)
08ca18f 
* New tutorial for pdal tindex

commit 3a91939
Author: ibell13 <[email protected]>
Date:   Thu Nov 14 14:09:04 2024 -0600

    add link in tutorial

commit 3450dda
Author: ibell13 <[email protected]>
Date:   Wed Nov 13 12:43:23 2024 -0600

    add parquet format

commit cc845ac
Author: ibell13 <[email protected]>
Date:   Tue Nov 12 14:55:41 2024 -0600

    tutorial - add images

commit 5c2e4c6
Author: ibell13 <[email protected]>
Date:   Tue Nov 12 14:27:09 2024 -0600

    tindex tutorial - draft 2

commit fe33e2d
Author: ibell13 <[email protected]>
Date:   Thu Nov 7 23:26:49 2024 -0600

    initial powershell additions

commit ffbc849
Author: ibell13 <[email protected]>
Date:   Mon Nov 4 16:54:00 2024 -0600

    _toc.yml: link tutorial

commit 3e81991
Author: ibell13 <[email protected]>
Date:   Mon Nov 4 16:40:35 2024 -0600

    new TIndex tutorial - draft

* tutorial updates 1

* tutorial updates 2

* Squashed commit of the following:

commit c720e6d
Author: Howard Butler <[email protected]>
Date:   Mon Dec 23 12:41:31 2024 -0600

    Local documentation build conveniences (#4592)

    * A local build Makefile

    * add more stuff for local build to environment.yml

    * Add hobu logo stuff

    * remove conf.py – this is now generated by jupyterbook

    * drop conda-provided sed/graphviz/doxygen

    * a little doc for building locally

    * update substitutions

commit 74382cb
Author: Howard Butler <[email protected]>
Date:   Mon Dec 23 09:52:32 2024 -0600

    adapt standard gitignore for CMake files (#4596)

commit 9bab3a3
Author: Guilhem Villemin <[email protected]>
Date:   Tue Dec 17 18:13:22 2024 +0100

    Enhance `filters.gpstimeconvert` to make it streamable (#4568)

    * make filter streamable

    * fix typo in disclaimer

    * update doc

    * thow when invalid conversion

    * revert spurious edit

    * add missing Streamable

    * adding run method

    * add wrapped tolerance arg

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit 991e67c
Author: Howard Butler <[email protected]>
Date:   Tue Dec 17 09:51:10 2024 -0600

    Fix #4578, doc nit in readers.rdb (#4590)

commit 998be56
Author: Howard Butler <[email protected]>
Date:   Mon Dec 16 10:30:14 2024 -0600

    update community page with link to Matrix (#4587)

commit 2d4a3ce
Author: Andrew Bell <[email protected]>
Date:   Mon Dec 16 09:47:35 2024 -0500

    Make sure points are added to a new point view if a filter returns no views and (#4583)

    where_merge == 'true'.
    Close #4572

commit cda0631
Author: Andrew Bell <[email protected]>
Date:   Mon Dec 16 09:39:22 2024 -0500

    Write binary in binary mode. (#4584)

    Some general cleanup.

commit 213a6a6
Author: chambbj <[email protected]>
Date:   Thu Dec 12 12:13:34 2024 -0600

    Isolate dependencies and tasks for doc generation and preview (#4579)

commit 2271196
Author: Howard Butler <[email protected]>
Date:   Thu Dec 12 11:59:06 2024 -0600

    adapt GDAL's backport bot (#4580)

commit 97522d1
Author: chambbj <[email protected]>
Date:   Wed Dec 11 11:07:00 2024 -0600

    Use tabs for the OS variants of using Conda (#4575)

commit ab8a5d1
Author: chambbj <[email protected]>
Date:   Wed Dec 11 10:45:39 2024 -0600

    Rewicker the pixi config (#4571)

    * Rewicker the pixi config

    * Enable NITF and also mark this as a win-64 platform task

    * Use target_compile_definitions for the _XKEYCHECK_H fix

    * Add task for linux-64 configure, move docs, build, and test tasks

commit 88531fc
Author: Howard Butler <[email protected]>
Date:   Wed Dec 4 17:16:29 2024 -0600

    protect against including null strings when reading PDAL vlr types (#4566)

commit 9800957
Author: Howard Butler <[email protected]>
Date:   Wed Dec 4 15:23:11 2024 -0600

    bump master CMake version to 2.9.0 to eliminate confusion

commit d172b32
Author: chambbj <[email protected]>
Date:   Wed Dec 4 15:11:26 2024 -0600

    Add Pixi install instructions (#4565)

    * Add instructions for setting up a Pixi environment

    * Fix nested code block

commit de41500
Author: Andrew Bell <[email protected]>
Date:   Tue Dec 3 15:58:31 2024 -0500

    `filters.expression` support for filtering NaN and LOWEST/HIGHEST (#4561)

    * Checkpoint.

    * testing WIP

    * Checkpoint.

    * Don't call checkEnd() in MathParser.
    Add failure tests.

    * Remove dead comment.

    * Provide nan support.

    * Remove junk.

    * Fix typos.

    * Test.

    * Test.

    * Test.

    * Test 2.

    * Test 3.

    * Test 4.

    * Test 5.

    * bump ci

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit cc3e18f
Author: Guilhem Villemin <[email protected]>
Date:   Tue Dec 3 17:28:43 2024 +0100

    Extend `filters.gpstimeconvert` to handle day seconds.

    * convert to day seconds

    * typo

    * finish day conversion

    * fixes

    * restore conversion arg, mark it as deprecated

    * declare m_converison

    * fix type declaration

commit cf6f6ef
Author: Howard Butler <[email protected]>
Date:   Sun Dec 1 10:37:20 2024 -0600

    Validate checksums and version number in automated release CI (#4564)

    * add release version checks

    * set exit

    * dummy 7.7.7 release version

    * explicit verify of checksums

    * cd into correct dir

    * need conda for testing

    * syntax

    * break apart steps a bit

    * set path to environment

    * bad path :(

    * bad path again :(

    * dont build tests

    * check installed version version

    * use pdal-config for clean version string

    * only release for x.x.x

commit 3334e6e
Author: chambbj <[email protected]>
Date:   Fri Nov 22 15:52:28 2024 -0600

    Detect path to conf.py and set JSON/CSV paths relative (#4560)

    * Detect path to conf.py and set JSON/CSV paths relative

    * Try running it as a separate pre_build step

    * fix typo

    * jeez

    ---------

    Co-authored-by: Howard Butler <[email protected]>

commit eab55c5
Author: Howard Butler <[email protected]>
Date:   Wed Nov 20 16:18:38 2024 -0600

    Automate release creation (#4559)

    * automate release artifact generation

    * on push

    * fix syntax

    * test builds for any tag

    * explicit source tree build job

    * working-directory

    * only draft release for x.x.x tags

    * fix syntax

    * update HOWTORELEASE

* move file list up to cloud

* howard WIP

* Do not add prefixes to filenames that are isRemote

* update Makefile

* tindex tutorial WIP

* handle reader_args, let log() for each stage created to readers.tindex's log

* more tutorial updates

* add missing images

---------

Co-authored-by: ibell13 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

2.8.3

Development

Successfully merging this pull request may close these issues.

2 participants

@hobu @abellgithub