Skip to content

Fix CMake discrepancies#2

Merged
a0x8o merged 73 commits intoa0x8o:masterfrom
OSGeo:master
Jun 6, 2022
Merged

Fix CMake discrepancies#2
a0x8o merged 73 commits intoa0x8o:masterfrom
OSGeo:master

Conversation

@a0x8o
Copy link
Copy Markdown
Owner

@a0x8o a0x8o commented Jun 6, 2022

GPKG: writer: fix corruption when only a subset of all bands of a tile is flushed

If a block/tile flush happens when only a subset of all bands is
'dirty', and in particular if the alpha band is not yet written, then
the tile was wrongly considered as transparent.

The issue could happen even when writing all bands together, but under
strong pressure on the block cache, due to other threads doing
concurrent reads and forcing partial flushing.

GPKG: writer: write fully set tiles as soon as possible to decrease pressure on block cache

Docker: update Alpine based images to 3.16, and add Apache Arrow/Parquet to alpine-normal image [ci skip]

CMake: build iso8211 library conditionally to drivers requiring it, and no longer tag it as a driver

SAFE: for non-SLC products, do not report subdatasets that can't be opened

Fixes issue reported in https://lists.osgeo.org/pipermail/gdal-dev/2022-May/055882.html

With that fix, the only reported subdatasets now are:

SUBDATASET_1_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW_VH:AMPLITUDE
  SUBDATASET_1_DESC=Single band with IW swath and VH polarization and UNCALIB calibration
  SUBDATASET_2_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW_VV:AMPLITUDE
  SUBDATASET_2_DESC=Single band with IW swath and VV polarization and UNCALIB calibration
  SUBDATASET_3_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW:AMPLITUDE
  SUBDATASET_3_DESC=IW swath with all polarizations (VH+VV) as bands and UNCALIB calibration

mitab.rst: fix driver short name

Add perftests/computeminmax.py

Doc: build_hints: document discrepancies between name of CMake optioss controlling driver build and driver names (fixes OSGeo#5809)

RMF: fix potential heap-buffer-overflow in RMFDataset::WriteHeader. triggered by recent master commit. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47716

Update doc/source/build_hints.rst

GDALRasterBand::ComputeRasterMinMax(): add optimized implementation for Byte and UInt16 data types

re-use the optimizations of ComputeStatistics(), using SSE2 when available

Before:

$ python perftests/computeminmax.py
testByte(): 8.776
testUInt16(): 7.174
testInt16(): 8.210
testFloat32(): 7.846
testFloat64(): 9.407

After:

$ python perftests/computeminmax.py
testByte(): 0.316
testUInt16(): 0.779
testInt16(): 2.863
testFloat32(): 6.993
testFloat64(): 8.078

To be compared with timings of ComputeStatistics():

$ python ../perftests/computestatistics.py
testByte(): 0.483
testUInt16(): 1.262
testInt16(): 28.417
testFloat32(): 29.075
testFloat64(): 30.382

rouault and others added 30 commits May 26, 2022 18:36
…e is flushed

If a block/tile flush happens when only a subset of all bands is
'dirty', and in particular if the alpha band is not yet written, then
the tile was wrongly considered as transparent.

The issue could happen even when writing all bands together, but under
strong pressure on the block cache, due to other threads doing
concurrent reads and forcing partial flushing.
…pened

Fixes issue reported in https://lists.osgeo.org/pipermail/gdal-dev/2022-May/055882.html

With that fix, the only reported subdatasets now are:
```
  SUBDATASET_1_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW_VH:AMPLITUDE
  SUBDATASET_1_DESC=Single band with IW swath and VH polarization and UNCALIB calibration
  SUBDATASET_2_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW_VV:AMPLITUDE
  SUBDATASET_2_DESC=Single band with IW swath and VV polarization and UNCALIB calibration
  SUBDATASET_3_NAME=SENTINEL1_CALIB:UNCALIB:S1A_IW_GRDH_1SDV_20220529T045822_20220529T045847_043421_052F5A_01B5.SAFE/manifest.safe:IW:AMPLITUDE
  SUBDATASET_3_DESC=IW swath with all polarizations (VH+VV) as bands and UNCALIB calibration
```
…s controlling driver build and driver names (fixes #5809)
Co-authored-by: Adam J. Stewart <[email protected]>
…or Byte and UInt16 data types

re-use the optimizations of ComputeStatistics(), using SSE2 when available

Before:
```
$ python perftests/computeminmax.py
testByte(): 8.776
testUInt16(): 7.174
testInt16(): 8.210
testFloat32(): 7.846
testFloat64(): 9.407
```

After:
```
$ python perftests/computeminmax.py
testByte(): 0.316
testUInt16(): 0.779
testInt16(): 2.863
testFloat32(): 6.993
testFloat64(): 8.078
```

To be compared with timings of ComputeStatistics():
```
$ python ../perftests/computestatistics.py
testByte(): 0.483
testUInt16(): 1.262
testInt16(): 28.417
testFloat32(): 29.075
testFloat64(): 30.382
```
RMF: fix potential heap-buffer-overflow in RMFDataset::WriteHeader.
CMake: build iso8211 library conditionnaly to drivers requiring it, …
Doc: build_hints: document discrepancies between name of CMake options…
GPKG: writer: fix corruption when only a subset of all bands of a tile is flushed
MSGN: fix crash and do other changes to read HRV band (fixes #1040)
WMTS: add <Accept> and <ExtraQueryParameters> in local XML file to please difficult servers (fixes #5729)
SAFE: for non-SLC products, do not report subdatasets that can't be opened
…Time field (recently introduced issue, not in release)
…hAware=true property (fixes #5836)

This also requires the operating system to accept long paths, by setting
a LongPathsEnabled registry key to 1, e.g with the following Powershell command:
```
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force
```

See https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later
Arrow: fix reading geometry columns whose type is registered as an Arrow extension (fixes #5834)
GDALRasterBand::ComputeRasterMinMax(): add optimized implementation for Byte and UInt16 data types
aviborg and others added 29 commits June 3, 2022 16:29
NITF: Use correct index for image segment (fixes #5855)
… longer than the max indexed string, or ending with space
…ncated

OpenFileGDB: fix use of indexes on strings when the searched value is…
FileGDB: add MSCALE, MORIGIN and MTOLERANCE layer creation option (fixes #5858)
…RDER BY ... LIMIT ... OFFSET ...' where the ORDER BY column has an index
OpenFileGDB: take LIMIT and OFFSET into account in 'SELECT ... FROM ORDER BY ... LIMIT ... OFFSET ...' where the ORDER BY column has an index
COSAR: fix reading the last sample of each line (fixes #5622)
With this improvement, the ogr_bench program on the nz-building-outlines.gpkg
file mentioned in RFC 86 (#5830) runs
from 15.2 s down to 8.1 s
With this improvement, the ogr_bench program on the nz-building-outlines.gpkg
file mentioned in RFC 86 (#5830) runs
from 8.1 s (previous commit) to 7.5 s
…ariables (CID 1489577, 1489578) (master only)
Avoid issues with PROJJSON with id in members of datum ensemble
GPKG: optimize feature reading, in particular DateTime fields
Add OGR_G_ConcaveHull(), using GEOS >= 3.11 GEOSConcaveHull_r(), and map it to SWIG
@a0x8o a0x8o merged commit 8bf7382 into a0x8o:master Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CMake: mismatch between docs and driver flags

6 participants