Hi @sssangha, thanks for looking into this. I pulled PR #313 and reinstalled with python -m pip install .

I am now getting this error for three tracks that I've tested so far: 41, 48, and 19:

Thread count specified for gdal multiprocessing = 8
Rejected scene 9arc/productBoundingBox/20141219_20141008.json has only 24315.766499km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20141219_20141125.json has only 24315.766499km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20141220_20141009.json has only 213872.487194km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20141220_20141126.json has only 213872.487194km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20150112_20141101.json has only 24286.600098km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20150112_20141219.json has only 24286.600098km² overlap with bbox
Rejected scene 9arc/productBoundingBox/20150113_20141009.json has only 213734.748814km² overlap with bbox
/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/shapefile_util.py:71: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
  for polyobj in file_bbox:
Rejected scene 9arc/productBoundingBox/20150113_20141102.json has only 170739.425271km² overlap with bbox
Traceback (most recent call last):
  File "/home/rzinke/miniconda3/envs/insar/bin/ariaTSsetup.py", line 18, in <module>
    main(inps)
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/tsSetup.py", line 417, in main
    proj) = merged_productbbox(standardproduct_info.products[0],
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 493, in merged_productbbox   
    prods_bbox=open_shapefile(scene['productBoundingBox'][0], 0, 0)
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/shapefile_util.py", line 34, in open_shapefile
    file_bbox = file_bbox.GetLayerByIndex(lyrind).GetFeature(ftind)
AttributeError: 'NoneType' object has no attribute 'GetLayerByIndex'

The launch command is similar for each (I only change the bbox file and minimum overlap):

ariaTSsetup.py -f "AllProducts/*.nc" -b track_048_TibetDii.shp -d download -m download -nt 8 -mo 270000 -ml 3 -v

@rzinke and @bbuzz31 I have confirmed the job runs to completion now with this new patch.

Specifically, I found and fixed a bug whereby midnight crossings were not handled as intended. I have now refactored the code a bit behind this simple logic: if there is spatial overlap and the IFGs are within one day of eachother, that means they are of the same orbit and thus contiguous IFGs.

At the same time, I've made some ancillary pep8 updates that helped me debug and make the code a bit more readable.

Please confirm if this patch works as intended on your end!

It got a lot closer this time!

Now I am getting the error:

NETCDF:"/data/Tibet/Track041/AllProducts/S1-GUNW-A-R-041-tops-20180719_20180108-120249-00085E_00046N-PP-ab45-v2_0_5.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/data/Tibet/Track041/AllProducts/S1-GUNW-A-R-041-tops-20180719_20180108-120315-49678N_47644N-PP-6dd6-v2_0_4.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/data/Tibet/Track041/AllProducts/S1-GUNW-A-R-041-tops-20180719_20180108-120315-49678N_47644N-PP-6dd6-v2_0_4.nc":/science/grids/data/connectedComponents is GDAL compatible
Traceback (most recent call last):
  File "/home/rzinke/miniconda3/envs/insar/bin/ariaTSsetup.py", line 18, in <module>
    main(inps)
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/tsSetup.py", line 458, in main
    export_products(standardproduct_info.products[1],
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/extractProduct.py", line 669, in export_products
    product_stitch_overlap(unw_files,conn_files,prod_bbox_files,bounds,prods_TOTbbox, outFileUnw=outFileUnw,outFileConnComp= outFileConnComp, mask=mask,outputFormat = outputFormat,verbose=verbose)
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/unwrapStitching.py", line 1458, in product_stitch_overlap
    unw.UnwrapOverlap()
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/unwrapStitching.py", line 340, in UnwrapOverlap
    self.__calculateCyclesOverlap__()
  File "/home/rzinke/miniconda3/envs/insar/lib/python3.10/site-packages/ARIAtools/unwrapStitching.py", line 404, in __calculateCyclesOverlap__
    temp = connCompData2_temp.astype(np.int)-connCompData1.astype(np.int)
ValueError: operands could not be broadcast together with shapes (2229,4273) (2229,4274)

Looks like there needs to be a check to force the file sizes to be consistent.

Getting this same failure for another of the tracks, too.

Those tracks are T41 (the best-behaved track for Tibet) and T48 (the midnight-crosser).
AOI T19S succeeded with the bug fix.

@rzinke could you send exact command and shapefile bbox if necessary?

@sssangha is this related to the inconsistencies you were working on w.r.t. gdal.Warp?

Hey @bbuzz31,
Here are the commands for the TS preparation. Note that the data set is huge - 15K products * 50 Mb each = ~780GB.

ariaDownload.py -t 48 -b "23.0 55.0 63.0 117.0"

It looks like the error is occurring sometime around the interval 20180918_20180310, you might be able to save time by narrowing it down to that date range. Then again, the exact size of the file will likely depend on the intersection of the other product footprints.

The TSsetup command is

ariaTSsetup.py -f "/data/Tibet/Track048/AllProducts/*.nc" -w 9arc -b /data/Tibet/Track048/AOIs/track_048_TibetDii/track_048_TibetDii.shp -d download -m download -nt 8 -mo 270000 -ml 3 -v

The shapefile is included in the attached .zip.

track_048_TibetDii.zip

Let me know how I can help.

Hello @rzinke I was unable to replicate, but I had just tried it out with this IFG to conserve space. Could you perhaps share the following produceBoundingBox outputs in your workspace for testing: productBoundingBox/productBoundingBox*

@bbuzz31 I don't think it is related to that, especially because the other layers (e.g. amplitude) are generated fine. I wonder if it's perhaps an issue with the unwrap stitcher.

Here are the productBoundingBox files (attached
productBoundingBox.zip
)

Unwrap stitcher seems likely to me...
So the connectedComponent files are different shapes when they are differenced in the __calculateCyclesOverlap__ subroutine of UnwrapOverlap.

It looks like the connectedComponents get resampled by gdalWarp a few lines above, in lines 387-388. Seems to me we should be able to ensure the gdalWarp outputs are the same size as the inputs(?). For example, we could explicitly pass the output file size to the gdalWarp function in line 388... I wonder if there's a more elegant way to do this though, e.g., with TAP or some other flag. Do either of you know of a convenient argument for that, @sssangha or @bbuzz31 ?

Apologies, there was some confusion on my end to replicate the issue as I was working with track 48 products, but was able to debug with track 41 products using the following commands:

ariaDownload.py -t 41 -b '23.0 55.0 63.0 117.0' -nt 4 -i '20180719_20180108' -v
ariaExtract.py -f 'products/*.nc' -of ISCE -w test_dir -l 'amplitude,unwrappedPhase' -nt 8 -ml 3

I have resolved the issue by forcing consistent dimensions in the unwrapStitching code.

@rzinke I found this particular stitched pair to be of very poor quality. After running some quick tests to vet and hopefully merge these changes, could you also perhaps visually inspect pairs across a track or two? I'd be curious to get a gauge of the quality of the stitched IFGs. Perhaps it would be easiest to inspect the IFG stack through Mintpy. I had done something similar to this before for the CA tracks with David. Make sense?

That fixed it. ariaTSsetup.py has worked for all tracks I've tried so far (~10) including the ones that were formerly giving us trouble. Thanks!

P.S. I have not done any systematic QA on the stitched interferograms, though previous experience tells me the stitching quality varies greatly from one track to the next.

@yunjunz

With the mintpy conda install, the prep_*.py scripts wraps the python function as: sys.exit(main())

For all prep_*.py EXCEPT prep_aria.py, the main() function has no return, but for prep_aria.py it returns inps.outfile

sys.exit interprets the inps.outfile as a non-zero exit status, which causes circleci to stop.

Do you recall any particular reason that prep_aria has to return the inps.outfile? If not, I'll go ahead and issue a PR that removes it. Thanks!

@yunjunz

With the mintpy conda install, the prep_*.py scripts wraps the python function as: sys.exit(main())

For all prep_*.py EXCEPT prep_aria.py, the main() function has no return, but for prep_aria.py it returns inps.outfile

sys.exit interprets the inps.outfile as a non-zero exit status, which causes circleci to stop.

Do you recall any particular reason that prep_aria has to return the inps.outfile? If not, I'll go ahead and issue a PR that removes it. Thanks!

There is no reason, please feel free to go ahead to remove it.

@yunjunz
With the mintpy conda install, the prep_*.py scripts wraps the python function as: sys.exit(main())
For all prep_*.py EXCEPT prep_aria.py, the main() function has no return, but for prep_aria.py it returns inps.outfile
sys.exit interprets the inps.outfile as a non-zero exit status, which causes circleci to stop.
Do you recall any particular reason that prep_aria has to return the inps.outfile? If not, I'll go ahead and issue a PR that removes it. Thanks!

There is no reason, please feel free to go ahead to remove it.

Thanks! Any plans for a new conda release soon, or should I switch or circleci back to installing from source?

A new release of mintpy is planned before the end of next week (Aug 5), the conda-forge version usually is available within 12 hours after the github release.