add prep nisar #4

mirzaees · 2023-06-29T16:39:50Z

Description of proposed changes

Add prep_nisar.py script to read GUNW products. The inputs are a list of nisar*.h5 data cubes and the DEM.

Future to do list:

The water mask in the test products is all zero. To avoid masking the data, it was manually set to one. We need to remove manual part after correctly adding water mask to input data.
Tropospheric correction of Mintpy using pyaps does not perform well because of UTM coordinate system. We might read the correction layers from input data instead in the future. But for now, the height_correlation option works.
The x/y coordinates of data cube in radarGrid require to add a margin to cover the x/y coordinates of GUNW. For now the common bounding box to read all data is the area covered by both radarGrid and GUNW grid, meaning we cut the outer edges if needed.
The auto path for the input products is set to "../interferograms/*.h5". It might change based on where the GUNW products are located after processing with isce3

scottstanie

Just a few thoughts at first read-through, this will be great to have.
perhaps we can discuss what commonalities there are between this one and the OPERA prep script started here isce-framework/sweets#10 , but this one is clearly more built out so far

src/mintpy/prep_nisar.py

scottstanie · 2023-06-29T17:23:49Z

src/mintpy/prep_nisar.py

+    return bounds
+
+
+def bbox_to_utm(bbox, epsg_dst, epsg_src=4326):


is there anything like this already in Mintpy? It seems like i'd want something like that as a utility

Not sure, I rewrote this from COMPASS to avoid dependency

scottstanie · 2023-06-29T18:44:34Z

src/mintpy/prep_nisar.py

+    meta["Y_FIRST"] = y_origin
+    meta["X_STEP"] = pixel_width
+    meta["Y_STEP"] = pixel_height
+    meta["X_UNIT"] = meta["Y_UNIT"] = "meters"


I think for NISAR they might have a "units" attribute in every xCoordinates and yCoordinates dataset (and if not then it'll need to be added to be CF compliant)

It will be great to add that attribute, we can modify in the future

scottstanie · 2023-06-29T18:46:13Z

src/mintpy/prep_nisar.py

+    meta["X_FIRST"] = x_origin
+    meta["Y_FIRST"] = y_origin
+    meta["X_STEP"] = pixel_width
+    meta["Y_STEP"] = pixel_height


Two thoughts on this-

perhaps we could get these from like dataset.GetGeotransform(), if these are all geocoded. but if it's a mix of radar/geo, i can see why you'd do it this way

I think we need to add 1/2 a pixel to make these compliant with the top-left-edge convention, since I believe the xCoordinates refer to pixel centers. As in, x_origin = min(x_coordinates) - pixel_width/2, same for y

Thank you @scottstanie for reviewing and all the good suggestions.
For the first one I think it is a mixed data and might not be possible.
Second one, you are right I will check on that again

Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com>

updates: - [github.com/asottile/pyupgrade: v3.7.0 → v3.8.0](asottile/pyupgrade@v3.7.0...v3.8.0)

+ add an entry point for prep_nisar in setup.py + add subparser for prep_nisar in __main__.py

* Track and fix bug in gdal corner convention * readfile.read_gdal_vrt: add comments on GDAL convention * readfile.read_envi_hdr: add comments on ENVI convention --------- Co-authored-by: Zhang Yunjun <yunjunz@outlook.com>

updates: - [github.com/asottile/pyupgrade: v3.8.0 → v3.9.0](asottile/pyupgrade@v3.8.0...v3.9.0)

+ generalize gamma LUT file extension checking, by replacing the "full file ext checking" with "file ext ending chars checking", to handle both "utm*" and "eqa*" prefix, such as .UTM_TO_RDC, .eqa_2_rdc, as provided by the various gamma users (GAMMA is annoying), in the following scripts: - utils.readfile - utils.writefile - prep_gamma + utils.readfile.read_binary(): convert ~pi value to zero while read phase from cpx file - convert ~pi value to zero while read phase from cpx file, as it sometimes happens to gamma products, probably due to the limited numerical precision of the complex data file, such as .int file. - light refactoring while calling np.fromfile(), to improve the readability + utils.readfile.read_binary_file(): for roipac processor, which is the default one if not specified, honor the metadata file for the following attributes if they exist, instead of using the hardwired value, which is the same as the default ones: - INTERLEAVE - BANDS - BYTE_ORDER - DATA_TYPE + docs/api/attribute.md: add BYTE_ORDER for gamma + dosc/api/data_structure: add file name convention

+ objects/resample.py: fix a bug for `geocode.py --bbox` option while geocoding gamma products with customized SNWE from --bbox option, the output dest_box does not start from 0, thus, should use box[3/4] directly, instead of box[3]-box[1] and box[2]-box[0] + cli/subset: check input file exsitence + view.py --coord: update help msg to avoid potential confusions by updating utils.arg_utils.add_figure_argument() + view.py --coord: set the other choices as yx, instead of radar, and keep radar still in the loop for backward compatiability

updates: - [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](asottile/pyupgrade@v3.9.0...v3.10.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…nsarlab#1050) + save_hdfeos5: always write latitude/longitude dataset, and calculate based on metadata if missing in the geometry file, such as ARIA + cli/save_hdfeos5: check the coordinate of input file + improved UNIT and fig title handling for HDF-EOS5 product: - utils.readfile.read_attribute(): while grab the UNIT, use the last segment of dataset name if "/" is used - view.plot_subplot4figure(): use the last segment as the auto figure title, instead of the 2nd one, to be more generic + view: improved auto figsize for subplots when there is only one row of subplots

+ utils.utils0.py: add the following functions to convert coordinates between UTM and lat/lon: - utm2latlon() - latlon2utm() - utm_zone2epsg_code(): bugfix if the latitude letter is not N or S - get_lat_lon(): convert UTM to lat/lon to ensure lat/lon output for hyp3 products + add utm to the dependency files, since 1) it depends on numpy only; and 2) it's available on PyPI and conda-forge - requirements.txt - setup.py

Fix the previous confusion that UTM_ZONE should end with the hemisphere, not the latitude band, which is not part of UTM but part of MGRS. + utils.utils0.utm_zone2epsg_code(): revert back the south arg feeding from UTM_ZONE + utils.utils0.utm2latlon(): use the `northern` arg instead of the latitude band, to match the meaning of UTM_ZONE metadata + utils.utils0.latlon2utm(): return east/north only, for simplicity and style consistency with utm2latlon. + docs/api/attributes.UTM_ZONE: add more description and examples

+ utils.isce_utils: add/wrap filter_goldstein() from isce2 for easy reuse. + utils.isce_utils.unwrap_snaphu(): rename defo_max to max_defo to be more intuitive + cli/view.py: print out warning msg if --faultline is specified for radar-coded file + docs/api/data_structure.md: update syntax

+ tsview: add --figsize-img option to be able to customize the map figure size. Make this arg optional so that other existing functionalities won't be affected. + subset.read_subset_template2box: - support mix separator/delimiter of : and , so that values from isce-proc also work - replace try/except with if judgment

+ legacy/load2hdf5: add --compression option + view: add --faultline-min-dist option to trim the plotted fault lines + view: set the x-axis label to the top for horizontal colorbar at the bottom + utils.plot.auto_adjust_xaxis_date: consider axes width while determining every_year

…ab#1064) + utils.readfile.read_attribute(): translate DATA_TYPE value "ci2" to "float32" for roipac products + tests/smallbaselineApp: more comments on example usage

+ tropo_pyaps3: bugfix for hyp3 products in UTM coordinates: - calc_delay_timeseries(): remove the call of `ut.to_latlon()` while preparing the pixel-wised 2D lat/lon matrices, as `ut.get_lat_lon()` handled the UTM/latlon conversion already in insarlab#1052 + replace `ut.to_latlon()` with `ut.utm2latlon()` as the later supported more file types, in the following occurrences: - tropo_pyaps3.get_bounding_box() - utils.utils.prepare_geo_los_geometry() + utils.utils0.get_lat_lon(): print out UTM/latlon conversion message, to help future tracking/diagnose + prep_hyp3: convert UTM to lat/lon for LAT/LON_REF1/2/3/4

…nsarlab#1063) + objects.stack.ifgramStack.get_sequential_closure_phase(): bring back the spatial referencing to ARIA products, to avoid the abnormal closure phase calculation result, which is likely due to the phase stitching applied during ARIA products preparation in ARIA-tools.

+ add `argcomplete` to the dependency for tab completion capability. This package is light (only depends on pure Python), and available on pip and conda-forge (as noarch package). - requirements.txt: add argcomplete - setup.py: move argcomplete from optional to required + docs/installation.md: add "tab complete sub-commands" as another sub-section in Post-Installation Setup.

+ replace the np.rint() call with np.ceil() to ensure non-zero step size, which could happen for long time-series with highly redundant networks.

…sarlab#1068) + `utils.isce_utils.extract_multilook_number()`: improve robustness against the occasionally missing *.full.xml files for hgt/lat/lon/los during the isce2/topsStack processing by: - add shadowMask to the potential data file list - search both *.full.xml and *.full.vrt file + version: add version 1.5.2 tag

+ `dependency`: add `shapely` to `requirements.txt` and `setup.py` file, as it's used in `objects.euler_pole.py` to plot plate motion since insarlab#935. Note that 1) shapely is already a dependency for cartopy, so this won't really affect the mintpy installation; 2) shapely has been added explicitly to the dependency on conda-forge for mintpy (conda-forge/mintpy-feedstock#10) + `docs/installation/argcomplete`: replace `conda` with `brew` for bash installation, so that the bash is independent of the conda directory, which can be removed more easily/frequently than `/usr/local directory` from brew.

+ tsview: - timeseriesViewer.plot(): consider the SNWE for geocoded file when plotting in geo coordinate, for a more suited figure size - get_model_param_str(): do not show the intercept in the model fitting result, as it's not useful + view - update_inps_with_file_metadata(): add "_nws" suffix if showing a single subplot without whitespace - display pi symbol if min/max is very close 1e-3 via the following changes: - utils.plot.auto_adjust_colormap_lut_and_disp_limit(): convert near-pi value to pi if they are very close <1e-3 - utils.plot.plot_colorbar(): display pi symbol on the colorbar if displaying limits are very close to -pi/pi (<1e-3)

…pability (insarlab#1072) * utils0: fix typos * utils0: simplify look of `update_attribute_or_not` The bare `next` was acting as a `pass` or `continue` * fix more typos caught by vscode * use `ReadAsArray` after casting box to int This is fixed in gdal versions >3.7.0 OSGeo/gdal#8026 The issue came from passing `np.int64` objects to GDAL, which it wasnt handling properly. Regular python `int`s work find * remove unused import * gps: print the filename, instead of (possibly blank) dirname * convert `Path`s to str for readfile also fix grammar for file not found * remove 2nd version of wkt function

…sarlab#1077) + view.prep_slice(): support quoted lists of characters as inputs so that its parsing behavior is the same as the one in command line

+ `geocode`: support single input for the `mintpy.geocode.laloStep` option, as it is commonly used; and convert it into a list of two internally in the code. + `smallbaselineApp.run_load_data()`: tune the auto-skipping, by not adding the metadata from the custom template to the geometry HDF5 file, to avoid updating its file modification time. + `timeseries_rms`: auto adjust figure width for long TS + `utils.utils1.spatial_average()`: - always read the output from the text file, which is in 1e-4 precision, to ensure consistency. - use the dataset MODIFICATION_TIME for ifgramStack file, for a more representative run/skip checking.

+ support hdf5 input --> binary output + support binary input --> hdf5 output

…lab#1074) * prep_hyp3: support burst-wide interferogram from ASF hyp3-isce2 processor/instance * docs/dir_structure: add example directory structure from ASF hyp3_isce2 --------- Co-authored-by: Zhang Yunjun <yunjunz@outlook.com>

Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com>

* add modified `scripts/fix_typos.sh` from GDAL repo * run `fix_typos.sh` --------- Co-authored-by: Zhang Yunjun <yunjunz@outlook.com>

add prep nisar

89c1611

scottstanie reviewed Jun 29, 2023

View reviewed changes

Sara Mirzaee and others added 28 commits June 29, 2023 12:02

Use arbitrary bounding box in lat/lon

e054eff

Use speed of light from mintpy constants

a2ba92a

Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com>

remove extra line

dae973f

Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com>

Use earth radius from mintpy constants

b9910dc

Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com>

import constants from mintpy

2a765a6

Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com>

account for half a pixel difference in origin of the coordinates

530ca9d

correct x-first to be the upper left pixel

33e017c

pyupgrade: v3.7.0 → v3.8.0 (insarlab#1039)

7476d0f

updates: - [github.com/asottile/pyupgrade: v3.7.0 → v3.8.0](asottile/pyupgrade@v3.7.0...v3.8.0)

add prep_nisar to setup and __main__

ccff336

+ add an entry point for prep_nisar in setup.py + add subparser for prep_nisar in __main__.py

Update __main__.py

cb52331

pyupgrade: v3.8.0 → v3.9.0 (insarlab#1043)

a1db4bf

updates: - [github.com/asottile/pyupgrade: v3.8.0 → v3.9.0](asottile/pyupgrade@v3.8.0...v3.9.0)

pyupgrade: v3.9.0 → v3.10.1 (insarlab#1051)

3a646ae

updates: - [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](asottile/pyupgrade@v3.9.0...v3.10.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

read_attribute: translate DATA_TYPE ci2 to float32 for roipac (insarl…

2af3905

…ab#1064) + utils.readfile.read_attribute(): translate DATA_TYPE value "ci2" to "float32" for roipac products + tests/smallbaselineApp: more comments on example usage

fix zero step size bug in unwrap_error_phase_closure (insarlab#1067)

559b8d7

+ replace the np.rint() call with np.ceil() to ensure non-zero step size, which could happen for long time-series with highly redundant networks.

yunjunz and others added 10 commits August 15, 2023 13:00

view.prep_slice(): support quoted lists of characters as inputs (in…

f068ec5

…sarlab#1077) + view.prep_slice(): support quoted lists of characters as inputs so that its parsing behavior is the same as the one in command line

subset: support hdf5 <--> binary (insarlab#1076)

b800d64

+ support hdf5 input --> binary output + support binary input --> hdf5 output

Correct example command line

09daaac

Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com>

Run fix_typos.sh on repo to catch spelling errors (insarlab#1082)

7ff149b

* add modified `scripts/fix_typos.sh` from GDAL repo * run `fix_typos.sh` --------- Co-authored-by: Zhang Yunjun <yunjunz@outlook.com>

Merge branch 'main' into prep_script_for_NISAR

183819d

pre-commit suggestions

018ddbd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add prep nisar #4

add prep nisar #4

mirzaees commented Jun 29, 2023

scottstanie left a comment

scottstanie Jun 29, 2023

mirzaees Jun 29, 2023

scottstanie Jun 29, 2023

mirzaees Jun 29, 2023

scottstanie Jun 29, 2023 •

edited

Loading

mirzaees Jun 29, 2023

		return bounds


		def bbox_to_utm(bbox, epsg_dst, epsg_src=4326):

add prep nisar #4

Are you sure you want to change the base?

add prep nisar #4

Conversation

mirzaees commented Jun 29, 2023

scottstanie left a comment

Choose a reason for hiding this comment

scottstanie Jun 29, 2023

Choose a reason for hiding this comment

mirzaees Jun 29, 2023

Choose a reason for hiding this comment

scottstanie Jun 29, 2023

Choose a reason for hiding this comment

mirzaees Jun 29, 2023

Choose a reason for hiding this comment

scottstanie Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

mirzaees Jun 29, 2023

Choose a reason for hiding this comment

scottstanie Jun 29, 2023 •

edited

Loading