Skip to content

Comments

grass.script: Automatically parse JSON and CSV in parse_command#3687

Merged
petrasovaa merged 4 commits intoOSGeo:mainfrom
petrasovaa:improve-parse-command
May 8, 2024
Merged

grass.script: Automatically parse JSON and CSV in parse_command#3687
petrasovaa merged 4 commits intoOSGeo:mainfrom
petrasovaa:improve-parse-command

Conversation

@petrasovaa
Copy link
Contributor

@petrasovaa petrasovaa commented May 3, 2024

Given several tools have options format, this adds automatic parsing to parse_command:

>>> data = gs.parse_command("r.what", map='elevation@PERMANENT', coordinates=[641567, 223962], format='json')
[{'easting': 641567, 'northing': 223962, 'site_name': '', 'elevation@PERMANENT': {'value': 75.82444763183594}}]

>>> data = gs.parse_command("v.db.select", map='busroute1@PERMANENT', format='csv')
[{'cat': '1', 'ROUTE': '1'}, {'cat': '2', 'ROUTE': '1'}]
>>> pd.DataFrame(data)
  cat ROUTE
0   1     1
1   2     1

For csv, I use DictReader, which makes sense only when the csv has a header, but that seems like reasonable assumption, and then it works nicely with pandas.

Option delimiter does not fit well into it conceptually, so I deprecated it. Also, it's now taken into account only when parameter parse is not specified, it assumed that anyway.

@petrasovaa petrasovaa added this to the 8.4.0 milestone May 3, 2024
@petrasovaa petrasovaa requested a review from wenzeslaus May 3, 2024 19:39
@petrasovaa petrasovaa self-assigned this May 3, 2024
@github-actions github-actions bot added Python Related code is in Python libraries labels May 3, 2024
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great update overall. The heuristic should work well the format parameter which in combination with the parse_command function will almost always refer to output text format.

A short change for what it accomplishes!

Thanks for changing to csv.DictReader, the purpose here is convenience, so DictReader seems more appropriate than plain reader.

I agree with converting the generator to a list. json.loads also returns plain objects, so it is consistent with that.

>>> type(json.loads("""{"a": 1}"""))
<class 'dict'>
>>> type(json.loads("""[1,2]"""))
<class 'list'>

The delimiter change is a good clean up.

Please, add tests.

@github-actions github-actions bot added the tests Related to Test Suite label May 6, 2024
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tests! They nicely show how powerful this is.

@petrasovaa petrasovaa requested a review from wenzeslaus May 7, 2024 18:49
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Looking forward to using this. It will work great together with the JSON GSoC project.

@wenzeslaus wenzeslaus changed the title pythonlib/parse_command: automatically parse json and csv output grass.script: Automatically parse JSON and CSV output in parse_command May 8, 2024
@wenzeslaus wenzeslaus changed the title grass.script: Automatically parse JSON and CSV output in parse_command grass.script: Automatically parse JSON and CSV in parse_command May 8, 2024
@petrasovaa petrasovaa merged commit 79cf5fe into OSGeo:main May 8, 2024
@petrasovaa petrasovaa deleted the improve-parse-command branch May 8, 2024 01:46
HuidaeCho pushed a commit to HuidaeCho/grass that referenced this pull request May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

libraries Python Related code is in Python tests Related to Test Suite

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants