Repository navigation
Commit 8d79923
committed
Stop XML end events from repeating the text of nested elements
- In get_xml_iterator, the end event of an element that contains other
elements now has empty text. Before, it repeated the text of the last
element inside it. This applies to both the C and pure-Python parsers.
This dramatically reduces memory use for reading large XML files.
- For a VOTable BINARY or BINARY2 table, the end events of BINARY2, DATA
and TABLE elements no longer each hold a full copy of the base64
data stream.
- The C parser frees a text buffer that has grown past 1 MB once its
text is handed over, instead of keeping it until parsing ends.
- VOTable reading decodes the stream with binascii.a2b_base64 directly,
which skips a full copy of the stream text. Base64 text is always
ASCII, so valid files decode exactly as before. Only the error for a
malformed stream with a non-ASCII character changes: it is now
ValueError ("string argument should contain only ASCII characters")
instead of UnicodeEncodeError, which is a ValueError subclass.
- Reading the 2.5 GB file from issue astropy#8946 now needs about 6.8 GB of
memory at its peak, down from about 17 GB.
- Update the one test that expected the repeated text, which was just
a coverage test pinning current behavior from PR astropy#19922.
- Add tests for both parsers and for elements holding more than 1 MB
of text.1 parent 006f496 commit 8d79923
7 files changed
Lines changed: 105 additions & 3 deletions
File tree
- astropy
- io/votable
- utils/xml
- src
- tests
- docs/changes
- io.votable
- utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
3126 | 3127 | | |
3127 | 3128 | | |
3128 | 3129 | | |
3129 | | - | |
| 3130 | + | |
| 3131 | + | |
| 3132 | + | |
3130 | 3133 | | |
3131 | 3134 | | |
3132 | 3135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
272 | 272 | | |
273 | 273 | | |
274 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
275 | 307 | | |
276 | 308 | | |
277 | 309 | | |
| |||
500 | 532 | | |
501 | 533 | | |
502 | 534 | | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
503 | 539 | | |
504 | 540 | | |
505 | 541 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
9 | 11 | | |
10 | 12 | | |
11 | 13 | | |
| |||
130 | 132 | | |
131 | 133 | | |
132 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
0 commit comments