Repository navigation
Commit 1ad7032
authored
Grow VOTable result arrays in place instead of copying them (#20743)
* Grow VOTable result arrays in place instead of copying them
- While reading a TABLEDATA, BINARY or BINARY2 table, keep the values
and the missing-value flags in plain numpy arrays that can be resized
in place, and make the masked array once at the end. Before, the whole
table was copied each time the arrays filled up, and once more at the
end to trim them.
- Reading the 2.5 GB Gaia table from gh-8946 peaks at 5.1 GB instead of
6.8 GB, and takes 113s instead of 130s. That's a 25% reduction in
memory and a 15% speed increase.
- Result is the same: same type, dtype, mask, fill value and values.
- Add a test reading a 1000-row table in each format, which grows the
arrays past the first 512 rows and then trims them.
* Keep VOTable read arrays in a small builder class
- Replace the two helper functions with _MaskedArrayBuilder, which owns
the data and mask arrays, resizes them together and never hands them
out, so no view of them can outlive a refcheck=False resize. The
parsers set rows through set_row and set_rows.
- Test reading a variable-length column longer than the first
allocation, and tables whose nrows attribute is too small, correct
or too large, in each format.1 parent 531370c commit 1ad7032
3 files changed
Lines changed: 178 additions & 30 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
708 | 709 | | |
709 | 710 | | |
710 | 711 | | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
| 129 | + | |
130 | 130 | | |
131 | | - | |
132 | | - | |
133 | | - | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
134 | 140 | | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | 141 | | |
139 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
140 | 207 | | |
141 | 208 | | |
142 | 209 | | |
| |||
2988 | 3055 | | |
2989 | 3056 | | |
2990 | 3057 | | |
2991 | | - | |
2992 | | - | |
2993 | | - | |
| 3058 | + | |
2994 | 3059 | | |
2995 | 3060 | | |
2996 | 3061 | | |
2997 | 3062 | | |
2998 | 3063 | | |
2999 | 3064 | | |
3000 | 3065 | | |
3001 | | - | |
| 3066 | + | |
3002 | 3067 | | |
3003 | 3068 | | |
3004 | 3069 | | |
| |||
3076 | 3141 | | |
3077 | 3142 | | |
3078 | 3143 | | |
3079 | | - | |
3080 | | - | |
3081 | | - | |
3082 | | - | |
| 3144 | + | |
| 3145 | + | |
| 3146 | + | |
3083 | 3147 | | |
3084 | 3148 | | |
3085 | 3149 | | |
| |||
3091 | 3155 | | |
3092 | 3156 | | |
3093 | 3157 | | |
3094 | | - | |
3095 | | - | |
3096 | | - | |
3097 | | - | |
| 3158 | + | |
| 3159 | + | |
3098 | 3160 | | |
3099 | 3161 | | |
3100 | 3162 | | |
3101 | 3163 | | |
3102 | 3164 | | |
3103 | 3165 | | |
3104 | | - | |
| 3166 | + | |
3105 | 3167 | | |
3106 | 3168 | | |
3107 | 3169 | | |
| |||
3171 | 3233 | | |
3172 | 3234 | | |
3173 | 3235 | | |
3174 | | - | |
3175 | | - | |
3176 | | - | |
| 3236 | + | |
3177 | 3237 | | |
3178 | 3238 | | |
3179 | 3239 | | |
3180 | 3240 | | |
3181 | 3241 | | |
3182 | 3242 | | |
3183 | | - | |
| 3243 | + | |
3184 | 3244 | | |
3185 | 3245 | | |
3186 | 3246 | | |
3187 | 3247 | | |
3188 | | - | |
| 3248 | + | |
3189 | 3249 | | |
3190 | 3250 | | |
3191 | 3251 | | |
| |||
3230 | 3290 | | |
3231 | 3291 | | |
3232 | 3292 | | |
3233 | | - | |
3234 | | - | |
| 3293 | + | |
3235 | 3294 | | |
3236 | 3295 | | |
3237 | | - | |
| 3296 | + | |
| 3297 | + | |
3238 | 3298 | | |
3239 | 3299 | | |
3240 | 3300 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
0 commit comments