Skip to content

filter_file uses "surrogateescape" error handling #12765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tgamblin merged 4 commits into from
Oct 15, 2019
Merged

filter_file uses "surrogateescape" error handling #12765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0cd57d7
filter_file: added unit test to check for non ASCII encodings
alalazo Sep 8, 2019
58a0b7d
filter_file: use "surrogateescape" error handling to ignore unknown c…
alalazo Sep 8, 2019
aeae74f
filter_file: permit to stop filtering if a marker is found
alalazo Oct 14, 2019
852e811
Added unit test with a mixed text + binary file
alalazo Oct 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 35 additions & 3 deletions lib/spack/llnl/util/filesystem.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
import collections
import errno
import hashlib
import fileinput
import glob
import grp
import itertools
Expand Down Expand Up @@ -123,10 +122,15 @@ def filter_file(regex, repl, *filenames, **kwargs):
backup (bool): Make backup file(s) suffixed with ``~``. Default is True
ignore_absent (bool): Ignore any files that don't exist.
Default is False
stop_at (str): Marker used to stop scanning the file further. If a text
line matches this marker filtering is stopped and the rest of the
file is copied verbatim. Default is to filter until the end of the
file.
"""
string = kwargs.get('string', False)
backup = kwargs.get('backup', True)
ignore_absent = kwargs.get('ignore_absent', False)
stop_at = kwargs.get('stop_at', None)

# Allow strings to use \1, \2, etc. for replacement, like sed
if not callable(repl):
Expand Down Expand Up @@ -159,8 +163,36 @@ def groupid_to_group(x):
shutil.copy(filename, backup_filename)

try:
for line in fileinput.input(filename, inplace=True):
print(re.sub(regex, repl, line.rstrip('\n')))
extra_kwargs = {}
if sys.version_info > (3, 0):
extra_kwargs = {'errors': 'surrogateescape'}
Comment on lines +166 to +168
Copy link
Copy Markdown
Member Author

@alalazo alalazo Oct 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lee218llnl You can try to do something like the above also in #13479 to avoid the error with Python 2.6:

extra_kwargs = {}
if sys.version_info > (3, 0):
    extra_kwargs = {'encoding': 'utf-8', 'errors': 'surrogateescape'}


# Open as a text file and filter until the end of the file is
# reached or we found a marker in the line if it was specified
with open(backup_filename, mode='r', **extra_kwargs) as input_file:
with open(filename, mode='w', **extra_kwargs) as output_file:
# Using iter and readline is a workaround needed not to
# disable input_file.tell(), which will happen if we call
# input_file.next() implicitly via the for loop
for line in iter(input_file.readline, ''):
if stop_at is not None:
current_position = input_file.tell()
if stop_at == line.strip():
output_file.write(line)
break
filtered_line = re.sub(regex, repl, line)
output_file.write(filtered_line)
else:
current_position = None

# If we stopped filtering at some point, reopen the file in
# binary mode and copy verbatim the remaining part
if current_position and stop_at:
with open(backup_filename, mode='rb') as input_file:
input_file.seek(current_position)
with open(filename, mode='ab') as output_file:
output_file.writelines(input_file.readlines())

except BaseException:
# clean up the original file on failure.
shutil.move(backup_filename, filename)
Expand Down
Binary file added lib/spack/spack/test/data/filter_file/selfextract.bsx
View file
Open in desktop
Binary file not shown.
Loading