Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
26174d3
First commit -- began writing the FastBasic class
mdmueller Jun 19, 2014
8649b77
Fixed import problems, wrote skeleton of reading code
mdmueller Jun 19, 2014
d67d35b
Began work on Cython code, added file as an extension in setup_package
mdmueller Jun 19, 2014
848516a
Started work on C code set framework for tokenizer_t
mdmueller Jun 20, 2014
41ab8a3
Fixed indentation problems, started an error code system, began writi…
mdmueller Jun 20, 2014
1395ef4
Initial implementation of main tokenizing loop, got very basic test c…
mdmueller Jun 22, 2014
5f1f0bc
Added a couple subclasses of FastBasic, got header working using the …
mdmueller Jun 22, 2014
6854feb
Implemented data_start/header_start, created an enum for error codes
mdmueller Jun 22, 2014
f89760e
Initial type conversion and comment handling
mdmueller Jun 23, 2014
7ad6954
Did some work on handling too few/too many columns in data line
mdmueller Jun 23, 2014
0972509
Removed handling of non-line comments
mdmueller Jun 23, 2014
b07fd08
Removed row_positions and made the output_cols strings null-delimited
mdmueller Jun 23, 2014
e972b6f
Fixed problems with calling read() multiple times
mdmueller Jun 23, 2014
97f882f
Fixed errors with last line and too many data values in a row, worked…
mdmueller Jun 24, 2014
b303e40
Added FastBasic to the infrastructure of ascii.read
mdmueller Jun 24, 2014
da9ab25
Fixed parameter problem and started writing tests
mdmueller Jun 24, 2014
a9b78e3
Made tokenizer trip whitespace at the beginning of fields, fixed conv…
mdmueller Jun 26, 2014
5bdad7b
Implemented include_names/exclude_names handling and wrote tests
mdmueller Jun 26, 2014
5f13f74
Began work on handling quotes in tokenizer and fixed indentation
mdmueller Jun 27, 2014
17d63dc
Implemented data_end parameter and wrote more tests
mdmueller Jun 27, 2014
92ddfaa
Wrote implementation of fill_values, included a couple of tests
mdmueller Jun 27, 2014
b536a06
Made FastBasic, FastCsv use their respective procedures for dealing w…
mdmueller Jun 28, 2014
58bab50
Fixed memory reallocation problem for large files
mdmueller Jun 28, 2014
dab16de
Made type conversion more efficient by calling C functions
mdmueller Jun 28, 2014
e7bfeed
Moved field iteration code to C -- conversion is now much faster beca…
mdmueller Jun 30, 2014
cd0c7e2
Fixed problem with inconsistent row length handling
mdmueller Jun 30, 2014
594b5dd
Fixed Travis issues related to docs/Python 3
mdmueller Jun 30, 2014
3be6c47
Switched tabs to 4 spaces and made fast readers take precedence whene…
mdmueller Jul 1, 2014
5a12424
Fixed miscellaneous issues (such as w/conversion) and indentation
mdmueller Jul 1, 2014
db331e8
Added comments in Cython/C, replicated overflow behavior for numpy<1.6
mdmueller Jul 2, 2014
385b4e1
Implemented the parameter use_fast_reader and added some fixtures to …
mdmueller Jul 2, 2014
1ad985b
Added whitespace handling at the end of lines and parameters to contr…
mdmueller Jul 3, 2014
bb9bbf2
C parser now strips trailing whitespace in fields
mdmueller Jul 3, 2014
8ffca85
Implemented FastCommentedHeader class and handled a data_start issue
mdmueller Jul 3, 2014
3e2ee34
Implemented fast RDB reader
mdmueller Jul 4, 2014
1ef6825
Fixed up a couple //TODO issues
mdmueller Jul 4, 2014
3fa6c74
Began work on Cython table writing
mdmueller Jul 5, 2014
909629d
More work on the Cython writer including tests
mdmueller Jul 6, 2014
61760f1
Refactored fast writer, bypassed iter_str_vals for speed gains
mdmueller Jul 7, 2014
55a61df
Made writer handle Python lists rather than ndarrays to avoid repeate…
mdmueller Jul 7, 2014
80bcda0
More efficient detection of string values to strip using a cached lis…
mdmueller Jul 7, 2014
ed8c181
Fixed reading issue with whitespace before comment char, made tokeniz…
mdmueller Jul 8, 2014
16d8dc9
Fixed a few more issues, e.g. empty tables and relative data_start fo…
mdmueller Jul 8, 2014
6873138
Fixed typo in conversion code, wrote documentation
mdmueller Jul 9, 2014
629c1d8
Made fast writer the default, fixed dtype issue, cosmetic changes
mdmueller Jul 9, 2014
5f96565
Began writing multiprocessing code
mdmueller Jul 16, 2014
9c5f2b7
Made parser create separate tokenizers so process aren't sharing data
mdmueller Jul 16, 2014
75efd47
Began writing recombination code, fixed misc. issues
mdmueller Jul 17, 2014
e1c576a
Added reconversion for columns containing strings
mdmueller Jul 17, 2014
1aa098f
Fixed problems with raising exceptions in parsing processes
mdmueller Jul 19, 2014
9d41521
Added data_end handling
mdmueller Jul 20, 2014
f5543b4
Switched to fread()ing file chunks for each process
mdmueller Jul 26, 2014
78d9df1
Added file tests, fixed problems with file reading
mdmueller Jul 26, 2014
eac2169
Implemented parallel parameter, fixed tests and custom header handling
mdmueller Jul 27, 2014
72b35e6
Made convert_str use list instead of ndarray to fix unicode issues
mdmueller Jul 27, 2014
b6054bf
Implemented memory mapping
mdmueller Jul 27, 2014
9b0837d
Fixed line-by-line input in header and Windows line endings
mdmueller Jul 30, 2014
8db4819
Added parameter use_fast_converter to use xstrod
mdmueller Jul 31, 2014
7d7d98b
Switched to memory mapping wherever applicable
mdmueller Aug 1, 2014
018b5a9
Began writing Windows memory mapping code
mdmueller Aug 2, 2014
1bfe651
Fixed Windows issues with multiprocessing/memory mapping
mdmueller Aug 3, 2014
5ffacf0
Made parser share memmap char * pointer across processes
mdmueller Aug 5, 2014
35ed7c4
Added use_fast_reader/writer=None to test fixtures
mdmueller Aug 5, 2014
9b17764
Made converters avoid char* to bytes conversion for fill_values check…
mdmueller Aug 5, 2014
22819fd
Switched to only True/False for use_fast_reader/writer
mdmueller Aug 5, 2014
2fbbd4b
Reverted addition of use_fast_reader/writer=None to test fixtures
mdmueller Aug 5, 2014
d4cbc70
Fixed speed issue with output formatting in FastWriter
mdmueller Aug 6, 2014
d9fcd43
Made carriage return newline checking in tokenizer more efficient
mdmueller Aug 7, 2014
8670321
Moved include/exclude column handling to conversion step, restructure…
mdmueller Aug 7, 2014
dea990e
Restructured tests and fixed a str/unicode issue
mdmueller Aug 8, 2014
2a39790
Made string conversion more efficient
mdmueller Aug 8, 2014
91f5ec4
Added a separate section in the io.ascii docs for fast reading/writing
mdmueller Aug 11, 2014
0306a75
Mentioned possibility of parallel failure for quoted values in docs
mdmueller Aug 11, 2014
8c4a866
Added new subsection for parallel/fast conversion options
mdmueller Aug 11, 2014
c4e37f0
Improved fast converter, potential error no longer scales with the pr…
mdmueller Aug 12, 2014
c4906cd
Fixed line ending issue, added back reading tests, included new param…
mdmueller Aug 12, 2014
ab3ed48
Made xstrtod cache all possible powers of ten in memory
mdmueller Aug 12, 2014
255718b
Made splitlines method of FileString more efficient
mdmueller Aug 13, 2014
cadff1c
Made C string->Python string conversion work with Python 3
mdmueller Aug 13, 2014
13fc367
Fixed a couple issues with xstrtod()
mdmueller Aug 14, 2014
f195c07
Made splitlines() method return string, not bytes
mdmueller Aug 15, 2014
4c04555
Switched memory mapping over to Python with the mmap module
mdmueller Aug 15, 2014
b6eee1f
Separated buffer calls with mmap between PY2 and PY3
mdmueller Aug 15, 2014
f3651df
Fixed pep8 issues
mdmueller Aug 18, 2014
18b81a6
Rewrote splitlines() method in C for faster array access
mdmueller Aug 18, 2014
2ca26ab
Made splitlines() return string rather than bytes and updated noteboo…
mdmueller Aug 19, 2014
4097a1d
Replaced fast parsing options with fast_reader, fixed tests
mdmueller Aug 22, 2014
4b2e96b
Updated docs to reflect parameter change and added fast_writer
mdmueller Aug 22, 2014
c57fd8f
Made parsing fail given invalid read options and updated docs
mdmueller Aug 23, 2014
229efe0
Added changelog entry
mdmueller Aug 27, 2014
f29bdd2
Turned tokenizing macros into functions and fixed Travis issue
mdmueller Sep 8, 2014
6ff0349
Added calls to super() in fast reading __init__ methods
mdmueller Sep 9, 2014
81e0742
Mentioned that fast suboptions are false by default in a couple other…
mdmueller Sep 12, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ New Features
entirely by overriding inherited class attributes instead of setting
instance attributes in the Reader ``__init__`` method. [#2812]

- There is now a faster C/Cython engine available for reading and writing
simple ASCII formats like CSV. Both are enabled by default, and fast
reading will fall back on an ordinary reader in case of a parsing
failure. Their behavior can be altered with the parameter ``fast_reader``
in ``read`` and ``fast_writer`` in ``write``. [#2716]

- ``astropy.io.fits``

- ``astropy.io.misc``
Expand Down
7 changes: 7 additions & 0 deletions astropy/io/ascii/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from __future__ import absolute_import, division, print_function

from .core import (InconsistentTableError,
ParameterError,
NoType, StrType, NumType, FloatType, IntType, AllType,
Column,
BaseInputter, ContinuationLinesInputter,
Expand All @@ -23,6 +24,12 @@
Tab,
NoHeader,
CommentedHeader)
from .fastbasic import (FastBasic,
FastCsv,
FastTab,
FastNoHeader,
FastCommentedHeader,
FastRdb)
from .cds import Cds
from .latex import Latex, AASTex, latexdicts
from .html import HTML
Expand Down
40 changes: 39 additions & 1 deletion astropy/io/ascii/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@
# Global dictionary mapping format arg to the corresponding Reader class
FORMAT_CLASSES = {}

# Similar dictionary for fast readers
FAST_CLASSES = {}

class MaskedConstant(numpy.ma.core.MaskedConstant):
"""A trivial extension of numpy.ma.masked

Expand Down Expand Up @@ -65,6 +68,20 @@ class OptionalTableImportError(ImportError):
an ImportError.
"""

class ParameterError(NotImplementedError):
"""
Indicates that a reader cannot handle a passed parameter.

The C-based fast readers in ``io.ascii`` raise an instance of
this error class upon encountering a parameter that the
C engine cannot handle.
"""

class FastOptionsError(NotImplementedError):
"""
Indicates that one of the specified options for fast
reading is invalid.
"""

class NoType(object):
"""
Expand Down Expand Up @@ -718,6 +735,10 @@ def __init__(cls, name, bases, dct):
if format is None:
return

fast = dct.get('_fast')
if fast is not None:
FAST_CLASSES[format] = cls

FORMAT_CLASSES[format] = cls

io_formats = ['ascii.' + format] + dct.get('_io_registry_format_aliases', [])
Expand Down Expand Up @@ -1021,6 +1042,14 @@ def _get_reader(Reader, Inputter=None, Outputter=None, **kwargs):
because it depends only on the "core" module.
"""

from .fastbasic import FastBasic
if issubclass(Reader, FastBasic): # Fast readers handle args separately
if Inputter is not None:
kwargs['Inputter'] = Inputter
return Reader(**kwargs)

if 'fast_reader' in kwargs:
del kwargs['fast_reader'] # ignore fast_reader parameter for slow readers
reader_kwargs = dict([k, v] for k, v in kwargs.items() if k not in extra_reader_pars)
reader = Reader(**reader_kwargs)

Expand Down Expand Up @@ -1097,11 +1126,20 @@ def _get_reader(Reader, Inputter=None, Outputter=None, **kwargs):
'fill_exclude_names')


def _get_writer(Writer, **kwargs):
def _get_writer(Writer, fast_writer, **kwargs):
"""Initialize a table writer allowing for common customizations. This
routine is for internal (package) use only and is useful because it depends
only on the "core" module. """

from .fastbasic import FastBasic

if issubclass(Writer, FastBasic): # Fast writers handle args separately
return Writer(**kwargs)
elif fast_writer and 'fast_{0}'.format(Writer._format_name) in FAST_CLASSES:
# Switch to fast writer
kwargs['fast_writer'] = fast_writer
return FAST_CLASSES['fast_{0}'.format(Writer._format_name)](**kwargs)

writer_kwargs = dict([k, v] for k, v in kwargs.items() if k not in extra_writer_pars)
writer = Writer(**writer_kwargs)

Expand Down
Loading