class CSV
This class provides a complete interface to CSV files and data. It offers tools to enable you to read and write to and from Strings or IO objects, as needed.
Reading¶ ↑
From a File¶ ↑
A Line at a Time¶ ↑
CSV.foreach("path/to/file.csv") do |row| # use row here... end
All at Once¶ ↑
arr_of_arrs = CSV.read("path/to/file.csv")
From a String¶ ↑
A Line at a Time¶ ↑
CSV.parse("CSV,data,String") do |row| # use row here... end
All at Once¶ ↑
arr_of_arrs = CSV.parse("CSV,data,String")
Writing¶ ↑
To a File¶ ↑
CSV.open("path/to/file.csv", "wb") do |csv| csv << ["row", "of", "CSV", "data"] csv << ["another", "row"] # ... end
To a String¶ ↑
csv_string = CSV.generate do |csv| csv << ["row", "of", "CSV", "data"] csv << ["another", "row"] # ... end
Convert a Single Line¶ ↑
csv_string = ["CSV", "data"].to_csv # to CSV csv_array = "CSV,String".parse_csv # from CSV
Shortcut Interface¶ ↑
CSV { |csv_out| csv_out << %w{my data here} } # to $stdout CSV(csv = "") { |csv_str| csv_str << %w{my data here} } # to a String CSV($stderr) { |csv_err| csv_err << %w{my data here} } # to $stderr CSV($stdin) { |csv_in| csv_in.each { |row| p row } } # from $stdin
Advanced Usage¶ ↑
Wrap an IO Object¶ ↑
csv = CSV.new(io, options) # ... read (with gets() or each()) from and write (with <<) to csv here ...
CSV and Character Encodings (M17n or Multilingualization)¶ ↑
This new CSV parser is m17n savvy. The parser works in the Encoding of the IO or String object being read from or written to. Your data is never transcoded (unless you ask Ruby to transcode it for you) and will literally be parsed in the Encoding it is in. Thus CSV will return Arrays or Rows of Strings in the Encoding of your data. This is accomplished by transcoding the parser itself into your Encoding.
Some transcoding must take place, of course, to accomplish this
multiencoding support. For example, :col_sep,
:row_sep, and :quote_char must be transcoded to
match your data. Hopefully this makes the entire process feel transparent,
since CSV's defaults should just magically work for your data.
However, you can set these values manually in the target Encoding to avoid the translation.
It's also important to note that while all of CSV's core parser is now Encoding agnostic, some features are not. For example, the built-in converters will try to transcode data to UTF-8 before making conversions. Again, you can provide custom converters that are aware of your Encodings to avoid this translation. It's just too hard for me to support native conversions in all of Ruby's Encodings.
Anyway, the practical side of this is simple: make sure IO and String objects passed into CSV have the proper Encoding set and everything should just work. CSV methods that allow you to open IO objects (CSV::foreach(), ::open, ::read, and ::readlines) do allow you to specify the Encoding.
One minor exception comes when generating CSV into a String with an Encoding that is not ASCII compatible. There's no existing data for CSV to use to prepare itself and thus you will probably need to manually specify the desired Encoding for most of those cases. It will try to guess using the fields in a row of output though, when using ::generate_line or Array#to_csv().
I try to point out any other Encoding issues in the documentation of methods as they come up.
This has been tested to the best of my ability with all non-“dummy” Encodings Ruby ships with. However, it is brave new code and may have some bugs. Please feel free to report any issues you find with it.
Constants
- ConverterEncoding
The encoding used by all converters.
- Converters
This Hash holds the built-in converters of CSV that can be accessed by name. You can select Converters with #convert or through the
optionsHash passed to ::new.:integer-
Converts any field Integer() accepts.
:float-
Converts any field Float() accepts.
:numeric-
A combination of
:integerand:float. :date-
Converts any field Date.parse accepts.
:date_time-
Converts any field DateTime.parse accepts.
:all-
All built-in converters. A combination of
:date_timeand:numeric.
All built-in converters transcode field data to UTF-8 before attempting a conversion. If your data cannot be transcoded to UTF-8 the conversion will fail and the field will remain unchanged.
This Hash is intentionally left unfrozen and users should feel free to add values to it that can be accessed by all CSV objects.
To add a combo field, the value should be an Array of names. Combo fields can be nested with other combo fields.
- DEFAULT_OPTIONS
The options used when no overrides are given by calling code. They are:
:col_sep-
"," :row_sep-
:auto :quote_char-
'"' :field_size_limit-
nil :converters-
nil :unconverted_fields-
nil :headers-
false :return_headers-
false :header_converters-
nil :skip_blanks-
false :force_quotes-
false :skip_lines-
nil :liberal_parsing-
false
- DateMatcher
- DateTimeMatcher
A Regexp used to find and convert some common DateTime formats.
- FieldInfo
A FieldInfo Struct contains details about a field's position in the data source it was read from. CSV will pass this Struct to some blocks that make decisions based on field structure. See #convert_fields for an example.
index-
The zero-based index of the field in its row.
line-
The line of the data source this row is from.
header-
The header for the column, when available.
- HeaderConverters
This Hash holds the built-in header converters of CSV that can be accessed by name. You can select HeaderConverters with #header_convert or through the
optionsHash passed to ::new.:downcase-
Calls downcase() on the header String.
:symbol-
Leading/trailing spaces are dropped, string is downcased, remaining spaces are replaced with underscores, non-word characters are dropped, and finally to_sym() is called.
All built-in header converters transcode header data to UTF-8 before attempting a conversion. If your data cannot be transcoded to UTF-8 the conversion will fail and the header will remain unchanged.
This Hash is intentionally left unfrozen and users should feel free to add values to it that can be accessed by all CSV objects.
To add a combo field, the value should be an Array of names. Combo fields can be nested with other combo fields.
- VERSION
The version of the installed library.
Attributes
The encoded :col_sep used in parsing and writing. See ::new for details.
The limit for field size, if any. See ::new for details.
The line number of the last row read from this file. Fields with nested line-end characters will not affect this count.
The line number of the last row read from this file. Fields with nested line-end characters will not affect this count.
The encoded :quote_char used in parsing and writing. See ::new for details.
The encoded :row_sep used in parsing and writing. See ::new for details.
The regex marking a line as a comment. See ::new for details
Public Class Methods
This method is a convenience for building Unix-like filters for CSV data. Each row is yielded to the provided block
which can alter it as needed. After the block returns, the row is appended
to output altered or not.
The input and output arguments can be anything ::new accepts (generally String or IO objects). If not given, they default to
ARGF and $stdout.
The options parameter is also filtered down to ::new after some clever key parsing. Any
key beginning with :in_ or :input_ will have that
leading identifier stripped and will only be used in the
options Hash for the
input object. Keys starting with :out_ or
:output_ affect only output. All other keys are
assigned to both objects.
The :output_row_sep option defaults to
$INPUT_RECORD_SEPARATOR ($/).
# File lib/csv.rb, line 1098 def self.filter(input=nil, output=nil, **options) # parse options for input, output, or both in_options, out_options = Hash.new, {row_sep: $INPUT_RECORD_SEPARATOR} options.each do |key, value| case key.to_s when /\Ain(?:put)?_(.+)\Z/ in_options[$1.to_sym] = value when /\Aout(?:put)?_(.+)\Z/ out_options[$1.to_sym] = value else in_options[key] = value out_options[key] = value end end # build input and output wrappers input = new(input || ARGF, in_options) output = new(output || $stdout, out_options) # read, yield, write input.each do |row| yield row output << row end end
This method is intended as the primary interface for reading CSV files. You pass a path and any
options you wish to set for the read. Each row of file will
be passed to the provided block in turn.
The options parameter can be anything ::new understands. This method also
understands an additional :encoding parameter that you can use
to specify the Encoding of the data in the file
to be read. You must provide this unless your data is in Encoding.default_external.
CSV will use this to determine how to parse the
data. You may provide a second Encoding to
have the data transcoded as it is read. For example, encoding:
"UTF-32BE:UTF-8" would read UTF-32BE data from the file
but transcode it to UTF-8 before CSV parses it.
# File lib/csv.rb, line 1137 def self.foreach(path, **options, &block) return to_enum(__method__, path, options) unless block open(path, options) do |csv| csv.each(&block) end end
This method wraps a String you provide, or an empty default String, in a CSV object which is passed to the provided block. You can use the block to append CSV rows to the String and when the block exits, the final String will be returned.
Note that a passed String is modified by this method. Call dup() before passing if you need a new String.
The options parameter can be anything ::new understands. This method
understands an additional :encoding parameter when not passed
a String to set the base Encoding for the
output. CSV needs this hint if you plan to output
non-ASCII compatible data.
# File lib/csv.rb, line 1162 def self.generate(str=nil, **options) # add a default empty String, if none was given if str io = StringIO.new(str) io.seek(0, IO::SEEK_END) else encoding = options[:encoding] str = String.new str.force_encoding(encoding) if encoding end csv = new(str, options) # wrap yield csv # yield for appending csv.string # return final String end
This method is a shortcut for converting a single row (Array) into a CSV String.
The options parameter can be anything ::new understands. This method
understands an additional :encoding parameter to set the base
Encoding for the output. This method will try
to guess your Encoding from the first
non-nil field in row, if possible, but you may
need to use this parameter as a backup plan.
The :row_sep option defaults to
$INPUT_RECORD_SEPARATOR ($/) when calling this
method.
# File lib/csv.rb, line 1190 def self.generate_line(row, **options) options = {row_sep: $INPUT_RECORD_SEPARATOR}.merge(options) str = String.new if options[:encoding] str.force_encoding(options[:encoding]) elsif field = row.find { |f| not f.nil? } str.force_encoding(String(field).encoding) end (new(str, options) << row).string end
This method will return a CSV instance, just like ::new, but the instance will be cached and
returned for all future calls to this method for the same data
object (tested by Object#object_id) with the same
options.
If a block is given, the instance is passed to the block and the return value becomes the return value of the block.
# File lib/csv.rb, line 1058 def self.instance(data = $stdout, **options) # create a _signature_ for this method call, data object and options sig = [data.object_id] + options.values_at(*DEFAULT_OPTIONS.keys.sort_by { |sym| sym.to_s }) # fetch or create the instance for this signature @@instances ||= Hash.new instance = (@@instances[sig] ||= new(data, options)) if block_given? yield instance # run block, if given, returning result else instance # or return the instance end end
This constructor will wrap either a String or IO
object passed in data for reading and/or writing. In addition
to the CSV instance methods, several IO methods are delegated. (See ::open for a complete list.) If you pass
a String for data, you can later retrieve it (after writing to
it, for example) with CSV.string().
Note that a wrapped String will be positioned at the beginning (for reading). If you want it at the end (for writing), use ::generate. If you want any other positioning, pass a preset StringIO object instead.
You may set any reading and/or writing preferences in the
options Hash. Available options are:
:col_sep-
The String placed between each field. This String will be transcoded into the data's Encoding before parsing.
:row_sep-
The String appended to the end of each row. This can be set to the special
:autosetting, which requests that CSV automatically discover this from the data. Auto-discovery reads ahead in the data looking for the next"\r\n","\n", or"\r"sequence. A sequence will be selected even if it occurs in a quoted field, assuming that you would have the same line endings there. If none of those sequences is found,dataisARGF,STDIN,STDOUT, orSTDERR, or the stream is only available for output, the default$INPUT_RECORD_SEPARATOR($/) is used. Obviously, discovery takes a little time. Set manually if speed is important. Also note that IO objects should be opened in binary mode on Windows if this feature will be used as the line-ending translation can cause problems with resetting the document position to where it was before the read ahead. This String will be transcoded into the data's Encoding before parsing. :quote_char-
The character used to quote fields. This has to be a single character String. This is useful for application that incorrectly use
'as the quote character instead of the correct". CSV will always consider a double sequence of this character to be an escaped quote. This String will be transcoded into the data's Encoding before parsing. :field_size_limit-
This is a maximum size CSV will read ahead looking for the closing quote for a field. (In truth, it reads to the first line ending beyond this size.) If a quote cannot be found within the limit CSV will raise a MalformedCSVError, assuming the data is faulty. You can use this limit to prevent what are effectively DoS attacks on the parser. However, this limit can cause a legitimate parse to fail and thus is set to
nil, or off, by default. :converters-
An Array of names from the Converters Hash and/or lambdas that handle custom conversion. A single converter doesn't have to be in an Array. All built-in converters try to transcode fields to UTF-8 before converting. The conversion will fail if the data cannot be transcoded, leaving the field unchanged.
:unconverted_fields-
If set to
true, an unconverted_fields() method will be added to all returned rows (Array or CSV::Row) that will return the fields as they were before conversion. Note that:headerssupplied by Array or String were not fields of the document and thus will have an empty Array attached. :headers-
If set to
:first_rowortrue, the initial row of the CSV file will be treated as a row of headers. If set to an Array, the contents will be used as the headers. If set to a String, the String is run through a call of ::parse_line with the same:col_sep,:row_sep, and:quote_charas this instance to produce an Array of headers. This setting causes #shift to return rows as CSV::Row objects instead of Arrays and #read to return CSV::Table objects instead of an Array of Arrays. :return_headers-
When
false, header rows are silently swallowed. If set totrue, header rows are returned in a CSV::Row object with identical headers and fields (save that the fields do not go through the converters). :write_headers-
When
trueand:headersis set, a header row will be added to the output. :header_converters-
Identical in functionality to
:converterssave that the conversions are only made to header rows. All built-in converters try to transcode headers to UTF-8 before converting. The conversion will fail if the data cannot be transcoded, leaving the header unchanged. :skip_blanks-
When set to a
truevalue, CSV will skip over any empty rows. Note that this setting will not skip rows that contain column separators, even if the rows contain no actual data. If you want to skip rows that contain separators but no content, consider using:skip_lines, or inspecting fields.compact.empty? on each row. :force_quotes-
When set to a
truevalue, CSV will quote all CSV fields it creates. :skip_lines-
When set to an object responding to
match, every line matching it is considered a comment and ignored during parsing. When set to a String, it is first converted to a Regexp. When set tonilno line is considered a comment. If the passed object does not respond tomatch,ArgumentErroris thrown. :liberal_parsing-
When set to a
truevalue, CSV will attempt to parse input not conformant with RFC 4180, such as double quotes in unquoted fields.
See CSV::DEFAULT_OPTIONS for the default settings.
Options cannot be overridden in the instance methods for performance reasons, so be sure to set what you want here.
# File lib/csv.rb, line 1517 def initialize(data, col_sep: ",", row_sep: :auto, quote_char: '"', field_size_limit: nil, converters: nil, unconverted_fields: nil, headers: false, return_headers: false, write_headers: nil, header_converters: nil, skip_blanks: false, force_quotes: false, skip_lines: nil, liberal_parsing: false, internal_encoding: nil, external_encoding: nil, encoding: nil) raise ArgumentError.new("Cannot parse nil as CSV") if data.nil? # create the IO object we will read from @io = data.is_a?(String) ? StringIO.new(data) : data # honor the IO encoding if we can, otherwise default to ASCII-8BIT internal_encoding = Encoding.find(internal_encoding) if internal_encoding external_encoding = Encoding.find(external_encoding) if external_encoding if encoding encoding, = encoding.split(":", 2) if encoding.is_a?(String) encoding = Encoding.find(encoding) end @encoding = raw_encoding(nil) || internal_encoding || encoding || Encoding.default_internal || Encoding.default_external # # prepare for building safe regular expressions in the target encoding, # if we can transcode the needed characters # @re_esc = "\\".encode(@encoding).freeze rescue "" @re_chars = /#{%"[-\\]\\[\\.^$?*+{}()|# \r\n\t\f\v]".encode(@encoding)}/ @unconverted_fields = unconverted_fields # Stores header row settings and loads header converters, if needed. @use_headers = headers @return_headers = return_headers @write_headers = write_headers # headers must be delayed until shift(), in case they need a row of content @headers = nil init_separators(col_sep, row_sep, quote_char, force_quotes) init_parsers(skip_blanks, field_size_limit, liberal_parsing) init_converters(converters, :@converters, :convert) init_converters(header_converters, :@header_converters, :header_convert) init_comments(skip_lines) @force_encoding = !!encoding # track our own lineno since IO gets confused about line-ends is CSV fields @lineno = 0 # make sure headers have been assigned if header_row? and [Array, String].include? @use_headers.class and @write_headers parse_headers # won't read data for Array or String self << @headers end end
This method opens an IO object, and wraps that with CSV. This is intended as the primary interface for writing a CSV file.
You must pass a filename and may optionally add a
mode for Ruby's open(). You may also pass an optional Hash containing any options ::new understands as the final argument.
This method works like Ruby's open() call, in that it will pass a CSV object to a provided block and close it when the block terminates, or it will return the CSV object when no block is provided. (Note: This is different from the Ruby 1.8 CSV library which passed rows to the block. Use ::foreach for that behavior.)
You must provide a mode with an embedded Encoding designator unless your data is in Encoding.default_external.
CSV will check the Encoding of the underlying IO object (set by the mode you pass) to
determine how to parse the data. You may provide a second Encoding to have the data transcoded as it is read
just as you can with a normal call to IO.open. For example,
"rb:UTF-32BE:UTF-8" would read UTF-32BE data from
the file but transcode it to UTF-8 before CSV parses
it.
An opened CSV object will delegate to many IO methods for convenience. You may call:
-
binmode()
-
binmode?()
-
close()
-
close_read()
-
close_write()
-
closed?()
-
eof()
-
eof?()
-
external_encoding()
-
fcntl()
-
fileno()
-
flock()
-
flush()
-
fsync()
-
internal_encoding()
-
ioctl()
-
isatty()
-
path()
-
pid()
-
pos()
-
pos=()
-
reopen()
-
seek()
-
stat()
-
sync()
-
sync=()
-
tell()
-
to_i()
-
to_io()
-
truncate()
-
tty?()
# File lib/csv.rb, line 1264 def self.open(filename, mode="r", **options) # wrap a File opened with the remaining +args+ with no newline # decorator file_opts = {universal_newline: false}.merge(options) begin f = File.open(filename, mode, file_opts) rescue ArgumentError => e raise unless /needs binmode/ =~ e.message and mode == "r" mode = "rb" file_opts = {encoding: Encoding.default_external}.merge(file_opts) retry end begin csv = new(f, options) rescue Exception f.close raise end # handle blocks like Ruby's open(), not like the CSV library if block_given? begin yield csv ensure csv.close end else csv end end
This method can be used to easily parse CSV out of a
String. You may either provide a block which will be called
with each row of the String in turn, or just use the returned Array of
Arrays (when no block is given).
You pass your str to read from, and an optional
options containing anything ::new understands.
# File lib/csv.rb, line 1308 def self.parse(*args, &block) csv = new(*args) if block.nil? # slurp contents, if no block is given begin csv.read ensure csv.close end else # or pass each row to a provided block csv.each(&block) end end
This method is a shortcut for converting a single line of a CSV String into an Array. Note that if
line contains multiple rows, anything beyond the first row is
ignored.
The options parameter can be anything ::new understands.
# File lib/csv.rb, line 1328 def self.parse_line(line, **options) new(line, options).shift end
Use to slurp a CSV file into an Array of Arrays.
Pass the path to the file and any options ::new understands. This method also
understands an additional :encoding parameter that you can use
to specify the Encoding of the data in the file
to be read. You must provide this unless your data is in Encoding.default_external.
CSV will use this to determine how to parse the
data. You may provide a second Encoding to
have the data transcoded as it is read. For example, encoding:
"UTF-32BE:UTF-8" would read UTF-32BE data from the file
but transcode it to UTF-8 before CSV parses it.
# File lib/csv.rb, line 1343 def self.read(path, *options) open(path, *options) { |csv| csv.read } end
Alias for ::read.
# File lib/csv.rb, line 1348 def self.readlines(*args) read(*args) end
A shortcut for:
CSV.read( path, { headers: true, converters: :numeric, header_converters: :symbol }.merge(options) )
# File lib/csv.rb, line 1359 def self.table(path, **options) read( path, { headers: true, converters: :numeric, header_converters: :symbol }.merge(options) ) end
Public Instance Methods
The primary write method for wrapped Strings and IOs, row (an
Array or CSV::Row) is converted to CSV and appended to the data source. When a CSV::Row is passed, only the row's fields() are
appended to the output.
The data source must be open for writing.
# File lib/csv.rb, line 1680 def <<(row) # make sure headers have been assigned if header_row? and [Array, String].include? @use_headers.class and !@write_headers parse_headers # won't read data for Array or String end # handle CSV::Row objects and Hashes row = case row when self.class::Row then row.fields when Hash then @headers.map { |header| row[header] } else row end @headers = row if header_row? @lineno += 1 output = row.map(&@quote).join(@col_sep) + @row_sep # quote and separate if @io.is_a?(StringIO) and output.encoding != (encoding = raw_encoding) if @force_encoding output = output.encode(encoding) elsif (compatible_encoding = Encoding.compatible?(@io.string, output)) @io.set_encoding(compatible_encoding) @io.seek(0, IO::SEEK_END) end end @io << output self # for chaining end
You can use this method to install a CSV::Converters built-in, or provide a block that handles a custom conversion.
If you provide a block that takes one argument, it will be passed the field and is expected to return the converted value or the field itself. If your block takes two arguments, it will also be passed a CSV::FieldInfo Struct, containing details about the field. Again, the block should return a converted field or the field itself.
# File lib/csv.rb, line 1728 def convert(name = nil, &converter) add_converter(:@converters, self.class::Converters, name, &converter) end
Returns the current list of converters in effect. See ::new for details. Built-in converters will be returned by name, while others will be returned as is.
# File lib/csv.rb, line 1594 def converters @converters.map do |converter| name = Converters.rassoc(converter) name ? name.first : converter end end
Yields each row of the data source in turn.
Support for Enumerable.
The data source must be open for reading.
# File lib/csv.rb, line 1759 def each if block_given? while row = shift yield row end else to_enum end end
Returns true if all output fields are quoted. See ::new for details.
# File lib/csv.rb, line 1637 def force_quotes?() @force_quotes end
Identical to #convert, but for header rows.
Note that this method must be called before header rows are read to have any effect.
# File lib/csv.rb, line 1743 def header_convert(name = nil, &converter) add_converter( :@header_converters, self.class::HeaderConverters, name, &converter ) end
Returns the current list of converters in effect for headers. See ::new for details. Built-in converters will be returned by name, while others will be returned as is.
# File lib/csv.rb, line 1625 def header_converters @header_converters.map do |converter| name = HeaderConverters.rassoc(converter) name ? name.first : converter end end
Returns true if the next row read will be a header row.
# File lib/csv.rb, line 1785 def header_row? @use_headers and @headers.nil? end
Returns nil if headers will not be used, true if
they will but have not yet been read, or the actual headers after they have
been read. See ::new for details.
# File lib/csv.rb, line 1610 def headers @headers || true if @use_headers end
Returns a simplified description of the key CSV attributes in an ASCII compatible String.
# File lib/csv.rb, line 1959 def inspect str = ["<#", self.class.to_s, " io_type:"] # show type of wrapped IO if @io == $stdout then str << "$stdout" elsif @io == $stdin then str << "$stdin" elsif @io == $stderr then str << "$stderr" else str << @io.class.to_s end # show IO.path(), if available if @io.respond_to?(:path) and (p = @io.path) str << " io_path:" << p.inspect end # show encoding str << " encoding:" << @encoding.name # show other attributes %w[ lineno col_sep row_sep quote_char skip_blanks liberal_parsing ].each do |attr_name| if a = instance_variable_get("@#{attr_name}") str << " " << attr_name << ":" << a.inspect end end if @use_headers str << " headers:" << headers.inspect end str << ">" begin str.join('') rescue # any encoding error str.map do |s| e = Encoding::Converter.asciicompat_encoding(s.encoding) e ? s.encode(e) : s.force_encoding("ASCII-8BIT") end.join('') end end
Returns true if illegal input is handled. See ::new for details.
# File lib/csv.rb, line 1639 def liberal_parsing?() @liberal_parsing end
Slurps the remaining rows and returns an Array of Arrays.
The data source must be open for reading.
# File lib/csv.rb, line 1774 def read rows = to_a if @use_headers Table.new(rows) else rows end end
Returns true if headers will be returned as a row of results.
See ::new for details.
# File lib/csv.rb, line 1617 def return_headers?() @return_headers end
Rewinds the underlying IO object and resets CSV's lineno() counter.
# File lib/csv.rb, line 1664 def rewind @headers = nil @lineno = 0 @io.rewind end
The primary read method for wrapped Strings and IOs, a single row is pulled from the data source, parsed and returned as an Array of fields (if header rows are not used) or a CSV::Row (when header rows are used).
The data source must be open for reading.
# File lib/csv.rb, line 1796 def shift ######################################################################### ### This method is purposefully kept a bit long as simple conditional ### ### checks are faster than numerous (expensive) method calls. ### ######################################################################### # handle headers not based on document content if header_row? and @return_headers and [Array, String].include? @use_headers.class if @unconverted_fields return add_unconverted_fields(parse_headers, Array.new) else return parse_headers end end # # it can take multiple calls to <tt>@io.gets()</tt> to get a full line, # because of \r and/or \n characters embedded in quoted fields # in_extended_col = false csv = Array.new loop do # add another read to the line unless parse = @io.gets(@row_sep) return nil end if in_extended_col @line.concat(parse) else @line = parse.clone end parse.sub!(@parsers[:line_end], "") if csv.empty? # # I believe a blank line should be an <tt>Array.new</tt>, not Ruby 1.8 # CSV's <tt>[nil]</tt> # if parse.empty? @lineno += 1 if @skip_blanks next elsif @unconverted_fields return add_unconverted_fields(Array.new, Array.new) elsif @use_headers return self.class::Row.new(Array.new, Array.new) else return Array.new end end end next if @skip_lines and @skip_lines.match parse parts = parse.split(@col_sep, -1) if parts.empty? if in_extended_col csv[-1] << @col_sep # will be replaced with a @row_sep after the parts.each loop else csv << nil end end # This loop is the hot path of csv parsing. Some things may be non-dry # for a reason. Make sure to benchmark when refactoring. parts.each do |part| if in_extended_col # If we are continuing a previous column if part.end_with?(@quote_char) && part.count(@quote_char) % 2 != 0 # extended column ends csv.last << part[0..-2] if csv.last =~ @parsers[:stray_quote] raise MalformedCSVError, "Missing or stray quote in line #{lineno + 1}" end csv.last.gsub!(@double_quote_char, @quote_char) in_extended_col = false else csv.last << part << @col_sep end elsif part.start_with?(@quote_char) # If we are starting a new quoted column if part.count(@quote_char) % 2 != 0 # start an extended column csv << (part[1..-1] << @col_sep) in_extended_col = true elsif part.end_with?(@quote_char) # regular quoted column csv << part[1..-2] if csv.last =~ @parsers[:stray_quote] raise MalformedCSVError, "Missing or stray quote in line #{lineno + 1}" end csv.last.gsub!(@double_quote_char, @quote_char) elsif @liberal_parsing csv << part else raise MalformedCSVError, "Missing or stray quote in line #{lineno + 1}" end elsif part =~ @parsers[:quote_or_nl] # Unquoted field with bad characters. if part =~ @parsers[:nl_or_lf] raise MalformedCSVError, "Unquoted fields do not allow " + "\\r or \\n (line #{lineno + 1})." else if @liberal_parsing csv << part else raise MalformedCSVError, "Illegal quoting in line #{lineno + 1}." end end else # Regular ole unquoted field. csv << (part.empty? ? nil : part) end end # Replace tacked on @col_sep with @row_sep if we are still in an extended # column. csv[-1][-1] = @row_sep if in_extended_col if in_extended_col # if we're at eof?(), a quoted field wasn't closed... if @io.eof? raise MalformedCSVError, "Unclosed quoted field on line #{lineno + 1}." elsif @field_size_limit and csv.last.size >= @field_size_limit raise MalformedCSVError, "Field size exceeded on line #{lineno + 1}." end # otherwise, we need to loop and pull some more data to complete the row else @lineno += 1 # save fields unconverted fields, if needed... unconverted = csv.dup if @unconverted_fields # convert fields, if needed... csv = convert_fields(csv) unless @use_headers or @converters.empty? # parse out header rows and handle CSV::Row conversions... csv = parse_headers(csv) if @use_headers # inject unconverted fields and accessor, if requested... if @unconverted_fields and not csv.respond_to? :unconverted_fields add_unconverted_fields(csv, unconverted) end # return the results break csv end end end
Returns true blank lines are skipped by the parser. See ::new for details.
# File lib/csv.rb, line 1635 def skip_blanks?() @skip_blanks end
Returns true if unconverted_fields() to parsed results. See
::new for details.
# File lib/csv.rb, line 1604 def unconverted_fields?() @unconverted_fields end
Returns true if headers are written in output. See ::new for details.
# File lib/csv.rb, line 1619 def write_headers?() @write_headers end
Private Instance Methods
The actual work method for adding converters, used by both #convert and #header_convert.
This method requires the var_name of the instance variable to
place the converters in, the const Hash to lookup named converters in, and the normal
parameters of the #convert and #header_convert methods.
# File lib/csv.rb, line 2176 def add_converter(var_name, const, name = nil, &converter) if name.nil? # custom converter instance_variable_get(var_name) << converter else # named converter combo = const[name] case combo when Array # combo converter combo.each do |converter_name| add_converter(var_name, const, converter_name) end else # individual named converter instance_variable_get(var_name) << combo end end end
This method injects an instance variable unconverted_fields
into row and an accessor method for row called
unconverted_fields(). The variable is set to the contents of
fields.
# File lib/csv.rb, line 2263 def add_unconverted_fields(row, fields) class << row attr_reader :unconverted_fields end row.instance_variable_set(:@unconverted_fields, fields) row end
Processes fields with @converters, or
@header_converters if headers is passed as
true, returning the converted field set. Any converter that
changes the field into something other than a String halts the pipeline of
conversion for that field. This is primarily an efficiency shortcut.
# File lib/csv.rb, line 2199 def convert_fields(fields, headers = false) # see if we are converting headers or fields converters = headers ? @header_converters : @converters fields.map.with_index do |field, index| converters.each do |converter| break if headers && field.nil? field = if converter.arity == 1 # straight field converter converter[field] else # FieldInfo converter header = @use_headers && !headers ? @headers[index] : nil converter[field, FieldInfo.new(index, lineno, header)] end break unless field.is_a? String # short-circuit pipeline for speed end field # final state of each field, converted or original end end
Builds a regular expression in @encoding. All
chunks will be transcoded to that encoding.
# File lib/csv.rb, line 2286 def encode_re(*chunks) Regexp.new(encode_str(*chunks)) end
Builds a String in @encoding. All chunks will be
transcoded to that encoding.
# File lib/csv.rb, line 2294 def encode_str(*chunks) chunks.map { |chunk| chunk.encode(@encoding.name) }.join('') end
This method is an encoding safe version of Regexp.escape. It will escape any
characters that would change the meaning of a regular expression in the
encoding of str. Regular expression characters that cannot be
transcoded to the target encoding will be skipped and no escaping will be
performed if a backslash cannot be transcoded.
# File lib/csv.rb, line 2278 def escape_re(str) str.gsub(@re_chars) {|c| @re_esc + c} end
Stores the pattern of comments to skip from the provided options.
The pattern must respond to .match, else ArgumentError is raised. Strings are
converted to a Regexp.
See also ::new
# File lib/csv.rb, line 2161 def init_comments(skip_lines) @skip_lines = skip_lines @skip_lines = Regexp.new(Regexp.escape(@skip_lines)) if @skip_lines.is_a? String if @skip_lines and not @skip_lines.respond_to?(:match) raise ArgumentError, ":skip_lines has to respond to matches" end end
Loads any converters requested during construction.
If field_name is set :converters (the default)
field converters are set. When field_name is
:header_converters header converters are added instead.
The :unconverted_fields option is also activated for
:converters calls, if requested.
# File lib/csv.rb, line 2136 def init_converters(converters, ivar_name, convert_method) converters = case converters when nil then [] when Array then converters else [converters] end instance_variable_set(ivar_name, []) convert = method(convert_method) # load converters converters.each do |converter| if converter.is_a? Proc # custom code block convert.call(&converter) else # by name convert.call(converter) end end end
Pre-compiles parsers and stores them by name for access during reads.
# File lib/csv.rb, line 2104 def init_parsers(skip_blanks, field_size_limit, liberal_parsing) # store the parser behaviors @skip_blanks = skip_blanks @field_size_limit = field_size_limit @liberal_parsing = liberal_parsing # prebuild Regexps for faster parsing esc_row_sep = escape_re(@row_sep) esc_quote = escape_re(@quote_char) @parsers = { # for detecting parse errors quote_or_nl: encode_re("[", esc_quote, "\r\n]"), nl_or_lf: encode_re("[\r\n]"), stray_quote: encode_re( "[^", esc_quote, "]", esc_quote, "[^", esc_quote, "]" ), # safer than chomp!() line_end: encode_re(esc_row_sep, "\\z"), # illegal unquoted characters return_newline: encode_str("\r\n") } end
Stores the indicated separators for later use.
If auto-discovery was requested for @row_sep, this method will
read ahead in the @io and try to find one. ARGF,
STDIN, STDOUT, STDERR and any stream
open for output only with a default @row_sep of
$INPUT_RECORD_SEPARATOR ($/).
This method also establishes the quoting rules used for CSV output.
# File lib/csv.rb, line 2006 def init_separators(col_sep, row_sep, quote_char, force_quotes) # store the selected separators @col_sep = col_sep.to_s.encode(@encoding) @row_sep = row_sep # encode after resolving :auto @quote_char = quote_char.to_s.encode(@encoding) @double_quote_char = @quote_char * 2 if @quote_char.length != 1 raise ArgumentError, ":quote_char has to be a single character String" end # # automatically discover row separator when requested # (not fully encoding safe) # if @row_sep == :auto if [ARGF, STDIN, STDOUT, STDERR].include?(@io) or (defined?(Zlib) and @io.class == Zlib::GzipWriter) @row_sep = $INPUT_RECORD_SEPARATOR else begin # # remember where we were (pos() will raise an exception if @io is pipe # or not opened for reading) # saved_pos = @io.pos while @row_sep == :auto # # if we run out of data, it's probably a single line # (ensure will set default value) # break unless sample = @io.gets(nil, 1024) # extend sample if we're unsure of the line ending if sample.end_with? encode_str("\r") sample << (@io.gets(nil, 1) || "") end # try to find a standard separator if sample =~ encode_re("\r\n?|\n") @row_sep = $& break end end # tricky seek() clone to work around GzipReader's lack of seek() @io.rewind # reset back to the remembered position while saved_pos > 1024 # avoid loading a lot of data into memory @io.read(1024) saved_pos -= 1024 end @io.read(saved_pos) if saved_pos.nonzero? rescue IOError # not opened for reading # do nothing: ensure will set default rescue NoMethodError # Zlib::GzipWriter doesn't have some IO methods # do nothing: ensure will set default rescue SystemCallError # pipe # do nothing: ensure will set default ensure # # set default if we failed to detect # (stream not opened for reading, a pipe, or a single line of data) # @row_sep = $INPUT_RECORD_SEPARATOR if @row_sep == :auto end end end @row_sep = @row_sep.to_s.encode(@encoding) # establish quoting rules @force_quotes = force_quotes do_quote = lambda do |field| field = String(field) encoded_quote = @quote_char.encode(field.encoding) encoded_quote + field.gsub(encoded_quote, encoded_quote * 2) + encoded_quote end quotable_chars = encode_str("\r\n", @col_sep, @quote_char) @quote = if @force_quotes do_quote else lambda do |field| if field.nil? # represent +nil+ fields as empty unquoted fields "" else field = String(field) # Stringify fields # represent empty fields as empty quoted fields if field.empty? or field.count(quotable_chars).nonzero? do_quote.call(field) else field # unquoted field end end end end end
This method is used to turn a finished row into a CSV::Row. Header rows are also dealt with here,
either by returning a CSV::Row with identical
headers and fields (save that the fields do not go through the converters)
or by reading past them to return a field row. Headers are also saved in
@headers for use in future rows.
When nil, row is assumed to be a header row not
based on an actual row of the stream.
# File lib/csv.rb, line 2228 def parse_headers(row = nil) if @headers.nil? # header row @headers = case @use_headers # save headers # Array of headers when Array then @use_headers # CSV header String when String self.class.parse_line( @use_headers, col_sep: @col_sep, row_sep: @row_sep, quote_char: @quote_char ) # first row is headers else row end # prepare converted and unconverted copies row = @headers if row.nil? @headers = convert_fields(@headers, true) @headers.each { |h| h.freeze if h.is_a? String } if @return_headers # return headers return self.class::Row.new(@headers, row, true) elsif not [Array, String].include? @use_headers.class # skip to field row return shift end end self.class::Row.new(@headers, convert_fields(row)) # field row end
Returns the encoding of the internal IO object or the
default if the encoding cannot be determined.
# File lib/csv.rb, line 2302 def raw_encoding(default = Encoding::ASCII_8BIT) if @io.respond_to? :internal_encoding @io.internal_encoding || @io.external_encoding elsif @io.is_a? StringIO @io.string.encoding elsif @io.respond_to? :encoding @io.encoding else default end end