CSV

Working with CSV

CSV is the simplest format here and the one most likely to arrive subtly different from how it was written. Almost always because something opened it on the way.

There is no CSV standard. RFC 4180 is the nearest thing, and it is an Informational document written in 2005 to describe what people were already doing — not a specification anyone is obliged to follow. It says fields are separated by commas, that a field containing a comma or a quote gets wrapped in double quotes, and that a literal quote inside such a field is doubled. Plenty of real exporters do something else, and none of them are wrong in any enforceable sense. The W3C later tried to close the gap from a different direction with a model for tabular data on the web, which describes the metadata a CSV file cannot carry by itself — column types, units, relationships — and is worth a skim precisely because so little of it caught on.

That is why two files that both end in .csv can need entirely different parsers. A German or French export usually separates with semicolons, because in those locales the comma is the decimal point — so the same bytes read as one column or as three depending on what you tell the reader. Python's standard library takes the disagreement seriously enough to model it: its csv module is built around dialects, because there is no single set of rules to hard-code. Add the optional byte order mark at the front, the choice between CRLF and LF line endings, and the question of whether the first row is a header at all, and a format with one rule has four independent ways to be misread.

The bigger problem is what happens after the file is written. A spreadsheet opens CSV by guessing a type for every cell, and those guesses are lossy in ways that do not announce themselves: a leading zero on a reference code disappears, an identifier past fifteen digits is rounded, and anything shaped like a date becomes one. The most cited case is genetics — a 2016 study in Genome Biology found roughly a fifth of papers with supplementary gene lists carried errors introduced by a spreadsheet autocorrecting gene symbols into dates, and the naming committee eventually renamed the genes rather than expect the world to stop using spreadsheets.

None of that makes CSV a bad choice. It streams, it diffs line by line in version control, every language reads it without a dependency, and it does not carry a vendor's object model along with the data. It just means the format assumes both ends agreed on the details in advance, and the details are exactly what nobody writes down. If you have a file open now and want to know what a strict reader makes of it, CSV to Table shows the parse rather than a spreadsheet's interpretation of it.

CSV articles

CSV tools on this site

All categories