Video summary
Guillaume Plique, a researcher from the Media Lab in Paris, advocates for the widespread use of CSV files as a foundational data format for social sciences research. He explains that his lab, which brings together social scientists, engineers, and designers, has spent over a decade building open-source tools centered around this simple text-based format. Despite facing skepticism or even hostility from some who prefer more complex formats, Plique argues that CSV is essential because it is affordable, universally understandable, and free. He emphasizes that unlike hierarchical structures like JSON, tabular data represented in CSVs is immediately readable by almost anyone, making it an ideal bridge between the spreadsheet world used by researchers and the engineering world used by developers.
The core advantages of CSV extend beyond its simplicity to include its status as a truly open standard that requires no proprietary software to read or write. Plique highlights that because CSV is just plain text, it ensures long-term accessibility; even in fifty years, data stored in this format will likely remain readable, unlike binary formats such as Excel files which may become obsolete. Furthermore, the format is highly efficient and concise, naturally compressing well since headers are written only once at the beginning of the file rather than repeated for every row as seen in JSON. This efficiency allows researchers with limited hardware resources, who often lack powerful computers or GPUs, to process massive datasets, such as billions of tweets, using standard tools without needing complex database infrastructure.
To demonstrate the versatility of CSV, Plique showcases several custom tools developed by his team that operate directly on these files without requiring installation or a server. These include "Table-to-Net," which converts tabular data into network graphs for visualization, and "Takoyaki," a web-based tool designed to cluster data and identify clerical errors similar to features found in OpenRefine. The lab also created command-line utilities like "Minet" for downloading large lists of URLs and processed forks of existing tools to handle CSV data in unique ways, such as reading files in reverse byte order to resume interrupted processes efficiently. These examples illustrate how a simple format can support sophisticated workflows, from visualizing social networks to managing petabytes of data through clever engineering tricks that exploit the inherent properties of plain text.
In conclusion, Plique urges researchers and engineers to always bet on CSV for its robustness and interoperability, suggesting it as the default choice unless random access is specifically required, in which case a lightweight database like SQLite might be used. He presents CSV not just as a data container but as a "lingua franca" that fosters collaboration among diverse groups by enabling seamless data exchange between different tools and disciplines. By sticking to this austere yet powerful format, the social sciences community can maintain autonomy over their data, avoid vendor lock-in, and continue to innovate with open-source solutions that are accessible to everyone from students to senior researchers regardless of their technical background or available resources.
Read the full video transcript
Um, hello everyone. So, I hope everyone
hears me correctly. So, I'm a very
literal kind of person. So, this is a
CSV comp. I'm going to talk about CSV a
lot. Um, so I come from a very weird
little social sciences lab in Paris
which is named the media lab. Um, this
is a research lab that gathers a lot of
different profiles. So, researchers in
social sciences, uh, engineers such as
myself and designers. And so we try to
do a different kind of social science
and we have been building open-source
tools now for 10 years and those tools
are mostly geared towards uh social
sciences of course and the civil society
at large. So if you want you can check
up check out our GitHub and so I have a
CSV shaped question. So most if not all
our tools are basically using some kind
of CSV data at the end of the day. Uh
and we love CSV so much that we wrote a
love letter to CSV uh that you can read.
There is the URL here. And so the
question is why though? Why CSV? Because
I try to say to people on the web, I
love CSV and I was met with horror,
hatred, um guilt and so on. So um we are
using CSV basically because this is an
affordable uh understandable and free uh
data format. uh so much so that CSV uh
nowadays is a data lingua franka for our
work basically. So I I'm going to try
and explain to you why we are using CSV
so much in the social sciences.
So first of all there is really good
affordances. So basically everybody is
familiar with tabular data at some
level. Uh here you've got some kind of
spreadsheet very interesting one every
know everyone knows how to read that and
interpret this kind of information. If
you give this to someone, so JSON data,
nested data, har I can't say this word,
hierarchical data structures, uh they
will frowns. Uh this is not really
understandable. So this is easy to
understand. This is not. So we are going
to stick to table data. So we can like
basically interact with uh our users.
Um then CSV is actually quite simple as
a format, right? So the specifications
hold in the title. So comma separates
values. And I know sometimes you might
want to use semicolons, pipes, and so
on. I don't care. You use whatever you
want. It's not I'm not here to judge
you, right? So um it's very simple. Uh
everyone here knows it's a lie. It's a
bit more complicated, but you see the
point. Uh in any case, you can explain
CSV to almost anybody very simply in
mere minutes. Uh even more so the format
is so simple that you might invent it
yourself, right? Uh or I think you
should say you discovered CSV, right? I
think a lot of people here invented CSV
at some point in their life. That's all
right.
And then of course CSV is free and this
is very important. So nobody owns the
CSV format or at least I think so. uh it
is and it will always remain and be a
collective and free idea that was
thought by many and is used by many uh
and it is so free that he has it has no
real specification either all right I
know about RFC
yeah 4180
I don't care about it and I will fight
you for it um and so it's so free that
some scientific disciplines have their
own dialects okay so you can speak of
CSV dialect and I think it's very
important the term like is tied to
linguistics, right? So for instance, if
you work in bioinformatics, you might be
familiar with BCDF, GFF and so on. And
if you work in the web archiving space,
you might be um uh accustomed to CDX and
so on. All those are basically CSV
files, right? So if you know how to
handle CSV files, you can do a lot of
stuff.
Uh so and what's more, so it's free as
it as in it's not proprietary. Okay? So
you don't need some uh non-f free
software to be able to read, write,
process CSV uh data. So as a cheeky
counter point, if you try to open an
Excel file in your terminal or um text
editor, I know you all know that it will
happen badly and I know it's XLS, not
XLS6, but you see the point. Don't open
XLS files by yourself. It hurts.
And so it's simple, it's free. And
what's more, and this is like basically
a conclusion about that. It's just text,
right? And text is very simple. You can
write it yourself to some degree. Have
you already tried to write CSV yourself?
Yeah, I'm sure you had. Uh you can read
it yourself to some degree somehow. And
uh if I hand out some CSV to you, I give
you the file and I tell you this is some
data. you should be able to open it and
basically figure out what's going on
very simply, right? And in 50 years, you
should probably be still able to do so,
which is very important because in 50
years, I'm pretty sure that no one will
be able to read XLS
or at least I hope so.
And so this means that in the end CSV is
a bridge, right? So CSV uh in our lab
can be handled by both researchers,
students, engineers alike. everyone is
able to process uh produce CSV uh sheets
and so on and convert between CSV and
some other format very easily and so CSV
is a bridge between uh what I call the
spreadsheet world the spreadsheet world
and the engineering world. So CSV is a
good fit for both tiny and big data
problem uh which is very nice. So CSV at
the end creates the perfect condition
for a de facto auto organized interoper
interoperability of a lot of free tools
that are designed by people all around
the world. So this is a very good glue
of data to to make sure that a lot of
people are able to work together.
And so this was the the first main
point. CSV is very simple. CSV is free.
CSV is text and so on. And what's more
CSV is sober. And I'm not sure the
English term is very well chosen here
because the French meaning of the term
is a bit strange. But basically it's an
austere data format and it's a very
simple uh affordable data format that is
very concise uh compressed efficient
everything. Uh so it's not exactly a
chosen a chosen soiety here and I mean
soiety like in energy soiety.
So um I work in the social sciences and
basically we don't have money right. So
we don't have hardware, we don't have a
powerful computers, we don't have GPUs.
So it means that we have to get a little
bit creative to be sure that we can do
some stuff uh computationally wise.
So and what's more our users don't have
access to powerful um software or um
hardware. Uh researchers in social
sciences don't have powerful laptops.
Students even sometimes don't have any
laptops.
So uh we need to make it do. And so what
exactly makes CSV a good and and sober
sorry for the term uh data format.
So it's very it's a very succinct data
data format like by default naturally
because headers are only written once at
the beginning of the file. Uh if you
read some JSON file you will see that
the headers are like the keys are
repeated a lot. It does not mean that
you cannot compress it very well. But
basically CSV is like concise naturally.
Um the strings are optimally
represented. You will not like gain so
much except for the quote and the
numbers could be lighter but frankly
it's all right. So it's very succinct
and it's a very efficient uh format
also. So the question would be uh do you
think you can reliably collect billions
of tweets uh on the web without any
databases relying only on gigabytes of
CSV data and the answer is of course yes
you can uh at least before the evil dude
that that will not name arrived right so
in our lab we used to collect like a lot
of tweets for social sciences and we
just like basically dumped the tweets in
a tabular format within like heaps of
CSV data that were compressed afterwards
and that's all no database nothing just
row CSV and it worked very very very
well so it can be thought of as a
structured append only log format
basically
um so sorry for the engineering here but
it's very very efficient
and so this means ultimately that you
can take CSV and build a lot of
different tools around them and so I
will showcase some of them to you. So
some of the tools that we built in our
lab. So we built dedicated web UIs for
researchers and students. So this one is
called table to net. And so basically
you have a CSV of table data and you
want to convert it in some kind of
network or graph to use in gy or with
sigma. And so basically to map some
network. So uh you can use it in the
web. No installation required.
Uh this is one of our most uh old tool.
Uh this one is called takoyaki. So it's
a a twist on like what is called a
clerical clustering. So you are
clustering CSV data and table data to
find clerical errors to fix. Um so if
you are familiar with uh open refine
this is basically a twist on the cluster
and edit um feature. So still web UI no
installation required no server nothing
just your computer.
And then uh we also built common line
tools. So for instance here you have an
example of uh one tool that is named
minet that is running in the um in the
common line. So here we are taking CSV
in and we are outputting CSV out. So
here you've got an example of um a
common that is basically reading a very
large CSV of URLs and that is trying to
download them the fastest possible and
emitting a report like with uh what
happened and uh downloading the file
next to the the CSV file. So CSV in CSV
out. uh we've built other kind of uh CSV
data and we basically we love CSV so
much that we we built like a tool to
make sure that we can handle it from the
common line because we have a lot of CSV
data as as I told you we have like
billions of tweet on the server and so
we need to be able to process them
visualize them very quickly so uh we
built this which is a fork of a tool
that you might know which is named XSV
and we also build the libraries liaries
like Python libraries to be able to use
CSV in weird manners that are not
handled by the Python built-in modules.
And the last point I wanted to make was
that CSV also rewards out ofthe-box
engineering and like clever thinking
about how the format works. So I don't
know if you know this one, but CSV can
actually be read in reverse. So if you
feed the bytes in reverse, it's still
valid CSV. That's one of the only data
format I know that is able to do so. Uh
which means that if you read the file in
reverse and you like reverse the bytes
and reverse the the row order, hey,
you've read CSV in reverse. That's
crazy. Uh so here for instance, you've
got an example of that of in XAN. So the
XAN reverse uh file, the XAN reverse
command and the XAN tail common are able
to read a CSV file in constant time, in
constant memory and linear time, which
is stupid. the crazy and then uh you can
use this information to basically resume
some stuff. For instance, you you output
a log and then everything breaks
everything break and you need to come
back to it. So you read the file in
reverse, you find where you stopped and
you start back again.
Uh that's it. So my conclusion would be
always bet on CSV and I've made you a
little flowchart if you need a database.
All right? So if you need random access
use SQLite because you are a fancy
person. If you don't just use CSV all
right
and uh thank you for your time. There is
some links for you to peruse if you
want. So and I think I will finish here
somewhat in time. Thank you very much.