# JSON to CSV

This project consists on a set of tools in order to convert JSON items
to CSV.

## Version
version 1.0.1

## Project structure

For the time being, the project consists of two Python files:
* main.py
* json2csv\_lib.py

The directory thought for storing the input collections is:
* ./input/

At the current moment there is only one one file of ~200MB called
diada2016.json.

The resulting data is stored at:
* ./output/split\_tweets
* ./output/flattened\_tweets
* ./output/csv\_converted\_tweets

Unless specified otherwise using the -o (--output) argument.

## Example of use

Using the provided diada2016.json a basic example of use could be:

```
./main.py ./input/diada2016.json
```

The main.py file contains all the code related to initialization of the
program, global variables, and parsing the options and args passed by
the user. The options passed by the user will trigger one of the
different modes of execution and will determine the input and output
files or directories. The four accepted modes of execution are:

* "Split" a collection in individual tweets.
* "Flatten" either a single tweet or a collection.
* Convert from format flat-JSON to CSV, "flat2csv",
  while applying a filter to exclude some fields
* Apply all the previous steps in a single execution with "json2csv",
  which converts a JSON collection to multiple single CSVs.

The final resulting CSVs and the intermediate results are all stored in
the ./output directory and its subdirectories. However, this paths are
managed at main.py and could be changed at will.

### Filter (added at 0.3.0)

The filter is regex-based and can be customized at the main.py. No
filter (or an empty filter) will include all the information, and the
user can add any number and kind of regexes to exclude fields. The
filter is applied in the flatJSON to CSV conversion stage, and it will
compare every key in the flattened dictionary with all the elements on
the filter list.

### Command-line options and arguments (added at 0.4.0)
Now the main.py script accepts a few options and arguments that allow
the user to be more specific about what kind of execution he wants and
where to take the input from and where to store the output files.
For a complete specification of the options check "./main.py -h".

### Include/exclude filters (added at 0.5.0)
Now in addition to the filter implemented at v0.3.0, which was a
"excluding" filter (that is, the regexes written in the filter that
matched with a key were NOT written to the final CSV file and all the
rest were), now an "inclusive" filter has been added. This new filter
will only write to the CSV the key-value pairs where the key matches
with at least one of the regexes included in the filter. 
When the include filter is not empty, the exclude filter is ignored. To
convert a full file (all fields included) both filters should be empty.
To lighten the code files and reduce coupling the filters have been
moved from main.py to ./data/filters.py. Additional filters should be
written there too.
Finally, more code was added to the convert\_json2csv function in the
library ./lib/json2csv.py to fold the array fields and include them in a
unique field (excepting bounding\_box coordinates which were only folded
once and still have 4 fields for the four coordinate points).

#### Bugfix 0.5.1: CSV quoting
Fixed bug related to the lack of quotes in the CSV values, which made
the CSV formatters to interpret the produced documents wrong. For
example, a list [a, b, c, d] was interpreted as 4 different CSV values.
Now it is written "[a, b, c, d]" and interpreted correctly. This applies
to all the values because strings were troublesome too (booleans and
numbers probably didn't need that, but anyways).
Also, directory OLD removed and added a dir named samples to easily see
3 examples of files produced by the program (json, flat json and csv).
This is provided instead of the converted files in the output subtree
directory, which were probably too much (now ignored on .gitignore)

### Initial unit testing + major bugfix (0.6.0)
First test cases and tests implemented in a new directory specific for
testing. 
While working at it a tremendous bug was discovered: the splitting
operation wasn't performing as expected for some JSON objects. However,
the error was silent and didn't take place in the first tweets of the
collection, reason why it passed unadvertedly for a long time. This
error had to do with strings including braces, which had to be ignored
as they are not really part of the JSON structure. After that fix,
another bug appeared related to the escape character '\', which in some
files provoked an error when it appeared before a double quote.
The new algorithm manages properly escape characters and quotes and
splits the full 200MB collection correctly.

#### Minor refactoring (0.6.1)
Changed some names and code of the splitter, flattener and csv-converter
algorithms moved out to isolate the logic, strictly speaking, from the
rest of the I/O managing code and whatever else.

#### Added verbosity and minor refactoring (0.6.2)
Added a new mode, "verbose", activated with -v or --verbose.
Non-critical messages (excepting invocation failures that suppose
termination and so) will only be displayed if this flag is activated.
Some of the previous prints were put under verbose and some others were
added to provide more information about execution trace.
Also, minor refactoring was done in the main script.

#### Added silent mode (0.6.3)
Added a new mode, "silent", activated with -s or --silent. This mode
overrides verbose (and cancels it) if active. This mode not only
silences the messages provided with verbosity, but also the rest (error
messages and so) excepting the syntax help.

#### Significant number of unit tests added (0.6.4)
Lots of unittests have been added. Most of them are not purely "unit",
because they act over the main calls to split, flatten and convert to
csv, which in turn call to several complex json2csv library methods.
However, these tests check that the overall features work as desired.

#### New compression command-line option (0.6.5)
Now the user can opt to activate or deactivate the array compression
mode. The default mode (if no option is given) is no-compression, that
is, the arrays are written as flattened elements in the CSV file. If the
option -c is given, these arrays will be flattened 1 level.

#### New UPV-compatible and remove dollar options (0.6.6)
Now the user can opt to activate or deactivate a mode to remove the
dollar headed fields added by Mongo. These are fields like created\_at
where Mongo sometimes automatically added a dollar-termination with the
type of the field, like created\_at.$date. This mode removes the .$date
and similar teminations. Since this is not the basic mode of operations
and could induce bugs or unexpected behaviour for people working with
data from other origins, this functionality has been isolated. The
UPV-compatible mode is equivalent to activating -c & -r, that is, the
array-compression and remove-dollar options, simplified in a single
option just because this is the mode I'm going to use more frequently.

#### New filtering options (0.6.7)
Filters are no longer hardcoded. Now, if user doesn't specify anything
about filters, none will be applied. The user can specify the path to an 
exclude or an include filter using the -e/--exclude and the -i/--include
command-line options. Furthermore, the user can specify
-f/--default-filters to use the default filters included in data/filters

#### Overwrite prompts (0.6.8)
Now the user will be prompted whenever a collection with the same path
than an already existing one is going to be generated. The user will be
able to choose between overwriting or aborting the execution.

#### Bugfix (0.6.9)
Before this version, the array compression and remove dollar modes didn't apply
when there wasn't an include filter (that is, either with exclude filter or
without any filters at all). This has been fixed and now any combination of
filters and modes should work correctly.

### Added arg option to keep/discard the intermediate files (0.7.0)
The possibility of discarding or keeping the intermediate files (split and/or
flattened JSONs) has been added as an invocation argument (-k / --keep-files).
Before this version these files were always kept. Now they are discarded by
default and the -k options permits to keep them.
Before this version, the array compression and remove dollar modes didn't apply
when there wasn't an include filter (that is, either with exclude filter or
without any filters at all). This has been fixed and now any combination of
filters and modes should work correctly.

### Large changes in the directory structure of the output (0.8.0)
This patch includes a big reestructuration of the directory hierarchy produced
by the program. The three standard folders for output (split\_tweets,
flattened\_tweets and csv\_converted\_tweets) cease to exist. Now the only
preexistent folder for output is "output" itself. Everytime a collection is
split, flattened or converted, a new output folder with the name of that
collection is created inside "output". These collections can have up to three
subdirectories called "Split", "Flat" and "CSV", every one of which contains
the type of files indicated by its name.

#### Bugfixes (0.8.1)
Fixed some bugfixes related to the changes introduced at 0.8.0. E.g., in 0.8.0
one could not flatten the split directory of the same collection, because the
collection was removed before (the overwriting was performed over the top
directory of the collection, and thus the input for the operation was removed
before beginning to flatten). Tests have been adapted too

## Added integer argument to the option compression/-c (0.9.0 / 1.0.0)
Now the user cannot only specify if he wants to represent the arrays flattened
or compressed in the CSV file, but also which depth of compression. A level of
0 would equal no compression, a level 1 equals the old compression mode (and
the current UPV-compatible mode too), that is, to just unnest 1 level of
arrays, for example:

house[0].dweller[0]: "Ana"
house[0].dweller[1]: "Roberto"
house[1].dweller[0]: "Marina"

Would become:
house[0].dweller: "[Ana, Roberto]"
house[1].dweller: "[Marina]"

Now, it is also possible to select a compression of depth 2 or more, which
would provide the followint output:

house.dweller: "[[Ana, Roberto], [Marina]]"

Some minor bugs fixed and tests adapted to the new feature.
This is the state of the program desired in the beginning of the development,
and this version (with possible minor fixes) is the one expected for the 1.0.0
release.

#### Bugfix (1.0.1)
Fixed the array representation. Before this fix, the arrays of values were
represented as "[a, b, c]", which could lead to tremendous bugs due to
ambiguity (the example array could have three values, "a", "b" and "c", or only
one, "a, b, c", or other combinations). The fixed notation solves that
ambiguity, because now the arrays are represented in the form: ["a", "b", "c"]
