Name |
Last commit
|
Last update |
---|---|---|
lib | ||
.gitignore | ||
header.txt | ||
pymongoexport_csv.py | ||
pymongoexport_json.py | ||
requirements.txt | ||
utils.py |
Parallelized using multiprocessing library. I'm not really sure about the code being thread safe. I think we don't care if tweets are appended to the files in a different order, but the metadata files being corrupted would be problematic. In the first tests the metadata were fine, but I think this line is probably not thread safe (two threads could load try to update the old value at the same time, resulting in inconsistencies): """ metadata_file["files"][file_path]["count"] += increase """ Apart from that, code is much faster than before.
Name |
Last commit
|
Last update |
---|---|---|
lib | Loading commit data... | |
.gitignore | Loading commit data... | |
header.txt | Loading commit data... | |
pymongoexport_csv.py | Loading commit data... | |
pymongoexport_json.py | Loading commit data... | |
requirements.txt | Loading commit data... | |
utils.py | Loading commit data... |