compress broker workflow outputs #67

New Issue

claudio.atzori · 2020-12-09T09:26:05+01:00

claudio.atzori commented

2020-12-09 09:26:05 +01:00

I noticed the intermediate data produced by the broker related workflows is stored as non compressed. Given that the inspection tools supports on the fly decompression (HUE web UI, hdfs cmdline tool) I don't see a valid reason for not storing them in compressed format.

Many other worfklows implementation does the same using the gzip compression format, so please take inpiration from the existing classes.

Please note also that I'm assigning the task to the broker branch. Any change should be implemented there, tested, and only after a successful testing integrated in the master branch, possibly via a pull request.

I noticed the intermediate data produced by the broker related workflows is stored as non compressed. Given that the inspection tools supports on the fly decompression (HUE web UI, hdfs cmdline tool) I don't see a valid reason for not storing them in compressed format. Many other worfklows implementation does the same using the gzip compression format, so please take inpiration from the existing classes. Please note also that I'm assigning the task to the `broker` branch. Any change should be implemented there, tested, and only after a successful testing integrated in the master branch, possibly via a pull request.

claudio.atzori added the

enhancement

label 2020-12-09 09:26:05 +01:00

michele.artini was assigned by claudio.atzori

2020-12-09 09:26:05 +01:00

michele.artini commented

2020-12-10 14:51:33 +01:00

The fix has been committed on the broker branch

claudio.atzori commented

2020-12-10 14:57:32 +01:00

Side note: the fact that the Event records were not stored as compressed could have a role in the slowness of the procedure dedicated to partition them by opendoar ID. The bigger the data the more time it takes to move it around the cluster.

michele.artini referenced this issue

2020-12-15 09:02:55 +01:00

broker #78

claudio.atzori commented

2020-12-18 09:28:59 +01:00

Integrated in PR !78

@michele.artini We should monitor the execution of the paritioning by opendoarID operation to see if the compression introduces any significant advantage.

Integrated in PR !78 @michele.artini We should monitor the execution of the paritioning by opendoarID operation to see if the compression introduces any significant advantage.

claudio.atzori closed this issue

2020-12-18 09:28:59 +01:00

Sign in to join this conversation.