7096-fileGZip-collector-plugin #211

Adding FileGZipCollectorPlugin, FileCollectorPlugin and their respective tests.

The major remark I have for this PR is the mean of access to content of the files that the plugins are meant to collect in the aggregation task. In fact, we assume to use these plugins when we got the files that we want to include in the aggregation pipeline, without worrying about reimplementing or scripting the procedures for storing them in their respective MetadataStores. Therefore the underlying assumption is that those files are already stored on HDFS in some path (baseurl parameter).

Please see the comments inline.

The major remark I have for this PR is the mean of access to content of the files that the plugins are meant to collect in the aggregation task. In fact, we assume to use these plugins when we got the files that we want to include in the aggregation pipeline, without worrying about reimplementing or scripting the procedures for storing them in their respective MetadataStores. Therefore the underlying assumption is that those files are already stored on HDFS in some path (baseurl parameter). Please see the comments inline.

I have made the required changes to use org.apache.hadoop.fs.FileSystem for accessing the input files from HDFS.
Please, @claudio.atzori can you check that it is ok now ? Thanks

I have made the required changes to use `org.apache.hadoop.fs.FileSystem` for accessing the input files from HDFS. Please, @claudio.atzori can you check that it is ok now ? Thanks

Looks good now, thanks @schatz !

 @ -0,0 +17,4 @@
         log.info("baseUrl: {}", baseUrl);
         try {
             return new BufferedInputStream(new FileInputStream(baseUrl));

 @ -0,0 +19,4 @@
         log.info("baseUrl: {}", baseUrl);
         try {
             GZIPInputStream stream = new GZIPInputStream(new FileInputStream(baseUrl));

Labels Milestones

7096-fileGZip-collector-plugin #211

Reviewers

Step 1:

Step 2: