andreas.czerniak
/
BrBETA_dnet-hadoop
forked from D-Net/dnet-hadoop
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
707 Commits 23 Branches 0 Tags 90 MiB
Commit Graph

701 Commits

Author SHA1 Message Date
Michele Artini b35c59eb42 partial implementation of entities from db 2020-01-20 16:04:19 +01:00
Sandro La Bruzzo fa7504bf29 removed DLI stuff should be in a branch 2020-01-20 10:28:00 +01:00
Michele Artini 81f82b5d34 partial implementation of applications to migrate entities 2020-01-17 15:26:21 +01:00
Claudio Atzori 1cd6899480 merged from master 2020-01-17 14:25:57 +01:00
Claudio Atzori 749b0660ab instance URLs must be repeatable 2020-01-17 14:22:15 +01:00
Claudio Atzori 63c0db4ff8 instance URLs must be repeatable 2020-01-16 15:54:53 +02:00
Claudio Atzori 97c239ee0d WIP: trying to find a way to build the records for the index 2020-01-16 12:02:28 +02:00
miconis 4955be0197 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-01-14 15:03:44 +02:00
miconis f61adfc2bb minor changes 2020-01-14 15:03:27 +02:00
miconis 9bdcb02179 minor changes and update of the configuration for publications 2020-01-14 15:01:03 +02:00
Michele Artini f7b9a7a9af entity migration (partial implementation) 2020-01-10 15:55:23 +01:00
Michele Artini 7229fecbcf fix warnings in poms 2019-12-20 13:41:08 +01:00
Sandro La Bruzzo dd21db7036 fixed stuff 2019-12-18 16:28:22 +01:00
Claudio Atzori 7ba586d2e5 oozie workflow aimed to build the adjacency lists representation of the graph, needed to build the records to be indexed 2019-12-17 16:24:49 +01:00
Sandro La Bruzzo 76efcde4fd using new branch decisionTreeDedup 2019-12-13 12:20:35 +01:00
Sandro La Bruzzo b4392f9f43 implemented DedupRecord factory for missing entities 2019-12-13 09:40:02 +01:00
miconis 545e940007 implementation of the mergeFrom for the Datasources 2019-12-12 15:36:41 +01:00
Sandro La Bruzzo 39367676d7 implemented DedupRecord factory with the merge of project 2019-12-12 15:18:48 +01:00
Sandro La Bruzzo 6b45e37e22 implemented DedupRecord factory with the merge of organizations 2019-12-11 16:57:37 +01:00
Sandro La Bruzzo abd9034da0 implemented DedupRecord factory with the merge of publications 2019-12-11 15:43:24 +01:00
miconis 4b66b471a4 implementation of the sorting by trust mechanism and the merge of oaf entities 2019-12-10 14:57:16 +01:00
Sandro La Bruzzo cc63706347 Implemented deduplication on spark 2019-12-06 13:38:00 +01:00
Claudio Atzori 6a7bee5e43 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2019-11-14 15:43:07 +01:00
Claudio Atzori 0c4b316f82 align Result model with the latest OpenAIRE schema changes introduced in the protobuf model 2019-11-14 15:42:52 +01:00
Sandro La Bruzzo aad0cb40b7 Added schema Scholexplorer 2019-11-14 10:34:09 +01:00
Claudio Atzori 5711e75f67 use ${project.version} whenever possible 2019-11-08 17:41:51 +01:00
Claudio Atzori 245b4cbbb3 removed import limit 2019-11-08 17:41:01 +01:00
Claudio Atzori 7fe6835b47 [maven-release-plugin] prepare for next development iteration 2019-11-07 17:39:30 +01:00
Claudio Atzori 58918967d9 [maven-release-plugin] prepare release dhp-1.0.4 2019-11-07 17:39:27 +01:00
Claudio Atzori 2243089b78 Author PIDs include also provenance information 2019-11-07 17:38:37 +01:00
Claudio Atzori 5308f05a02 allow to speficy the target hive DB name in the infospace import workflow 2019-11-07 17:38:09 +01:00
Claudio Atzori a52d5bde4f simplified import procedure, maps the infospace as hive tables 2019-11-06 17:45:52 +01:00
Claudio Atzori 1e7a2ac41d align parmeter names, graph import procedure WIP 2019-11-04 17:41:01 +01:00
Claudio Atzori f39148dab8 [maven-release-plugin] prepare for next development iteration 2019-11-04 12:34:48 +01:00
Claudio Atzori 34b0e7b40a [maven-release-plugin] prepare release dhp-1.0.3 2019-11-04 12:34:46 +01:00
Claudio Atzori 439ad80d81 conversion utilities from protobuffer model to DHP model moved in dnet-mapreduce-jobs. Removed also the relative protobuf dependencies 2019-11-04 12:33:23 +01:00
Claudio Atzori 32ed4ae8d6 conversion utilities from protobuffer model to DHP model moved in dnet-mapreduce-jobs. Removed also the relative protobuf dependencies 2019-11-04 12:28:56 +01:00
Sandro La Bruzzo fd0ad82111 [maven-release-plugin] prepare for next development iteration 2019-10-31 12:08:51 +01:00
Sandro La Bruzzo f224613b40 [maven-release-plugin] prepare release dhp-1.0.2 2019-10-31 12:08:49 +01:00
Sandro La Bruzzo e13c30cc96 [maven-release-plugin] rollback the release of dhp-1.0.2 2019-10-31 12:07:04 +01:00
Sandro La Bruzzo 4da5239203 [maven-release-plugin] prepare release dhp-1.0.2 2019-10-31 12:06:14 +01:00
Sandro La Bruzzo db8b346edd [maven-release-plugin] rollback the release of 1.0.1 2019-10-31 11:49:05 +01:00
Sandro La Bruzzo fc80052173 [maven-release-plugin] prepare for next development iteration 2019-10-31 11:47:42 +01:00
Sandro La Bruzzo 3150c7ce6d [maven-release-plugin] prepare release 1.0.1 2019-10-31 11:47:40 +01:00
Sandro La Bruzzo fe2bd4df72 changed info about git 2019-10-31 11:46:07 +01:00
Sandro La Bruzzo 18ec8e8147 moved protoutils function to dhp-schemas 2019-10-31 11:31:37 +01:00
Sandro La Bruzzo 997e57d45b Added entity filter to spark class 2019-10-30 12:19:03 +01:00
Sandro La Bruzzo a336956708 added defautl property to job 2019-10-30 12:01:42 +01:00
Claudio Atzori 78b5b57e86 trying to make the spark action to be run as spark2 2019-10-29 18:56:34 +01:00
Claudio Atzori c8bb81cd9a align dependencies with IIS cluster 2019-10-29 18:10:20 +01:00
Claudio Atzori 5e32a4066a depending on dnet-openaire-data-protos:3.9.5-proto250 2019-10-28 18:17:32 +01:00
Sandro La Bruzzo fe62ccd6dd implemented oozie wf 2019-10-28 12:12:50 +01:00
Sandro La Bruzzo 06912fd0d3 fixed test 2019-10-28 12:06:30 +01:00
Sandro La Bruzzo 9ee4e5a196 remove a bit of syntactic sugar on the object inheritance :( 2019-10-25 18:10:30 +02:00
Sandro La Bruzzo c74335ebc7 resolved conflict 2019-10-25 14:34:50 +02:00
Sandro La Bruzzo 8c902c500a minor fix 2019-10-25 14:33:54 +02:00
miconis 9fa5aebe9c minor changes 2019-10-25 12:52:28 +02:00
miconis 551eda1600 dataset, orp and software mapping implemented. addition of test resources for results. implementation of tests to check the result of the mapping 2019-10-25 12:48:25 +02:00
Sandro La Bruzzo eef14fade3 fixed conflict 2019-10-25 11:58:20 +02:00
Sandro La Bruzzo 0ea7e861ab added organizations test 2019-10-25 11:56:28 +02:00
miconis 4908165e05 implementation of the createPublication method to map publications 2019-10-25 11:54:14 +02:00
miconis df37bd6aaf placeholders for setters in createpublication 2019-10-25 10:57:19 +02:00
Claudio Atzori 331b853ad6 Merge branch 'master' of https://code-repo.d3science.org/D-Net/dnet-hadoop 2019-10-25 10:55:43 +02:00
Claudio Atzori 4eaff36ea6 a bit of syntactic sugar on the object inheritance 2019-10-25 10:55:35 +02:00
Sandro La Bruzzo c8d6d6bbd1 implemented organization mapping 2019-10-25 10:23:51 +02:00
miconis b525b54130 starting implementing the createPublication class 2019-10-25 09:55:31 +02:00
Claudio Atzori b0aa7cd7fb fluent setters 2019-10-25 09:53:08 +02:00
Claudio Atzori 4b331790e7 resolved conflicts 2019-10-25 09:45:12 +02:00
Claudio Atzori c929c1dfac more proto 2 graph model mappings 2019-10-25 09:25:36 +02:00
Sandro La Bruzzo 09ffda03a2 removed circular dependencies 2019-10-25 09:24:18 +02:00
Sandro La Bruzzo a10d071cf4 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:55:44 +02:00
Sandro La Bruzzo 3a8bb11695 mapped first part 2019-10-24 17:55:40 +02:00
Claudio Atzori d46371ceab Merge branch 'master' of https://code-repo.d2science.org/D-Net/dnet-hadoop 2019-10-24 17:43:55 +02:00
Claudio Atzori 0d88f9a6a4 added mapping for projects 2019-10-24 17:43:42 +02:00
Sandro La Bruzzo 2dd9572f41 added Mapping of OriginalDescription 2019-10-24 17:36:44 +02:00
miconis 351d850ad3 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:29:07 +02:00
miconis b66a7e3030 publication test added 2019-10-24 17:29:01 +02:00
Sandro La Bruzzo 6c32d418ac added conversion of ExtraInfo 2019-10-24 17:26:55 +02:00
Claudio Atzori 5f339a2c24 added mappings for basic types 2019-10-24 17:21:45 +02:00
Claudio Atzori 52abfcfac7 Field<T> is an actual class, fluent setters 2019-10-24 17:17:12 +02:00
Sandro La Bruzzo 9d04111391 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:05:52 +02:00
Sandro La Bruzzo 0902bac7dd fixed conflict 2019-10-24 17:05:42 +02:00
Claudio Atzori d8bfaa3687 added mapping for relations 2019-10-24 17:04:13 +02:00
Sandro La Bruzzo d2965636e0 created test for convert json into new OAF data model 2019-10-24 17:02:35 +02:00
Claudio Atzori 79c4f1bbd8 Protobuf to internal graph model, early steps 2019-10-24 16:56:13 +02:00
Claudio Atzori d38aeb8c6e DataInfo.provenanceaction not repeatable, fluent setters 2019-10-24 16:55:38 +02:00
Sandro La Bruzzo 5744a64478 added module dhp=graph-mapper 2019-10-24 16:00:28 +02:00
Sandro La Bruzzo ed14a40890 removed unuesd directory 2019-10-24 11:40:47 +02:00
Sandro La Bruzzo 25a62b79e5 added new model for information space dataframes 2019-10-24 11:39:41 +02:00
Sandro La Bruzzo 5a8a323f2a dhp-collection-worker integrated in dhp-workflows 2019-10-24 11:36:59 +02:00
Sandro La Bruzzo c8e3e4d7c3 Refactoring dependencies versions 2019-10-24 10:20:31 +02:00
Claudio Atzori dd1d6fcb01 moved libs in main pom file 2019-10-18 10:50:55 +02:00
Claudio Atzori 176a13601b commented out maven plugin for integration tests 2019-10-18 10:50:32 +02:00
Claudio Atzori 3f08ed94e3 replacing iis references with dhp 2019-10-18 10:49:34 +02:00
Claudio Atzori e97656e915 fixed sandbox directory assignment 2019-10-18 10:48:02 +02:00
Claudio Atzori 0c284e0a51 doc 2019-10-18 09:42:41 +02:00
Claudio Atzori c7654b6fe3 renamed collection & transformation oozie workflow files 2019-10-18 09:42:20 +02:00
Claudio Atzori 44d7e85797 imported oozie-installer.markdown docs from https://github.com/openaire/iis/blob/master/iis-wf/docs/oozie-installer.markdown 2019-10-17 18:43:43 +02:00
Claudio Atzori 27db5afdad integrating the oozie workflow build/deploy/run mechanism, took inspiration from iis 2019-10-17 18:38:30 +02:00
Sandro La Bruzzo bbb87d0e3d implemented saxonHE on transformation spark job 2019-10-10 11:33:51 +02:00
Sandro La Bruzzo 4b8c7c279d Added documentation on a class, and reused ArgumetApplicationParser on dhp-aggregation 2019-10-07 17:02:53 +02:00
Sandro La Bruzzo a423a6ebfd Created a generic Argument parser to be used in all modules 2019-10-03 12:22:44 +02:00
Sandro La Bruzzo b259cd0bd8 changed wrong version 2019-10-02 15:54:01 +02:00
Sandro La Bruzzo 785b9c7cda removed dhp-mdstore-manager from pom since it was migrated to another repository 2019-10-02 15:51:04 +02:00
Sandro La Bruzzo 2e6c05a0e1 removed dhp-applications/dhp-mdstore-manager-app since it was migrated to another repository 2019-10-02 15:50:04 +02:00
Sandro La Bruzzo 27ba72da96 review readme 2019-10-02 15:26:58 +02:00
Sandro La Bruzzo d584c79bf7 test commit 2019-10-02 15:12:09 +02:00
luosolo b877bf8a40 test commit 2019-10-02 15:08:21 +02:00
luosolo a36ca1b7e9 fixed poms 2019-10-02 14:59:02 +02:00
luosolo 5b48bb9be1 Removed springboot dependencies, is useless 2019-10-02 14:45:12 +02:00
Sandro La Bruzzo 53ec9bccca changed the implemetation of RabitMQ Comunication 2019-04-16 12:28:01 +02:00
Sandro La Bruzzo 403c13eebf Implemented message manager, Fixed bug on collection worker, implemented Collecion and Transform spark job 2019-04-11 15:39:29 +02:00
enricoottonello 58c4e1f725 cleaned properties 2019-04-09 12:17:03 +02:00
Sandro La Bruzzo 9294851a6c implemented comunication layer using rabbitMq between oozie node and Dnet 2019-04-05 12:19:25 +02:00
Sandro La Bruzzo 3f4ba71bbd resolved conflicts 2019-04-03 16:12:57 +02:00
Sandro La Bruzzo ded6aef5e1 moved collector worker 2019-04-03 16:05:16 +02:00
enricoottonello 2f79eb930a added apidescriptor 2019-04-03 16:03:44 +02:00
Sandro La Bruzzo c2ecbf5572 moved collector worker 2019-04-03 16:03:36 +02:00
enricoottonello b316467608 added common module 2019-04-03 10:53:54 +02:00
Michele Artini 1a1ec7da8e Merge branch 'master' of https://github.com/dnet-team/dnet-hadoop.git 2019-04-01 10:40:59 +02:00
Michele Artini f8c8b6669f ui 2019-04-01 10:40:09 +02:00
Sandro La Bruzzo 0b503949de Merge branch 'master' of https://github.com/dnet-team/dnet-hadoop 2019-03-25 15:19:08 +01:00
Sandro La Bruzzo b2cd76831e added common cli 2019-03-25 15:18:48 +01:00
Sandro La Bruzzo 12c65eab4c implemented command line 2019-03-25 15:18:31 +01:00
Michele Artini b1200b6f46 ui 2019-03-25 11:54:18 +01:00
Michele Artini 6f813fbc8e ui 2019-03-22 15:15:57 +01:00
Michele Artini e2f1013b3d logs and ui 2019-03-22 11:41:00 +01:00
Michele Artini 405f418dfc fix a conflict 
Merge branch 'master' of https://github.com/dnet-team/dnet-hadoop
 2019-03-22 10:24:42 +01:00
Michele Artini 55aa7e1f26 new info 2019-03-22 10:16:47 +01:00
Enrico Ottonello 3b5f291df2 added angular client 2019-03-22 10:13:56 +01:00
Enrico Ottonello 639483090a added swagger comments to input parameters 2019-03-22 10:11:46 +01:00
Enrico Ottonello 042d477878 added versions support 2019-03-20 15:36:23 +01:00
Sandro La Bruzzo 859957d0fd Merge branch 'master' of https://github.com/dnet-team/dnet-hadoop 2019-03-20 09:49:53 +01:00
Enrico Ottonello 46bc3e1f53 added view on mdstore and transaction tables 2019-03-19 13:34:40 +01:00
Enrico Ottonello cefcb38e62 added module dhp-applications 2019-03-18 16:02:23 +01:00
enrico ed484bf24e Merge branch 'master' of https://github.com/dnet-team/dnet-hadoop.git 2019-03-18 15:38:04 +01:00
Enrico Ottonello 126d89ed38 first implementation of dhp-mdstore-manager-app 2019-03-18 15:30:07 +01:00
Michele De Bonis 2af1d61d43 test 2019-03-18 11:36:10 +01:00
Sandro La Bruzzo 6156562893 Added test 2019-03-18 10:47:28 +01:00
Sandro La Bruzzo 49d8cc716e added oozie wf 2019-03-18 10:46:07 +01:00
Sandro La Bruzzo c0da3da4c4 addedd common modules for CollectionBuilder in spark 2019-03-18 10:45:30 +01:00
Sandro La Bruzzo e67d9ee1a9 added first implementation of dnet-workflows 2019-03-18 10:44:35 +01:00
Sandro La Bruzzo 37b84b6afa Added description of the module 2019-03-13 14:53:35 +01:00
luosolo 1eb0281b38 refactored structure of the project 2019-03-13 14:43:20 +01:00
luosolo c10770cd3e added module that allows to collect data into HDFS 2019-03-12 15:40:55 +01:00
luosolo fe949276f0 deleted test to verify if I can push something 2018-01-16 14:31:26 +01:00
luosolo 8e03de306c added test 2018-01-16 14:26:22 +01:00
Claudio Atzori f072ed91b2 first commit 2018-01-16 14:21:13 +01:00
Claudio Atzori 2c27a2762d added common ignores 2018-01-16 14:19:39 +01:00
Claudio Atzori 47c4d75733 added common ignores 2018-01-16 14:17:06 +01:00
Claudio Atzori c877e5031e added common ignores 2018-01-16 14:16:00 +01:00