przemyslaw.jacewicz
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
957 Commits 16 Branches 0 Tags 76 MiB
Commit Graph

833 Commits

Author SHA1 Message Date
Enrico Ottonello d4e9075f22 Merge branch 'doiboost' of https://code-repo.d4science.org/D-Net/dnet-hadoop into doiboost 2020-05-18 19:51:36 +02:00
Enrico Ottonello fc80e8c7de added accumulator; last modified date of the record is added to saved data; lambda file is partitioned into 20 parts before starting downloading 2020-05-18 19:51:29 +02:00
Claudio Atzori f3bc8aed31 lifted memory requirements for country propagation wf 2020-05-18 15:29:10 +02:00
Miriam Baglioni b71fbb68b1 removed the removeOutputDir command from code. Reltions are written in Append. The erase of the output dir ment to remove all the relations computed in the prevoius steps 2020-05-18 13:57:20 +02:00
Miriam Baglioni 629af7cb79 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-05-18 13:07:36 +02:00
Claudio Atzori ef9a9a9f1a remove the outout path when starting 2020-05-15 22:34:19 +02:00
Enrico Ottonello 0b29bb7e3b spark job to download orcid record modified after a fixed date 2020-05-15 19:49:26 +02:00
Claudio Atzori 7838f2c63f init the empty list for author pids mapped from OAF 2020-05-15 17:06:01 +02:00
Claudio Atzori 82b615ab33 NPE check 2020-05-15 16:04:46 +02:00
Claudio Atzori 7a89507ab1 code formatting 2020-05-15 15:16:54 +02:00
Miriam Baglioni 5ec8c49ad5 removed serialization points 2020-05-15 12:49:58 +02:00
Claudio Atzori 1d35836a58 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-05-15 12:26:31 +02:00
Claudio Atzori cfc8948717 fixed mapping OdfToGraph: pick the correct element to map author pids and author affiliations; extended mapping Oaf2Graph: added support for author pids 2020-05-15 12:26:16 +02:00
Michele Artini 2a4e68a292 events recognition 2020-05-15 12:25:37 +02:00
Claudio Atzori a832658296 code formatting 2020-05-15 10:21:09 +02:00
Claudio Atzori 50d6a2ad3c added output directory removal in the blacklist spark actions; included common global properties in blacklist's workflow.xml 2020-05-15 09:53:37 +02:00
Claudio Atzori 18f46e47b9 added relations to the graph2hive import workflow 2020-05-15 09:34:48 +02:00
Claudio Atzori 9d028ffe1c cleanup 2020-05-15 09:28:55 +02:00
Claudio Atzori fd62359538 cleanup 2020-05-15 09:28:15 +02:00
Claudio Atzori eb64335a54 parallel implementation for graph Hive importer 2020-05-15 09:05:26 +02:00
Miriam Baglioni 94571c9a51 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-05-14 18:29:55 +02:00
Miriam Baglioni f25db01664 changed in the constant from propagationconstants to modelconstants 2020-05-14 18:29:24 +02:00
Miriam Baglioni d05630d979 removed the constants added in ModelConstants 2020-05-14 18:22:50 +02:00
Claudio Atzori f044d09315 revised mapping: more accurate mapping for name/surname from datacite format; improved mapping of null values 2020-05-14 15:07:24 +02:00
Miriam Baglioni e7eb4f377e Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2020-05-14 10:34:17 +02:00
Miriam Baglioni 8828458acf minor changes 2020-05-14 10:34:12 +02:00
Claudio Atzori ab37953332 added global properties in wf definitions to avoid repeating name-node and job-tracker in the (many) distcp actions; reintroduced output directory removal at the beginning of each spark action 2020-05-14 10:25:41 +02:00
Claudio Atzori 12bfa6702e Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-05-13 17:01:17 +02:00
Claudio Atzori 5ecacad70a fixed default resource typing in Oaf/Odf mapping 2020-05-13 17:01:11 +02:00
Enrico Ottonello 12756f9d41 multithread (4 threads) test to feed elastic search 2020-05-13 16:11:40 +02:00
Michele Artini c0265213a0 partial implementation 2020-05-13 12:00:27 +02:00
Sandro La Bruzzo a92ee0f41e Merge remote-tracking branch 'origin/master' into doiboost 2020-05-13 10:38:13 +02:00
Sandro La Bruzzo d876f47d06 next step of MAG conversion implemented 2020-05-13 10:38:04 +02:00
Claudio Atzori 1ddd33de41 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-05-13 09:04:41 +02:00
Claudio Atzori 85f3c55992 fixed node names in blacklist workflow 2020-05-13 09:04:33 +02:00
Miriam Baglioni 43f127448d changed the package name from dhp-propagation to dhp-enrichment for the preparation phase of funding propagation 2020-05-12 18:24:26 +02:00
Enrico Ottonello 08040cef80 spark action to analyze orcid lambda file 2020-05-12 16:57:43 +02:00
Claudio Atzori ec0782e582 renamed jar containing the bulktagging and propagation workflows from dhp-[bulktagging|propagation] to dhp-enrichment; adjusted xml formatting 2020-05-12 15:49:28 +02:00
Miriam Baglioni 1547ca7e15 added blacklist step to the end of the provision wf 2020-05-12 12:17:27 +02:00
Miriam Baglioni 14979f299e changed the configuration factory 2020-05-12 11:28:38 +02:00
Miriam Baglioni f8aef6161a minor modification 2020-05-12 11:28:07 +02:00
Miriam Baglioni 7387f3449a changed the route to find the verb resolver classes 2020-05-12 11:27:38 +02:00
Miriam Baglioni 7687519f00 merged conflicts with upstream branch 2020-05-12 10:03:44 +02:00
Miriam Baglioni 8ffc050b8a fixed problem in communityconfigurationfactory test 2020-05-12 10:01:09 +02:00
Claudio Atzori 527e8169a8 adjusted paths pointing to test configurations, cleanup 2020-05-11 18:17:05 +02:00
Claudio Atzori f9a62ba63b added wf nodes to copy entities to the output path 2020-05-11 18:16:39 +02:00
Miriam Baglioni ad63effb4e removed deletion of working dir 2020-05-11 17:48:22 +02:00
Claudio Atzori c6b028f2af code formatting 2020-05-11 17:38:08 +02:00
Claudio Atzori 6d0b11252e bulktagging wfs moved into common dhp-enrichment module 2020-05-11 17:32:06 +02:00
Enrico Ottonello 3b1a68cbf5 elastic search feed test 2020-05-11 14:53:52 +02:00
Enrico Ottonello f53e42bda7 merged 2020-05-11 14:49:28 +02:00
Enrico Ottonello 7990894454 different date format in lambda file parsing 2020-05-11 14:41:11 +02:00
Sandro La Bruzzo 0c6774e4da updated pom version 2020-05-11 14:35:14 +02:00
Miriam Baglioni bbc9b4f329 removed unused imports 2020-05-11 14:28:55 +02:00
Miriam Baglioni 757bae53ea removed unusefule serialization points 2020-05-11 14:28:37 +02:00
Miriam Baglioni b35d57a1ac added resources for test 2020-05-11 14:15:30 +02:00
Miriam Baglioni e563e65335 moved check from join to method 2020-05-11 14:11:44 +02:00
Sandro La Bruzzo b90609848b Merge remote-tracking branch 'origin/master' into doiboost 2020-05-11 14:08:31 +02:00
Sandro La Bruzzo 4062eafbdb merged from branch 2020-05-11 14:08:16 +02:00
Miriam Baglioni f5d785e096 used the DbClient moved in dhp-common 2020-05-11 13:59:42 +02:00
Miriam Baglioni 112b2cb3c3 added the test class 2020-05-11 13:58:58 +02:00
Miriam Baglioni 9a7ae523c9 update to version 1.2.1-SNAPSHOT 2020-05-11 13:57:47 +02:00
Miriam Baglioni 2abb84877d Merge branch 'master' into blacklist 2020-05-11 10:37:49 +02:00
Miriam Baglioni b0f0b24263 update to version 1.2.1-SNAPSHOT 2020-05-11 10:37:31 +02:00
Miriam Baglioni a7e91e23ba update to versione 1.2.1-SNAPSHOT 2020-05-11 10:34:30 +02:00
Miriam Baglioni bb59bdd60f merge upstream 2020-05-11 10:33:17 +02:00
Miriam Baglioni 5e3548add6 - 2020-05-11 10:33:08 +02:00
Miriam Baglioni dc8c8fa480 changed the version 2020-05-11 10:20:48 +02:00
Miriam Baglioni 871e079b45 merged with master 2020-05-11 10:20:00 +02:00
Claudio Atzori 60c40618d3 [maven-release-plugin] prepare for next development iteration 2020-05-11 10:17:14 +02:00
Claudio Atzori c267d958d5 [maven-release-plugin] prepare release dhp-1.2.0 2020-05-11 10:17:10 +02:00
Miriam Baglioni 622ba87ec2 changed the version 2020-05-11 10:10:36 +02:00
Miriam Baglioni 391b2399cc merge upstream 2020-05-11 10:08:51 +02:00
Claudio Atzori 42f1a2bf94 bumped project version to 1.2.0-SNAPSHOT 2020-05-11 10:05:57 +02:00
Sandro La Bruzzo 1412158a6f merged from branch 2020-05-11 09:45:50 +02:00
Miriam Baglioni 32301451ec merge upstream 2020-05-11 09:42:23 +02:00
Miriam Baglioni 7e66bc2527 fix a typo in the compression keyword and added some logging info in the spark job 2020-05-11 09:40:58 +02:00
Sandro La Bruzzo 1662f221f5 added test class 2020-05-11 09:39:11 +02:00
Sandro La Bruzzo 2b48a2c32c Merge branch 'doiboost' of code-repo.d4science.org:D-Net/dnet-hadoop into doiboost 2020-05-11 09:38:36 +02:00
Sandro La Bruzzo 4cebca09d2 start implementing MAG mapping 2020-05-11 09:38:27 +02:00
Spyros Zoupanos ae0f535c73 Fixing hardcoded reference to main openAIRE graph db 2020-05-09 22:34:48 +03:00
Claudio Atzori fd519df616 new rels produced by dedup workflow must be unique 2020-05-08 19:00:38 +02:00
Claudio Atzori 0ccc864ad9 [maven-release-plugin] prepare for next development iteration 2020-05-08 17:01:31 +02:00
Claudio Atzori 6e47c724c6 [maven-release-plugin] prepare release dhp-1.1.7 2020-05-08 17:01:27 +02:00
Claudio Atzori 5b28bb4131 code formatting 2020-05-08 16:49:47 +02:00
Claudio Atzori 8fd1952f16 code formatting 2020-05-08 16:01:09 +02:00
miconis 3420998bb4 reltype set in mergerels 2020-05-08 15:43:30 +02:00
Enrico Ottonello b9d126dd1f formatting modified after commit 2020-05-08 14:54:37 +02:00
Enrico Ottonello 7e1c987370 Merge branch 'doiboost' of https://code-repo.d4science.org/D-Net/dnet-hadoop into doiboost 2020-05-08 14:49:50 +02:00
Enrico Ottonello 9d812788e4 added job to download from orcid the records modified after a fixed date, the info are taken from last_modified.csv on hdfs 2020-05-08 14:49:39 +02:00
Miriam Baglioni 9a29ab7508 got back to the readPath we have before 2020-05-08 13:08:56 +02:00
Miriam Baglioni 28556507e7 - 2020-05-08 12:54:52 +02:00
Claudio Atzori b2192fdcdc simplified reset_outputpath nodes across the workflows, applied common xml formatting 2020-05-08 12:33:31 +02:00
Miriam Baglioni 4c94231cad merge with master fork 2020-05-08 12:25:57 +02:00
Miriam Baglioni 9b4c0d4b3a - 2020-05-08 11:51:45 +02:00
Miriam Baglioni 53952707b6 modified test because of new step of data preparation. It now expects to find ResultCountrySet serialization nstead of DatasourceCountry 2020-05-08 11:49:19 +02:00
Claudio Atzori 62ea19f1d3 introduced mapping for ExternalReferences, made urls defined within an instance unique 2020-05-08 09:43:26 +02:00
Claudio Atzori 8c67073a07 force speculative execution to false 2020-05-08 09:42:21 +02:00
Miriam Baglioni d6b9de9f46 Merge branch 'master' of https://code-repo.d4science.org/miriam.baglioni/dnet-hadoop 2020-05-07 18:22:59 +02:00
Miriam Baglioni f95d288681 fixed swithch of parameters 2020-05-07 18:22:32 +02:00