przemyslaw.jacewicz
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
1,312 Commits 16 Branches 0 Tags 76 MiB
Commit Graph

217 Commits

Author SHA1 Message Date
Claudio Atzori 1767dfaa3f method can be protected, it is meant to be used only in tests 2020-03-27 14:31:26 +01:00
Sandro La Bruzzo 15d9106b3f FIxed merge of dhp dedup 2020-03-27 13:48:44 +01:00
Sandro La Bruzzo 8c9a56a0c8 refactored package name 2020-03-27 13:19:33 +01:00
Sandro La Bruzzo a9935f80d4 refactor class name and workflow name for graph mapper, added javadoc 2020-03-27 13:16:24 +01:00
Claudio Atzori 673e744649 moved openaire specific implementations under dedicated package eu.dnetlib.dhp.oa 2020-03-27 10:42:17 +01:00
Claudio Atzori 098fabab3f reorganizing content under dhp-workflows/dhp-graph-mapper 2020-03-26 19:44:19 +01:00
Claudio Atzori 77c4294924 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2020-03-26 18:26:52 +01:00
Claudio Atzori 43cbcda7ef unit test for SparkGraphImporterJob 2020-03-26 18:26:40 +01:00
Sandro La Bruzzo 0cd022ad6a merge with master 2020-03-26 14:08:29 +01:00
Claudio Atzori 2180cc4fe7 more fields included in result view definition 2020-03-25 11:21:46 +01:00
Claudio Atzori 8b0ba3d76a posprocessing script correctly run as hive2 action 2020-03-23 17:40:39 +01:00
Claudio Atzori 658d40ccbe WIP trying to use hive2 actions 2020-03-23 11:14:54 +01:00
Sandro La Bruzzo 0594b92a6d implemented relation with dataset 2020-03-19 11:11:07 +01:00
Claudio Atzori abe8fb69a2 added global properties, moved postprocessing script inside the oozie_app directory 2020-03-18 15:43:54 +01:00
Claudio Atzori 8fe7ae1482 xml formatting 2020-03-13 15:53:56 +01:00
Sandro La Bruzzo addaaa091f migrate relation from RDD to Dataset 2020-03-13 09:13:20 +01:00
Claudio Atzori 7b6f0c8756 reading graph dump as text files, encoded as newline-delimited JSON records, as indicated in the wiki 2020-03-10 17:19:17 +01:00
Claudio Atzori 0233987603 introduced post processing step following the hive DB creation/population 2020-03-04 10:56:50 +01:00
Claudio Atzori 9af3e904be close the SparkSession at the end 2020-03-04 10:53:31 +01:00
Claudio Atzori 25ceec29ab code formatting 2020-03-04 10:44:24 +01:00
Claudio Atzori 60bc2b1a20 drop the hive DB before populating it from scratch 2020-02-27 10:10:55 +01:00
Sandro La Bruzzo 2b8675462f refactoring code 2020-02-19 10:07:08 +01:00
Claudio Atzori 1b18fd4d54 sync with master branch 2020-02-17 13:49:46 +01:00
Sandro La Bruzzo 76ee85141a added oozie job for DNET migration and implemented Spark job for extracting entities 2020-02-17 12:31:44 +01:00
Claudio Atzori 1fee6e2b7e implemented XML records construction and serialization, indexing WIP 2020-02-13 16:53:27 +01:00
Sandro La Bruzzo 19a80e4638 implemented workfow for aggregation and generation of infospace graph 2020-01-24 09:58:55 +01:00
Michele Artini b35c59eb42 partial implementation of entities from db 2020-01-20 16:04:19 +01:00
Sandro La Bruzzo abd9034da0 implemented DedupRecord factory with the merge of publications 2019-12-11 15:43:24 +01:00
miconis 4b66b471a4 implementation of the sorting by trust mechanism and the merge of oaf entities 2019-12-10 14:57:16 +01:00
Claudio Atzori 245b4cbbb3 removed import limit 2019-11-08 17:41:01 +01:00
Claudio Atzori 5308f05a02 allow to speficy the target hive DB name in the infospace import workflow 2019-11-07 17:38:09 +01:00
Claudio Atzori a52d5bde4f simplified import procedure, maps the infospace as hive tables 2019-11-06 17:45:52 +01:00
Claudio Atzori 1e7a2ac41d align parmeter names, graph import procedure WIP 2019-11-04 17:41:01 +01:00
Claudio Atzori 32ed4ae8d6 conversion utilities from protobuffer model to DHP model moved in dnet-mapreduce-jobs. Removed also the relative protobuf dependencies 2019-11-04 12:28:56 +01:00
Sandro La Bruzzo 997e57d45b Added entity filter to spark class 2019-10-30 12:19:03 +01:00
Sandro La Bruzzo a336956708 added defautl property to job 2019-10-30 12:01:42 +01:00
Claudio Atzori 78b5b57e86 trying to make the spark action to be run as spark2 2019-10-29 18:56:34 +01:00
Claudio Atzori c8bb81cd9a align dependencies with IIS cluster 2019-10-29 18:10:20 +01:00
Sandro La Bruzzo fe62ccd6dd implemented oozie wf 2019-10-28 12:12:50 +01:00
Sandro La Bruzzo 9ee4e5a196 remove a bit of syntactic sugar on the object inheritance :( 2019-10-25 18:10:30 +02:00
miconis 9fa5aebe9c minor changes 2019-10-25 12:52:28 +02:00
miconis 551eda1600 dataset, orp and software mapping implemented. addition of test resources for results. implementation of tests to check the result of the mapping 2019-10-25 12:48:25 +02:00
Sandro La Bruzzo eef14fade3 fixed conflict 2019-10-25 11:58:20 +02:00
Sandro La Bruzzo 0ea7e861ab added organizations test 2019-10-25 11:56:28 +02:00
miconis 4908165e05 implementation of the createPublication method to map publications 2019-10-25 11:54:14 +02:00
miconis df37bd6aaf placeholders for setters in createpublication 2019-10-25 10:57:19 +02:00
Sandro La Bruzzo c8d6d6bbd1 implemented organization mapping 2019-10-25 10:23:51 +02:00
miconis b525b54130 starting implementing the createPublication class 2019-10-25 09:55:31 +02:00
Claudio Atzori 4b331790e7 resolved conflicts 2019-10-25 09:45:12 +02:00
Claudio Atzori c929c1dfac more proto 2 graph model mappings 2019-10-25 09:25:36 +02:00
Sandro La Bruzzo 09ffda03a2 removed circular dependencies 2019-10-25 09:24:18 +02:00
Sandro La Bruzzo a10d071cf4 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:55:44 +02:00
Sandro La Bruzzo 3a8bb11695 mapped first part 2019-10-24 17:55:40 +02:00
Claudio Atzori d46371ceab Merge branch 'master' of https://code-repo.d2science.org/D-Net/dnet-hadoop 2019-10-24 17:43:55 +02:00
Claudio Atzori 0d88f9a6a4 added mapping for projects 2019-10-24 17:43:42 +02:00
Sandro La Bruzzo 2dd9572f41 added Mapping of OriginalDescription 2019-10-24 17:36:44 +02:00
miconis 351d850ad3 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:29:07 +02:00
miconis b66a7e3030 publication test added 2019-10-24 17:29:01 +02:00
Sandro La Bruzzo 6c32d418ac added conversion of ExtraInfo 2019-10-24 17:26:55 +02:00
Claudio Atzori 5f339a2c24 added mappings for basic types 2019-10-24 17:21:45 +02:00
Sandro La Bruzzo 9d04111391 Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop 2019-10-24 17:05:52 +02:00
Sandro La Bruzzo 0902bac7dd fixed conflict 2019-10-24 17:05:42 +02:00
Claudio Atzori d8bfaa3687 added mapping for relations 2019-10-24 17:04:13 +02:00
Sandro La Bruzzo d2965636e0 created test for convert json into new OAF data model 2019-10-24 17:02:35 +02:00
Claudio Atzori 79c4f1bbd8 Protobuf to internal graph model, early steps 2019-10-24 16:56:13 +02:00
Claudio Atzori d38aeb8c6e DataInfo.provenanceaction not repeatable, fluent setters 2019-10-24 16:55:38 +02:00
Sandro La Bruzzo 5744a64478 added module dhp=graph-mapper 2019-10-24 16:00:28 +02:00