 Claudio Atzori
|
044d3a0214
|
fixed query used to load datasources in the Graph
|
2020-09-24 13:48:58 +02:00 |
|
Claudio Atzori
|
42f55395c8
|
fixed order of the ISSNs returned by the SQL query
|
2020-09-24 12:09:58 +02:00 |
|
Claudio Atzori
|
9a7e72d528
|
using concat_ws to join textual columns from PSQL. When using || to perform the concatenation, Null columns makes the operation result to be Null
|
2020-09-24 10:42:47 +02:00 |
|
Claudio Atzori
|
9e3e93c6b6
|
setting the correct issn type in the datasource.journal element
|
2020-09-24 10:39:16 +02:00 |
 Miriam Baglioni
|
39eb8ab25b
|
changed the dump to move from h2020programme to h2020classification
|
2020-09-23 17:33:00 +02:00 |
|
Miriam Baglioni
|
e2ceefe9be
|
-
|
2020-09-14 14:33:28 +02:00 |
|
Miriam Baglioni
|
1f893e63dc
|
-
|
2020-09-14 14:33:10 +02:00 |
|
Claudio Atzori
|
8a523474b7
|
code formatting
|
2020-09-07 11:40:16 +02:00 |
|
Miriam Baglioni
|
8694bb9b31
|
refactoring due to compilation
|
2020-08-24 17:07:34 +02:00 |
|
Miriam Baglioni
|
8a069a4fea
|
-
|
2020-08-24 17:01:30 +02:00 |
|
Miriam Baglioni
|
34fa96f3b1
|
-
|
2020-08-24 17:00:20 +02:00 |
|
Miriam Baglioni
|
5fb2949cb8
|
added utils methods
|
2020-08-24 17:00:09 +02:00 |
|
Miriam Baglioni
|
2a540b6c01
|
added constants for the pid graph dump
|
2020-08-24 16:55:35 +02:00 |
|
Miriam Baglioni
|
40c8d2de7b
|
test resources for the dump of the pids graph
|
2020-08-24 16:50:39 +02:00 |
|
Miriam Baglioni
|
bef79d3bdf
|
first attempt to the dump of pids graph
|
2020-08-24 16:49:38 +02:00 |
|
Miriam Baglioni
|
85203c16e3
|
merge branch with master
|
2020-08-19 11:49:03 +02:00 |
|
Miriam Baglioni
|
1c593a9cfe
|
-
|
2020-08-19 11:29:51 +02:00 |
|
Miriam Baglioni
|
e42b2f5ae2
|
-
|
2020-08-19 11:29:09 +02:00 |
|
Miriam Baglioni
|
f81ee22418
|
changed to mirror the changes in the model (Instance, CommunityInstance, GraphResult)
|
2020-08-19 11:28:26 +02:00 |
|
Miriam Baglioni
|
387be43fd4
|
changed to discriminate if dumping all the results type together or each one in its own archive
|
2020-08-19 11:25:27 +02:00 |
|
Miriam Baglioni
|
c5858afb88
|
added parameter to guide the dump for the result (resultAggregation). true if all the result types should be dump together, false otherwise.
|
2020-08-19 11:24:14 +02:00 |
|
Miriam Baglioni
|
5570678c65
|
changed parameter name from hfdsNameNode to nameNode
|
2020-08-19 10:59:26 +02:00 |
|
Miriam Baglioni
|
dc5096a327
|
refactoring due to compilation
|
2020-08-19 10:57:36 +02:00 |
|
Miriam Baglioni
|
09f5b92763
|
added specific reference to class
|
2020-08-14 20:00:09 +02:00 |
|
Miriam Baglioni
|
37e7c43652
|
changed parameter name from hdfsNaemNode to nameNode
|
2020-08-14 18:18:25 +02:00 |
|
Miriam Baglioni
|
a5043de5da
|
added method to get the mapped instance
|
2020-08-13 18:45:50 +02:00 |
|
Miriam Baglioni
|
fcd10f452c
|
changed because of #40 (comment)
|
2020-08-13 12:55:32 +02:00 |
|
Miriam Baglioni
|
bfd1fcde6d
|
removed not useful method and changed because of #40 (comment) and #40 (comment)
|
2020-08-13 12:14:37 +02:00 |
|
Miriam Baglioni
|
7fd8397123
|
apply changes in #40 (comment)
|
2020-08-13 12:13:15 +02:00 |
|
Miriam Baglioni
|
753d448cc9
|
apply changes in #40 (comment)
|
2020-08-13 12:12:58 +02:00 |
|
Miriam Baglioni
|
c0e071fa26
|
apply changes in #40 (comment)
|
2020-08-13 12:12:40 +02:00 |
|
Miriam Baglioni
|
526db915bc
|
apply changes in #40 (comment)
|
2020-08-13 12:12:16 +02:00 |
|
Miriam Baglioni
|
b0fab0d138
|
apply changes in #40 (comment)
|
2020-08-13 12:11:57 +02:00 |
|
Miriam Baglioni
|
1b6320b251
|
apply changes in #40 (comment)
|
2020-08-13 12:11:41 +02:00 |
|
Miriam Baglioni
|
743d31be22
|
apply changes in #40 (comment)
|
2020-08-13 12:11:22 +02:00 |
|
Miriam Baglioni
|
65b48df652
|
apply changes in #40 (comment)
|
2020-08-13 12:11:06 +02:00 |
|
Miriam Baglioni
|
90b54d3efb
|
apply changes in #40 (comment)
|
2020-08-13 12:08:24 +02:00 |
|
Miriam Baglioni
|
69bbb9592a
|
apply changes in #40 (comment)
|
2020-08-13 12:07:39 +02:00 |
|
Miriam Baglioni
|
945323299a
|
apply changes in #40 (comment)
|
2020-08-13 12:07:24 +02:00 |
|
Miriam Baglioni
|
e04c993247
|
apply changes in #40 (comment)
|
2020-08-13 12:07:07 +02:00 |
|
Miriam Baglioni
|
ed0812d0ce
|
apply changes in #40 (comment)
|
2020-08-13 12:06:49 +02:00 |
|
Miriam Baglioni
|
d55cfe0ea5
|
apply changes in #40 (comment)
|
2020-08-13 12:06:20 +02:00 |
|
Miriam Baglioni
|
80866bec7d
|
apply changes in #40 (comment)
|
2020-08-13 12:06:05 +02:00 |
|
Miriam Baglioni
|
1400978c0a
|
apply changes in #40 (comment)
|
2020-08-13 12:05:44 +02:00 |
|
Miriam Baglioni
|
7b941a2e0a
|
apply changes in #40 (comment)
|
2020-08-13 12:05:17 +02:00 |
|
Miriam Baglioni
|
f7474f50fe
|
apply changes in #40 (comment)
|
2020-08-13 12:04:52 +02:00 |
|
Miriam Baglioni
|
367203f412
|
apply changes in #40 (comment)
|
2020-08-13 12:04:33 +02:00 |
|
Miriam Baglioni
|
3ab4809d31
|
apply changes in #40 (comment)
|
2020-08-13 12:04:10 +02:00 |
|
Miriam Baglioni
|
235d4e4d6e
|
moved Context as relevant for Communities dump
|
2020-08-12 18:16:45 +02:00 |
|
Miriam Baglioni
|
7400cd019d
|
removed not needed variable
|
2020-08-12 10:03:33 +02:00 |
|
Miriam Baglioni
|
98d28bab5c
|
fixed missing _ in context nsprefix
|
2020-08-12 10:00:18 +02:00 |
|
Miriam Baglioni
|
2d67476417
|
merge branch with master
|
2020-08-11 15:46:04 +02:00 |
|
Miriam Baglioni
|
0603ec4757
|
changed test to upload the dump for covid-19 community
|
2020-08-11 15:43:25 +02:00 |
|
Miriam Baglioni
|
acb0926b2e
|
json schemas for the dumped entities and relation
|
2020-08-11 15:39:48 +02:00 |
|
Miriam Baglioni
|
ff52c51f92
|
added the communityMapPath parameter and removed the isLookUpUrl parameter
|
2020-08-11 15:39:22 +02:00 |
|
Miriam Baglioni
|
6f43acda5e
|
added the maketar and send to zenodo step. Adjusted wf parameters
|
2020-08-11 15:38:20 +02:00 |
|
Miriam Baglioni
|
ddc19de2e9
|
removed the isLookUpUrl among the parameters
|
2020-08-11 15:37:47 +02:00 |
|
Miriam Baglioni
|
592a8ea573
|
added parameter file for maketar class
|
2020-08-11 15:37:14 +02:00 |
|
Miriam Baglioni
|
77a0951b32
|
added the make archive step in the workflow
|
2020-08-11 15:32:32 +02:00 |
|
Miriam Baglioni
|
cf4d918787
|
added description, changed parameter name and added method
|
2020-08-11 15:27:31 +02:00 |
|
Miriam Baglioni
|
dc5fc5366d
|
Creation of an archive for each related dump part
|
2020-08-11 15:26:06 +02:00 |
|
Miriam Baglioni
|
0ce49049d6
|
added description
|
2020-08-11 15:25:11 +02:00 |
|
Miriam Baglioni
|
9bae991167
|
added description of the class
|
2020-08-11 11:20:43 +02:00 |
|
Miriam Baglioni
|
341dc59ead
|
removed the repartition(1). Added code for the creation of an archive containing all the parts dumped for each community
|
2020-08-11 11:18:58 +02:00 |
|
Miriam Baglioni
|
1991a49f70
|
removed reference to isLookUp to get the communityMap
|
2020-08-10 18:02:56 +02:00 |
|
Miriam Baglioni
|
fe88904df0
|
changed the wf definition
|
2020-08-10 12:01:14 +02:00 |
|
Miriam Baglioni
|
87856467e2
|
removed isLookUpUrl and added code to read from HDSF the communitymap
|
2020-08-10 11:38:41 +02:00 |
|
Miriam Baglioni
|
1cf7043e26
|
removed isLookUoUrl from the parameters
|
2020-08-10 11:38:03 +02:00 |
|
Miriam Baglioni
|
46986aae2d
|
added the new parameter for newdeposion/newversion and concept_record_id
|
2020-08-07 18:00:06 +02:00 |
|
Miriam Baglioni
|
3aedfdf0d6
|
added option to do a new deposition or new version of an old deposition
|
2020-08-07 17:49:14 +02:00 |
|
Miriam Baglioni
|
1b3ad1bce6
|
filter out authors pid (only orcid). Added check to get unique provenance for context id. filtr out countries with code UNKNOWN
|
2020-08-07 17:48:18 +02:00 |
|
Miriam Baglioni
|
5ceb8c5f0a
|
moved constants from graph.Constants
|
2020-08-07 17:46:47 +02:00 |
|
Miriam Baglioni
|
6c65c93c0e
|
refactoring
|
2020-08-07 17:45:35 +02:00 |
|
Miriam Baglioni
|
68adf86fe4
|
refactoring
|
2020-08-07 17:43:20 +02:00 |
|
Miriam Baglioni
|
26d2ad6ebb
|
refactoring
|
2020-08-07 17:41:56 +02:00 |
|
Miriam Baglioni
|
9675af7965
|
refactoring
|
2020-08-07 17:41:07 +02:00 |
|
Miriam Baglioni
|
346a91f4d9
|
Added constants
|
2020-08-07 17:35:39 +02:00 |
|
Miriam Baglioni
|
d52b0e1797
|
no use of IsLookUp. The query is done once and its result stored on HDFS. The path to the result is given instead of the isLookUpUrl
|
2020-08-07 17:34:40 +02:00 |
|
Miriam Baglioni
|
ae1b7fbfdb
|
changed method signature from set of mapkey entries to String representing path on file system where to find the map
|
2020-08-07 17:32:27 +02:00 |
|
Miriam Baglioni
|
545ea9f77e
|
moved in common. Zenodo response model and APIClient to deposit in Zenodo
|
2020-08-07 16:44:51 +02:00 |
|
Miriam Baglioni
|
da9b012c15
|
fixed dewcription
|
2020-08-06 11:55:44 +02:00 |
|
Miriam Baglioni
|
6dbadcf181
|
the new schema for the dumped result
|
2020-08-06 11:05:56 +02:00 |
 Sandro La Bruzzo
|
4fb1821fab
|
Merge branch 'master' of code-repo.d4science.org:D-Net/dnet-hadoop
|
2020-08-06 10:28:31 +02:00 |
|
Sandro La Bruzzo
|
9d9e9edbd2
|
improved extractEntity Relation workflows using dataset
|
2020-08-06 10:28:24 +02:00 |
|
Miriam Baglioni
|
14eda4f46e
|
added method to try to put inputstream to zenodo
|
2020-08-05 14:18:25 +02:00 |
|
Miriam Baglioni
|
e737a47270
|
added classes to try to send input stream to zenodo for the upload
|
2020-08-05 14:17:40 +02:00 |
|
Miriam Baglioni
|
873e9cd50c
|
changed hadoop setting to connect to s3
|
2020-08-04 15:37:25 +02:00 |
 Alessia Bardi
|
a29565ff57
|
code formatting
|
2020-08-04 12:55:27 +02:00 |
|
Alessia Bardi
|
01db29e208
|
fixes redmine issue #5846: datacite and its different namespace declarations
|
2020-08-04 12:53:48 +02:00 |
|
Alessia Bardi
|
b4e4e5f858
|
do not duplicate result PIDs
|
2020-08-04 12:52:14 +02:00 |
|
Miriam Baglioni
|
5b651abf82
|
merge branch with master
|
2020-08-04 10:14:07 +02:00 |
|
Miriam Baglioni
|
901ae37f7b
|
added step to workflow
|
2020-08-03 18:12:54 +02:00 |
|
Miriam Baglioni
|
e43aeb139a
|
added new property file and changed some parameter to old files
|
2020-08-03 18:07:28 +02:00 |
|
Miriam Baglioni
|
aa9f3d9698
|
changed logic for save in s3 directly
|
2020-08-03 18:06:18 +02:00 |
|
Miriam Baglioni
|
d465f0eec9
|
added fulltext to result
|
2020-08-03 18:03:27 +02:00 |
|
Miriam Baglioni
|
c892c7dfa7
|
changed to query for community map just once and save the result for remaining executions
|
2020-08-03 17:56:31 +02:00 |
 Michele Artini
|
652b13abb6
|
Merge branch 'master' into nsprefix_blacklist
|
2020-07-31 07:58:37 +02:00 |
|
Claudio Atzori
|
cd631bb5bc
|
defaults fixed in the cleaning workflow forces result.publisher to NULL when result.publisher.value in empty
|
2020-07-30 17:03:53 +02:00 |
|
Miriam Baglioni
|
57c87b7653
|
re-implemented to fix issue on not serializable Set<String> variable
|
2020-07-30 16:43:43 +02:00 |
|
Miriam Baglioni
|
ef8e5957b5
|
added specific directory where to save results
|
2020-07-30 16:42:46 +02:00 |
|
Miriam Baglioni
|
75f3361c85
|
-
|
2020-07-30 16:41:31 +02:00 |
|
Miriam Baglioni
|
3f695b25fa
|
refactoring
|
2020-07-30 16:40:15 +02:00 |
|
Miriam Baglioni
|
e623f12bef
|
refactoring
|
2020-07-30 16:32:59 +02:00 |
|
Miriam Baglioni
|
ff7d05abb4
|
added support class to store the couple organizationId representativeId gaot from sql query on hive
|
2020-07-30 16:32:04 +02:00 |
|
Miriam Baglioni
|
cf6d80b2ab
|
added command to close the writer
|
2020-07-30 16:31:22 +02:00 |
|
Miriam Baglioni
|
f985bca37b
|
added USER_CLAIM constant value
|
2020-07-30 16:25:26 +02:00 |
|
Claudio Atzori
|
4bbfcf1ac6
|
Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop
|
2020-07-30 16:25:06 +02:00 |
|
Claudio Atzori
|
4ff8007518
|
added function to set the missing vocabulary names, used in the cleaning workflow as a pre-cleaning step
|
2020-07-30 16:24:39 +02:00 |
|
Miriam Baglioni
|
6f1c40a933
|
-
|
2020-07-30 16:24:28 +02:00 |
|
Miriam Baglioni
|
2b66a93f9e
|
added property file that was missing
|
2020-07-30 16:24:17 +02:00 |
|
Michele Artini
|
bdece15ca0
|
blacklist of nsprefix
|
2020-07-30 16:13:38 +02:00 |
Sandro La Bruzzo
|
c97c8f0c44
|
implemented new oozie job to extract entities in a separate dataset
|
2020-07-30 12:13:58 +02:00 |
|
Sandro La Bruzzo
|
3010a362bc
|
updated changing in the workflow of provision in the phase of aggregation. Removed serialization in JSON RDD and used spark Dataset
|
2020-07-30 09:25:56 +02:00 |
|
Sandro La Bruzzo
|
16ae3c9ccf
|
updated changing in the workflow of provision in the phase of aggregation. Removed serialization in JSON RDD and used spark Dataset
|
2020-07-30 09:25:32 +02:00 |
|
Miriam Baglioni
|
76bcab98ce
|
added code to filter out null originalId from the dump
|
2020-07-29 18:28:21 +02:00 |
|
Miriam Baglioni
|
86bab79512
|
-
|
2020-07-29 18:20:22 +02:00 |
|
Miriam Baglioni
|
31791dcf3d
|
fixed wrong property file path name
|
2020-07-29 18:20:08 +02:00 |
|
Miriam Baglioni
|
9e722aa1ef
|
-
|
2020-07-29 18:00:08 +02:00 |
|
Miriam Baglioni
|
d22f106f27
|
added constant to identify datasource associated to funders
|
2020-07-29 17:56:55 +02:00 |
|
Miriam Baglioni
|
40e194fe2f
|
added check to not dump datasources related to funders
|
2020-07-29 17:56:18 +02:00 |
|
Miriam Baglioni
|
b48934f6df
|
changed the workflow name
|
2020-07-29 17:43:43 +02:00 |
|
Miriam Baglioni
|
074e9ab75e
|
refactoring
|
2020-07-29 17:42:50 +02:00 |
|
Miriam Baglioni
|
8ad8dac7d4
|
merge branch with fork master
|
2020-07-29 17:38:28 +02:00 |
|
Miriam Baglioni
|
9fa82dc93b
|
fixed issue
|
2020-07-29 17:36:16 +02:00 |
|
Miriam Baglioni
|
8907648d6a
|
-
|
2020-07-29 17:35:47 +02:00 |
|
Miriam Baglioni
|
40a8dafbdc
|
-
|
2020-07-29 17:30:44 +02:00 |
|
Miriam Baglioni
|
6d0f08277b
|
classes to implement the dump of the whole graph.
|
2020-07-29 17:03:19 +02:00 |
|
Miriam Baglioni
|
8d4327b292
|
input parameters and workflow definition for the dump of the whole graph
|
2020-07-29 17:00:34 +02:00 |
|
Miriam Baglioni
|
b5f995ab12
|
refactoring
|
2020-07-29 16:59:48 +02:00 |
|
Miriam Baglioni
|
f7a87cc447
|
added new constants value
|
2020-07-29 16:58:40 +02:00 |
|
Miriam Baglioni
|
b71d12cf26
|
refactoring
|
2020-07-29 16:52:44 +02:00 |
|
Miriam Baglioni
|
a8d65b68cb
|
changed to delete the part to check if it was a test or a real execution
|
2020-07-29 16:47:57 +02:00 |
|
Miriam Baglioni
|
3ec2392904
|
Added new class to move the place the split is effectively run
|
2020-07-29 16:46:50 +02:00 |
|
Miriam Baglioni
|
178c2729a7
|
changed the path to reach the java class to be executed
|
2020-07-29 12:29:51 +02:00 |
|
Miriam Baglioni
|
437ac12139
|
removed unused parameter
|
2020-07-29 12:28:16 +02:00 |
|
Miriam Baglioni
|
6c2223d1fc
|
added code to get the openaire id for contexts
|
2020-07-24 17:30:15 +02:00 |
|
Miriam Baglioni
|
afd54c1684
|
removed not needed upload and refactoring
|
2020-07-24 17:28:56 +02:00 |
|
Miriam Baglioni
|
7b0569d989
|
changed to map also the result associated to the whole graph
|
2020-07-24 17:28:11 +02:00 |
|
Miriam Baglioni
|
082225ad61
|
-
|
2020-07-24 17:27:26 +02:00 |
|
Miriam Baglioni
|
968c59d97a
|
added teh logic to dump also the products for the whole graph. They will miss collected from and context information that will be materialized as new relations
|
2020-07-24 17:25:19 +02:00 |
|
Miriam Baglioni
|
332258d199
|
split the classes related to the communities dump and to the whole graph dump
|
2020-07-24 17:21:48 +02:00 |
|
Claudio Atzori
|
56bbfdc65d
|
introduced parameter 'numParitions', driving the hive DB table data partitioning. Currently specified only for table 'project'
|
2020-07-23 08:54:10 +02:00 |
|
Claudio Atzori
|
ebf60020ac
|
map results as OPRs in case of missing //CobjCategory/@type and the vocabulary dnet:result_typologies doesn't resolve the super type
|
2020-07-20 19:01:10 +02:00 |
|
Miriam Baglioni
|
355d7e426e
|
added dumo for project - not finished
|
2020-07-20 18:54:43 +02:00 |
|
Miriam Baglioni
|
a2f01e5259
|
added getter and setter
|
2020-07-20 18:54:17 +02:00 |
|
Miriam Baglioni
|
40bbe94f7c
|
merge with master fork
|
2020-07-20 18:10:03 +02:00 |
|
Miriam Baglioni
|
23160b4d29
|
realignment of the workflow classes with the changes in the structure of the module
|
2020-07-20 18:04:30 +02:00 |
|
Miriam Baglioni
|
08dbd99455
|
changed to dump the whole results graph by usign classes already implemented for communities. Added class to dump also organization
|
2020-07-20 17:54:28 +02:00 |
|
Miriam Baglioni
|
e47ea9349c
|
extended some types by adding provenance as the couple (provenance, trust) and moved some classes to be used by the complete graph dump also
|
2020-07-20 17:46:27 +02:00 |
|
Claudio Atzori
|
32f5e466e3
|
imports cleanup
|
2020-07-20 17:42:58 +02:00 |