D-Net
/
dnet-hadoop
13
4
Fork 10
Code Issues 14 Pull Requests 8 Projects 2 Releases 1 Wiki Activity
3,704 Commits 108 Branches 18 Tags 647 MiB
Commit Graph

3226 Commits

Author SHA1 Message Date
Serafeim Chatzopoulos d0b84d3297 Add FileCollectorPlugin and respective test 2022-04-07 15:06:38 +03:00
Serafeim Chatzopoulos bc1bf55507 Add AbstractSplittedRecordPlugin 2022-04-07 14:33:04 +03:00
Serafeim Chatzopoulos e612489670 Add fileGZip collector plugin and respective test 2022-04-06 19:12:44 +03:00
Claudio Atzori 4190c9f6bc [graph raw] avoid NPEs importing datasource consent fields 2022-04-06 15:34:31 +02:00
Miriam Baglioni e77d104951 [OC] added / to workflow path 2022-04-05 15:07:11 +02:00
Claudio Atzori 873369af1c Merge pull request '[stats wf] added apcs in monitor db' (#207) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: #207
 2022-03-29 15:40:20 +02:00
Antonis Lempesis 7112806a73 views cannot be stored as parquet... 2022-03-29 16:37:29 +03:00
Antonis Lempesis fff0b3cc19 added apcs in monitor db 2022-03-29 14:15:31 +03:00
Claudio Atzori de85367695 Merge pull request '[stats wf] fix: views cannot be stored as parquet...' (#206) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: #206
 2022-03-29 12:51:02 +02:00
Antonis Lempesis ee24f3eb2c views cannot be stored as parquet... 2022-03-29 13:47:48 +03:00
Sandro La Bruzzo 1b11010169 minor fix 2022-03-29 10:59:14 +02:00
Claudio Atzori 0a0ae84c22 [graph raw] DOI based instance URLs on https 2022-03-29 10:52:58 +02:00
Claudio Atzori 9fa3dd78fe Merge pull request '[stats wf] various fixes, organization ids for inst. dashboard' (#205) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: #205
 2022-03-28 22:03:49 +02:00
Claudio Atzori 96aa2a5d0d Merge branch 'beta' into instance_group_by_url 2022-03-28 09:23:52 +02:00
Claudio Atzori 741bc99c47 Merge branch 'beta' into datasource_pdf_consent 2022-03-28 09:20:48 +02:00
Claudio Atzori 61319b2e83 updated dhp-schema version; set entity-level dataInfo before & after merging the fields from the group of duplicates 2022-03-25 16:38:33 +01:00
Antonis Lempesis d8503cd191 added moooar organizations 2022-03-24 14:02:36 +02:00
Miriam Baglioni 7b8f85692e [Enrichment country] fixed issues with parameters and workflow args 2022-03-23 17:20:23 +01:00
Claudio Atzori 48d32466e4 instances grouped by URL expose only one refereed 2022-03-23 14:52:03 +01:00
Claudio Atzori f10066547b increased spark.sql.shuffle.partitions in affiliation_from_semrel_propagation 2022-03-23 12:22:26 +01:00
Claudio Atzori 43733c1a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-03-23 12:14:27 +01:00
Antonis Lempesis 62f91b0869 cleanup 2022-03-22 16:17:49 +02:00
Antonis Lempesis 2e8394ecf8 creating aaall tables as parquet 2022-03-22 16:16:08 +02:00
Antonis Lempesis dcfbeb8142 yet more typos 2022-03-21 12:36:03 +02:00
Miriam Baglioni 89fd275480 [HostedByMap] added left over from PR and fixed issue on workflow 2022-03-21 09:54:45 +01:00
miconis c959639bd5 dependency updated to the new pace-core version 2022-03-15 16:33:03 +01:00
Claudio Atzori f430029596 cleanup 2022-03-11 14:28:28 +01:00
Miriam Baglioni 12de9acb0d [Country Propagation] left out from previous commit 2022-03-11 14:17:02 +01:00
Miriam Baglioni 2fbb35ade5 mergin with branch beta 2022-03-11 13:58:10 +01:00
Miriam Baglioni 4437f9345d [Country Propagation] left out from previous commit 2022-03-11 13:57:47 +01:00
Miriam Baglioni 2b643059fa [Country Propagation] changed the logic to get the collectedfrom at the result level. To fix issue when no instance is created for a result that should have the country associated. Change the code to use spark instead of hive to prepare the data needed for the propagation step. Added new tests for the intermediate steps and new verification for the propagation itself 2022-03-11 13:56:48 +01:00
Claudio Atzori f25407bbe2 added mapping for datasource consent fields to integrate them in the graph 2022-03-11 09:32:42 +01:00
Miriam Baglioni 2c5087d55a [HostedByMap] download of doaj from json, modification of test resources, deletion of class no more needed for the CSV download 2022-03-04 15:18:21 +01:00
Miriam Baglioni 5d608d6291 [HostedByMap] changed the model to include also oaStart date and review process that could be possibly used in the future 2022-03-04 11:06:09 +01:00
Miriam Baglioni b7c2340952 [HostedByMap - DOIBoost] changed to use code moved to common since used also from hostedbymap now 2022-03-04 11:05:23 +01:00
Miriam Baglioni 8a41f63348 [HostedByMap] update to download the json instead of the csv 2022-03-04 10:38:43 +01:00
Miriam Baglioni 44b0c03080 [HostedByMap] update to download the json instead of the csv 2022-03-04 10:37:59 +01:00
Antonis Lempesis ad78e505da yet another fix 2022-03-03 12:28:12 +02:00
Miriam Baglioni 3970651ee1 Merge pull request 'fixed query after the change in the indicator table' (#200) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: #200
 2022-03-02 14:05:58 +01:00
Antonis Lempesis efeeebfee1 fixed query after the change in the indicator table 2022-03-02 13:29:25 +02:00
Claudio Atzori 1932a65d1c Merge pull request '[Stats wf] sprint 6 indicators' (#198) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: #198
 2022-02-25 12:09:18 +01:00
Miriam Baglioni f5b0a6f89c [master to beta] fixed issues in test files 2022-02-25 10:21:57 +01:00
miconis 8991d097b4 bug fix in the DedupRecordFactory, DataInfo set before merge 2022-02-24 17:13:12 +01:00
Claudio Atzori a87c070447 conflicts resolved, merged from beta 2022-02-24 12:51:31 +01:00
Claudio Atzori 86cdb7a38f [provision] serialize measures defined on the result level 2022-02-23 15:54:18 +01:00
Alessia Bardi 9d6203f79b test mapping datasource 2022-02-23 15:00:53 +01:00
Antonis Lempesis 3b92a2ab9c added the rest of spring 6 in monitor db 2022-02-23 12:05:57 +02:00
Antonis Lempesis 87c91f70a2 added sprint 6 indicators to monitor db 2022-02-22 14:41:48 +02:00
Claudio Atzori 5226d0a100 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-02-18 15:21:07 +01:00
Claudio Atzori 99f5b14469 [graph raw] invisible records stored among the raw graph rather than the claimed subgraph 2022-02-18 15:20:57 +01:00