sabeel
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
3,884 Commits 55 Branches 18 Tags 649 MiB
Commit Graph

3386 Commits

Author SHA1 Message Date
Claudio Atzori 30105f0722 Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-04-22 11:22:21 +02:00
Sandro La Bruzzo a82ec3aaaf code formatter 2022-04-22 11:08:13 +02:00
Sandro La Bruzzo aa12429f50 Modified last intersection since we lost many titles. 2022-04-22 11:05:08 +02:00
Miriam Baglioni 7cb7066472 [EoscTag] first "rough" implementation 2022-04-22 10:44:17 +02:00
Sandro La Bruzzo d660895b30 fixed wrong mapping type of dataset 2022-04-21 20:41:13 +02:00
Miriam Baglioni e0915061c2 [Clean Context] fixed issue in param name 2022-04-21 16:32:40 +02:00
Miriam Baglioni 6dc68c48e0 [EOSCTag] - 2022-04-21 16:19:04 +02:00
Miriam Baglioni 9a961a0092 [Clean Context] fixed issue in param name 2022-04-21 15:12:24 +02:00
Claudio Atzori 29150a5d0c code formatting 2022-04-21 13:31:56 +02:00
Miriam Baglioni 5b7d9e741c [Clean Context] added logic to cleaning workflow to accomodate also context cleaning 2022-04-21 13:02:14 +02:00
Miriam Baglioni ccba1a3db1 [Clean Context] added logic to cleaning workflow to accomodate also context cleaning 2022-04-21 13:00:06 +02:00
Miriam Baglioni 20de75ca64 [Measures] removed typo 2022-04-21 12:14:03 +02:00
Miriam Baglioni bebb2a0560 Merge branch 'eosc_dimitris' of https://code-repo.d4science.org/D-Net/dnet-hadoop into eosc_dimitris 2022-04-21 12:10:19 +02:00
Miriam Baglioni b61efd613b [Measures] addressed comments in the PR 2022-04-21 12:09:37 +02:00
Miriam Baglioni d012d125d7 [EOSCTag] - 2022-04-21 12:02:09 +02:00
Claudio Atzori 88acad76f9 Merge branch 'beta' into eosc_dimitris 2022-04-21 12:00:03 +02:00
Claudio Atzori eabb40fccc Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-04-21 11:42:43 +02:00
Miriam Baglioni c304657d91 [Measures] put the logic in common, no need to change the schema 2022-04-21 11:27:26 +02:00
Sandro La Bruzzo d580e15442 Modified last intersection since we lost many titles. 
this is my last resource, after that, I've to  change my job
 2022-04-21 11:06:08 +02:00
Miriam Baglioni 5295effc96 [Measures] fixed issue 2022-04-20 16:20:40 +02:00
Miriam Baglioni a38f0f5ea7 mergin with branch beta 2022-04-20 15:44:18 +02:00
Miriam Baglioni dbfbe8841a [Clean Context] changed the description in input parameters 2022-04-20 15:41:03 +02:00
Miriam Baglioni 5feae77937 [Measures] last changes to accomodate tests 2022-04-20 15:13:09 +02:00
Miriam Baglioni 869407c6e2 [Measures] added new measure (usagecounts) as action set. Measure added at the level of the result. Ref #7587 2022-04-20 14:02:05 +02:00
Antonis Lempesis b7cd2c6ca1 added open citations 2022-04-20 14:46:55 +03:00
Michele Artini c96a8613f8 update SQL queries 2022-04-20 12:07:49 +02:00
Michele Artini 4314db55c8 migration to services: update sql queries 2022-04-19 15:05:02 +02:00
Miriam Baglioni 0012e57bf9 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2022-04-14 14:14:44 +02:00
Miriam Baglioni c5a863132c [BulkTagging] revert it 2022-04-14 14:14:13 +02:00
Sandro La Bruzzo d5b29d96a7 fix merging in crossrefAggregator which creates dataInfo null 2022-04-14 11:07:04 +02:00
Miriam Baglioni 8e8933d41a [BulkTagging] added fix if result.dataInfo is null 2022-04-14 09:04:24 +02:00
Claudio Atzori b93a141d6c [Doiboost] fixed fundingReference extraction from the Crossref records 2022-04-12 10:26:05 +02:00
Claudio Atzori 73c172926a [Doiboost] fixed fundingReference extraction from the Crossref records 2022-04-12 10:25:42 +02:00
Claudio Atzori 48b580b45c [graph enrichment] fixed country_propagation oozie workflow definition, parameter saveGraph is not needed anymore by the SparkCountryPropagationJob 2022-04-11 08:52:36 +02:00
Claudio Atzori 21f32b83c6 [graph enrichment] fixed country_propagation oozie workflow definition, parameter saveGraph is not needed anymore by the SparkCountryPropagationJob 2022-04-11 08:52:12 +02:00
Claudio Atzori 4eff7856f5 Merge pull request '[stats-wf] computing stats in each step' (#210) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#210
 2022-04-08 14:21:01 +02:00
Serafeim Chatzopoulos d0b84d3297 Add FileCollectorPlugin and respective test 2022-04-07 15:06:38 +03:00
Serafeim Chatzopoulos bc1bf55507 Add AbstractSplittedRecordPlugin 2022-04-07 14:33:04 +03:00
Claudio Atzori c26222623f [maven-release-plugin] prepare for next development iteration 2022-04-07 13:32:22 +02:00
Claudio Atzori 86585a6b27 [maven-release-plugin] prepare release dhp-1.2.4 2022-04-07 13:32:19 +02:00
Claudio Atzori ad85d88eaf [maven-release-plugin] rollback the release of dhp-1.2.4 2022-04-07 13:28:35 +02:00
Claudio Atzori 598e11dfd7 [maven-release-plugin] prepare for next development iteration 2022-04-07 13:27:02 +02:00
Claudio Atzori db3d9877a5 [maven-release-plugin] prepare release dhp-1.2.4 2022-04-07 13:26:58 +02:00
Claudio Atzori 3bba6d6e38 [maven-release-plugin] rollback the release of dhp-1.2.4 2022-04-07 12:23:17 +02:00
Claudio Atzori 2ac2d928bd [maven-release-plugin] prepare for next development iteration 2022-04-07 12:18:47 +02:00
Claudio Atzori 85bc722ff4 [maven-release-plugin] prepare release dhp-1.2.4 2022-04-07 12:18:43 +02:00
Claudio Atzori bc05b6168a [maven-release-plugin] rollback the release of dhp-1.2.4 2022-04-07 11:49:06 +02:00
Claudio Atzori 505420fd61 [maven-release-plugin] prepare for next development iteration 2022-04-07 11:34:06 +02:00
Claudio Atzori 66e718981e [maven-release-plugin] prepare release dhp-1.2.4 2022-04-07 11:34:02 +02:00
Serafeim Chatzopoulos e612489670 Add fileGZip collector plugin and respective test 2022-04-06 19:12:44 +03:00
Claudio Atzori 4190c9f6bc [graph raw] avoid NPEs importing datasource consent fields 2022-04-06 15:34:31 +02:00
Claudio Atzori 05fafa1408 [graph raw] avoid NPEs importing datasource consent fields 2022-04-06 15:23:50 +02:00
Antonis Lempesis c442c91f89 computing stats in each step 2022-04-06 12:40:02 +03:00
Claudio Atzori 8c457f1b2c conflicts resolved, merged from beta 2022-04-06 10:27:52 +02:00
Miriam Baglioni e77d104951 [OC] added / to workflow path 2022-04-05 15:07:11 +02:00
Miriam Baglioni 79336d46c5 [Clean Context] first naive implementation of a functionality to clean not wanted contextes from one result. This implementation simply verifies the main title of the results start with a given string 2022-04-04 15:52:31 +02:00
Claudio Atzori 873369af1c Merge pull request '[stats wf] added apcs in monitor db' (#207) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#207
 2022-03-29 15:40:20 +02:00
Antonis Lempesis 7112806a73 views cannot be stored as parquet... 2022-03-29 16:37:29 +03:00
Antonis Lempesis fff0b3cc19 added apcs in monitor db 2022-03-29 14:15:31 +03:00
Claudio Atzori de85367695 Merge pull request '[stats wf] fix: views cannot be stored as parquet...' (#206) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#206
 2022-03-29 12:51:02 +02:00
Antonis Lempesis ee24f3eb2c views cannot be stored as parquet... 2022-03-29 13:47:48 +03:00
Sandro La Bruzzo 1b11010169 minor fix 2022-03-29 10:59:14 +02:00
Claudio Atzori 0a0ae84c22 [graph raw] DOI based instance URLs on https 2022-03-29 10:52:58 +02:00
Claudio Atzori 9fa3dd78fe Merge pull request '[stats wf] various fixes, organization ids for inst. dashboard' (#205) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#205
 2022-03-28 22:03:49 +02:00
Claudio Atzori 96aa2a5d0d Merge branch 'beta' into instance_group_by_url 2022-03-28 09:23:52 +02:00
Claudio Atzori 741bc99c47 Merge branch 'beta' into datasource_pdf_consent 2022-03-28 09:20:48 +02:00
Claudio Atzori 61319b2e83 updated dhp-schema version; set entity-level dataInfo before & after merging the fields from the group of duplicates 2022-03-25 16:38:33 +01:00
Antonis Lempesis d8503cd191 added moooar organizations 2022-03-24 14:02:36 +02:00
Miriam Baglioni 7b8f85692e [Enrichment country] fixed issues with parameters and workflow args 2022-03-23 17:20:23 +01:00
Claudio Atzori 48d32466e4 instances grouped by URL expose only one refereed 2022-03-23 14:52:03 +01:00
Claudio Atzori f10066547b increased spark.sql.shuffle.partitions in affiliation_from_semrel_propagation 2022-03-23 12:22:26 +01:00
Claudio Atzori 43733c1a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-03-23 12:14:27 +01:00
Antonis Lempesis 62f91b0869 cleanup 2022-03-22 16:17:49 +02:00
Antonis Lempesis 2e8394ecf8 creating aaall tables as parquet 2022-03-22 16:16:08 +02:00
Antonis Lempesis dcfbeb8142 yet more typos 2022-03-21 12:36:03 +02:00
Miriam Baglioni 89fd275480 [HostedByMap] added left over from PR and fixed issue on workflow 2022-03-21 09:54:45 +01:00
miconis c763aded70 dependency updated to the new pace-core version 2022-03-16 16:41:50 +01:00
miconis c959639bd5 dependency updated to the new pace-core version 2022-03-15 16:33:03 +01:00
Miriam Baglioni 0f7d8ca2e0 [HostedByMap] change on master to align to PR 201 on beta merged as 9f3036c847 2022-03-11 15:16:02 +01:00
Claudio Atzori f430029596 cleanup 2022-03-11 14:28:28 +01:00
Miriam Baglioni 12de9acb0d [Country Propagation] left out from previous commit 2022-03-11 14:17:02 +01:00
Miriam Baglioni 2fbb35ade5 mergin with branch beta 2022-03-11 13:58:10 +01:00
Miriam Baglioni 4437f9345d [Country Propagation] left out from previous commit 2022-03-11 13:57:47 +01:00
Miriam Baglioni 2b643059fa [Country Propagation] changed the logic to get the collectedfrom at the result level. To fix issue when no instance is created for a result that should have the country associated. Change the code to use spark instead of hive to prepare the data needed for the propagation step. Added new tests for the intermediate steps and new verification for the propagation itself 2022-03-11 13:56:48 +01:00
Claudio Atzori f25407bbe2 added mapping for datasource consent fields to integrate them in the graph 2022-03-11 09:32:42 +01:00
Miriam Baglioni 2c5087d55a [HostedByMap] download of doaj from json, modification of test resources, deletion of class no more needed for the CSV download 2022-03-04 15:18:21 +01:00
Miriam Baglioni 5d608d6291 [HostedByMap] changed the model to include also oaStart date and review process that could be possibly used in the future 2022-03-04 11:06:09 +01:00
Miriam Baglioni b7c2340952 [HostedByMap - DOIBoost] changed to use code moved to common since used also from hostedbymap now 2022-03-04 11:05:23 +01:00
Miriam Baglioni 8a41f63348 [HostedByMap] update to download the json instead of the csv 2022-03-04 10:38:43 +01:00
Miriam Baglioni 44b0c03080 [HostedByMap] update to download the json instead of the csv 2022-03-04 10:37:59 +01:00
Antonis Lempesis ad78e505da yet another fix 2022-03-03 12:28:12 +02:00
Miriam Baglioni 3be8737c32 [graph-stats] fixed query after the change in the indicator table related to PR#200 2022-03-02 14:09:05 +01:00
Miriam Baglioni 3970651ee1 Merge pull request 'fixed query after the change in the indicator table' (#200) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#200
 2022-03-02 14:05:58 +01:00
Antonis Lempesis efeeebfee1 fixed query after the change in the indicator table 2022-03-02 13:29:25 +02:00
Claudio Atzori 580d904aae manually merging PR#199 D-Net/dnet-hadoop#199 2022-02-25 12:22:50 +01:00
Claudio Atzori 1932a65d1c Merge pull request '[Stats wf] sprint 6 indicators' (#198) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#198
 2022-02-25 12:09:18 +01:00
Miriam Baglioni f5b0a6f89c [master to beta] fixed issues in test files 2022-02-25 10:21:57 +01:00
miconis 8991d097b4 bug fix in the DedupRecordFactory, DataInfo set before merge 2022-02-24 17:13:12 +01:00
miconis fe1c966cbf Merge branch 'master_202203' of code-repo.d4science.org:D-Net/dnet-hadoop into master_202203 2022-02-24 17:08:38 +01:00
miconis b0f369dc78 bug fix in the DedupRecordFactory, DataInfo set before merge 2022-02-24 17:08:24 +01:00
Miriam Baglioni 859cb7ac9d [DoiBoost AR] changed test resource to be sure the result will always have EMBARGO as value for AccessRight 2022-02-24 16:55:32 +01:00
Miriam Baglioni a40b59b7d5 [ResultToOrgFromInstRepoTest] fixed issue in model of the input resources 2022-02-24 16:05:57 +01:00
Claudio Atzori 66c09b1bc7 code formatting 2022-02-24 12:58:07 +01:00
Claudio Atzori a87c070447 conflicts resolved, merged from beta 2022-02-24 12:51:31 +01:00
Claudio Atzori 86cdb7a38f [provision] serialize measures defined on the result level 2022-02-23 15:54:18 +01:00
Alessia Bardi 9d6203f79b test mapping datasource 2022-02-23 15:00:53 +01:00
Antonis Lempesis 3b92a2ab9c added the rest of spring 6 in monitor db 2022-02-23 12:05:57 +02:00
Antonis Lempesis 87c91f70a2 added sprint 6 indicators to monitor db 2022-02-22 14:41:48 +02:00
Claudio Atzori 5226d0a100 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-02-18 15:21:07 +01:00
Claudio Atzori 99f5b14469 [graph raw] invisible records stored among the raw graph rather than the claimed subgraph 2022-02-18 15:20:57 +01:00
Claudio Atzori 401dd38074 code formatting 2022-02-18 15:19:15 +01:00
Claudio Atzori cf8443780e added processingchargeamount to the result view 2022-02-18 15:17:48 +01:00
Sandro La Bruzzo 891781ee3f Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta 2022-02-18 11:11:32 +01:00
Sandro La Bruzzo d3f03abd51 fixed wrong json path 2022-02-18 11:11:17 +01:00
Claudio Atzori 89c7313fc5 Merge branch 'beta' into hierarchical_orgs_relations 2022-02-17 10:30:04 +01:00
dimitrispie 58c59f46eb Added Sprint 6 2022-02-17 10:21:09 +02:00
Antonis Lempesis 393a4ee956 fixed yet another typo... 2022-02-15 12:56:50 +02:00
Sandro La Bruzzo 3aa2020b24 added script to regenerate hostedBy Map following instruction defined on ticket #7539 
updated hosted By Map
 2022-02-15 11:05:27 +01:00
Miriam Baglioni be64055cfe [OpenCitation] changed the name of destination folders 2022-02-14 15:49:44 +01:00
Miriam Baglioni 1490867cc7 [OpenCitation] cleaning of the COCI model 2022-02-14 14:52:12 +01:00
Miriam Baglioni c191080965 mergin with branch beta 2022-02-14 14:49:39 +01:00
Alessia Bardi 600ede1798 serialisation of APCs int he XML records 2022-02-11 11:00:20 +01:00
Miriam Baglioni 5c4043dba8 [OpenCitation] refactoring 2022-02-08 16:23:05 +01:00
Miriam Baglioni 759ed519f2 [OpenCitation] added logic to avoid the genration of self citations relations 2022-02-08 16:15:34 +01:00
Miriam Baglioni b071f8e415 [OpenCitation] change to extract in json format each folder just onece 2022-02-08 15:37:28 +01:00
Miriam Baglioni fbc28ee8c3 [OpenCitation] change the integration logic to consider dois with commas inside 2022-02-07 18:32:08 +01:00
Miriam Baglioni 78be2975f0 [stats-wf]fixed another typo related to PR#193 2022-02-07 11:22:08 +01:00
Miriam Baglioni 1f8302dc37 Merge pull request '[stats-wf]fixed yet another typo' (#193) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#193
 2022-02-07 11:19:26 +01:00
Antonis Lempesis 5f762cbd09 fixed yet another typo 2022-02-07 12:09:12 +02:00
Alessia Bardi ac8b8f224f Merge branch 'beta' into extendResult 2022-02-04 16:43:27 +01:00
Miriam Baglioni 493caef358 [stats-wf]fixed the result_result table related to PR#191 2022-02-04 14:51:25 +01:00
Miriam Baglioni 0547fd6ee7 Merge pull request '[stats-wf]fixed the result_result table' (#191) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#191
 2022-02-04 14:47:31 +01:00
Antonis Lempesis ae633c566b fixed the result_result table 2022-02-04 15:04:19 +02:00
Miriam Baglioni aae667e6b6 [APC at the result level] added the APC at the level of the result and modified test class 2022-02-04 12:34:25 +01:00
Sandro La Bruzzo bcfdf9a0d7 iis repository with https 2022-02-03 16:49:31 +01:00
Miriam Baglioni 3c60e53a96 [stats-wf]fixed the result_result creation for monitor PR#190 on beta 2022-02-03 14:47:08 +01:00
Miriam Baglioni 89922156c9 Merge pull request '[stats-wf]fixed the result_result creation for monitor' (#190) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#190
 2022-02-03 13:00:56 +01:00
Antonis Lempesis c2b44530a3 typo... 2022-02-03 13:44:07 +02:00
Antonis Lempesis dbd2646d59 fixed the result_result creation for monitor 2022-02-03 12:37:10 +02:00
Alessia Bardi 2e215abfa8 test for instances with URLs for OpenAPC 2022-02-02 17:27:44 +01:00
Miriam Baglioni 37784209c9 [dhp-schemas-] updated the version of dhp-schema to 2.10.27 for APC name and id modification 2022-02-02 12:46:31 +01:00
Miriam Baglioni 73eba34d42 [UnresolvedEntities] Changed the way to merge the unresolved because the new merge removed the dataInfo from the merged result. Added also data info for subjects 2022-02-01 08:38:41 +01:00
Miriam Baglioni dce7f5fea8 [BULK TAGGING] changed to fix issue that should have been fixed already 2022-01-31 08:20:28 +01:00
Claudio Atzori 8eb75ca169 adapted GenerateEntitiesApplicationTest behaviour 2022-01-27 16:24:37 +01:00
Claudio Atzori af61e44acc ported changes to the GraphCleaningFunctionsTest from 8de9788308 2022-01-27 16:19:14 +01:00
Claudio Atzori 1322379741 Merge branch 'beta' into delegated_authorities 2022-01-25 14:28:25 +01:00
Claudio Atzori 59a250337c [graph resolution] drop output path at the beginning 2022-01-24 18:02:39 +01:00
Claudio Atzori 97ad94d7d9 [graph resolution] drop output path at the beginning 2022-01-24 18:02:07 +01:00
Claudio Atzori 8de9788308 applied fix for avoiding ruling out the invisible (APC) records during the graph cleaning 2022-01-24 11:29:22 +01:00
Claudio Atzori 2f385b3ac6 updated dnet workflow profile definitions 2022-01-21 13:59:46 +01:00