sabeel
/
dnet-hadoop
forked from D-Net/dnet-hadoop
1
0
Fork 0
Code Pull Requests Activity
4,014 Commits 55 Branches 18 Tags 649 MiB
Commit Graph

3506 Commits

Author SHA1 Message Date
Claudio Atzori 612b7a5530 Merge branch 'beta' into tagEosc 2022-07-25 14:12:59 +02:00
Claudio Atzori c3ede1b379 Merge branch 'beta' into pubmed_update 2022-07-25 14:10:22 +02:00
Miriam Baglioni 144c103b67 [EOSC Context Tagging] add check to avoid the insertion of the context if already present 2022-07-25 13:52:45 +02:00
Enrico Ottonello 657b0208a2 multiple works download (<=100) for single request 2022-07-25 12:37:39 +02:00
Miriam Baglioni d091866e48 [EOSC Context Tagging] refactoring 2022-07-25 11:12:22 +02:00
Miriam Baglioni 5968ec018d [Clean Country] modified workflow and added param file 2022-07-22 16:48:38 +02:00
Miriam Baglioni a12d28c644 [Clean Country] added logic not to remove country from result if it exist a hosting datasource with that country. Moreover the country will be removed only if added with propagation 2022-07-22 16:23:12 +02:00
Miriam Baglioni 2c933f1158 mergin with branch beta 2022-07-22 14:57:41 +02:00
Miriam Baglioni 06a95daf60 [EOSC context TAG] refactoring after compilation 2022-07-22 14:57:06 +02:00
Miriam Baglioni ffb0ce3fb9 mergin with branch beta 2022-07-22 14:55:55 +02:00
Miriam Baglioni 627332526b [EOSC context TAG] workflow start from reset_outputpath action 2022-07-22 14:55:11 +02:00
Miriam Baglioni 7a1c1b6f53 [EOSC context TAG] Add test class and resourcesK 2022-07-22 14:36:02 +02:00
Sandro La Bruzzo ddc414b258 fixed wrong json param 2022-07-22 09:43:15 +02:00
Miriam Baglioni 317a4a56ef [EOSC context TAG] first implementation of the logic to tag results imported from datasources registered in the EOSC 2022-07-21 17:37:48 +02:00
Miriam Baglioni 3be036f290 [EOSC TAG] refactoring after compilation 2022-07-21 14:45:43 +02:00
Miriam Baglioni e61b8e6b03 mergin with branch beta 2022-07-21 14:43:23 +02:00
Miriam Baglioni 56d09e6348 [EOSC TAG] before adding the tag added a step to verify the same tag is not already present 2022-07-21 14:36:48 +02:00
Miriam Baglioni 5143a80232 [EOSC TAG] modification of test class to align with new element 2022-07-21 11:56:51 +02:00
Sandro La Bruzzo 5f651f2316 changed filter relation on SubRelType 2022-07-21 10:11:48 +02:00
Miriam Baglioni 438abdf96f [EOSC TAG] adding eosc interoperability guidelines in the specific element in the result. Removed from subjects. Removed also the deletion of EOSC Jupyter Notebook from subject since now the criteria are searchd for in a different place 2022-07-20 18:07:54 +02:00
Miriam Baglioni 65cc736e2f [Clean Country] first implementation to remove country NL from results collected from NARCIS when doi starts with mendely prefix 2022-07-20 17:05:56 +02:00
Sandro La Bruzzo 5b76321d9c implemented oozie workflow to generate scholix dump filtering relclass semantic 2022-07-20 16:34:32 +02:00
Claudio Atzori 1138b2ac8e code formatting 2022-07-19 14:15:49 +02:00
Sandro La Bruzzo 00168303db Added unit test to verify the generation in the OriginalID the old openaire Identifier generated by OAI 2022-07-14 10:19:59 +02:00
Sandro La Bruzzo 0a4f4d98fa added PMCId to PmArticle 2022-07-13 15:27:17 +02:00
Claudio Atzori 0c1cfee396 mapping oaf:fulltext elements in the result.fulltext field 2022-07-11 17:34:59 +02:00
Miriam Baglioni fae681fea1 [Country Propagation] add check to avoid NPE on datasource.getDatasourceType().getClassis() 2022-07-03 17:39:58 +02:00
Miriam Baglioni c09fcdb40b Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-07-01 12:38:03 +02:00
Claudio Atzori 0cb1c70788 code formatting 2022-07-01 10:44:08 +02:00
Claudio Atzori 4ec13e2b66 Merge branch 'master' into dump_new_funded_products 2022-07-01 10:30:28 +02:00
Claudio Atzori 072f192853 include the class information in the measure XML serialization 2022-07-01 09:54:56 +02:00
Claudio Atzori a88103bcf9 [action manager] added more testing 2022-07-01 09:06:59 +02:00
Claudio Atzori 7da24c1dec added more logging 2022-06-28 13:47:49 +02:00
Miriam Baglioni ee1f1eeca2 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-06-28 11:06:32 +02:00
Miriam Baglioni 71744a1f52 [DUMP DELTA PROJECTS] refactoring 2022-06-27 18:07:58 +02:00
Miriam Baglioni 1d1fe3b151 [DUMP DELTA PROJECTS] refactoring 2022-06-27 18:04:59 +02:00
Claudio Atzori a8773af0cb Merge branch 'beta' into project_organization_contribution 2022-06-27 09:37:40 +02:00
Claudio Atzori 4829b96bb5 Merge branch 'beta' into author_name_particles 2022-06-27 09:37:03 +02:00
Claudio Atzori 5130eac247 mapping by participant project contribution 2022-06-24 17:16:42 +02:00
Claudio Atzori 929b145130 code formatting 2022-06-21 23:07:06 +02:00
Miriam Baglioni edddfc6c63 [DUMP DELTA PROJECTS] adding test and resource 2022-06-21 18:28:53 +02:00
Miriam Baglioni f561f13dd9 [Funder Products Dump] fixed names of parameters in workflow 2022-06-21 18:18:17 +02:00
Miriam Baglioni ff74e73369 [DUMP NEW FUNDED PRODUCTS] change in resources 2022-06-21 18:02:51 +02:00
Miriam Baglioni b98f904d48 [Funder Products Dump] new way to avoid using hive 2022-06-21 17:52:27 +02:00
Miriam Baglioni 7423577a08 [Graph DUMP] add code to produce the delta of new projects with respect to the previous delta/dump 2022-06-21 14:51:38 +02:00
Claudio Atzori b295a40d9c restored use of name_particles when parsing author names 2022-06-16 12:20:43 +02:00
Claudio Atzori c7b09c6225 Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-06-16 09:28:50 +02:00
Claudio Atzori e03c0c7794 Merge branch 'beta' into oaf_relation_mapping 2022-06-16 09:27:01 +02:00
Claudio Atzori 06b5533d4c Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-06-16 09:22:16 +02:00
Claudio Atzori 4c8e820ff0 mapping relationship from trasformed records based on oaf:relation 2022-06-14 08:49:02 +02:00
Alessia Bardi 88d531dc91 exclude FAIRsharing records from Datacite 2022-06-13 16:17:17 +02:00
Claudio Atzori 116902c028 mapping relationship from trasformed records based on oaf:relation 2022-06-13 14:31:48 +02:00
Claudio Atzori b8cda65487 code formatting 2022-06-13 09:20:03 +02:00
Michele Artini 634869ce95 deleted hierarchical rels from ror action set 2022-06-13 09:12:21 +02:00
Alessia Bardi 922c6d66ef Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-06-10 17:29:15 +02:00
Alessia Bardi 68bd58d6a4 tests for ROHub 2022-06-10 17:29:11 +02:00
Miriam Baglioni b229c6e7af Merge pull request 'beta' (#218) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#218
 2022-06-10 11:03:48 +02:00
Antonis Lempesis ab18c9daa9 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta 2022-06-09 15:48:21 +03:00
Antonis Lempesis 574492c659 removed double result_apc table creation from monitor 2022-06-09 15:48:13 +03:00
Michele Artini b94a791bc5 unit tests to transform cnr explora 2022-06-09 12:25:34 +02:00
Miriam Baglioni 4b6913787b [DOI-BOOST] added one method in test of crossref mapping to aof and one resource. Related to ticket 7807 2022-06-08 14:55:19 +02:00
Antonis Lempesis db088cc69c fixed *_organization tables 2022-06-07 04:04:28 +03:00
Miriam Baglioni 31d4557e8d Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2022-06-06 11:52:29 +02:00
Claudio Atzori 5c2949a864 Merge pull request '[stats wf] added open citations & more orgs in monitor, removed collab indicator' (#213) from antonis.lempesis/dnet-hadoop:beta into beta 
Reviewed-on: D-Net/dnet-hadoop#213
 2022-05-20 11:38:43 +02:00
Miriam Baglioni 5e0b8f9b5f [CountryPropagation] refactoring 2022-05-20 09:15:53 +02:00
Miriam Baglioni c298c148cb [CountryPropagation] fix NPE issue 2022-05-20 09:11:46 +02:00
Miriam Baglioni eaf9385ae5 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-05-17 15:09:37 +02:00
Miriam Baglioni f5207885e3 [EOSCTag] changed code to remove EOSC Jupyter Notebook and modified test to exclude galaxy + software from the tagging for Galaxy 2022-05-17 15:09:22 +02:00
Claudio Atzori d098ad0d93 [hb patch] updated map 2022-05-16 15:54:04 +02:00
Claudio Atzori 1dda11e031 [hb patch] updated map 2022-05-16 15:53:27 +02:00
Claudio Atzori 8dd5517548 code formatting 2022-05-16 14:35:24 +02:00
Claudio Atzori 52cb086506 [graph grouping] drop relation target path before copying from source 2022-05-16 12:08:36 +02:00
Claudio Atzori 6442763f97 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-05-16 12:07:45 +02:00
Claudio Atzori 997c50078e [graph grouping] drop relation target path before copying from source 2022-05-16 12:07:40 +02:00
Sandro La Bruzzo c1971d52c4 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta 2022-05-16 10:30:35 +02:00
Sandro La Bruzzo 4c50f35c8b update publication Date format 2022-05-16 10:29:36 +02:00
Michele Artini 46c07e0724 deleted hierarchical rels from ror action set 2022-05-16 09:39:54 +02:00
Claudio Atzori 6031acb2e3 [openorgs] fixed parent/child query, using the correct semantic labels 2022-05-16 09:20:48 +02:00
Claudio Atzori 0dc33ea391 [openorgs] fixed parent/child query, using the correct semantic labels 2022-05-16 09:20:30 +02:00
Antonis Lempesis 3fc9efeab6 fixed typo, addded open citations and apcs in monitor 2022-05-13 14:28:13 +03:00
Miriam Baglioni e4eac1d20b [EOSC TAG] added code to remove EOSC Jupyter Notebook from subjects and put EOSC as classid in the qualifier 2022-05-13 11:01:33 +02:00
Sandro La Bruzzo 22f65680b9 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta 2022-05-11 15:30:12 +02:00
Sandro La Bruzzo ca8d26bcb4 added better filter for openCitations 2022-05-11 15:29:57 +02:00
Claudio Atzori 5d3b4a9c25 [graph merge beta] merge datasource originalid, collectedfrom, and pid lists 2022-05-11 14:13:06 +02:00
Antonis Lempesis 23334479bb removed yet another collab, added more orgs in monitor 2022-05-11 13:05:52 +03:00
Claudio Atzori 2a8e0fb72f [openorgs] mapping parent/child relations without massaging the semantic labels 2022-05-10 08:45:53 +02:00
Claudio Atzori 77bc9863e9 [openorgs] mapping parent/child relations without massaging the semantic labels 2022-05-09 16:06:04 +02:00
Claudio Atzori 378020e30a [eosc_services] unit test adaptation 2022-05-09 16:05:06 +02:00
Miriam Baglioni 89657a0b78 [UsageCount] refactoring 2022-05-09 14:43:27 +02:00
Miriam Baglioni a056f59c6e [UsageCount] make it as an action set as it should be, plus changed the test to make them work as well now 2022-05-09 12:51:35 +02:00
Antonis Lempesis 61b4c19e65 restored indi_result_org_country_collab, removed indi_result_org_collab 2022-05-06 12:52:10 +03:00
Antonis Lempesis cfbbcaf7c4 commented out indi_result_org_country_collab 2022-05-06 12:49:36 +03:00
Claudio Atzori 658450d9a3 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta 2022-05-05 11:38:08 +02:00
Claudio Atzori 846975c886 [eosc_services] using the correct 'keyword' subject type, as declared in the dnet:subject_classification_typologies vocabulary 2022-05-05 11:37:58 +02:00
Miriam Baglioni 8a72de4011 [EOSCTag] modified workflow to execute all the steps and not only the last one 2022-05-04 10:10:56 +02:00
Miriam Baglioni bd1108f98b mergin with branch beta 2022-05-04 10:06:56 +02:00
Miriam Baglioni 3aeedd931a [EOSCTag] fixed issue in case description is null. Modified test resources and classes 2022-05-04 10:06:38 +02:00
Claudio Atzori da611cfbbd [eosc_services] resolved merge conflicts 2022-05-03 13:37:15 +02:00
Claudio Atzori 9e12cb3c92 EOSC Services - removed field knowledgegraph; depending on the released schema module 2022-05-03 11:55:45 +02:00
Miriam Baglioni a21fe310e5 [EOSCTag] last test and change in the implementation to search in title and descriptio 2022-05-02 17:43:20 +02:00
Claudio Atzori 2ade69dea6 EOSC Services - minor 2022-05-02 17:03:31 +02:00
Claudio Atzori b6a7ff3a99 EOSC Services - removed fields from mapping, testing preparation 2022-05-02 15:52:33 +02:00
Miriam Baglioni e37177e1ce mergin with branch beta 2022-05-02 12:31:50 +02:00
Claudio Atzori a8c51f6f16 EOSC Services - fixed query and testing preparation 2022-05-02 11:09:03 +02:00
Claudio Atzori 05c1ea92e9 EOSC Services - added Service-specific fields in the XML record serialization 2022-04-29 15:56:55 +02:00
Claudio Atzori f5f532d134 EOSC Services - ongoing update 2022-04-29 12:25:24 +02:00
Serafeim Chatzopoulos 623f7be26d Fix reading files from HDFS in FileCollector & FileGZipCollector plugins 2022-04-28 16:31:11 +03:00
Claudio Atzori 5ffc24d1ba EOSC Services - ongoing update 2022-04-26 16:18:41 +02:00
Sandro La Bruzzo 78015a5733 Merge branch 'beta' of code-repo.d4science.org:D-Net/dnet-hadoop into beta 2022-04-26 09:56:34 +02:00
Sandro La Bruzzo 8c22e5c30a added fix to include date array with only year or year and month 2022-04-26 09:56:27 +02:00
Claudio Atzori 81c4496d32 Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-04-26 09:02:15 +02:00
Miriam Baglioni e342ec93f0 [EOSCTag] prepared resources for test 2022-04-22 18:35:37 +02:00
Miriam Baglioni 88562c0930 [EOSC TAG] added test for galaxy for title and description criterias 2022-04-22 18:35:03 +02:00
Miriam Baglioni dfbd2bcbea [EOSC TAG] added logic in case subject is null 2022-04-22 18:34:03 +02:00
Miriam Baglioni 27c85e901a [EOSCTag] added resources and finalized test for Jupyter Notebook tagging 2022-04-22 17:38:10 +02:00
Miriam Baglioni 87bff36d9e mergin with branch beta 2022-04-22 15:52:34 +02:00
Miriam Baglioni 911ce0780a Merge branch 'cleancontext' of https://code-repo.d4science.org/D-Net/dnet-hadoop into cleancontext 2022-04-22 15:41:42 +02:00
Miriam Baglioni 19d90658fc [Clean Context] added description to parameters 2022-04-22 15:41:23 +02:00
Claudio Atzori 54162f5c4f Merge branch 'beta' into cleancontext 2022-04-22 11:49:33 +02:00
Miriam Baglioni bbb77052d3 [EOSCTag] first test 2022-04-22 11:32:57 +02:00
Claudio Atzori 30105f0722 Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-04-22 11:22:21 +02:00
Sandro La Bruzzo a82ec3aaaf code formatter 2022-04-22 11:08:13 +02:00
Sandro La Bruzzo aa12429f50 Modified last intersection since we lost many titles. 2022-04-22 11:05:08 +02:00
Miriam Baglioni 7cb7066472 [EoscTag] first "rough" implementation 2022-04-22 10:44:17 +02:00
Sandro La Bruzzo d660895b30 fixed wrong mapping type of dataset 2022-04-21 20:41:13 +02:00
Miriam Baglioni e0915061c2 [Clean Context] fixed issue in param name 2022-04-21 16:32:40 +02:00
Miriam Baglioni 6dc68c48e0 [EOSCTag] - 2022-04-21 16:19:04 +02:00
Miriam Baglioni 9a961a0092 [Clean Context] fixed issue in param name 2022-04-21 15:12:24 +02:00
Claudio Atzori 29150a5d0c code formatting 2022-04-21 13:31:56 +02:00
Miriam Baglioni 5b7d9e741c [Clean Context] added logic to cleaning workflow to accomodate also context cleaning 2022-04-21 13:02:14 +02:00
Miriam Baglioni ccba1a3db1 [Clean Context] added logic to cleaning workflow to accomodate also context cleaning 2022-04-21 13:00:06 +02:00
Miriam Baglioni 20de75ca64 [Measures] removed typo 2022-04-21 12:14:03 +02:00
Miriam Baglioni bebb2a0560 Merge branch 'eosc_dimitris' of https://code-repo.d4science.org/D-Net/dnet-hadoop into eosc_dimitris 2022-04-21 12:10:19 +02:00
Miriam Baglioni b61efd613b [Measures] addressed comments in the PR 2022-04-21 12:09:37 +02:00
Miriam Baglioni d012d125d7 [EOSCTag] - 2022-04-21 12:02:09 +02:00
Claudio Atzori 88acad76f9 Merge branch 'beta' into eosc_dimitris 2022-04-21 12:00:03 +02:00
Claudio Atzori eabb40fccc Merge branch 'beta' into 7096-fileGZip-collector-plugin 2022-04-21 11:42:43 +02:00
Miriam Baglioni c304657d91 [Measures] put the logic in common, no need to change the schema 2022-04-21 11:27:26 +02:00
Sandro La Bruzzo d580e15442 Modified last intersection since we lost many titles. 
this is my last resource, after that, I've to  change my job
 2022-04-21 11:06:08 +02:00
Miriam Baglioni 5295effc96 [Measures] fixed issue 2022-04-20 16:20:40 +02:00
Miriam Baglioni a38f0f5ea7 mergin with branch beta 2022-04-20 15:44:18 +02:00
Miriam Baglioni dbfbe8841a [Clean Context] changed the description in input parameters 2022-04-20 15:41:03 +02:00
Miriam Baglioni 5feae77937 [Measures] last changes to accomodate tests 2022-04-20 15:13:09 +02:00
Miriam Baglioni 869407c6e2 [Measures] added new measure (usagecounts) as action set. Measure added at the level of the result. Ref #7587 2022-04-20 14:02:05 +02:00
Antonis Lempesis b7cd2c6ca1 added open citations 2022-04-20 14:46:55 +03:00
Michele Artini c96a8613f8 update SQL queries 2022-04-20 12:07:49 +02:00
Michele Artini 4314db55c8 migration to services: update sql queries 2022-04-19 15:05:02 +02:00
Miriam Baglioni 0012e57bf9 Merge branch 'master' of https://code-repo.d4science.org/D-Net/dnet-hadoop 2022-04-14 14:14:44 +02:00
Miriam Baglioni c5a863132c [BulkTagging] revert it 2022-04-14 14:14:13 +02:00
Sandro La Bruzzo d5b29d96a7 fix merging in crossrefAggregator which creates dataInfo null 2022-04-14 11:07:04 +02:00