dnet-hadoop

Commit Graph

Select branches

Hide Pull Requests

beta

convert_hive_to_spark_actions

deduptesting

dhp_oaf_model

dump

forceOrginalId_claims

graph_cleaning

graph_db

hive

master

monitor_dbs

observatory_country

organization_pids

promotion-prod-only

stable_ids

unique_field_in_lists

usage-stats-export-wf-v2

wf-promotion

#1

1.0.1

archive/bipFinder_master_test

archive/dedupTest

archive/islookup_timeout

archive/junit5

archive/master_pre_stable_ids

dhp-1.0.2

dhp-1.0.3

dhp-1.0.4

dhp-1.1.5

dhp-1.1.6

dhp-1.1.7

dhp-1.2.0

dhp-1.2.1

dhp-1.2.2

dhp-1.2.3

dhp-1.2.4

eed6f21025 broadcasting moar beta Antonis Lempesis 2024-10-10 01:48:49 +0300
6f426383e6 added broadcast hints Antonis Lempesis 2024-10-10 01:27:52 +0300
a3f1be5857 Analyzing correct table Antonis Lempesis 2024-10-08 23:43:50 +0300
a0a559eb81 Analyzing correct table Antonis Lempesis 2024-10-08 13:29:28 +0300
d9a8ead200 ANALYZE TABLE is not supported on views. Antonis Lempesis 2024-10-08 01:17:32 +0300
9c9b871b2b deleted duplicate statement Antonis Lempesis 2024-10-08 00:45:46 +0300
f89a701467 deleted duplicate statement Antonis Lempesis 2024-10-08 00:27:06 +0300
413f8928cd it's late... Antonis Lempesis 2024-10-07 23:49:15 +0300
d6f2d06a9c finished the statement... Antonis Lempesis 2024-10-07 23:07:55 +0300
d645f869bc it's analyZe not analySe you moron Antonis Lempesis 2024-10-04 11:04:24 +0300
e780b15619 it's analyZe not analySe you moron Antonis Lempesis 2024-10-04 00:43:14 +0300
d0a96c8a07 using refereed for result_peerreviewed Antonis Lempesis 2024-10-03 23:26:21 +0300
b6e9ffb7e3 computing stats after every table creation Antonis Lempesis 2024-10-03 10:57:10 +0300
2ed9c5504c computing stats after every table creation Antonis Lempesis 2024-10-03 10:55:56 +0300
f3c179658a datasource table creation split in steps Antonis Lempesis 2024-09-30 17:12:21 +0300
619aa34a15 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into beta Antonis Lempesis 2024-09-23 15:25:59 +0300
dbea7a4072 removed duplicate line Antonis Lempesis 2024-09-23 14:57:11 +0300
c9241dba0d Merge pull request 'convert_hive_to_spark_actions' (#1) from convert_hive_to_spark_actions into beta Antonis Lempesis 2024-09-23 13:53:28 +0200
37ad259296 cleanup Antonis Lempesis 2024-09-05 16:02:44 +0300
b64c144abf added new institutions Antonis Lempesis 2024-09-05 16:00:09 +0300
d0590e0e49 added latest institutions Antonis Lempesis 2024-07-23 15:17:15 +0300
7d2c0a3723 added new institutions Antonis Lempesis 2024-07-23 15:10:17 +0300
e9686365a2 Improve performance of creating the "result_fos" table, by using a temp-table to cache data, which is requested multiple times. #1 convert_hive_to_spark_actions Lampros Smyrnaios 2024-07-03 20:24:36 +0300
ce0aee21cc Improve performance of transferring the stats-DBs to another cluster and querying the DBs' tables, by ordering Spark to create up to 100 files per table, instead of thousands. Lampros Smyrnaios 2024-07-03 20:15:33 +0300
7b7dd32ad5 - Fix placement of some "set mapred.job.queue.name=analytics" statements and remove their unused "/*EOS*/" indicator. - Add stacktrace-info to failed actions. Lampros Smyrnaios 2024-07-03 19:53:24 +0300
7ce051d766 - Update the remaining hive-actions to spark-actions. - Update the version of shell-actions. - Fix missing "/*EOS*/" indicators. Lampros Smyrnaios 2024-07-03 19:49:19 +0300
aa4d7d5e20 Prioritize the rest of the stats-queries over other tasks on the cluster, by putting them in the "analytics" queue. Lampros Smyrnaios 2024-07-03 19:14:25 +0300
54e11b6a43 Improve performance and efficiency by rewriting the creation process of "publication", "project", "dataset", "datasource", "software", "otherresearchproduct" and "result" tables, to be performed in a single query, for each one. Lampros Smyrnaios 2024-07-03 13:03:15 +0300
fe2275a9b0 Merge branch 'beta' of https://code-repo.d4science.org/antonis.lempesis/dnet-hadoop into convert_hive_to_spark_actions Lampros Smyrnaios 2024-06-25 20:17:47 +0300
66cd28f70a - Fix not using the "export HADOOP_USER_NAME" statement in "createPDFsAggregated.sh", which caused permission-issues when creating tables with Impala. - Remove unused "--user" parameter in "impala-shell" calls. - Code polishing. Lampros Smyrnaios 2024-06-20 14:33:46 +0300
c6b1ab2a18 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-20 14:33:05 +0300
d35edac212 [IrishFunderList]make changed according to 9635 comment 20, 21, 22 and 23 Miriam Baglioni 2024-06-20 12:28:28 +0200
6421f8fece Merge remote-tracking branch 'origin/beta' into beta Miriam Baglioni 2024-06-19 11:12:15 +0200
ac270f795b [IrishFunderList]make changed according to 9635 comment 14, 15 and 16 Miriam Baglioni 2024-06-19 11:11:52 +0200
236aed8954 Merge remote-tracking branch 'origin/beta' into beta Lampros Smyrnaios 2024-06-18 17:12:35 +0300
dd541f8cf5 Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster.' (#447) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-06-18 15:52:30 +0200
ff335578ea Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Lampros Smyrnaios 2024-06-18 14:52:31 +0300
285416c74e Merge branch 'beta' into beta Lampros Smyrnaios 2024-06-18 13:50:38 +0200
3095047e5e Miscellaneous updates to the copying operation to Impala Cluster: - Fix not breaking out of the VIEWS-infinite-loop when the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" is set to "false". - Exit the script when no HDFS-active-node was found, independently of the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR". - Fix view_name-recognition in a log-message, by using the more advanced "Perl-Compatible Regular Expressions" in "grep". - Add error-handling for "compute stats" errors. Lampros Smyrnaios 2024-06-18 14:40:41 +0300
0456f1b788 Merge remote-tracking branch 'origin/beta' into beta Antonis Lempesis 2024-06-14 15:11:30 +0300
38636942c7 filtering out deletedbyinference and invinsible results from accessroute Antonis Lempesis 2024-06-14 15:11:19 +0300
d942a1101b Miscellaneous updates to the copying operation to Impala Cluster: - Show some counts and the elapsed time for various sub-tasks. - Code polishing. Lampros Smyrnaios 2024-06-14 12:14:38 +0300
9bf2bda1c6 Fix: next returned a null value at end of stream Giambattista Bloisi 2024-06-12 13:28:51 +0200
d90cb099b8 Fix for paginationStart parameter management Giambattista Bloisi 2024-06-11 20:23:44 +0200
11fe3a4fe0 [graph resolution] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-11 14:21:01 +0200
a8d68c9d29 avoid NPEs Claudio Atzori 2024-06-11 12:40:50 +0200
8fe934810f Merge remote-tracking branch 'origin/beta' into beta Miriam Baglioni 2024-06-11 10:28:51 +0200
9da006e98c [SDGFoSActionSet]remove datainfo for the result. It is not needed (qualifier.classid = UPDATE) useless since subject do not go at the level of the instance Miriam Baglioni 2024-06-11 10:28:32 +0200
85c1eae7e0 Fixes for pagination strategy looping at end of download Giambattista Bloisi 2024-06-10 19:03:30 +0200
b0eba210c0 [actionset promotion] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-10 16:15:07 +0200
3776327a8c hostedby patching to work with the updated Crossref contents, resolved conflict Claudio Atzori 2024-06-10 15:22:33 +0200
0139f23d66 Merge pull request 'organization type from OpenOrgs' (#445) from import_openorg_type into beta Claudio Atzori 2024-06-07 12:17:31 +0200
c726572418 changed some parameters in OSF test Michele Artini 2024-06-07 12:03:26 +0200
ec79405cc9 [graph raw] set organization type from openorgs Claudio Atzori 2024-06-07 11:30:31 +0200
1477406ecc [bulkTag] fixed issue that made project disappear in graph_10_enriched Miriam Baglioni 2024-06-05 16:20:40 +0200
92c3abd5a4 [graph cleaning] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-05-29 14:36:49 +0200
ce2364743a applying changes from PR#442: Fix for missing collectedfrom after dedup Claudio Atzori 2024-06-06 10:43:43 +0200
f70dc76b61 minor Claudio Atzori 2024-06-06 10:43:10 +0200
73bd1938a5 [graph2hive] use sparkExecutorMemory to define also the memoryOverhead Claudio Atzori 2024-06-05 12:17:35 +0200
da5c1e73a4 Merge pull request 'Irish oaipmh exporter' (#443) from irish-oaipmh-exporter into beta Claudio Atzori 2024-06-05 10:55:09 +0200
81090ad593 [IE OAIPHM] added oozie workflow, minor changes, code formatting Claudio Atzori 2024-06-05 10:03:33 +0200
a02f3f0d2b code formatting Claudio Atzori 2024-05-30 10:21:18 +0200
eadfd8d71d Merge pull request 'Updated XMLIterator for splitting on different nodes' (#436) from dblp_collection_plugin into beta Alessia Bardi 2024-05-29 16:05:06 +0200
05ee783c07 Merge branch 'beta' into dblp_collection_plugin Alessia Bardi 2024-05-29 16:04:39 +0200
fe9fb59c90 Merge pull request 'Rest collector plugin on hadoop supports a new param to pass request headers' (#441) from rest-collector-request-header-map into beta Alessia Bardi 2024-05-29 15:54:39 +0200
c272c4ad68 code formatting Claudio Atzori 2024-05-29 15:50:07 +0200
c5f4da16a4 Merge branch 'beta' into rest-collector-request-header-map Alessia Bardi 2024-05-29 15:46:23 +0200
1b165a14a0 Rest collector plugin on hadoop supports a new param to pass request headers Alessia 2024-05-29 15:41:36 +0200
e996787be2 OSF test Michele Artini 2024-05-29 15:05:17 +0200
62716141c5 Merge pull request 'Miscellaneous updates to the copying operation to Impala Cluster' (#440) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-05-29 14:34:51 +0200
5d85b70e1f [NOAMI] removed Ireland funder id 501100011103. ticket 9635 Miriam Baglioni 2024-05-29 11:55:00 +0200
a644a6f4fe Catch Spark-sql errors and show a log with the statement that failed. Lampros Smyrnaios 2024-05-29 12:10:11 +0300
e3f28338c1 Miscellaneous updates to the copying operation to Impala Cluster: - Assign the WRITE and EXECUTE permissions to the DBs' HDFS-directories, in order to be able to create tables on top of them, in the Impala Cluster. - Make sure the "copydb" function returns early, when it encounters a fatal error, while respecting the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" config. Lampros Smyrnaios 2024-05-28 17:51:45 +0300
73316d8c83 Add jaxb and jaxws dependencies when compiling with spark-34 profile as they are required to run with jdk > 8 Giambattista Bloisi 2024-05-28 14:14:51 +0200
75d5ddb999 Update to include a blackList that filters out the results we know are wrongly associated to IE - update workflow definition - the blacklist parameter Miriam Baglioni 2024-05-24 16:01:19 +0200
87c9c61b41 Update to include a blackList that filters out the results we know are wrongly associated to IE - refactoring Miriam Baglioni 2024-05-24 15:23:42 +0200
b55fed09f8 Update to include a blackList that filters out the results we know are wrongly associated to IE Miriam Baglioni 2024-05-24 12:28:24 +0200
107d958b89 [org dedup] avoid NPEs in SparkPrepareNewOrgs Claudio Atzori 2024-05-26 21:23:30 +0200
3a7a6ecc32 [org dedup] avoid NPEs in SparkPrepareOrgRels Claudio Atzori 2024-05-26 16:48:11 +0200
1af4224d3d [org dedup] avoid NPEs in SparkPrepareOrgRels Claudio Atzori 2024-05-26 15:43:24 +0200
0d5bdb2db0 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Claudio Atzori 2024-05-27 11:59:02 +0200
66548e6a83 Merge pull request 'changes in copy script' (#438) from antonis.lempesis/dnet-hadoop:beta into beta Claudio Atzori 2024-05-27 11:54:03 +0200
888637773c Add missing "/*EOS*/" comments. Lampros Smyrnaios 2024-05-27 12:34:49 +0300
e0ac494859 Merge branch 'beta' into convert_hive_to_spark_actions Lampros Smyrnaios 2024-05-27 12:27:40 +0300
15b54a345a added fos lvl4 Antonis Lempesis 2024-05-24 13:21:28 +0300
b48ed6e617 Change configuration in the copy-operation to Impala Cluster: Set the "SHOULD_EXIT_WHOLE_SCRIPT_UPON_ERROR" parameter to "false". Lampros Smyrnaios 2024-05-23 16:58:12 +0300
68322843e2 Small updates to the copy-operation to Impala Cluster: - Add a configuration-"switch" to control whether the script exits upon an error or not. - Allow the script to exit when a table could not be created. - Show the elapsed time for processing each database. Lampros Smyrnaios 2024-05-23 15:07:49 +0300
c7b32bbacc Update CopyDataToImpalaCluster: Update the code of acquiring the entities from Ocean cluster, through hive, in order to optimize the process and account for additional reserved keywords in Impala. Lampros Smyrnaios 2024-05-23 13:00:19 +0300
1b2357e10a Merge pull request 'Changes in maven poms to build and test the project using Spark 3.4.x and scala 2.12' (#327) from spark34-integration into beta Giambattista Bloisi 2024-05-23 09:20:28 +0200
f1fe363b19 merged again from beta (I hope for the last time) Sandro La Bruzzo 2024-05-22 11:08:52 +0200
66c1ffc866 merged again from beta (I hope for the last time) Sandro La Bruzzo 2024-05-22 11:02:46 +0200
1ea67eba82 Merge branch 'beta' of https://code-repo.d4science.org/D-Net/dnet-hadoop into beta Claudio Atzori 2024-05-21 13:48:48 +0200
f9fb2fef6e Merge pull request 'Modification of Microsoft Academic Graph Mapping' (#435) from mag_only_doi into beta Claudio Atzori 2024-05-21 13:48:42 +0200
834461ba26 [graph provision]fixed wf definition, revised serialization of the usage counts measures Claudio Atzori 2024-05-21 13:47:05 +0200
e8a61d5dd5 removed plugin, use only FileGZip plugin Sandro La Bruzzo 2024-05-21 13:45:29 +0200
ca9414b737 Implement multiple node name splitter on GZipCollectorPlugin and all nodes that use XMLIterator. If the splitter name contains is a comma separated values it splits for all the values Sandro La Bruzzo 2024-05-21 09:11:13 +0200
032bcc8279 since last beta workflow we decide to introduce in the graph only MAG item with DOI and set them invisible ( this should be the same behaviour of the previous DOIBoost mapping). This commit apply this type of mapping Sandro La Bruzzo 2024-05-20 09:24:15 +0200
103e2652b3 merged beta Sandro La Bruzzo 2024-05-17 14:43:07 +0200
a87f9ea643 fixed scholexplorer bug Sandro La Bruzzo 2024-05-17 14:16:43 +0200
6efab4d88e fixed scholexplorer bug Sandro La Bruzzo 2024-05-16 16:19:18 +0200