Open Citation integration #401

miriam.baglioni · 2024-03-13T14:26:57+01:00

miriam.baglioni commented

2024-03-13 14:26:57 +01:00

New implementation for the integration of the new dump of Open Citation. We need to change the implementation because OC has changed the information in the dataset: they use now internal OC identifiers to link citing and cited results instead of pids (DOI, PMID etc)

We also need to download a correspondence file between OC internal identifiers and pids.

OC also uses isbn and issn as identifiers for the citing/cited resource. For now this types of identifiers are not considered. We can reconsider them once we have inserted venues.

Parameters to be passed to the workflow:

1. inputPath = the path where the downloaded new OC dumps are stored
2. outputPathExtraction = not volatile path to store the result of the citation extraction remapped to the pids. In this path all the new extracted relations will be appended. The AS is created starting from this path
3. outputPath = the Action Set path
4. resumeFrom = step where to resume the execution.
5. filelist = The list of files are split by ; Each element represents the URL to be used to download the dump and the date of the dump as in the OC folder. The URL and the date are split by @. Example https://figshare.com/ndownloader/files/42777190@2023-05-20; ... 
All the dumps at the URL will be downloaded and stored in inputPath/date
6. filecorrispondence = the URL of the file storing the correspondence between the OC internal identifiers and the pids. Also in this case it will be formatted as `URL@omid.zip` (https://figshare.com/ndownloader/files/43491411@omid.zip)
7. delimiter = the delimiter in the csv to be used to split the information (,)

Possible values for the resumeFrom parameter:

DownloadDump to download the new OC dump files and the correspondence file
ExtractContent to extract the zip from the dump
ReadContent to read the content in the new dump and create the corresponding json file
MapContent to map the internal OC identifiers in the corresponding pids
CreateAS to create the action set
The default step is the deletion of the inputPath before downloading the new dumped files

New implementation for the integration of the new dump of Open Citation. We need to change the implementation because OC has changed the information in the dataset: they use now internal OC identifiers to link citing and cited results instead of pids (DOI, PMID etc) We also need to download a correspondence file between OC internal identifiers and pids. OC also uses isbn and issn as identifiers for the citing/cited resource. For now this types of identifiers are not considered. We can reconsider them once we have inserted venues. Parameters to be passed to the workflow: ``` 1. inputPath = the path where the downloaded new OC dumps are stored 2. outputPathExtraction = not volatile path to store the result of the citation extraction remapped to the pids. In this path all the new extracted relations will be appended. The AS is created starting from this path 3. outputPath = the Action Set path 4. resumeFrom = step where to resume the execution. 5. filelist = The list of files are split by ; Each element represents the URL to be used to download the dump and the date of the dump as in the OC folder. The URL and the date are split by @. Example https://figshare.com/ndownloader/files/42777190@2023-05-20; ... All the dumps at the URL will be downloaded and stored in inputPath/date 6. filecorrispondence = the URL of the file storing the correspondence between the OC internal identifiers and the pids. Also in this case it will be formatted as `URL@omid.zip` (https://figshare.com/ndownloader/files/43491411@omid.zip) 7. delimiter = the delimiter in the csv to be used to split the information (,) ``` Possible values for the `resumeFrom` parameter: * `DownloadDump` to download the new OC dump files and the correspondence file * `ExtractContent` to extract the zip from the dump * `ReadContent` to read the content in the new dump and create the corresponding json file * `MapContent` to map the internal OC identifiers in the corresponding pids * `CreateAS` to create the action set * The default step is the deletion of the inputPath before downloading the new dumped files

claudio.atzori was assigned by miriam.baglioni

2024-03-13 14:26:57 +01:00

miriam.baglioni added 3 commits 2024-03-13 14:26:58 +01:00

fd34372c40 [OCNEW] first implementation

48c052215c [OC New] last fix

5a32bb9578 [OC New] last fix

claudio.atzori added 1 commit 2024-03-25 16:10:26 +01:00

6c3b692f60 integrated minor change from beta branch

claudio.atzori added 1 commit 2024-03-25 16:10:33 +01:00

74e5d05577 Merge branch 'beta' into ocnew

claudio.atzori merged commit fa4b3e6d2b into beta

2024-03-25 16:10:41 +01:00