diff --git a/geoportal-service/docs/architecture.rst b/geoportal-service/docs/architecture.rst index 803a91b..02997b4 100644 --- a/geoportal-service/docs/architecture.rst +++ b/geoportal-service/docs/architecture.rst @@ -4,7 +4,7 @@ Architecture ############### -.. figure:: _static/imgs/architecture.png +.. figure :: _static/imgs/architecture.png :alt: CMS Suite overall concept gCube CMS suite : General Architecture @@ -39,15 +39,15 @@ It exploits the infrastructure StorageHub for archiving FileSets and the Informa Interface ========= -.. note:: The service interface is available for testing here. +.. note :: The service interface is available for testing here. -.. alert:: You need gCube credentials in order to send requests to the service. +.. alert :: You need gCube credentials in order to send requests to the service. Projects ******** -BASE PATH : ''../projects/{ucid}/'' +BASE PATH : **../projects/{ucid}/** Projects interface offers the following methods over a collection (see {ucid} in path) @@ -57,73 +57,237 @@ Projects interface offers the following methods over a collection (see {ucid} in * STEP invocation * Project configuration, reporting the generated resources for the collection and how to access them. + FileSet Management ================== +Clients can register Filesets with JSON requests with the following fields : + +* **fieldName** : target field name +* **parentPath** : the target field's parent +* **fieldDefinitionPath** : JSONPath of the target field definition in UCD schema +* **streams** : list of JSON objects representing the fileset's payload and filenames. Payload can be specified either as **gCube Storage Hub volatile area ID** or as **URL** +* **clashOptions** : one of **REPLACE_EXISTING**, **MERGE_EXISTING**, **APPEND** +* **toSetAccess** [optional]: to set access policy for the fileset. Default is taken from the UCD. +* **attributes** [optional] : additional attributes to set to the FileSet. + +Following example is taken from *Concessioni* use case submission : + + + + + Querying ======== +The service supports querying against Projects collections. Clients can submit query requests as a JSON object with the following fields : + +* **ordering** [optional] : bean with both **direction** and **fields** list +* **paging** [optional]: bean with both **offset** and **limit** +* **projection** [optional] (NB see mongoDB aggregation pipelines): the project fields to be returned +* **filtering** [optional] (NB see mongoDB aggregation pipelines): the condition to evaluate +Following are examples of queries towards collections (in some of the following example we consider structure and lifecycle from *"Concessioni"* UCD) : +Query **lifecycleInformation** ordered by submission +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + { + "ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]}, + "projection" :{"_lifecycleInformation._phase":1,"_theDocument.nome":1,"_lifecycleInformation._lastOperationStatus":1,"_lifecycleInformation._errorMessages":1,"_info._creationInfo._instant":1} + } + +Query **filesets** ordered by submission +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + { + "ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]}, + "projection" :{"_theDocument.abstractRelazione.fileset._payloads" : 1,"_theDocument.immaginiRappresentative.fileset._payloads" : 1, + "_theDocument.pianteFineScavo.fileset._payloads" : 1,"_theDocument.posizionamentoScavo.fileset._payloads" : 1} + } + +Query **projects** with "follow" relationship ordered by *name* +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:: + { + "ordering" :{"direction" : "ASCENDING", "fields" :["_theDocument.nome"]}, + "filter" : {"_relationships._relationshipName" : {"$eq":"follows"}} + } STEP Invocation =============== +Lifecycle STEPs can be invoked by submitting JSON requests with the following fields : + +* **stepID** : the ID of the STEP to be executed. +* **options** : the parameters needed for the STEP exewcution. + +.. alert :: STEP ID needs to be the same expected by the configured LifeCycle Manager + +.. note :: You can check the supported STEPs along with its parameters, just by interacting with the **Plugins** interface described below. + Project configuration ===================== - -UCDs -**** - -Capabilities -************ +Some resources are generated by the service in the normal execution of Projects lifecycle (e.g. indexes). +The service exposes the following informations in order to allow clients (e.g. GUIs) to properly consume these resources. +.. alert :: Format of the resource representation vary according to their nature in order to adapt to the needs of the collection's lifecycle. +* **indexes** : A set of generated indexes (e.g. SDI centroid layers) +* **archives** : A set of generated archive (e.g. Document archive) along with related information. +* **errorMessages** : A set of error messages reporting errors in generating the current response +* **warningMessages** : A set of warning messages reporting unexpected behaviour in generating the current response +* **status** : the global status. -============== +.. note :: In the current implementation all collection uses the same **DOCUMENT-STORE-COLLECTION** (based on MongoDB) and **W-STORAGE** (based on gCube StorageHub). + +.. note :: **DOCUMENT-STORE-COLLECTION** archive also reports aggregated Projects count by PHASE and STATUS. + +Following is an example of the returned information +:: + { + "profile_id": "profiledConcessioni", + "context": "/pred4s/preprod/preVRE", + "last_updated_time": "2022-11-17T14:58:59.026", + "indexes": [ + {"_type": "GIS-CENTROIDS", + "layer": {"_type": "gcube-sdi-layer", + "_platformInfo": [{ + "_type": "Geoserver", + "workspace": "profiledconcessioni_prevre", + "storeName": "profiledconcessioni_prevre_centroids", + "_host": "geoserver1-t.pre.d4science.org", + "layerName": "profiledconcessioni_prevre_centroids" + }], + "_bbox": {"_maxX": 180.0,"_minX": -180.0,"_maxY": 90.0,"_minY": -90.0}, + "_ogcLinks": { + "wms": "https://geoserver1-t.pre.d4science.org/geoserver/profiledconcessioni_prevre/wms?service=WMS&version=1.1.0&request=GetMap&layers=profiledconcessioni_prevre:profiledconcessioni_prevre_centroids&styles=&bbox=-180.000000,-90.000000,180.000000,90.000000&srs=EPSG:4326&format=application/openlayers&width=400&height=400" + } + }, + "indexName": "profiledconcessioni_prevre_centroids", + "records": 1, + "crossReferencedLayers": {}, + "flag": "public" + }], + "archives": [ + { + "_type": "DOCUMENT-STORE-COLLECTION", + "count": 225, + "collection_name": "profiledConcessioni", + "countByPhase": [ + {"_id": {"phase": "DRAFT","status": "OK"},"count": 19}, + {"_id": {"phase": "PUBLISHED","status": "OK"},"count": 3}, + {"_id": {"phase": "Pending Approval","status": "ERROR"},"count": 11}, + ... + ] + }, + { + "_type": "W-STORAGE", + "folder_id": "ded73475-f9b1-46c8-94f8-5b35c60667ce" + } + ], + "errorMessages": null, + "warningMessages": null, + "status": "OK" + } + +************** UCD Management -============== +************** + +Current approach uses the class *org.gcube.application.geoportal.service.engine.providers.ucd.SingleISResourceUCDProvider* as a UCD Provider. + +This class asks the *gCube Information System* a **Generic Resource** with the following coordinates : + +* SecondaryType : **CMS** +* Name : **UCDs** + +It expects the body to be like the following example +:: + + + + -================== +****************** Project Management -================== +****************** +Projects metadata are stored as JSON document in a Document Store DB (powered by MongoDB) called Project archive. The core logic of Project management lies in the class *org.gcube.application.geoportal.service.engine.mongo.ProfiledMongoManager*, which is the only component in charge of accessing the Project archive, in order to grant robustness via isolation. -Project archive -*************** +.. note :: In order to execute both lifecyle **EVENTs** and **STEPs**, the service locates the configured LifecycleManager handler and passes to them requests with all the required information (check **Plugin Framework** below). Locking ======= - -Fileset management -****************** - - -STEP Execution -************** +*ProfiledMongoManager* class implements the following lock mechanism for all writing executions (here represented for the STEP execution scenario for completeness). +.. figure :: _static/imgs/step_execution.png + :alt: Projects locking mechanism **************** Plugin Framework **************** +In order to isolate logic, Plugins are discovered via reflection. +Plugins are thus expected to implement the proper interface, choosing between the ones defined in the framework and listed below. + +.. alert :: Plugins **SHOULD** depend **only** by the Plugin Framework and their specific dependencies. + +The framework provides also : +* Definition of all involved **Requests**, **Response** and **Faults** beans +* Basic plugin implementations for common use cases, that can be extended for custom solutions (e.g. AbstractLifeCycleManager, SingleLifecycleManager, Default3PhaseManager) +* Common utilities for plugin discovery (e.g. for Lifecycle Manager to invoke Materializers and Indexers) +* **Engine Provider** in order to allow plugins to access implementations provided by the service +* Common utilities (e.g. serialization) + =============== Engine Provider =============== +*org.gcube.application.cms.implementations.ImplementationProvider* class represents a common point of implementations reference for both the service and plugins. + +It exposes registered implementations by asking for an Engine (generic interface *org.gcube.application.cms.caches.Engine*) responsible to provide a specific object class. + +The service (or a startup environment, e.g. for testing purposes) is expected to register the implemented Engines at startup. + +.. note ::Some of the provided engines also implements **Cache** (generic cache) and **AbstractScopedMap** (per context cache). + + +In the following list there are all **Engine** implementations provided by the service and available to Plugins : + +* MongoClientProvider provided class T : org.gcube.application.geoportal.service.model.internal.db.Mongo +* ProjectAccessProvider provided class T : org.gcube.application.cms.implementations.ProjectAccess +* StorageClientProvider provided class T : org.gcube.application.geoportal.common.utils.StorageUtils +* SingleISResourceUCDProviderp rovided class T : org.gcube.application.geoportal.service.engine.providers.ucd.ProfileMap +* PluginManager provided class T : org.gcube.application.geoportal.service.engine.providers.PluginManager.PluginMap +* UCDManager provided class T : org.gcube.application.geoportal.service.engine.mongo.UCDManagerI +* ConfigurationCacheprovided class T : org.gcube.application.geoportal.service.engine.providers.ConfigurationCache.ConfigurationMap + ================= Plugin Interfaces ================= +The following is a list of defined plugin interfaces, which implementations are expected to implement in order to make themself available. + + +Following diagram shows the interfaces defined in the framework (package *org.gcube.application.cms.plugins*) + +.. figure :: _static/imgs/plugin_interfaces.png + :alt: Plugin interfaces + + +.. note :: **Plugin** interface as root of the above hierarchy grants common behaviour to all plugins, i.e. the exposition of PluginDescriptor object, which information are returned by the service Plugin REST interface. + + ========================== Plugin Development support ==========================