gcube-cms-suite/geoportal-service/docs/architecture.rst

.. _architecture:

###############
Architecture
###############

.. figure :: _static/imgs/architecture.png
   :alt: CMS Suite overall concept

gCube CMS suite : General Architecture

The conceptual architecture behind gCube CMS Suite is schematized in the figure above.

Here we describe the rationale behind the architecture : 

* Clients perform both CRUD and Query operations against the service both via using the client library or by directly interacting with the service REST interface. 
* They interact with the service producing the custom metadata document (stored in a Document Store DB) and registering related Filesets (stored in gCube StorageHub).
* Clients sends steps (pre configured in the collection UCD) requests to the service, which in turn delegate Lifecycle operations to configured implementations.
* Clients may also directly consume materialized projects in ad hoc engines (e.g. SDI or DBMS). For this purpose, the service exposes information needed by the client in order to properly consume the project collection.


In the figure below we further illustrate the various components involved in the typical execution of Projects' lifecycle

.. figure :: _static/imgs/execution_schema.png
   :alt: Step execution



*******
Service
*******

gCube CMS Service is a gCube REST-like Application developped on top of gCube SmartGears.

It implements the core business logic of gCube CMS Suite, exposing methods for the management of Projects, Use Case Descriptors and available server plugins.

It is built on top of Jersey Framework, it uses a MongoDB Document Store for JSON archiving and Querying.
It exploits the infrastructure StorageHub for archiving FileSets and the Information System in order to discover available resources.


=========
Interface
=========

.. note :: The service interface is available for testing here.

.. alert :: You need gCube credentials in order to send requests to the service.


Projects
********

BASE PATH : **../projects/{ucid}/**

Projects interface offers the following methods over a collection (see {ucid} in path)

* CRUD operations (create, deleteById, getById, updateById)
* FileSet Management
* Query operations (powered by Mongo DB capabiliteis) allowing for filtering, pagination, projection. 
* STEP invocation
* Project configuration, reporting the generated resources for the collection and how to access them.


FileSet Management
==================

Clients can register Filesets with JSON requests with the following fields : 

* **fieldName** : target field name 
* **parentPath** : the target field's parent
* **fieldDefinitionPath** : JSONPath of the target field definition in UCD schema
* **streams** : list of JSON objects representing the fileset's payload and filenames. Payload can be specified either as **gCube Storage Hub volatile area ID** or as **URL**
* **clashOptions** : one of **REPLACE_EXISTING**, **MERGE_EXISTING**, **APPEND**
* **toSetAccess** [optional]: to set access policy for the fileset. Default is taken from the UCD.
* **attributes** [optional] : additional attributes to set to the FileSet.

Following example is taken from *Concessioni* use case submission : 





Querying
========

The service supports querying against Projects collections. Clients can submit query requests as a JSON object with the following fields : 

* **ordering** [optional] : bean with both **direction** and **fields** list
* **paging** [optional]: bean with both **offset** and **limit**
* **projection** [optional] (NB see mongoDB aggregation pipelines): the project fields to be returned
* **filtering** [optional] (NB see mongoDB aggregation pipelines): the condition to evaluate  


Following are examples of queries towards collections (in some of the following example we consider structure and lifecycle from *"Concessioni"* UCD) : 

Query **lifecycleInformation** ordered by submission 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
   {
       "ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]},
       "projection" :{"_lifecycleInformation._phase":1,"_theDocument.nome":1,"_lifecycleInformation._lastOperationStatus":1,"_lifecycleInformation._errorMessages":1,"_info._creationInfo._instant":1}
   }

Query **filesets** ordered by submission
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
   {
       "ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]},
       "projection" :{"_theDocument.abstractRelazione.fileset._payloads" : 1,"_theDocument.immaginiRappresentative.fileset._payloads" : 1,
       "_theDocument.pianteFineScavo.fileset._payloads" : 1,"_theDocument.posizionamentoScavo.fileset._payloads" : 1}
   }

Query **projects** with "follow" relationship ordered by *name*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:: 
   {
    "ordering" :{"direction" : "ASCENDING", "fields" :["_theDocument.nome"]},
    "filter" : {"_relationships._relationshipName" : {"$eq":"follows"}}
   }


STEP Invocation
===============

Lifecycle STEPs can be invoked by submitting JSON requests with the following fields : 

* **stepID** : the ID of the STEP to be executed. 
* **options** : the parameters needed for the STEP exewcution.

.. alert :: STEP ID needs to be the same expected by the configured LifeCycle Manager

.. note :: You can check the supported STEPs along with its parameters, just by interacting with the **Plugins** interface described below. 


Project configuration
=====================

Some resources are generated by the service in the normal execution of Projects lifecycle (e.g. indexes).
The service exposes the following informations in order to allow clients (e.g. GUIs) to properly consume these resources.


.. alert :: Format of the resource representation vary according to their nature in order to adapt to the needs of the collection's lifecycle.

* **indexes** : A set of generated indexes (e.g. SDI centroid layers)
* **archives** : A set of generated archive (e.g. Document archive) along with related information.
* **errorMessages** : A set of error messages reporting errors in generating the current response
* **warningMessages** : A set of warning messages reporting unexpected behaviour in generating the current response
* **status** : the global status.

.. note :: In the current implementation all collection uses the same **DOCUMENT-STORE-COLLECTION** (based on MongoDB) and **W-STORAGE** (based on gCube StorageHub).

.. note :: **DOCUMENT-STORE-COLLECTION** archive also reports aggregated Projects count by PHASE and STATUS.

Following is an example of the returned information 
:: 
      {
          "profile_id": "profiledConcessioni",
          "context": "/pred4s/preprod/preVRE",
          "last_updated_time": "2022-11-17T14:58:59.026",
          "indexes": [
              {"_type": "GIS-CENTROIDS",
                  "layer": {"_type": "gcube-sdi-layer",
                      "_platformInfo": [{
                              "_type": "Geoserver",
                              "workspace": "profiledconcessioni_prevre",
                              "storeName": "profiledconcessioni_prevre_centroids",
                              "_host": "geoserver1-t.pre.d4science.org",
                              "layerName": "profiledconcessioni_prevre_centroids"
                          }],
                      "_bbox": {"_maxX": 180.0,"_minX": -180.0,"_maxY": 90.0,"_minY": -90.0},
                      "_ogcLinks": {
                          "wms": "https://geoserver1-t.pre.d4science.org/geoserver/profiledconcessioni_prevre/wms?service=WMS&version=1.1.0&request=GetMap&layers=profiledconcessioni_prevre:profiledconcessioni_prevre_centroids&styles=&bbox=-180.000000,-90.000000,180.000000,90.000000&srs=EPSG:4326&format=application/openlayers&width=400&height=400"
                      }
                  },
                  "indexName": "profiledconcessioni_prevre_centroids",
                  "records": 1,
                  "crossReferencedLayers": {},
                  "flag": "public"
              }],
          "archives": [
              {
                  "_type": "DOCUMENT-STORE-COLLECTION",
                  "count": 225,
                  "collection_name": "profiledConcessioni",
                  "countByPhase": [
                      {"_id": {"phase": "DRAFT","status": "OK"},"count": 19},
                      {"_id": {"phase": "PUBLISHED","status": "OK"},"count": 3},
                      {"_id": {"phase": "Pending Approval","status": "ERROR"},"count": 11},
                      ...
                  ]
              },
              {
                  "_type": "W-STORAGE",
                  "folder_id": "ded73475-f9b1-46c8-94f8-5b35c60667ce"
              }
          ],
          "errorMessages": null,
          "warningMessages": null,
          "status": "OK"
      }

**************
UCD Management
**************

Current approach uses the class *org.gcube.application.geoportal.service.engine.providers.ucd.SingleISResourceUCDProvider* as a UCD Provider. 

This class asks the *gCube Information System* a **Generic Resource** with the following coordinates : 

* SecondaryType : **CMS**
* Name : **UCDs**

It expects the body to be like the following example 
::
         <UCDs>
            <record label="Human readable name" ucdUrl="some url" defaultForRegex="[opt] regex"/>
            <record label="Another readable name" ucdUrl="some other url" defaultForRegex="[opt] regex"/>
         </UCDs>


******************
Project Management
******************

Projects metadata are stored as JSON document in a Document Store DB (powered by MongoDB) called Project archive. The core logic of Project management lies in the class *org.gcube.application.geoportal.service.engine.mongo.ProfiledMongoManager*, which is the only component in charge of accessing the Project archive, in order to grant robustness via isolation.

.. note :: In order to execute both lifecyle **EVENTs** and **STEPs**, the service locates the configured LifecycleManager handler and passes to them requests with all the required information (check **Plugin Framework** below).

Locking
=======

*ProfiledMongoManager* class implements the following lock mechanism for all writing executions (here represented for the STEP execution scenario for completeness).


.. figure :: _static/imgs/step_execution.png
   :alt: Projects locking mechanism

****************
Plugin Framework
****************

In order to isolate logic, Plugins are discovered via reflection.
Plugins are thus expected to implement the proper interface, choosing between the ones defined in the framework and listed below.

.. alert :: Plugins **SHOULD** depend **only** by the Plugin Framework and their specific dependencies.

The framework provides also : 
* Definition of all involved **Requests**, **Response** and **Faults** beans
* Basic plugin implementations for common use cases, that can be extended for custom solutions (e.g. AbstractLifeCycleManager, SingleLifecycleManager, Default3PhaseManager)
* Common utilities for plugin discovery (e.g. for Lifecycle Manager to invoke Materializers and Indexers)
* **Engine Provider** in order to allow plugins to access implementations provided by the service 
* Common utilities (e.g. serialization)

===============
Engine Provider
===============

*org.gcube.application.cms.implementations.ImplementationProvider* class represents a common point of implementations reference for both the service and plugins. 

It exposes registered implementations by asking for an Engine (generic interface *org.gcube.application.cms.caches.Engine*<T>) responsible to provide a specific object class.

The service (or a startup environment, e.g. for testing purposes) is expected to register the implemented Engines at startup.

.. note ::Some of the provided engines also implements **Cache** (generic cache) and **AbstractScopedMap** (per context cache).


In the following list there are all **Engine<T>** implementations provided by the service and available to Plugins : 

* MongoClientProvider provided class T : org.gcube.application.geoportal.service.model.internal.db.Mongo
* ProjectAccessProvider provided class T : org.gcube.application.cms.implementations.ProjectAccess
* StorageClientProvider provided class T : org.gcube.application.geoportal.common.utils.StorageUtils
* SingleISResourceUCDProviderp rovided class T : org.gcube.application.geoportal.service.engine.providers.ucd.ProfileMap
* PluginManager provided class T : org.gcube.application.geoportal.service.engine.providers.PluginManager.PluginMap
* UCDManager provided class T : org.gcube.application.geoportal.service.engine.mongo.UCDManagerI
* ConfigurationCacheprovided class T : org.gcube.application.geoportal.service.engine.providers.ConfigurationCache.ConfigurationMap


=================
Plugin Interfaces
=================

The following is a list of defined plugin interfaces, which implementations are expected to implement in order to make themself available.


Following diagram shows the interfaces defined in the framework (package *org.gcube.application.cms.plugins*)

.. figure :: _static/imgs/plugin_interfaces.png
   :alt: Plugin interfaces


.. note :: **Plugin** interface as root of the above hierarchy grants common behaviour to all plugins, i.e. the exposition of PluginDescriptor object, which information are returned by the service Plugin REST interface.


==========================
Plugin Development support
==========================

In order to facilitate the development of plugins, the following solutions may be of use to the developer : 

* extension of an already provided LifecycleManager in order to override default behaviour (e.g. as per *Concessioni* UCD, see suite module concessioni-lifecycle)

* extension of *org.gcube.application.cms.plugins.AbstractPlugin* class in order to leverage on common behaiovur (e.g. configuration management, error handling)

* extension of *org.gcube.application.cms.caches.AbstractScopedMap* or *org.gcube.application.cms.caches.TimedMap* for easy implementation of caches (respectively by context and global)

* leverage on module *cms-test-commons* for exploitation of context management, basic plugin testing (both static and in a gCube context)


**************
Client Library
**************