gcube-cms-suite/geoportal-service/docs/architecture.rst

Architecture

gCube CMS suite : General Architecture

The conceptual architecture behind gCube CMS Suite is schematized in the figure above.

Here we describe the rationale behind the architecture :

• Clients perform both CRUD and Query operations against the service both via using the client library or by directly interacting with the service REST interface.
• They interact with the service producing the custom metadata document (stored in a Document Store DB) and registering related Filesets (stored in gCube StorageHub).
• Clients sends steps (pre configured in the collection UCD) requests to the service, which in turn delegate Lifecycle operations to configured implementations.
• Clients may also directly consume materialized projects in ad hoc engines (e.g. SDI or DBMS). For this purpose, the service exposes information needed by the client in order to properly consume the project collection.

In the figure below we further illustrate the various components involved in the typical execution of Projects' lifecycle

Service

gCube CMS Service is a gCube REST-like Application developped on top of gCube SmartGears.

It implements the core business logic of gCube CMS Suite, exposing methods for the management of Projects, Use Case Descriptors and available server plugins.

It is built on top of Jersey Framework, it uses a MongoDB Document Store for JSON archiving and Querying. It exploits the infrastructure StorageHub for archiving FileSets and the Information System in order to discover available resources.

Interface

Projects

BASE PATH : ../projects/{ucid}/

Projects interface offers the following methods over a collection (see {ucid} in path)

• CRUD operations (create, deleteById, getById, updateById)
• FileSet Management
• Query operations (powered by Mongo DB capabiliteis) allowing for filtering, pagination, projection.
• STEP invocation
• Project configuration, reporting the generated resources for the collection and how to access them.

FileSet Management

Clients can register Filesets with JSON requests with the following fields :

• fieldName : target field name
• parentPath : the target field's parent
• fieldDefinitionPath : JSONPath of the target field definition in UCD schema
• streams : list of JSON objects representing the fileset's payload and filenames. Payload can be specified either as gCube Storage Hub volatile area ID or as URL
• clashOptions : one of REPLACE_EXISTING, MERGE_EXISTING, APPEND
• toSetAccess [optional]: to set access policy for the fileset. Default is taken from the UCD.
• attributes [optional] : additional attributes to set to the FileSet.

Following example is taken from Concessioni use case submission :

Querying

The service supports querying against Projects collections. Clients can submit query requests as a JSON object with the following fields :

• ordering [optional] : bean with both direction and fields list
• paging [optional]: bean with both offset and limit
• projection [optional] (NB see mongoDB aggregation pipelines): the project fields to be returned
• filtering [optional] (NB see mongoDB aggregation pipelines): the condition to evaluate

Following are examples of queries towards collections (in some of the following example we consider structure and lifecycle from "Concessioni" UCD) :

Query lifecycleInformation ordered by submission

::

{

"ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]}, "projection" :{"_lifecycleInformation._phase":1,"_theDocument.nome":1,"_lifecycleInformation._lastOperationStatus":1,"_lifecycleInformation._errorMessages":1,"_info._creationInfo._instant":1}

}

Query filesets ordered by submission

::

{

"ordering" :{"direction" : "DESCENDING", "fields" :["_info._creationInfo._instant"]}, "projection" :{"_theDocument.abstractRelazione.fileset._payloads" : 1,"_theDocument.immaginiRappresentative.fileset._payloads" : 1, "_theDocument.pianteFineScavo.fileset._payloads" : 1,"_theDocument.posizionamentoScavo.fileset._payloads" : 1}

}

Query projects with "follow" relationship ordered by name

::

{

"ordering" :{"direction" : "ASCENDING", "fields" :["_theDocument.nome"]}, "filter" : {"_relationships._relationshipName" : {"$eq":"follows"}}

}

STEP Invocation

Lifecycle STEPs can be invoked by submitting JSON requests with the following fields :

• stepID : the ID of the STEP to be executed.
• options : the parameters needed for the STEP exewcution.

Project configuration

Some resources are generated by the service in the normal execution of Projects lifecycle (e.g. indexes). The service exposes the following informations in order to allow clients (e.g. GUIs) to properly consume these resources.

• indexes : A set of generated indexes (e.g. SDI centroid layers)
• archives : A set of generated archive (e.g. Document archive) along with related information.
• errorMessages : A set of error messages reporting errors in generating the current response
• warningMessages : A set of warning messages reporting unexpected behaviour in generating the current response
• status : the global status.

Following is an example of the returned information :: { "profile_id": "profiledConcessioni", "context": "/pred4s/preprod/preVRE", "last_updated_time": "2022-11-17T14:58:59.026", "indexes": [ {"_type": "GIS-CENTROIDS", "layer": {"_type": "gcube-sdi-layer", "_platformInfo": [{ "_type": "Geoserver", "workspace": "profiledconcessioni_prevre", "storeName": "profiledconcessioni_prevre_centroids", "_host": "geoserver1-t.pre.d4science.org", "layerName": "profiledconcessioni_prevre_centroids" }], "_bbox": {"_maxX": 180.0,"_minX": -180.0,"_maxY": 90.0,"_minY": -90.0}, "_ogcLinks": { "wms": "https://geoserver1-t.pre.d4science.org/geoserver/profiledconcessioni_prevre/wms?service=WMS&version=1.1.0&request=GetMap&layers=profiledconcessioni_prevre:profiledconcessioni_prevre_centroids&styles=&bbox=-180.000000,-90.000000,180.000000,90.000000&srs=EPSG:4326&format=application/openlayers&width=400&height=400" } }, "indexName": "profiledconcessioni_prevre_centroids", "records": 1, "crossReferencedLayers": {}, "flag": "public" }], "archives": [ { "_type": "DOCUMENT-STORE-COLLECTION", "count": 225, "collection_name": "profiledConcessioni", "countByPhase": [ {"_id": {"phase": "DRAFT","status": "OK"},"count": 19}, {"_id": {"phase": "PUBLISHED","status": "OK"},"count": 3}, {"_id": {"phase": "Pending Approval","status": "ERROR"},"count": 11}, ... ] }, { "_type": "W-STORAGE", "folder_id": "ded73475-f9b1-46c8-94f8-5b35c60667ce" } ], "errorMessages": null, "warningMessages": null, "status": "OK" }

UCD Management

Current approach uses the class org.gcube.application.geoportal.service.engine.providers.ucd.SingleISResourceUCDProvider as a UCD Provider.

This class asks the gCube Information System a Generic Resource with the following coordinates :

• SecondaryType : CMS
• Name : UCDs

It expects the body to be like the following example :: <UCDs> <record label="Human readable name" ucdUrl="some url" defaultForRegex="[opt] regex"/> <record label="Another readable name" ucdUrl="some other url" defaultForRegex="[opt] regex"/> </UCDs>

Project Management

Projects metadata are stored as JSON document in a Document Store DB (powered by MongoDB) called Project archive. The core logic of Project management lies in the class org.gcube.application.geoportal.service.engine.mongo.ProfiledMongoManager, which is the only component in charge of accessing the Project archive, in order to grant robustness via isolation.

Locking

ProfiledMongoManager class implements the following lock mechanism for all writing executions (here represented for the STEP execution scenario for completeness).

Plugin Framework

In order to isolate logic, Plugins are discovered via reflection. Plugins are thus expected to implement the proper interface, choosing between the ones defined in the framework and listed below.

The framework provides also : * Definition of all involved Requests, Response and Faults beans * Basic plugin implementations for common use cases, that can be extended for custom solutions (e.g. AbstractLifeCycleManager, SingleLifecycleManager, Default3PhaseManager) * Common utilities for plugin discovery (e.g. for Lifecycle Manager to invoke Materializers and Indexers) * Engine Provider in order to allow plugins to access implementations provided by the service * Common utilities (e.g. serialization)

Engine Provider

org.gcube.application.cms.implementations.ImplementationProvider class represents a common point of implementations reference for both the service and plugins.

It exposes registered implementations by asking for an Engine (generic interface org.gcube.application.cms.caches.Engine<T>) responsible to provide a specific object class.

The service (or a startup environment, e.g. for testing purposes) is expected to register the implemented Engines at startup.

In the following list there are all Engine<T> implementations provided by the service and available to Plugins :

• MongoClientProvider provided class T : org.gcube.application.geoportal.service.model.internal.db.Mongo
• ProjectAccessProvider provided class T : org.gcube.application.cms.implementations.ProjectAccess
• StorageClientProvider provided class T : org.gcube.application.geoportal.common.utils.StorageUtils
• SingleISResourceUCDProviderp rovided class T : org.gcube.application.geoportal.service.engine.providers.ucd.ProfileMap
• PluginManager provided class T : org.gcube.application.geoportal.service.engine.providers.PluginManager.PluginMap
• UCDManager provided class T : org.gcube.application.geoportal.service.engine.mongo.UCDManagerI
• ConfigurationCacheprovided class T : org.gcube.application.geoportal.service.engine.providers.ConfigurationCache.ConfigurationMap

Plugin Interfaces

The following is a list of defined plugin interfaces, which implementations are expected to implement in order to make themself available.

Following diagram shows the interfaces defined in the framework (package org.gcube.application.cms.plugins)

Plugin Development support

In order to facilitate the development of plugins, the following solutions may be of use to the developer :

• extension of an already provided LifecycleManager in order to override default behaviour (e.g. as per Concessioni UCD, see suite module concessioni-lifecycle)
• extension of org.gcube.application.cms.plugins.AbstractPlugin class in order to leverage on common behaiovur (e.g. configuration management, error handling)
• extension of org.gcube.application.cms.caches.AbstractScopedMap or org.gcube.application.cms.caches.TimedMap for easy implementation of caches (respectively by context and global)
• leverage on module cms-test-commons for exploitation of context management, basic plugin testing (both static and in a gCube context)

Client Library