"<strong>gCube CMS</strong> is a software application that allows users to collaborate in the creation, editing, and production of complex digital components called <strong> Projects</strong>."
]
},
{
"cell_type": "markdown",
"id": "be945c2c-1b00-44a9-890f-bde5be45ad38",
"metadata": {},
"source": [
"The application is built on top of <strong>gCube Infrastructure</strong> features, for maximum integration with <strong>Virtual Research Environments (VREs)</strong> and supports : \n",
"- <strong>Complex Data Management</strong> : Projects are made of core Metadata, custom Metadata, multiple linked FileSets\n",
"- <strong>Versioning, Workflows, Access Policies</strong> integrated with the Infrastructure User Role Management\n",
"The application is comprised of various components, including : \n",
"- <strong>GUIs</strong> to facilitate end users in managing and consuming Projects\n",
"- <strong>Service</strong> implemented the core business logic\n",
"- <strong>Plugins</strong> set of modular logic implementations (both general-purpose and custom-made) aimed at dealing with Projects Lifecycle Phases and Events\n",
"<strong>Projects</strong> are the main entities in gCube CMS. They represent a complex document comprising of its metadata, its datasets, and related information aimed at properly consume them.\\ \n",
"The present section guides through the management of CMS Projects, explaining it's model and all supported operations."
]
},
{
"cell_type": "markdown",
"id": "be47ca27-fc7b-42f5-9a54-0d01d59221e3",
"metadata": {},
"source": [
"<strong>Projects </strong> enacapsulate user's JSON Document in a richer format comprising : \n",
"- <strong>Core Metadata</strong> : Metadata information used by the service to manage the project lifecycle\n",
"- <strong>The Document</strong> : A JSON Object compliant to user's defined schema, reporting both \n",
" - The actual metadata regarding the managed documents\n",
" - Attachments references to payloads and generated materializations\n",
"\n",
"Projects are linked to <strong>Use Case Descriptors (UCD)</strong>, which define how its Document contents are gonna be treated in the system.\n",
"\n",
"More detailed informations can be found at https://gcube.wiki.gcube-system.org/gcube/GeoPortal_Service\n",
"\n",
"<strong>NB</strong>: In this notebook we are going to use the UCD \"basicUCD\". If you want to learn more about UCDs, please refer to wiki pages or related notebooks.\n",
"\n",
"<strong>NB</strong>: User can experiment with different UCDs but they need to be published in the current infrastructure's Context."
]
},
{
"cell_type": "markdown",
"id": "0a949821-435a-4b33-8ed0-2e45049be441",
"metadata": {
"tags": []
},
"source": [
"## Create New\n",
"Start This tutorial by creating your new Project."
]
},
{
"cell_type": "markdown",
"id": "ecabb6d7-d86c-4682-8c01-5024e7bf76a1",
"metadata": {},
"source": [
"In this section we are going to <strong>create</strong> a new simple document and inspect the result. \n",
"You will learn the basic sections of a Project Metadata and their meaning.\n",
"\n",
"It is sufficient to perform a <strong>post</strong> operation with our document as the body.\n",
"\n",
"The REST base endpoint for managing projects is <em>SERVICE_BASE_ENDPOINT</em>/<strong>projects</strong>/UCID where:\\\n",
"- SERVICE_BASE_ENDPOINT should be discovered from the infrastructure's Information System\n",
"- UCID is the ID of the Use Case Descriptor to be used with this project"
"The field <strong>_spatialReference</strong> contains a GEOJSON geometry identifying the Project."
]
},
{
"cell_type": "markdown",
"id": "454d5dab-73aa-4fe7-ac44-f42aff1ba679",
"metadata": {},
"source": [
"E.g. like the following : \n",
"\n",
"```json\n",
"\"_spatialReference\":\n",
" {\n",
" \"type\": \"Point\",\n",
" \"bbox\":\n",
" [\n",
" 8.621178639172953,\n",
" 40.62975046683799,\n",
" 40.630257904721645,\n",
" 8.62091913167495\n",
" ],\n",
" \"coordinates\":\n",
" [\n",
" -32.00907926554869,\n",
" -32.00883133516304\n",
" ]\n",
" }\n",
"```\n"
]
},
{
"cell_type": "markdown",
"id": "a533d6f9-6b3d-429f-acfd-d35c6f1f83c1",
"metadata": {},
"source": [
"### Temporal Reference \n",
"TBD"
]
},
{
"cell_type": "markdown",
"id": "83850b57-8d2c-47e5-ba48-c213f53493fa",
"metadata": {
"tags": []
},
"source": [
"## Editing a project\n",
"Edit the Document section of a Project. "
]
},
{
"cell_type": "markdown",
"id": "e38ae8ec-6aa9-4932-b052-69f8a6a9132a",
"metadata": {},
"source": [
"The Document section is the actual document that needs to be managed, and the most basic operation is updating it.\\\n",
"To do this it's sufficient to perform a <strong>put</strong> operation with our document as the body, specifying our project's <strong>ID</strong>\n",
"\n",
"The endpoint for this operation is going to be <em>SERVICE_BASE_ENDPOINT</em>/projects/UCID/<strong>ID</strong> where:\n",
"- SERVICE_BASE_ENDPOINT should be discovered from the infrastructure's Information System\n",
"- UCID is the ID of the Use Case Descriptor to be used with this project\n",
"Projects' documents can contain multiple collections of files called FileSet."
]
},
{
"cell_type": "markdown",
"id": "11dfedd2-bebd-420e-87b5-06b37bdd9327",
"metadata": {},
"source": [
"We use the concept of <strong>FileSet</strong> to identify a collection of files that should be managed together (e.g. GIS shape file along with its SHX file).\\\n",
"In this section we are going to <strong>register a FileSet</strong> in our project.\\\n",
"\n",
"The operations triggers :\n",
"- the upload and archive of passed data streams\n",
"- registration of the file in \"_theDocument\", precisely at the JSON path declared in the request\n",
"\n",
"<strong>NB<strong> Please beware that FileSet JSON paths <strong>MUST</strong> be deined in our UCD schema section."
]
},
{
"cell_type": "markdown",
"id": "e2bdb963-6746-4131-9607-de84b1f8a035",
"metadata": {},
"source": [
"In order to perform this operation we are going to create a JSON object request with at least the following information : \n",
"- <strong>fieldDefinitionPath</strong> : a JSON path pointing to the field definition in UCD schema [NB root is \"schema\"]\n",
"- <strong>parentPath</strong> : a JSON path pointing to the parent of our target element [NB root is \"_theDocument\"]\n",
"- <strong>fieldName</strong> : the target element name at which the FileSet is going to be registered\n",
"- <strong>streams</strong> : a collection of JSON objects, each containing\n",
" - <strong>id</strong> : the infrastructure <strong>STORAGE volatile ID</strong> of our File\n",
" - <strong>url</strong> : [Alternative to ID] the url at which our File can be downloaded.\n",
"Once our request is ready we can send it to server with a <strong>POST</strong> request at <em>SERVICE_BASE_ENDPOINT</em>/projects/UCID/registerFiles/<strong>ID</strong> where:\\\n",
"- SERVICE_BASE_ENDPOINT should be discovered from the infrastructure's Information System\n",
"- UCID is the ID of the Use Case Descriptor to be used with this project\n",
"- altering a Project lifecycle information (reporting outcome, status, messages etc.)\n",
"\n",
"STEPS are implemented in <strong>LifecycleManager</strong> plugins.\n",
"The suite comes with a default set of a pre defined, general-purpose plugins that can help manage the user's project.\n",
"This set can be extended for custom implementations.\n",
"\n",
"<strong>NB</strong> : Projects lifecycles are configured in linked Use Case Descriptors.\\\n",
"Each Project starts in <strong>Draft</strong> PHASE and then changes according to configured Lifecycle Manager."
]
},
{
"cell_type": "markdown",
"id": "e288d77d-70f6-48bc-b78b-16e673d38100",
"metadata": {},
"source": [
"Once our request is ready we can send it to server with a <strong>POST</strong> request at <em>SERVICE_BASE_ENDPOINT</em>/projects/UCID/step/<strong>STEPID/<\\strong><strong>ID</strong> where:\\\n",
"- SERVICE_BASE_ENDPOINT should be discovered from the infrastructure's Information System\n",
"- UCID is the ID of the Use Case Descriptor to be used with this project\n",
"- ID is our project Unique Identifier\n",
"- ID is our project Unique Identifier\n",
"\n",
"Depending on implementations and configurations, invoked STEP may expect/require some additional parameters.\\\n",
"These can be specified in the request body as a simple JSON object.\n",
"Please refere to specific plugin documentation for details."
"Learn how Filesets are materialized into enhanced resources (e.g. GIS Layers, DBMS Tables, Algorithms..)"
]
},
{
"cell_type": "markdown",
"id": "9d9c127c-ff7c-4b29-8a4d-df693cbb6288",
"metadata": {},
"source": [
"As discussed above, Fileset may need to be managed in order to : \n",
"- materialize them in dedicated engines in order to properyl consume them (e.g. GIS, DBs, Catalogues, Data Analytics..)\n",
"- perform data analytics tasks\n",
"\n",
"If these processes generates some kind of consumable resources, they are registered in the Fileset as <strong>Materializations</strong>\n",
"\n",
"If the above execution went well (check for errors in the displayed lifecycle information), our fileset should have been materialized as a layer in our infrastructure SDI.\\\n",
"See below the enhanced fileset with new generated information about our new layer.\n",
"\n",
"<strong>NB</strong> Each FileSet Materialization type has a different structure, in order to adapt to the needs of the application."
"The service allows for both access and querying against the internal Document archive."
]
},
{
"cell_type": "markdown",
"id": "c3990c67-f239-4b27-9136-7ea198f907bb",
"metadata": {
"tags": []
},
"source": [
"### Querying"
]
},
{
"cell_type": "markdown",
"id": "32631682-7928-4432-8cfd-fbfb8fa738f2",
"metadata": {},
"source": [
"<strong>Queries</strong> can be perfomed with a <strong>POST</strong> HTTP Request at <em>SERVICE_BASE_ENDPOINT</em>/projects/<em>UCID</em>/<strong>query</strong> with a JSON body with the following fields : \n",
"- <strong>filter</strong> : the JSON filter to apply (see MongoDB documentation for more details)\n",
"- <strong>projection</strong> : the target format to use to represent the returned JSON objects (see MongoDB documentation for more details)\n",
"As standard REST agreement, Projects can be obtained by their ID with a <strong>POST</strong> HTTP Request at <em>SERVICE_BASE_ENDPOINT</em>/projects/<em>UCID</em>/<strong>ID</strong>"
"These resources are not linked to one particular Project, but are related to the management of a specific UCD.\\\n",
"Applications that need to access these resources (e.g. GUIs) can access a Configuration report at <em>SERVICE_BASE_ENDPOINT</em>/<strong>projects</strong>/UCID/configuration."
"Use Case Descriptors (<strong>UCDs</strong>) are JSON documents containing the configuration settings needed for the management of our Projects, from the creation to their consumption.\n",
"\n",
"In this section we are going to inspect both generic configurations (e.g. Data Access Policies) and some common Plugin configuration (e.g. Lifecycle Management)"
"<strong>Data Access Policies</strong> determine both <em>READ</em> and <em>WRITE</em> authorization on Projects, based on the user's <strong>Role</strong>\n",
"\n",
"Section <em>_dataAccessPolicies</em> of UCD contains a list of <strong>Rules<strong> in the following format : \n",
"- <strong>_policy</strong> [Mandatory] : contains read and write operation policy for the present Rule (allowed values are <strong>any</strong>,<strong>own</strong> and <strong>none</strong>)\n",
"- <strong>_roles</strong> [Mandatory] : Collections of users roles for witch the present Rule should be applied\n",
"- <strong>_enforcer</strong> : Additional conditions that should be always applied when accessing data (allowed value is <strong>filter</strong>)\n",
"\n",
"E.g. \n",
"The following rule : \n",
"- Applies as default (no roles are specified)\n",
"- Denies any WRITE operation\n",
"- Allows read operations to any Project\n",
"- Filters any access to Projects selecting only the one which Phase is Published\n",
"The CMS service itself only deals with generic logic of Projects lifecycle management. The actual hard work is performed by the <strong>Plugins</strong> configured in the linked <strong>Use Case Descriptor</strong>."
]
},
{
"cell_type": "markdown",
"id": "d51c4141-9139-48f9-938c-43823fede4a9",
"metadata": {},
"source": [
"In more details, the the basic service logic relies on a <strong>Lifecycle Manager</strong> for the execution of <strong>STEPS</strong> and <strong>EVENT</strong> handling, which in turn <em>may</em> exploit other plugins for its purposes.\\\n",
"Projects and document are only actually updated by the service logic, granting full centralized error management.\n",
"In this section we are going to introduce some of the most common plugins that can be configured in UCDs.\\\n",
"<strong>NB : Plugin set can be expanded by the community both from scratch and by extending the already implemented behaviour</strong>"
]
},
{
"cell_type": "markdown",
"id": "13fe4942-e7cb-4590-b76d-5d8448041eae",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"## Lifecycle Managers\n",
"<strong>Lifecycle Managers</strong> are the main responsible for the execution of a <strong>Project workflow</strong>."
]
},
{
"cell_type": "markdown",
"id": "aa53f525-f9ba-4ea4-b14d-d66433c3a93f",
"metadata": {},
"source": [
"They define the support for <strong>STEPs</strong>, <strong>EVENTs</strong> and the resulting document <strong>PHASE</strong>, so basically they define the Workflow itself.\n",
"\n",
"However customizable, workflows shares some basic behaviour : \n",
"- Each Project starts in <strong>DRAFT</strong> PHASE, then it evolves depending on the configured Lifecycle Manager.\n",
"- The following EVENTS are always triggered (even if they can be implemented as NO-OP) :\n",
" - ON INIT\n",
" - ON UPDATE\n",
" - ON DELETE\n",
" - ON DELETE FILESET\n",
"\n",
"<strong>NB</strong> : ROLE based access to STEPs is entrusted to Lifecycle Managers. Provided implementations exploit the feature <strong>BasicRoleManager</strong> documented below.\n",
"\n",
"The service is distributed with some default implementations in order to provide support for common situations."
]
},
{
"cell_type": "markdown",
"id": "c37b85e1-f5fd-48e9-8c78-6ca507c8f11b",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"### DEFAULT-SINGLE-STEP\n",
"<strong>DEFAULT-SINGLE-STEP</strong> Lifecycle Manager is a simple workflow implementation consisting of a single STEP \"Publish\". "
"It is built on top of <strong>DEFAULT-SINGLE-STEP</strong>, extending its behavior in order to : \n",
"- Manage different indexes (restricted and public) for documets in PHASES \"Awaiting Approval\" and \"Published\"\n",
"- Allow for iterative editing cycle between submitting user and moderators\n",
"\n",
"It exploits the feature <strong>BasicRoleManager</strong> documented below."
]
},
{
"cell_type": "markdown",
"id": "0a408c93-621b-4f77-ab47-f3c1941bec6b",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"### Basic Role Manager\n",
"\n",
"Basic role manager is an internal feature of the CMS service. It restricts execution of STEPs based on user Role.\n",
"It reads the Lifecycle manager configuration expecting a collection of rules like in the following example."
]
},
{
"cell_type": "markdown",
"id": "d11d003c-a83a-45bb-80fb-ec6749d65a94",
"metadata": {},
"source": [
"The following configuration allows users with <em>Editor</em> role to only <em>SUBMIT</em> projects, while <em>Moderator</em> and <em>DataAdmin</em> can also <em>PUBLISH</em>\n",
"Indexer plugins are in charge of managing resources (<strong>Indexes</strong>)that contain references to the Projects in order to provide some kind of browsing experience through them.\\\n",
"They use the information on the selected Project to manage Index entries and (optionally) generate <strong>spatialReferences</strong> and <strong>temporalReferences</strong>."
]
},
{
"cell_type": "markdown",
"id": "bbdb399a-9c07-49bf-877f-2b5d284f0bd7",
"metadata": {},
"source": [
"### SDI-Indexer-Plugin\n",
"This plugin manages centroids layers in the SDI GeoServer, generating the Project spatialReference.\n",
"\n",
"It expects following <strong>UCD Handler Configuration</strong> : \n",
"- <strong>bboxEvaluation</strong> Collection of JSONPaths to use in order to evaluate the Project centroid\n",
"- <strong>explicitFieldMapping</strong> Collection of JSON Object defining the centroid record fields along with JSONPath for their evaluation\n",
"- <strong>additionalLayers</strong> Collection of JSON Objects defining additional layers to use for cross-reference querying of the centroid layers\n",
"\n",
"A typycal configuration might be as following : \n",
"It also expects the Lifecycle manager to provide the following call Parameters : \n",
"- Indexing request\n",
" - indexName : must be unique (table in postgis)\n",
" - worksapce : GeoServer Workspace\n",
" - centroidRecord : optionally declare the record to index instead of evaluating it"
]
},
{
"cell_type": "markdown",
"id": "889e6e56-1d5c-48ad-ad09-c8a6861315ca",
"metadata": {
"tags": []
},
"source": [
"## Materializer Plugins\n",
"Materializer plugins are in charge of generating resources representing <strong>FileSets</strong>, and producing their references to be set in the Document itself."
]
},
{
"cell_type": "markdown",
"id": "22799978-5c63-487d-badc-3a1adff27e13",
"metadata": {
"tags": []
},
"source": [
"### SDI-Indexer-Plugin\n",
"This plugin orchestrate the creation and publication of <strong>GIS Layers</strong> in the infrastructure's SDI, registering FileSets in its specific target engine (depending on data formats)."
]
},
{
"cell_type": "markdown",
"id": "afe3d19c-7984-4418-88a5-3157d415adff",
"metadata": {},
"source": [
"Currently supported data formats are : \n",
"- SHP : shape files (along with SHX files if present)\n",
"\n",
"It expects the following <strong>UCD Handler Configuration</strong> : \n",
"- <strong>registeredFileSetPaths</strong> Collection of JSONPaths pairs pointing to the FileSets to be Materialized by this plugin.\n",
"\n",
"A typycal configuration might be as following : \n",
"The service allows for inspaction and management of installed plugins."
]
},
{
"cell_type": "markdown",
"id": "4666c8ce-d45f-4bb4-9f0b-76a646b7bb6d",
"metadata": {},
"source": [
"Installed plugins descriptors can be obtained at SERVICE_BASE_ENDPOINT/<strong>plugins</strong> and SERVICE_BASE_ENDPOINT/plugins/<strong>pluginID</strong>"
