ckanext-d4science_theme

This is a CKAN extension that implements the D4Science theme used by D4Science Catalogues.

The extension enriches the platform with advanced features: it allows saving datasets with duplicate fields, improves data visibility control, guides users in compiling metadata through customized profiles, automatically validates entered data, and supports content moderation.

Table of Contents

• Features
• Installation
• Configuration
• Major Changes from 2.6 to 2.10
  • Routing changes
  • Privatedatasets integration
  • Duplicated extras keys
  • List of other plugins that needed to be customized
• Updated User Interface
• Styling
• Applying SCSS Changes
• Images and Logos
• Custom Icons for Types in the HomePage filtering bar
• Customizing access to dataset's resources
• Template Helpers
• Enhancements
  • Spatial Query Lazy Loading Component
    • Overview
    • Purpose
    • Key Features
    • Dependencies
    • URL Parameters

Features

• D4Science Theme Template: Implements the look and feel of D4Science Catalogues.
• Duplicated Extras Keys: Allows saving multiple extra fields with the same key but different values.
• Namespace Support for Extras: Merges attributes with the same namespace into tables on the dataset detail page.
• Improved Visibility Management: Integrated private datasets functionality with searchable metadata for unauthorized users.
• Custom Entity Icons: Support for custom icons for types, groups, and organizations.
• Dynamic Homepage Facets: Home page filtering bar with customizable icons (Image or Font Awesome).
• Resource Access Control: Fine-grained control over resource access based on organization and dataset type.
• Spatial Query Optimization: Lazy loading for Mapbox maps to reduce API costs.
• QR Code Generation: Generates QR codes for dataset items in D4Science Catalogues.

Installation

To install the ckanext-d4science_theme, use the docker-ckan repository, which includes this extension by default.

Configuration

These are the main options that can be set in the .ini file:

Option	Description	Default
ckan.d4science_theme.systemtypefield	The field used to store the dataset type.	system:type
ckan.d4science_theme.namespace_separator	The character used to separate namespace from fieldname in extras.	:
ckan.d4science_theme.ic_proxy_url	URL for the IC Proxy service.	https://registry.d4science.org/icproxy/gcube/service
ckan.d4science_theme.application_token	Token for D4Science Infrastructure interaction.	None
ckan.d4science_theme.namespaces_generic_resource_id	Resource ID for loading namespaces from IS.	23d827cd-ba8e-4d8c-9ab4-6303bdb7d1db
ckan.focused_dataset_types	Mapping of dataset types to image filenames (separated by , and :).	None
ckan.focused_dataset_types_icons	Mapping of dataset types to Font Awesome classes.	None
ckanext.d4science_theme.public_dataset.public_access_to_resources	JSON string defining resource access policies.	{}
ckanext.d4science_theme.d4science_gateway_url	Base URL for the D4Science gateway.	https://services.d4science.org

Major Changes from 2.6 to 2.10

This plugin was migrated from CKAN version 2.6 to CKAN 2.10.5.
To do so, the base infrastructure of the plugin had to change:

• The code was updated from Python 2 to Python 3;
• The Pylons framework was replaced with Flask.

The functionality of the privatedatasets plugin was updated and integrated directly into this theme.

Changes in details

As the first step, the code was updated using the automatic python command 2to3 that converted each file from python2 syntax to python3.
Then, each function that was listed as deprecated by the ckan official CHANGELOG, got removed and substitued with the corrisponding new one. For example the methods: before_create, before_view, etc...were changed into before_dataset_create, before_dataset_view, etc...

Routing changes

Ckan changed its routing structure from Pylons to Flask. Pylons uses controllers to manage the view and routing of the ckan application, Flask uses Blueprints and rules to manage the views, instead.
The old "controller structure" has changed into "views structure".

Ckan 2.6

def before_map(self, map):  
  [...]
  map.connect('/organization_vre', controller='ckanext.d4science_theme.controllers.organization:OrganizationVREController', action='index')
  map.connect('/organization_vre/{id}', controller='ckanext.d4science_theme.controllers.organization:OrganizationVREController', action='read')
  [...]
  return map

#in controllers folder
class OrganizationVREController( toolkit.DefaultOrganizationForm, toolkit.DefaultGroupForm):

Ckan 2.10.5

def get_blueprint(self): 
  [...]
  for url, endpoint, view_func in rules:
    blueprint.add_url_rule(url, endpoint, view_func)

  blueprint.register_blueprint(organization_vre_blueprint)
        
  return blueprint

#in views folder
organization_vre_blueprint = Blueprint('organization_vre', __name__)

@organization_vre_blueprint.route('/organization_vre', strict_slashes=False)
def index():
  [...]

Privatedatasets integration

The plugin privatedatasets was originally used to make "searchable" datasets with the attribute Private set to true. So even if a user is not logged or is not a part of the dataset's organization can see the dataset in the list of all available datasets, but without the possibility to access it, unless he logs into the site. If a user doesn't have the permission to access the dataset a pop-up will be shown requesting the login. After CKAN 2.9 the method that retrives all the dataset has changed, making it even more robust. So in order to maintain this feature, a custom package_search method was developed. It works like this:

1. Ckan checks if the user is anonymous or he isn't part of the organization/group, if it is the case, the function removes the permission check from the context and add a facet to search all the datasets with the parameter searchable == True;

2. Then the "original" package_search from CKAN is called.

Duplicated extras keys

D4Science_Theme Plugin offers the possibility to save multiple keys in the extras field with the same name, but different values. This is useful if a dataset as more than one author for example.

The extras fields are also used to define custom type and give a better data visualization for every single dataset. This is done by using special attributes declared inside the custom extras field.

To make a type, the user has to write system:type as a key and the associated value will be the type. (that can be used as a facet in the package_search).

To create a table that merges attributes on the dataset detail page, users must save the extra fields using a common namespace, such as namespace:attribute_key.

List of other plugins that needed to be customized

D4science personalized other plugins, so they were updated to CKAN 2.10 too.

• DCat
• GeoNetwork
• Harvest
• Hierarchy
• Oaipmh
• Spatial

The customizations mainly focus on fixing harvesting functions: extras fields and searchable must be imported.

Updated User Interface

The look and feel of the catalog have been almost completely redesigned compared to the previous version.

Starting with the homepage, the search bar is now prominently featured, and a tabbed structure has been introduced to display various organizations, types, tags, and more.

Other important UI changes can be found in the dataset card within the Organizations section.

In addition to the title, description, and system type already present before, it is now possible to view the publication date and the author when the dataset is not private.

The most significant change, which deeply modifies CKAN’s default structure, has been implemented in the dataset detail page.

Typically, all catalog pages in CKAN follow a structure defined within the page.html file (in the templates folder). However, for the dataset detail page (templates/package), a new custom_page.html has been created, defining a different structure from the standard one by repositioning snippets in alternative locations.

The main modified files include:

• custom_page.html
• read_base.html
• /snippet/info.html
• and others.

Styling

The website’s styling is managed through SCSS files located in assets/scss.

Within this directory, the _variables.scss file defines the color variables used in d4science_theme.scss, where all the site styles are structured and categorized with section comments.

Applying SCSS Changes

To apply SCSS modifications to the site, use the VSCode extension "Live Sass Compiler" (formerly "Watch Sass") and activate it. This generates a CSS file from the SCSS source.

Through the webassets.yml file and its definition within base.html, both the stylesheets and JavaScript scripts located in assets/js/d4science_scripts.js are selected for use.

Images and Logos

Images, logos, and other visual assets are stored in the /public folder.

The plugin supports custom icons for entities "types", "groups", and "organizations" by placing image files in specific subdirectories under /public/images/{entity}/icon/:

• Types: /public/images/types/icon/
• Groups: /public/images/groups/icon/
• Organizations: /public/images/types/organizations/icon

The icon resolution is handled by the helper method d4science_get_url_to_icon_for_ckan_entity, which automatically searches for matching icon files based on the entity name.

Naming Convention: The icon filename should match the entity name (type, group, or organization name) with .png extension.

Example:

• For a type named dataset: place dataset.png in /public/images/types/icon/
• For a group named climate: place climate.png in /public/images/groups/icon/
• For a organization named my_org: place my_org.png in /public/images/organizations/icon/

Naming Convention: The icon filename should match the entity name (type, group, or organization name) with a supported image extension (e.g., .svg, .png).

Custom Icons for Types in the HomePage filtering bar

The plugin is able to set custom icons for every types in the catalogue. The are two approach to this. If both configuration are present, the plugin will always choose the image one.

Image approach

Populate the ckan.focused_dataset_types with pairs formed by <type:image_file_name> separeted by [,].

The files MUST be putted inside the /catalog/icons/tags-icons/ path.

Example:

CKAN__FOCUSED_DATASET_TYPES="dataset:database.svg, service:service_tag.svg, deliverable:papers-text.svg"

This line will associate the dataset type with database.svg, the same will happen to service and deliverable types.

Icons approach

Populate the ckan.focused_dataset_types_icons with pairs formed by <type:icon_class> separeted by [,].

The icons can be found on Font Awesome: you need to copy the only the "class" value.

Example:

CKAN__FOCUSED_DATASET_TYPES_ICONS="dataset:fa-solid fa-database, deliverable:fa-solid fa-newspaper"

This line will associate the dataset type with a database icon, the same will happen to the deliverable type with a newspaper icon.

Customizing access to dataset's resources

The D4Science plugin allows an organization to grant access to all of its resources or for resources of specific types of datasets for external/anonymous users.

If no configuration is provided, the default policy is: all denied.

The levels of permission are structured as follows:

• organization level: all the resources of every type of dataset can be access or not.
• dataset type level: all the resource of the specified datasets can be access or not. If an organization has other types of dataset not specified in the production.ini the default policy will be applied for them. For example: if an organization allows the access to all the resources of the "type1" datasets and doesn't specified anything else, the access to the resources of the "type2" datasets will be denied.

In order to customize this feature, the user has to edit the json of CKANEXT__D4SCIENCE_THEME__PUBLIC_DATASET__PUBLIC_ACCESS_TO_RESOURCES variable.

Pay attention to spaces, also if both level are specified for a single organization the dataset type level will be used.

Example:

CKANEXT__D4SCIENCE_THEME__PUBLIC_DATASET__PUBLIC_ACCESS_TO_RESOURCES={"test2": True, "open_science.system_type.policy": True, "fao": False, "fao.system_type.F2DSItem": True}

From the example:

• the organization test2 grants access to all the resources of every type of its datasets;
• the organization open_science grants access to the resources of all the datasets with the policy type. The access for the resources of every others datasets with different type will be denied.
• the organization fao has definied both permissions, although the organization level is set to false, the plugin will grant access to the resources of the F2DSItem type datasets.

Template Helpers

The following helpers are available for use in templates:

Helper Name	Description
d4science_theme_markdown_extract_html	Extracts plain text from markdown, optionally allowing HTML.
d4science_theme_get_systemtype_value_from_extras	Retrieves the system type from dataset extras.
d4science_get_extras_indexed_for_namespaces	Groups extras based on their namespace.
d4science_get_url_to_icon_for_ckan_entity	Resolves the URL for custom entity icons.
d4science_get_qrcode_for_url	Generates a base64 encoded QR code for a given URL.
d4science_get_site_statistics	Returns counts for datasets, groups, organizations, and types.
d4science_check_resource_access	Checks if a user can access a specific resource based on configuration.
d4science_is_spatial_map_mapbox	Returns True if the spatial map is configured with Mapbox.

Enhancements

Spatial Query Lazy Loading Component

Overview

This template extends the standard spatial query functionality to provide lazy-loading capabilities for map widgets. It optimizes performance by deferring resource-intensive map initialization until actually needed, particularly when using Mapbox as a basemap provider.

Purpose

• Displays a spatial filter map widget for defining geographic search boundaries
• Implements lazy-loading for Mapbox-based maps to minimize API calls and costs
• Provides direct map display for non-Mapbox providers (e.g., OpenStreetMap)
• Maintains backward compatibility with existing spatial filtering features

Key Features

Lazy Loading Strategy

• Mapbox Integration: Map content loads via AJAX only when the user clicks "Show Spatial Filter" button
• Active Filter Detection: If ext_bbox parameter exists in URL, map loads immediately on page load
• Cost Optimization: Avoids unnecessary Mapbox API initialization until user interaction

Dynamic Behavior

• Toggle button to show/hide spatial filter interface
• Automatic state preservation based on URL parameters
• AJAX-based snippet loading from custom endpoint
• Dynamic CKAN module re-initialization after content injection

Non-Mapbox Providers

• Direct map rendering without toggle functionality
• Suitable for free mapping providers (OpenStreetMap)
• Loads spatial query assets on page initialization

Dependencies

• CKAN spatial extension assets (JS/CSS)
• Custom endpoint: d4s.d4s_spatial.spatial_query_snippet
• CKAN module initialization system
• Helper function: h.d4science_is_spatial_map_mapbox()

URL Parameters

• ext_bbox: Indicates active spatial filter; triggers immediate map display

Author

• Francesco Mangiacrapa (ORCID) Computer Scientist at ISTI-CNR Infrascience Group

This approach significantly improves the user experience by preventing unnecessary spatial computations during page initialization.