ckanext-d4science_theme

The CKAN extension that implements the D4Science theme template used by D4Science Catalogues.
The most important features are the possibility to save dataset with duplicated keys in the extra fields section and an improved managment of datasets visibility.

Requirements

None.

Installation

To install ckanext-d4science_theme:

Put these lines in the Dockerfile where you put all the other plugins that you use in your CKAN application:

pip3 install --no-cache-dir -e git+https://code-repo.d4science.org/CatalogueRevamping/harvester-d4science.git@fix_validation#egg=ckanext-harvest && \
pip3 install --no-cache-dir -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt && \

pip3 install --no-cache-dir -e git+https://code-repo.d4science.org/CatalogueRevamping/spatial-d4science.git@15d275586fbcaf4f2ec3c55d8020134927406745#egg=ckanext-spatial && \
pip3 install --no-cache-dir -r ${APP_DIR}/src/ckanext-spatial/pip-requirements.txt && \

pip3 install --no-cache-dir -e git+https://code-repo.d4science.org/CatalogueRevamping/ckanext-keycloak.git@d4s_custom#egg=ckanext-keycloak && \

pip3 install --no-cache-dir -r ${APP_DIR}/src/ckanext-keycloak/requirements.txt && \
pip3 install --no-cache-dir -e git+https://code-repo.d4science.org/CatalogueRevamping/geonetwork-d4science.git@d4science_custom#egg=ckanext-geonetwork && \

pip3 install --no-cache-dir -e git+https://code-repo.d4science.org/CatalogueRevamping/ckanext-d4science_theme.git@v1.0.4#egg=ckanext-d4science_theme && \
pip3 install --no-cache-dir -r ${APP_DIR}/src/ckanext-d4science-theme/requirements.txt

This will install the plugin and all the other ones that are required in order to work properly.

Major Changes from 2.6 to 2.10

This plugin migrate from Ckan version 2.6 to Ckan 2.10.5.
To do so, the base infrastructure of the plugin had to change:

• The code was updated from Python2 to Python3;
• Pylons framework got changed with Flask.

The plugin privatedataset was updated and inclued too.

Changes in details

As the first step, the code was updated using the automatic python command 2to3 that converted each file from python2 syntax to python3.
Then, each function that was listed as deprecated by the ckan official CHANGELOG, got removed and substitued with the corrisponding new one. For example the methods: before_create, before_view, etc...were changed into before_dataset_create, before_dataset_view, etc...

Routing changes

Ckan changed its routing structure from Pylons to Flask. Pylons uses controllers to manage the view and routing of the ckan application, Flask uses Blueprints and rules to manage the views, instead.
The old "controller structure" has changed into "views structure".

Ckan 2.6

def before_map(self, map):  
  [...]
  map.connect('/organization_vre', controller='ckanext.d4science_theme.controllers.organization:OrganizationVREController', action='index')
  map.connect('/organization_vre/{id}', controller='ckanext.d4science_theme.controllers.organization:OrganizationVREController', action='read')
  [...]
  return map

#in controllers folder
class OrganizationVREController( toolkit.DefaultOrganizationForm, toolkit.DefaultGroupForm):

Ckan 2.10.5

def get_blueprint(self): 
  [...]
  for url, endpoint, view_func in rules:
    blueprint.add_url_rule(url, endpoint, view_func)

  blueprint.register_blueprint(organization_vre_blueprint)
        
  return blueprint

#in views folder
organization_vre_blueprint = Blueprint('organization_vre', __name__)

@organization_vre_blueprint.route('/organization_vre', strict_slashes=False)
def index():
  [...]

Privatedatasets integration

The plugin privatedatasets was originally used to make "searchable" datasets with the attribute Private set to true. So even if a user is not logged or is not a part of the dataset's organization can see the dataset in the list of all available datasets, but without the possibility to access it, unless he logs into the site. If a user doesn't have the permission to access the dataset a pop-up will be shown requesting the login. After CKAN 2.9 the method that retrives all the dataset has changed, making it even more robust. So in order to maintain this feature, a custom package_search method was developed. It works like this:

1. Ckan checks if the user is anonymous or he isn't part of the organization/group, if it is the case, the function removes the permission check from the context and add a facet to search all the datasets with the parameter searchable == True;

2. Then the "original" package_search from CKAN is called.

Duplicated extras keys

D4Science_Theme Plugin offers the possibility to save multiple keys in the extras field with the same name, but different values. This is useful if a dataset as more than one author for example.

The extras fields are also used to define custom type and give a better data visualization for every single dataset. This is done by using special attributes declared inside the custom extras field.

To make a type, the user has to write system:type as a key and the associated value will be the type. (that can be used as a facet in the package_search).

To make a table that merges some attributes inside the dataset detail page, the user must save all the extras fields with the same namespace, like this namespace:attribute_key.

List of other plugins that needed to be customized

D4science personalized other plugins, so they were updated to CKAN 2.10 too.

• DCat
• GeoNetwork
• Harvest
• Hierarchy
• Oaipmh
• Spatial

The customizations mainly focus on fixing harvesting functions: extras fields and searchable must be imported.

Updated User Interface

The look and feel of the catalog have been almost completely redesigned compared to the previous version.

Starting with the homepage, the search bar is now prominently featured, and a tabbed structure has been introduced to display various organizations, types, tags, and more.

Other important UI changes can be found in the dataset card within the Organizations section.

In addition to the title, description, and system type already present before, it is now possible to view the publication date and the author when the dataset is not private.

The most significant change, which deeply modifies CKAN’s default structure, has been implemented in the dataset detail page.

Typically, all catalog pages in CKAN follow a structure defined within the page.html file (in the templates folder). However, for the dataset detail page (templates/package), a new custom_page.html has been created, defining a different structure from the standard one by repositioning snippets in alternative locations.

The main modified files include:

• custom_page.html
• read_base.html
• /snippet/info.html
• and others.

Styling

The website’s styling is managed through SCSS files located in assets/scss.

Within this directory, the _variables.scss file defines the color variables used in d4science_theme.scss, where all the site styles are structured and categorized with section comments.

Applying SCSS Changes

To apply SCSS modifications to the site, use the VSCode extension "Live Sass Compiler" (formerly "Watch Sass") and activate it. This generates a CSS file from the SCSS source.

Through the webassets.yml file and its definition within base.html, both the stylesheets and JavaScript scripts located in assets/js/d4science_scripts.js are selected for use.

Images and Logos

Images, logos, and other visual assets are stored in the /public folder.

Custom Icons for Types

The plugin is able to set custom icons for every types in the catalogue. The are two approach to this. If both configuration are present, the plugin will always choose the image one.

Image approach

Populate the ckan.focused_dataset_types with pairs formed by <type:image_file_name> separeted by [,].

The files MUST be putted inside the /catalog/icons/tags-icons/ path.

Example:

CKAN__FOCUSED_DATASET_TYPES="dataset:database.svg, service:service_tag.svg, deliverable:papers-text.svg"

This line will associate the dataset type with database.svg, the same will happen to service and deliverable types.

Icons approach

Populate the ckan.focused_dataset_types_icons with pairs formed by <type:icon_class> separeted by [,].

The icons can be found on Font Awesome: you need to copy the only the "class" value.

Example:

CKAN__FOCUSED_DATASET_TYPES_ICONS="dataset:fa-solid fa-database, deliverable:fa-solid fa-newspaper"

This line will associate the dataset type with a database icon, the same will happen to the deliverable type with a newspaper icon.

Customizing access to dataset's resources

The D4Science plugin allows an organization to grant access to all of its resources or for resources of specific types of datasets for external/anonymous users.

If no configuration is provided, the default policy is: all denied.

The levels of permission are structured as follows:

• organization level: all the resources of every type of dataset can be access or not.
• dataset type level: all the resource of the specified datasets can be access or not. If an organization has other types of dataset not specified in the production.ini the default policy will be applied for them. For example: if an organization allows the access to all the resources of the "type1" datasets and doesn't specified anything else, the access to the resources of the "type2" datasets will be denied.

In order to customize this feature, the user has to edit the json of CKANEXT__D4SCIENCE_THEME__PUBLIC_DATASET__PUBLIC_ACCESS_TO_RESOURCES variable.

Pay attention to spaces, also if both level are specified for a single organization the dataset type level will be used.

Example:

CKANEXT__D4SCIENCE_THEME__PUBLIC_DATASET__PUBLIC_ACCESS_TO_RESOURCES={"test2": True, "open_science.system_type.policy": True, "fao": False, "fao.system_type.F2DSItem": True}

From the example:

• the organization test2 grants access to all the resources of every type of its datasets;
• the organization open_science grants access to the resources of all the datasets with the policy type. The access for the resources of every others datasets with different type will be denied.
• the organization fao has definied both permissions, although the organization level is set to false, the plugin will grant access to the resources of the F2DSItem type datasets.