Updated to GA4, see #25253

Allow configurable value for recent view days when getting data from Google Analytics API

MANIFEST.in

@@ -1,4 +1,4 @@
 include README.rst
 include LICENSE.txt
 include requirements.txt
-recursive-include ckanext/googleanalytics *.html *.js
+recursive-include ckanext/googleanalytics *.html *.js *.json *.css *.yml *.mo

README.md

@@ -1,5 +1,4 @@
-CKAN Google Analytics Extension
-===============================
+# CKAN Google Analytics Extension
 
 **Status:** Production
 
@@ -8,8 +7,8 @@ CKAN Google Analytics Extension
 A CKAN extension that both sends tracking data to Google Analytics and
 retrieves statistics from Google Analytics and inserts them into CKAN pages.
 
-Features
---------
+## Features
+
 
 * Puts the Google Analytics asynchronous tracking code into your page headers
   for basic Google Analytics page tracking.
@@ -24,26 +23,22 @@ Features
 * Add Google Analytics Event Tracking function that can be used in any exstension
   to create your custom events tracking.
 
-  ``ckanext.googleanalytics.plugin._post_analytics``
+		ckanext.googleanalytics.plugin._post_analytics
+
+## Installation
 
-Installation
-------------
 
 1. Install the extension as usual, e.g. (from an activated virtualenv):
 
-    ::
-
-    $ pip install -e  git+https://github.com/ckan/ckanext-googleanalytics.git#egg=ckanext-googleanalytics
-    $ pip install -r ckanext-googleanalytics/requirements.txt
+		$ pip install -e  git+https://github.com/ckan/ckanext-googleanalytics.git#egg=ckanext-googleanalytics
+		$ pip install -r ckanext-googleanalytics/requirements.txt
 
 2. Edit your development.ini (or similar) to provide these necessary parameters:
 
-    ::
-
-      googleanalytics.id = UA-1010101-1
-      googleanalytics.account = Account name (i.e. data.gov.uk, see top level item at https://www.google.com/analytics)
-      googleanalytics.username = googleaccount@gmail.com
-      googleanalytics.password = googlepassword
+		googleanalytics.id = UA-1010101-1
+		googleanalytics.account = Account name (i.e. data.gov.uk, see top level item at https://www.google.com/analytics)
+		googleanalytics.username = googleaccount@gmail.com
+        googleanalytics.password = googlepassword
 
    Note that your password will probably be readable by other people;
    so you may want to set up a new gmail account specifically for
@@ -52,22 +47,20 @@ Installation
 3. Edit again your configuration ini file to activate the plugin
    with:
 
-   ::
-
-      ckan.plugins = googleanalytics
+		ckan.plugins = googleanalytics
 
    (If there are other plugins activated, add this to the list.  Each
    plugin should be separated with a space).
 
 4. Finally, there are some optional configuration settings (shown here
-   with their default settings)::
+   with their default settings)
 
-      googleanalytics_resource_prefix = /downloads/
-      googleanalytics.domain = auto
-      googleanalytics.track_events = false
-      googleanalytics.fields = {}
-      googleanalytics.enable_user_id = false
-      googleanalytics.download_handler = ckan.views.resource:download
+		googleanalytics_resource_prefix = /downloads/
+		googleanalytics.domain = auto
+        googleanalytics.track_events = false
+        googleanalytics.fields = {}
+        googleanalytics.enable_user_id = false
+        googleanalytics.download_handler = ckan.views.resource:download
 
    ``resource_prefix`` is an arbitrary identifier so that we can query
    for downloads in Google Analytics.  It can theoretically be any
@@ -99,12 +92,12 @@ Installation
    and some of your plugins redefines `resource.download`
    route(ckanext-cloudstorage, for example), you can specify which
    function must be called instead of `ckan.views.resource:download`
-   via `ckanext.googleanalytics.download_handler` config variable. For ckanext-cloudstorage you can use::
+   via `ckanext.googleanalytics.download_handler` config variable. For ckanext-cloudstorage you can use:
 
-     ckanext.googleanalytics.download_handler = ckanext.cloudstorage.views:download
+		ckanext.googleanalytics.download_handler = ckanext.cloudstorage.views:download
+
+# Domain Linking
 
-Domain Linking
---------------
 
 This plugin supports cross-domain tracking using Googles' site linking feature.
 
@@ -112,18 +105,18 @@ To use this, set the ``googleanalytics.linked_domains`` configuration option to
 
 See `Googles' documentation<https://support.google.com/analytics/answer/1034342?hl=en>`_ for more information
 
-Setting Up Statistics Retrieval from Google Analytics
------------------------------------------------------
+# Setting Up Statistics Retrieval from Google Analytics
+
 
 1. Run the following command from ``src/ckanext-googleanalytics`` to
    set up the required database tables (of course, altering the
-   ``--config`` option to point to your site config file)::
+   ``--config`` option to point to your site config file):
 
-       paster initdb --config=../ckan/development.ini
+		paster initdb --config=../ckan/development.ini
 
 2. Optionally, add::
 
-       googleanalytics.show_downloads = true
+		googleanalytics.show_downloads = true
 
    to your CKAN ini file. If ``show_downloads`` is set, a download count for
    resources will be displayed on individual package pages.
@@ -137,7 +130,7 @@ Setting Up Statistics Retrieval from Google Analytics
 6. Import Google stats by running the following command from
    ``src/ckanext-googleanalytics``::
 
-       paster loadanalytics credentials.json --config=../ckan/development.ini
+		paster loadanalytics credentials.json --config=../ckan/development.ini
 
    (Of course, pointing config at your specific site config and credentials.json at the
    key file obtained from the authorization step)
@@ -157,8 +150,8 @@ Setting Up Statistics Retrieval from Google Analytics
    remember to run it by hand, or your statistics won't get updated.
 
 
-Authorization
---------------
+## Authorization
+
 
 Before ckanext-googleanalytics can retrieve statistics from Google Analytics, you need to set up the OAUTH details which you can do by following the `instructions <https://developers.google.com/analytics/devguides/reporting/core/v3/quickstart/service-py>`_ the outcome of which will be a file with authentication key. These steps are below for convenience:
 
@@ -175,17 +168,16 @@ Before ckanext-googleanalytics can retrieve statistics from Google Analytics, yo
 6. Find "User management" button in corresponding column. Add service account using Service account ID(email) generated in 3rd step and grant "Read" role to it.
 
 
-Testing
--------
+## Testing
+
 
 There are some very high-level functional tests that you can run using::
 
-  (pyenv)~/pyenv/src/ckan$ nosetests --ckan ../ckanext-googleanalytics/tests/
+	(pyenv)~/pyenv/src/ckan$ nosetests --ckan ../ckanext-googleanalytics/tests/
 
 (note -- that's run from the CKAN software root, not the extension root)
 
-Future
-------
+## Future
 
 This is a bare-bones, first release of the software.  There are
 several directions it could take in the future.

ckanext/googleanalytics/assets/googleanalytics_event_tracking.js

@@ -1,5 +1,5 @@
 // Add Google Analytics Event Tracking to resource download links.
-this.ckan.module("google-analytics", function(jQuery, _) {
+ckan.module("google-analytics", function(jQuery, _) {
   "use strict";
   return {
     options: {

ckanext/googleanalytics/assets/webassets.yml

@@ -1,5 +1,8 @@
 event_tracking:
   filters: rjsmin
   output: ckanext-googleanalytics/event_tracking.js
+  extra:
+    preload:
+      - base/main
   contents:
     - googleanalytics_event_tracking.js

ckanext/googleanalytics/cli.py

@@ -15,25 +15,24 @@ import ckan.plugins.toolkit as tk
 log = logging.getLogger(__name__)
 PACKAGE_URL = "/dataset/"  # XXX get from routes...
 DEFAULT_RESOURCE_URL_TAG = "/downloads/"
+DEFAULT_RECENT_VIEW_DAYS = 14
 
 RESOURCE_URL_REGEX = re.compile("/dataset/[a-z0-9-_]+/resource/([a-z0-9-_]+)")
 DATASET_EDIT_REGEX = re.compile("/dataset/edit/([a-z0-9-_]+)")
 
 
 def get_commands():
-    return [
-        googleanalytics
-    ]
+    return [googleanalytics]
 
 
 @click.group(short_help=u"GoogleAnalytics commands")
 def googleanalytics():
     pass
 
+
 @googleanalytics.command()
 def init():
-    """Initialise the local stats database tables
-    """
+    """Initialise the local stats database tables"""
     model.Session.remove()
     model.Session.configure(bind=model.meta.engine)
     dbutil.init_tables()
@@ -73,6 +72,14 @@ def _resource_url_tag():
     )
 
 
+def _recent_view_days():
+    return tk.asint(
+        tk.config.get(
+            "googleanalytics.recent_view_days", DEFAULT_RECENT_VIEW_DAYS
+        )
+    )
+
+
 ###############################################################################
 #                                     xxx                                     #
 ###############################################################################
@@ -129,10 +136,10 @@ def internal_save(packages_data, summary_date):
                 SELECT sum(count)
                 FROM tracking_summary t2
                 WHERE t1.url = t2.url
-                AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - 14
+                AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - %s
              ) + t1.count
              WHERE t1.running_total = 0 AND tracking_type = 'resource';"""
-    engine.execute(sql)
+    engine.execute(sql, _recent_view_days())
 
     # update summary totals for pages
     sql = """UPDATE tracking_summary t1
@@ -146,12 +153,12 @@ def internal_save(packages_data, summary_date):
                 SELECT sum(count)
                 FROM tracking_summary t2
                 WHERE t1.package_id = t2.package_id
-                AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - 14
+                AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - %s
              ) + t1.count
              WHERE t1.running_total = 0 AND tracking_type = 'page'
              AND t1.package_id IS NOT NULL
              AND t1.package_id != '~~not~found~~';"""
-    engine.execute(sql)
+    engine.execute(sql, _recent_view_days())
 
 
 def bulk_import(service, profile_id, start_date=None):
@@ -246,8 +253,7 @@ def get_ga_data_new(service, profile_id, start_date=None, end_date=None):
 
 
 def save_ga_data(packages_data):
-    """Save tuples of packages_data to the database
-    """
+    """Save tuples of packages_data to the database"""
     for identifier, visits in list(packages_data.items()):
         recently = visits.get("recent", 0)
         ever = visits.get("ever", 0)
@@ -280,10 +286,13 @@ def save_ga_data(packages_data):
 
 
 def ga_query(
-    service, profile_id, query_filter=None, from_date=None, metrics=None,
+    service,
+    profile_id,
+    query_filter=None,
+    from_date=None,
+    metrics=None,
 ):
-    """Execute a query against Google Analytics
-    """
+    """Execute a query against Google Analytics"""
     now = datetime.datetime.now()
     to_date = now.strftime("%Y-%m-%d")
     if isinstance(from_date, datetime.date):
@@ -322,7 +331,7 @@ def get_ga_data(service, profile_id, query_filter):
        {'identifier': {'recent':3, 'ever':6}}
     """
     now = datetime.datetime.now()
-    recent_date = now - datetime.timedelta(14)
+    recent_date = now - datetime.timedelta(_recent_view_days())
     recent_date = recent_date.strftime("%Y-%m-%d")
     floor_date = datetime.date(2005, 1, 1)
     packages = {}

ckanext/googleanalytics/commands.py

@@ -9,20 +9,21 @@ import time
 from pylons import config as pylonsconfig
 from ckan.lib.cli import CkanCommand
 import ckan.model as model
+from ckan.plugins.toolkit import asint
 
 from . import dbutil
 
 log = logging.getLogger("ckanext.googleanalytics")
 PACKAGE_URL = "/dataset/"  # XXX get from routes...
 DEFAULT_RESOURCE_URL_TAG = "/downloads/"
+DEFAULT_RECENT_VIEW_DAYS = 14
 
 RESOURCE_URL_REGEX = re.compile("/dataset/[a-z0-9-_]+/resource/([a-z0-9-_]+)")
 DATASET_EDIT_REGEX = re.compile("/dataset/edit/([a-z0-9-_]+)")
 
 
 class InitDB(CkanCommand):
-    """Initialise the local stats database tables
-    """
+    """Initialise the local stats database tables"""
 
     summary = __doc__.split("\n")[0]
     usage = __doc__
@@ -63,6 +64,11 @@ class LoadAnalytics(CkanCommand):
         self.resource_url_tag = self.CONFIG.get(
             "googleanalytics_resource_prefix", DEFAULT_RESOURCE_URL_TAG
         )
+        self.recent_view_days = asint(
+            self.CONFIG.get(
+                "googleanalytics.recent_view_days", DEFAULT_RECENT_VIEW_DAYS
+            )
+        )
 
         # funny dance we need to do to make sure we've got a
         # configured session
@@ -121,10 +127,10 @@ class LoadAnalytics(CkanCommand):
                     SELECT sum(count)
                     FROM tracking_summary t2
                     WHERE t1.url = t2.url
-                    AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - 14
+                    AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - %s
                  ) + t1.count
                  WHERE t1.running_total = 0 AND tracking_type = 'resource';"""
-        engine.execute(sql)
+        engine.execute(sql, self.recent_view_days)
 
         # update summary totals for pages
         sql = """UPDATE tracking_summary t1
@@ -138,12 +144,12 @@ class LoadAnalytics(CkanCommand):
                     SELECT sum(count)
                     FROM tracking_summary t2
                     WHERE t1.package_id = t2.package_id
-                    AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - 14
+                    AND t2.tracking_date <= t1.tracking_date AND t2.tracking_date >= t1.tracking_date - %s
                  ) + t1.count
                  WHERE t1.running_total = 0 AND tracking_type = 'page'
                  AND t1.package_id IS NOT NULL
                  AND t1.package_id != '~~not~found~~';"""
-        engine.execute(sql)
+        engine.execute(sql, self.recent_view_days)
 
     def bulk_import(self):
         if len(self.args) == 3:
@@ -262,8 +268,7 @@ class LoadAnalytics(CkanCommand):
             log.info("Saved %s records from google" % len(packages_data))
 
     def save_ga_data(self, packages_data):
-        """Save tuples of packages_data to the database
-        """
+        """Save tuples of packages_data to the database"""
         for identifier, visits in list(packages_data.items()):
             recently = visits.get("recent", 0)
             ever = visits.get("ever", 0)
@@ -304,8 +309,7 @@ class LoadAnalytics(CkanCommand):
         metrics=None,
         sort=None,
     ):
-        """Execute a query against Google Analytics
-        """
+        """Execute a query against Google Analytics"""
         if not to_date:
             now = datetime.datetime.now()
             to_date = now.strftime("%Y-%m-%d")
@@ -345,7 +349,7 @@ class LoadAnalytics(CkanCommand):
            {'identifier': {'recent':3, 'ever':6}}
         """
         now = datetime.datetime.now()
-        recent_date = now - datetime.timedelta(14)
+        recent_date = now - datetime.timedelta(self.recent_view_days)
         recent_date = recent_date.strftime("%Y-%m-%d")
         floor_date = datetime.date(2005, 1, 1)
         packages = {}

ckanext/googleanalytics/controller.py

@@ -15,6 +15,7 @@ from ckan.controllers.api import ApiController
 
 from ckan.exceptions import CkanVersionException
 import ckan.plugins.toolkit as tk
+
 try:
     tk.requires_ckan_version("2.9")
 except CkanVersionException:

ckanext/googleanalytics/plugin/__init__.py

@@ -47,7 +47,9 @@ class AnalyticsPostThread(threading.Thread):
             log.debug("Sending API event to Google Analytics: " + data)
             # send analytics
             res = requests.post(
-                "http://www.google-analytics.com/collect", data, timeout=10,
+                "http://www.google-analytics.com/collect",
+                data,
+                timeout=10,
             )
             # signals to queue job is done
             self.queue.task_done()
@@ -148,6 +150,10 @@ class GoogleAnalyticsPlugin(GAMixinPlugin, p.SingletonPlugin):
             "googleanalytics_fields": str(self.googleanalytics_fields),
             "googleanalytics_linked_domains": self.googleanalytics_linked_domains,
         }
+        #return p.toolkit.render_snippet(
+        #    "googleanalytics/snippets/googleanalytics_header.html", data
+        #)
+        # Updated to GA4 #25196
         return p.toolkit.render_snippet(
-            "googleanalytics/snippets/googleanalytics_header.html", data
+            "googleanalytics/snippets/gtag_header.html", data
         )

ckanext/googleanalytics/plugin/flask_plugin.py

@@ -6,6 +6,7 @@ import ckan.plugins as plugins
 from ckanext.googleanalytics.views import ga
 from ckanext.googleanalytics.cli import get_commands
 
+
 class GAMixinPlugin(plugins.SingletonPlugin):
     plugins.implements(plugins.IBlueprint)
     plugins.implements(plugins.IClick)

ckanext/googleanalytics/templates/googleanalytics/snippets/gtag_header.html

@@ -0,0 +1,9 @@
+<!-- Google tag (gtag.js) -->
+<script async src="https://www.googletagmanager.com/gtag/js?id={{googleanalytics_id}}"></script>
+<script>
+  window.dataLayer = window.dataLayer || [];
+  function gtag(){dataLayer.push(arguments);}
+  gtag('js', new Date());
+
+  gtag('config', '{{googleanalytics_id}}');
+</script>

ckanext/googleanalytics/views.py

@@ -14,7 +14,7 @@ import ckan.views.resource as resource
 
 from ckan.common import g
 
-CONFIG_HANDLER_PATH = 'googleanalytics.download_handler'
+CONFIG_HANDLER_PATH = "googleanalytics.download_handler"
 
 log = logging.getLogger(__name__)
 ga = Blueprint("google_analytics", "google_analytics")
@@ -40,7 +40,9 @@ def action(logic_function, ver=api.API_MAX_VERSION):
 
 
 ga.add_url_rule(
-    "/api/action/<logic_function>", methods=["GET", "POST"], view_func=action,
+    "/api/action/<logic_function>",
+    methods=["GET", "POST"],
+    view_func=action,
 )
 ga.add_url_rule(
     u"/<int(min=3, max={0}):ver>/action/<logic_function>".format(
@@ -57,11 +59,9 @@ def download(id, resource_id, filename=None, package_type="dataset"):
         handler = import_string(handler_path, silent=True)
     else:
         handler = None
-        log.warning((
-            'Missing {} config option.'
-        ).format(CONFIG_HANDLER_PATH))
+        log.warning(("Missing {} config option.").format(CONFIG_HANDLER_PATH))
     if not handler:
-        log.debug('Use default CKAN callback for resource.download')
+        log.debug("Use default CKAN callback for resource.download")
         handler = resource.download
     _post_analytics(
         g.user,
@@ -71,8 +71,11 @@ def download(id, resource_id, filename=None, package_type="dataset"):
         resource_id,
     )
     return handler(
-        package_type=package_type, id=id,
-        resource_id=resource_id, filename=filename)
+        package_type=package_type,
+        id=id,
+        resource_id=resource_id,
+        filename=filename,
+    )
 
 
 ga.add_url_rule(

pyproject.toml

@@ -1,3 +1,63 @@
 [tool.black]
 line-length = 79
-include = '\.py$'
+
+[tool.towncrier]
+issue_format = ""
+directory = "changes"
+package = "ckanext.googleanalytics"
+package_dir = "ckanext"
+filename = "CHANGELOG.rst"
+name = "ckanext-googleanalytics"
+
+[tool.pyright]
+pythonVersion = "3.8"
+include = ["ckanext/googleanalytics"]
+exclude = [
+    "**/test*",
+    "**/migration",
+]
+ignore = [
+       "ckan"
+]
+strict = []
+
+strictParameterNoneValue = true # type must be Optional if default value is None
+
+reportFunctionMemberAccess = true # non-standard member accesses for functions
+reportMissingImports = true
+reportMissingModuleSource = true
+reportMissingTypeStubs = false
+reportImportCycles = false
+reportUnusedImport = false
+reportUnusedClass = true
+reportUnusedFunction = true
+reportUnusedVariable = false
+reportDuplicateImport = true
+reportOptionalSubscript = true
+reportOptionalMemberAccess = true
+reportOptionalCall = true
+reportOptionalIterable = true
+reportOptionalContextManager = true
+reportOptionalOperand = true
+reportTypedDictNotRequiredAccess = false # We are using Context in a way that conflicts with this check
+reportConstantRedefinition = false
+reportIncompatibleMethodOverride = false
+reportIncompatibleVariableOverride = true
+reportOverlappingOverload = true
+reportUntypedFunctionDecorator = false
+reportUnknownParameterType = false # it creates a lot of noise
+reportUnknownArgumentType = false
+reportUnknownLambdaType = false
+reportMissingTypeArgument = true
+reportInvalidTypeVarUse = true
+reportCallInDefaultInitializer = true
+reportUnknownVariableType = false
+reportUntypedBaseClass = false # ignore it because we are relying on untyped CKAN
+reportUnnecessaryIsInstance = true
+reportUnnecessaryCast = true
+reportUnnecessaryComparison = true
+reportAssertAlwaysTrue = true
+reportSelfClsParameterName = true
+reportUnusedCallResult = false # allow function calls for side-effect only (like logic.check_acces)
+
+useLibraryCodeForTypes = true

setup.py

@@ -2,7 +2,7 @@ import os
 from setuptools import setup, find_packages
 HERE = os.path.dirname(__file__)
 
-version = "2.0.3"
+version = "2.0.7"
 
 extras_require = {}
 _extras_groups = [
@@ -12,12 +12,16 @@ for group, filepath in _extras_groups:
     with open(os.path.join(HERE, filepath), 'r') as f:
         extras_require[group] = f.readlines()
 
+# Get the long description from the relevant file
+with open(os.path.join(HERE, 'README.md'), encoding='utf-8') as f:
+    long_description = f.read()
+
 setup(
     name="ckanext-googleanalytics",
     version=version,
     description="Add GA tracking and reporting to CKAN instance",
-    long_description="""\
-	""",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
     classifiers=[],  # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
     keywords="",
     author="Seb Bacon",