lsmyrnaios
/
UrlsController
1
0
Fork 2
Code Issues Pull Requests Releases 31 Wiki Activity
The Controller app of the PDF Aggregation Service.
474 Commits 4 Branches 31 Tags 1.3 MiB
Java 98.6%
Shell 1.3%
Go to file
Lampros Smyrnaios c843880878 Update the "passSecurityChecks" method to accept the IPV6 loopback uncompressed address. 2026-01-02 16:53:17 +02:00
documentation - Add large-scale documentation files. 2025-12-18 14:12:22 +02:00
gradle/wrapper - Show warn-message when the stats-queries failed because the tables are in a "merging"-state. 2025-11-18 13:03:21 +02:00
prometheus - Improve error-handling in "UrlsCOntrollerApplication.gentleAppShutdown()". 2025-12-09 17:11:02 +02:00
src/main Update the "passSecurityChecks" method to accept the IPV6 loopback uncompressed address. 2026-01-02 16:53:17 +02:00
.gitignore springified project 2022-01-30 22:15:13 +02:00
Dockerfile Downgrade to Java-21 as code from hadoop-libraries still call deprecated (and removed in Java-25) Java-APIs, resulting in some runtime errors. 2025-10-24 14:00:57 +03:00
LICENSE - Use "snake_case" names for custom Prometheus-metrics. 2025-01-18 23:42:23 +02:00
README.md - Add large-scale documentation files. 2025-12-18 14:12:22 +02:00
build.gradle - Increase intentional waiting between heavy queries in "ParquetFileUtils.compactTable()" to make the transaction more robust. 2025-12-31 02:06:30 +02:00
docker-compose.yml - Improve error-handling in "UrlsCOntrollerApplication.gentleAppShutdown()". 2025-12-09 17:11:02 +02:00
gradle.properties - Improve text-layout in README. 2024-12-19 23:00:05 +02:00
gradlew - Fix missing "execute"-permission for "gradlew"-scripts. 2025-12-11 16:38:44 +02:00
gradlew.bat - Fix missing "execute"-permission for "gradlew"-scripts. 2025-12-11 16:38:44 +02:00
installAndRun.sh - Fix missing "execute"-permission for "gradlew"-scripts. 2025-12-11 16:38:44 +02:00
settings.gradle - Add the "isControllerAlive"-endpoint. 2021-09-23 15:08:52 +03:00
shutdownService.sh - Show warn-message when the stats-queries failed because the tables are in a "merging"-state. 2025-11-18 13:03:21 +02:00

README.md

PDF Aggregation Service - Controller

Jenkins build status

The UrlsController is the central component of the PDF Aggregation Service, a distributed system designed for large-scale collection and processing of full-text content from the web.

This service acts as a coordinator, generating and distributing crawling assignments to a fleet of remote "Worker" nodes. It then orchestrates the retrieval, storage, and metadata processing of the collected content, leveraging a powerful data stack that includes Apache Impala, Parquet, and HDFS.

Key Features

  • Distributed Crawling: Manages a fleet of remote workers to efficiently crawl and retrieve full-text content (PDFs and HTML).
  • Bulk Import: Supports the ingestion of large, pre-existing datasets from various sources.
  • Scalable Data Processing: Built on a foundation of Apache Impala and Parquet for efficient, large-scale data storage and querying.
  • Asynchronous and Robust: Designed for high-throughput, asynchronous processing to handle long-running tasks without blocking.
  • Comprehensive Monitoring: Exposes a rich set of metrics for real-time monitoring with Prometheus and Grafana.

Documentation

For a deeper understanding of the system, please refer to the following documents:

Implementation Notes

  • Data Transfer: The service uses Facebook's Zstandard compression algorithm for efficient transfer of full-text files.
  • File Naming Convention: Uploaded full-text files follow the naming scheme: datasourceID/recordID::fileHash.pdf.