Changelog

Pelican Releases

Every Pelican release, with notes grouped by version.

V7.26

Breaking Changes & Upgrade Notes for Admins

  • [All]: Clients and Servers are now separate binaries, with the client binary now less than ⅓ the size of the server binary. #3343
  • [All]: Durations in config params must now include units #3396
  • [Origin/Cache]: XRootD bumped to version 5.9.2-1.1 #3379
  • [Origin/Cache]: XRootD-S3-HTTP plugin bumped to version v0.6.7 to fix a bug with S3 Listings #3381

New Features and Enhancements

  • [Client]: Added expiration flag in token creation CLI #3339
  • [Client]: Added a new client-only container image #3383
  • [Origin]: Added a multi-user version of the POSIXv2 backend #3214
  • [Web-UI]: Admins can now edit users and groups in the UI

Bugs Fixed

  • [All] Security Improvements #3313

Pelican v7.26.0

This release focuses on code cleanup and security changes. It has some underlying CLI upgrades and includes upgrades of dependency plugins. The major change is that the server and client binaries are now separated into the pelican and pelican-server binaries.

Breaking Changes & Upgrade Notes for Admins

  • [All]: Clients and Servers are now separate binaries, with the client binary now less than ⅓ the size of the server binary. #3343
  • [All]: Durations in config params must now include units #3396
  • [Origin/Cache]: XRootD bumped to version 5.9.2-1.1 #3379
  • [Origin/Cache]: XRootD-S3-HTTP plugin bumped to version v0.6.7 to fix a bug with S3 Listings #3381

New Features and Enhancements

  • [Client]: Added expiration flag in token creation CLI #3339
  • [Client]: Added a new client-only container image #3383
  • [Origin]: Added a multi-user version of the POSIXv2 backend #3214
  • [Web-UI]: Admins can now edit users and groups in the UI

Bugs Fixed

  • [All] Security Improvements #3313

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.25.2...v7.26.0

V7.25

Breaking Changes & Upgrade Notes for Admins

  • [Cache]: [Admin Action Required] Override the new storage-health defaults on caches: set Monitoring.StorageWarningThreshold: 92 and Monitoring.StorageCriticalThreshold: 97. The defaults added in #3241 (80/90) sit below typical cache purge watermarks, causing the Director to filter the cache before it can purge. The knobs were unhidden in #3453; defaults will be fixed in a follow-up release.
  • [Origin]: Hardened the POSC plugin to prevent startup across filesystem boundaries in #3269; Added POSC bypass filter for monitoring paths in #3297

New Features and Enhancements

  • [Origin]: POSIX origins now validate filesystem permissions against namespace capabilities at startup in #2881
  • [Origin]: Added an embedded OAuth2 issuer in #3151
  • [Origin]: Third-party copies are now enabled in origins out-of-the-box in #3148
  • [Origin]: Integrated the N2N plugin to automatically map FederationPrefix to StoragePrefix for XRootD storage backends in #3111
  • [Servers]: Added periodic SQLite database backups with compression and encryption in #3154
  • [Servers]: Added a web read-only mode (Server.WebReadOnly) to restrict mutating operations via the web UI in #3205
  • [Client]: Improved object ls output with sorted entries, terminal-aware column formatting, and clear directory distinction in #3250
  • [Client]: Transfers now continue with explicitly specified caches (-c/Client.PreferredCaches) when the Director
  • [Director]: Director redirect information is now probabilistically added to plugin output class ads in #3177
  • [Registry]: Displayed registration type explicitly in Registry’s WebUI in #3218
  • [Origin/Cache]: Presented the registered server name provided in Origin and Cache’ WebUI #3220

Bugs Fixed

  • [Client]: The client now accurately reports when a token is not sent in #3223
  • [Client]: Preserved PreferredCaches ordering against responsiveness-based reordering in sortAttempts in #3253
  • [Client]: Fixed token discovery logic in #2846
  • [Client]: Fixed checksum verification for mismatched algorithm types in #3174
  • [Director]: The Director no longer stats more caches than it intends to provide for sorting in #3179
  • [Cache]: Caches now always use the federation host for the metadata discovery endpoint in #3268
  • [Cache]: Disabled client x509 by default, as most CAs will no longer issue certificates with both client/server auth in the EKU #3202
  • [Origin]: Fixed Globus origin to correctly handle non-existent directories in #3186, rejected multiple exports #3170, and OAuth flow requires Server.EnableUI = true #3144
  • [Origin]: StoragePrefix and FederationPrefix are no longer validated the same way in #3267
  • [Servers]: Configuration values marked as hidden are now hidden in the config editor webUI in #3222
  • [Servers]: Downtime scheduling now prevents an end time before the start time in #3243
  • [Director]: Fixed HA director cascade-failure bugs in peer discovery and self-advertisement #3452
  • [Director]: Backported director-related JWT clock-skew fixes (premature advertisement expiry) #3473
  • [Director]: Stopped iterating the live serverAds cache under its own lock to avoid lock contention #3513
  • [Origin]: Amended various storage parameter defaults in #3454

New Contributors

  • @Macman1234 made their first contribution in #3098

Dependency Updates

  • [All]: Upgraded the XRootD build dependency from 5.9.1 to 5.9.2
  • [All]: Upgraded lestrrat-go/jwx/v2 to v2.1.6 to pick up jwt.WithResetValidators

Bugs Fixed

  • [Director]: Fixed HA director cascade-failure bugs in peer discovery and self-advertisement #3452
  • [Director]: Backported director-related JWT clock-skew fixes (premature advertisement expiry) #3473
  • [Director]: Stopped iterating the live serverAds cache under its own lock to avoid lock contention #3513
  • [Origin]: Amended various storage parameter defaults in #3454

Dependency Updates

  • [All]: Upgraded the XRootD build dependency from 5.9.1 to 5.9.2
  • [All]: Upgraded lestrrat-go/jwx/v2 to v2.1.6 to pick up jwt.WithResetValidators

Pelican v7.25.0

This release introduces an embedded OAuth2 issuer as the OA4MP alternative third-party copy enabled by default for origins, a web read-only mode for servers, and periodic encrypted database backups. It also includes a broad set of improvements to the client, UI, director reliability, security, and federation operations.

Breaking Changes & Upgrade Notes for Admins

  • [Cache]: [Admin Action Required] Override the new storage-health defaults on caches: set Monitoring.StorageWarningThreshold: 92 and Monitoring.StorageCriticalThreshold: 97. The defaults added in #3241 (80/90) sit below typical cache purge watermarks, causing the Director to filter the cache before it can purge. The knobs were unhidden in #3453; defaults will be fixed in a follow-up release.
  • [Origin]: Hardened the POSC plugin to prevent startup across filesystem boundaries in #3269; Added POSC bypass filter for monitoring paths in #3297

New Features and Enhancements

  • [Origin]: POSIX origins now validate filesystem permissions against namespace capabilities at startup in #2881
  • [Origin]: Added an embedded OAuth2 issuer in #3151
  • [Origin]: Third-party copies are now enabled in origins out-of-the-box in #3148
  • [Origin]: Integrated the N2N plugin to automatically map FederationPrefix to StoragePrefix for XRootD storage backends in #3111
  • [Servers]: Added periodic SQLite database backups with compression and encryption in #3154
  • [Servers]: Added a web read-only mode (Server.WebReadOnly) to restrict mutating operations via the web UI in #3205
  • [Client]: Improved object ls output with sorted entries, terminal-aware column formatting, and clear directory distinction in #3250
  • [Client]: Transfers now continue with explicitly specified caches (-c/Client.PreferredCaches) when the Director is unreachable in #3028
  • [Director]: Director redirect information is now probabilistically added to plugin output class ads in #3177
  • [Registry]: Displayed registration type explicitly in Registry’s WebUI in #3218
  • [Origin/Cache]: Presented the registered server name provided in Origin and Cache’ WebUI #3220

Bugs Fixed

  • [Client]: The client now accurately reports when a token is not sent in #3223
  • [Client]: Preserved PreferredCaches ordering against responsiveness-based reordering in sortAttempts in #3253
  • [Client]: Fixed token discovery logic in #2846
  • [Client]: Fixed checksum verification for mismatched algorithm types in #3174
  • [Director]: The Director no longer stats more caches than it intends to provide for sorting in #3179
  • [Cache]: Caches now always use the federation host for the metadata discovery endpoint in #3268
  • [Cache]: Disabled client x509 by default, as most CAs will no longer issue certificates with both client/server auth in the EKU #3202
  • [Origin]: Fixed Globus origin to correctly handle non-existent directories in #3186, rejected multiple exports #3170, and OAuth flow requires Server.EnableUI = true #3144
  • [Origin]: StoragePrefix and FederationPrefix are no longer validated the same way in #3267
  • [Servers]: Configuration values marked as hidden are now hidden in the config editor webUI in #3222
  • [Servers]: Downtime scheduling now prevents an end time before the start time in #3243

New Contributors

  • @Macman1234 made their first contribution in #3098

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.24.3...v7.25.0

V7.24

Breaking Changes & Upgrade Notes for Admins

  • [Cache/Origin]: Caches and Origins have a new dependency requirement on xrdhttp-pelican >= v0.0.10

New Features and Enhancements

  • [Client]: Made significant updates to error classifications in Clients to provide users with more actionable/intelligible errors in #3089
  • [Client]: Introduced a Client API Server for Pelican that exposes Client functionality through a RESTful API over Unix domain sockets, enabling programmatic access to Pelican transfer capabilities without directly invoking the CLI in #2780
  • [Client]: Improved performance of the Client's object prestage command when using Caches that support the pre-stage API in #3014
  • [Client]: Introduced a --dry-run option for the Client's object commands that lets users preview what actions the Client will take before moving data in #2865
  • [Servers]: Added CLI tooling to create API keys for interacting with Pelican servers in #2958
  • [Director]: Made discrepancies between federation metadata hosted at the Discovery URL and federation Directors more obvious and less prone to drift in #3009 and #3008
  • [Servers]: Added the ability for Pelican servers to analyze/report on various storage metrics, including disk consumption by Origin exports and health monitoring for service log files in #2957 and #2967
  • [Cache/Origin]: Added support for PKCS#11 to protect TLS credentials from unanticipated XRootD security vulnerabilities in #2948, #3055, and #3072
  • [Cache/Origin]: Caches and Origins now check for installation of a minimum required XRootD version on startup and cause an error if no such version is found in #3092

Bugs Fixed

  • [Client]: Fixed a progress bar bug that caused degraded performance with recursive Client actions (e.g. pelican object get --recursive) on namespaces containing ~100k+ objects in #2975
  • [Client]: Made it more explicit in Client checksum operations when a checksum type other than what was requested is returned by a server in #2847
  • [Director]: Fixed a bug that prevented the Director from logging errors or recording failure metrics when some Cache Director tests failed in #3099
  • [Cache]: Patched instantaneous client timeout bug in Cache middleware (xrdcl-pelican) in #3217
  • [Origin]: Restored missing sssd dependency in Origin containers in #3312
  • [Origin]: Patched Origin multiuser plugin for containers to fix segfault bug in #3303

Pelican v7.24.3

Contains bugfixes for Caches and Origins

Bugs Fixed

  • [Cache]: Patched instantaneous client timeout bug in Cache middleware (xrdcl-pelican) in #3217
  • [Origin]: Restored missing sssd dependency in Origin containers in #3312
  • [Origin]: Patched Origin multiuser plugin for containers to fix segfault bug in #3303

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.24.2…v7.24.3

This patch release resolves a known vulnerability.

Pelican v7.24.0

This release introduces a substantial set of new features requested by users and server administrators, with development accelerated through AI-assisted engineering workflows.

Breaking Changes & Upgrade Notes for Admins

  • [Cache/Origin]: Caches and Origins have a new dependency requirement on xrdhttp-pelican >= v0.0.10

New Features and Enhancements

  • [Client]: Made significant updates to error classifications in Clients to provide users with more actionable/intelligible errors in #3089
  • [Client]: Introduced a Client API Server for Pelican that exposes Client functionality through a RESTful API over Unix domain sockets, enabling programmatic access to Pelican transfer capabilities without directly invoking the CLI in #2780
  • [Client]: Improved performance of the Client's object prestage command when using Caches that support the pre-stage API in #3014
  • [Client]: Introduced a --dry-run option for the Client's object commands that lets users preview what actions the Client will take before moving data in #2865
  • [Servers]: Added CLI tooling to create API keys for interacting with Pelican servers in #2958
  • [Director]: Made discrepancies between federation metadata hosted at the Discovery URL and federation Directors more obvious and less prone to drift in #3009 and #3008
  • [Servers]: Added the ability for Pelican servers to analyze/report on various storage metrics, including disk consumption by Origin exports and health monitoring for service log files in #2957 and #2967
  • [Cache/Origin]: Added support for PKCS#11 to protect TLS credentials from unanticipated XRootD security vulnerabilities in #2948, #3055, and #3072
  • [Cache/Origin]: Caches and Origins now check for installation of a minimum required XRootD version on startup and cause an error if no such version is found in #3092

Bugs Fixed

  • [Client]: Fixed a progress bar bug that caused degraded performance with recursive Client actions (e.g. pelican object get --recursive) on namespaces containing ~100k+ objects in #2975
  • [Client]: Made it more explicit in Client checksum operations when a checksum type other than what was requested is returned by a server in #2847
  • [Director]: Fixed a bug that prevented the Director from logging errors or recording failure metrics when some Cache Director tests failed in #3099

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.23.0...v7.24.0

V7.23

New Features and Enhancements

  • [All]: Non-root users who do not provide configuration in ~/.config/pelican will now fallback to look at system configuration in /etc/pelican #2990
  • [Client]: Added --direct flag to allow for object retrieval directly from an origin #2864
  • [Client]: Implemented a golang io.FS interface in the client #2962
  • [Plugin]: The plugin can now retrieve Pelican configuration variables from HTCondor ClassAds #2891
  • [Plugin]: There is now a CacheHit field in transfer output ClassAd #2952
  • [Origin]: Full support for Globus origins - now allows for ls and writes #2769
  • [Cache/Origin]: Now allows for admin groups #2863
  • [Servers]: Added the ability to temporarily change the log level on running servers #2896
  • [Servers]: Added support for GitHub OAuth2 authentication #2901

Bugs Fixed

  • [Client]: Implemented retries for some types of client ls failures #2910
  • [Client]: Pelican sync no longer creates directories with incorrect permissions
  • [Plugin]: PELICAN_CLIENT_PREFERREDCACHES environment variable no longer ignored by the plugin #2828
  • [Director]: Fixed race condition in director advertise shutdown #2922
  • [Director]: Fixed frequent panics and restarts due to two Prometheus endpoints #2936
  • [Cache/Origin]: Origins and Caches no longer need Registry acknowledgement to go into downtime #2942

This patch release resolves a known vulnerability.

Pelican v7.23.0

This release brings new features and enhancements - such as write support for Globus origins and a golang io.FS interface. It also includes some behind the scenes reliability improvements.

New Features and Enhancements

  • [All]: Non-root users who do not provide configuration in ~/.config/pelican will now fallback to look at system configuration in /etc/pelican #2990
  • [Client]: Added --direct flag to allow for object retrieval directly from an origin #2864
  • [Client]: Implemented a golang io.FS interface in the client #2962
  • [Plugin]: The plugin can now retrieve Pelican configuration variables from HTCondor ClassAds #2891
  • [Plugin]: There is now a CacheHit field in transfer output ClassAd #2952
  • [Origin]: Full support for Globus origins - now allows for ls and writes #2769
  • [Cache/Origin]: Now allows for admin groups #2863
  • [Servers]: Added the ability to temporarily change the log level on running servers #2896
  • [Servers]: Added support for GitHub OAuth2 authentication #2901

Bugs Fixed

  • [Client]: Implemented retries for some types of client ls failures #2910
  • [Client]: Pelican sync no longer creates directories with incorrect permissions
  • [Plugin]: PELICAN_CLIENT_PREFERREDCACHES environment variable no longer ignored by the plugin #2828
  • [Director]: Fixed race condition in director advertise shutdown #2922
  • [Director]: Fixed frequent panics and restarts due to two Prometheus endpoints #2936
  • [Cache/Origin]: Origins and Caches no longer need Registry acknowledgement to go into downtime #2942

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.22.1...v7.23.0

V7.22

Breaking Changes & Upgrade Notes for Admins

  • [Origin]: For admins using the Multi-User feature, checksums will now be written to the xattrs of uploaded files

New Features and Enhancements

  • [**Client/Origin **] New collection management CLI commands and revamped collections authorization #2971
  • [All] A new web interface for managing groups and user membership #2740
  • [Cache] A new Site-Local cache mode #2798
  • [All] Prometheus data retention policy is now configurable #2779
  • [Cache/Origin] New health metrics for config update and XRootD self-tests #2666
  • [Client] More robust error classifications in client #2825
  • [Director/Origin] Director can now scrape metrics from brokered Origins #2824
  • [Origin]: Enhanced authorization template form with new multi-select fields for improved value selection in #2968

Bugs Fixed

  • [Client] Fixed authentication loop issue in the client #2790
  • [Director] Fixed namespace TTL cache behavior in the Director #2821
  • [Origin]: Fixed multi-user plugin configuration that was preventing proper initialization with checksumming in #2978 and #3041
  • [Director]: Fix director's stat feature to no longer "leak" goroutines #2935
  • [All]: Updated Docker image build process to use stock AlmaLinux base image for cleaner builds in #2992
  • [All]: Added bsdtar to work around ARM64 emulation bug with standard tar command in #3069

This patch release resolves a known vulnerability.

Pelican v7.22.1

This release fixes critical bugs in the multi-user plugin configuration and prevents potential deadlocks in stat routines.

Breaking Changes & Upgrade Notes for Admins

  • [Origin]: For admins using the Multi-User feature, checksums will now be written to the xattrs of uploaded files

New Features and Enhancements

  • [Origin]: Enhanced authorization template form with new multi-select fields for improved value selection in #2968

Bugs Fixed

  • [Origin]: Fixed multi-user plugin configuration that was preventing proper initialization with checksumming in #2978 and #3041
  • [Director]: Fix director's stat feature to no longer "leak" goroutines #2935
  • [All]: Updated Docker image build process to use stock AlmaLinux base image for cleaner builds in #2992
  • [All]: Added bsdtar to work around ARM64 emulation bug with standard tar command in #3069

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.22.0...v7.22.1

Pelican v7.22.0

This release brings a broad set of improvements to reliability, observability, and authorization, offering meaningful quality-of-life upgrades for operators and users.

Breaking Changes & Upgrade Notes for Admins

Several config parameters (e.g., Debug, various DbLocation keys) have been formally deprecated or removed. The central services (Registry & Director) must be upgraded before Origins and Caches. The ordering between Registry and Director upgrade doesn't matter.

New Features and Enhancements

  • [**Client/Origin **] New collection management CLI commands and revamped collections authorization #2971
  • [All] A new web interface for managing groups and user membership #2740
  • [Cache] A new Site-Local cache mode #2798
  • [All] Prometheus data retention policy is now configurable #2779
  • [Cache/Origin] New health metrics for config update and XRootD self-tests #2666
  • [Client] More robust error classifications in client #2825
  • [Director/Origin] Director can now scrape metrics from brokered Origins #2824

Bugs Fixed

  • [Client] Fixed authentication loop issue in the client #2790
  • [Director] Fixed namespace TTL cache behavior in the Director #2821

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.21.1..v7.22.0

V7.21

Breaking Changes & Upgrade Notes for Admins

  • [Cache/Origin] Enabled auto shutdown a Cache/Origin server if its scitokens.cfg file and/or Authfile are not updating for a given time (1h by default) #2666
  • [All] Deprecated Debug param and modified default client/server log levels #2750
  • [Servers] Switched to installing the pelican-server binary in all server images #2725
  • [Servers] Updated Goreleaser config to retain symbols in release builds #2667
  • [Registry] Added cleanup mechanism for stale entries in servers and services tables upon upgrade #2745; Additionally, the creation of new stale entries will be prevented #2748, #2739

New Features and Enhancements

  • [All] Supported Dark Mode in Pelican Docs #2651
  • [All] Added versions to Pelican Docs #2749
  • [Client] Improved transfer error handling #2742, #2733, #2757
  • [Client] Added Permission Denied error to Client/Plugin #2669
  • [Client] Logged checksum error only when checksum is required but failing to get #2693
  • [Servers] Improved downtimes UI #2636
  • [Registry] Added Deletion Confirm Button #2670
  • [Registry] Supported Autofill Security Contact if the registration is from Origin #2630
  • [Client] Supported depth limited recursive listings in pelican object ls command #2720
  • [Client] Checked local filesystem before contacting director on upload #2677
  • [Servers] Made OIDC scopes configurable #2726

Bugs Fixed

  • [Client]: If the download fails for some reason (ex 404), no half-done file remains on disk #2701
  • [Cache/Origin] Fixed a bug preventing Origins with Logging.Level: Warn from starting #2712
  • [Director] Fixed a JSON marshalling bug that causes Origin/Cache-set downtimes stuck in Director when no active downtime #2684
  • [Cache] Fixed repetitive error log message for the XrdCurlStats monitoring #2775
  • [Origin]: Fixed a bug that prevented Origins from parsing some custom authfile attributes in #2778
  • [Cache/Origin] Trigger degraded state with a configurable ratio of server concurrency #2801
  • [Client] Make tls.AlertError retryable #2813
  • [All] Fixed 404 error when switching to other Pelican Docs versions from v7.21 #2804
  • [Director] Fixed a TTL cache bug that prevents the eviction of stale JWKS #2821
  • [Origin] Restore the ability to view an Origin's directory listing directly in the browser #2842

This patch release resolves a known vulnerability.

Pelican v7.21.2

Bugs Fixed

  • [Cache/Origin] Trigger degraded state with a configurable ratio of server concurrency #2801
  • [Client] Make tls.AlertError retryable #2813
  • [All] Fixed 404 error when switching to other Pelican Docs versions from v7.21 #2804
  • [Director] Fixed a TTL cache bug that prevents the eviction of stale JWKS #2821
  • [Origin] Restore the ability to view an Origin's directory listing directly in the browser #2842

Pelican v7.21.1

Pelican v7.21.1 addresses a build infrastructure issue that was preventing Docker images from being built successfully due to insufficient disk space in the GitHub Actions runner. No changes were made on Pelican client and server.

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.21.0…v7.21.1

Pelican v7.21.0

This release comes with UX improvements on Pelican Client, HTCondor Plugin, and Servers. It also provides a short circuit mechanism for Caches/Origins with outdated XRootD config files, and other useful updates.

Breaking Changes & Upgrade Notes for Admins

This release further standardizes using a Federation's "Discovery URL" as the root of trust when the Federation central services generate authZ tokens – this may cause issues if the Director's private signing key does NOT match one of the keys exported by the Discovery URL's issuer metadata. To check for metadata alignment, visit the /.well-known/pelican-configuration path at your Discovery URL and follow the jwks_uri link. Compare the keys here with the keys from the jwks_uri link at your Director's /.well-known/openid-configuration path. The Director's keys should be in the list you accessed via the Discovery URL.

  • [Cache/Origin] Enabled auto shutdown a Cache/Origin server if its scitokens.cfg file and/or Authfile are not updating for a given time (1h by default) #2666
  • [All] Deprecated Debug param and modified default client/server log levels #2750
  • [Servers] Switched to installing the pelican-server binary in all server images #2725
  • [Servers] Updated Goreleaser config to retain symbols in release builds #2667
  • [Registry] Added cleanup mechanism for stale entries in servers and services tables upon upgrade #2745; Additionally, the creation of new stale entries will be prevented #2748, #2739

New Features and Enhancements

  • [All] Supported Dark Mode in Pelican Docs #2651
  • [All] Added versions to Pelican Docs #2749
  • [Client] Improved transfer error handling #2742, #2733, #2757
  • [Client] Added Permission Denied error to Client/Plugin #2669
  • [Client] Logged checksum error only when checksum is required but failing to get #2693
  • [Servers] Improved downtimes UI #2636
  • [Registry] Added Deletion Confirm Button #2670
  • [Registry] Supported Autofill Security Contact if the registration is from Origin #2630
  • [Client] Supported depth limited recursive listings in pelican object ls command #2720
  • [Client] Checked local filesystem before contacting director on upload #2677
  • [Servers] Made OIDC scopes configurable #2726

Bugs Fixed

  • [Client]: If the download fails for some reason (ex 404), no half-done file remains on disk #2701
  • [Cache/Origin] Fixed a bug preventing Origins with Logging.Level: Warn from starting #2712
  • [Director] Fixed a JSON marshalling bug that causes Origin/Cache-set downtimes stuck in Director when no active downtime #2684
  • [Cache] Fixed repetitive error log message for the XrdCurlStats monitoring #2775
  • [Origin]: Fixed a bug that prevented Origins from parsing some custom authfile attributes in #2778

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.20.2...v7.21.0

V7.20

New Features and Enhancements

  • [Client]: Modified the Client command pelican object ls to produce single column outputs for better piping with grep in #2621
  • [Plugin]: Enabled the HTCondor plugin to run condor_reconfig automatically whenever it's updated in #2624
  • [Origin]: Gave Origin admins the ability to configure the issuer URL set in the Director's X-Pelican-Token-Generation response header (used by Clients for bootstrapping automatic token generation) in #2598
  • [Cache]: Made a variety of new XRootD metrics available via Prometheus to aid in debugging server performance in #2597 and #2588
  • [Director]: Changed the Director's UI to distinguish between a Cache/Origin's XRootD health tests and its overall server health status in #2625

Bugs Fixed

  • [Client]: Fixed various authorization errors related to listings and recursive deletes in #2499 and #2579
  • [Client]: Fixed a bug that prevented pelican origin token create from using custom paths for private keys in #2641
  • [Client]: Fixed a Client deadlock that occurred in some transfers in #2655
  • [Servers]: Fixed an RPM packaging bug that overrode the OSDF's discovery URL in #2612
  • [Cache]: Updated containerized Cache dependencies to fix a variety of deadlocks/segfault in #2611 and #2615
  • [Director]: Fixed a bug in the Director's "adaptive sort" algorithm that resulted in sending Clients to some broken Caches in #2647
  • [Cache]: Upgraded the containerized xrdcl-pelican dependency to v1.5.6 to fix a Cache thread deadlock in #2688
  • [Director]: Fixed a bug that prevented expired downtimes set by Origin/Cache admins from clearing in #2684
  • [Origin]: Bumped the containerized xrootd-s3-http version to fix a bug in S3 Origins that could cause server crashes in #2718
  • [Cache/Origin]: Changed the way authfile entries are sorted to prevent applying access policies from public namespaces to private namespaces when the public namespace is a prefix of the private (e.g. /foo is public and /foo-auth is private) in #2705
  • [Cache/Origin]: Fixed a bug that prevented Caches and Origins from starting when Pelican's global logging level was set to Warn in #2712
  • [Director]: Made small changes to the Director's adaptive sort algorithm to help stabilize load distribution around closer caches in #2716
  • [Origin]: Fixed a bug that prevented Origins from parsing some custom authfile attributes in #2778

New Contributors

  • @fallow64 made their first contribution to Pelican documentation in #2511

Pelican v7.20.3

This release comes with a bugfix for custom XRootD authfiles at Origins.

Bugs Fixed

  • [Origin]: Fixed a bug that prevented Origins from parsing some custom authfile attributes in #2778

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.20.2…v7.20.3

Pelican v7.20.2

This release comes with general Cache/Origin stability bugfixes, tunes the Director's adaptive sorting algorithm and provides an important security update for Caches/Origins.

Bugs Fixed

  • [Origin]: Bumped the containerized xrootd-s3-http version to fix a bug in S3 Origins that could cause server crashes in #2718
  • [Cache/Origin]: Changed the way authfile entries are sorted to prevent applying access policies from public namespaces to private namespaces when the public namespace is a prefix of the private (e.g. /foo is public and /foo-auth is private) in #2705
  • [Cache/Origin]: Fixed a bug that prevented Caches and Origins from starting when Pelican's global logging level was set to Warn in #2712
  • [Director]: Made small changes to the Director's adaptive sort algorithm to help stabilize load distribution around closer caches in #2716

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.20.1…v7.20.2

Pelican v7.20.1

Pelican v7.20.1 fixes a Cache bug that results in unresponsive Caches, as well as a Director bug that prevents some server downtimes from clearing.

Bugs Fixed

  • [Cache]: Upgraded the containerized xrdcl-pelican dependency to v1.5.6 to fix a Cache thread deadlock in #2688
  • [Director]: Fixed a bug that prevented expired downtimes set by Origin/Cache admins from clearing in #2684

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.20.0…v7.20.1

Pelican v7.20.0

Pelican v7.20 brings a variety of Client usability improvements and bugfixes, server UI upgrades and Cache stability/debugging enhancements.

Breaking Changes & Upgrade Notes for Admins

No special upgrade considerations.

New Features and Enhancements

  • [Client]: Modified the Client command pelican object ls to produce single column outputs for better piping with grep in #2621
  • [Plugin]: Enabled the HTCondor plugin to run condor_reconfig automatically whenever it's updated in #2624
  • [Origin]: Gave Origin admins the ability to configure the issuer URL set in the Director's X-Pelican-Token-Generation response header (used by Clients for bootstrapping automatic token generation) in #2598
  • [Cache]: Made a variety of new XRootD metrics available via Prometheus to aid in debugging server performance in #2597 and #2588
  • [Director]: Changed the Director's UI to distinguish between a Cache/Origin's XRootD health tests and its overall server health status in #2625

Bugs Fixed

  • [Client]: Fixed various authorization errors related to listings and recursive deletes in #2499 and #2579
  • [Client]: Fixed a bug that prevented pelican origin token create from using custom paths for private keys in #2641
  • [Client]: Fixed a Client deadlock that occurred in some transfers in #2655
  • [Servers]: Fixed an RPM packaging bug that overrode the OSDF's discovery URL in #2612
  • [Cache]: Updated containerized Cache dependencies to fix a variety of deadlocks/segfault in #2611 and #2615
  • [Director]: Fixed a bug in the Director's "adaptive sort" algorithm that resulted in sending Clients to some broken Caches in #2647

New Contributors

  • @fallow64 made their first contribution to Pelican documentation in #2511

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.19.0...v7.20.0

V7.19

New Features and Enhancements

  • [Client] Prevent overwriting of objects on upload #2480
  • [Logging] Logs now don’t include ANSI color codes when going to non-terminals #2529
  • [Client] New error classification for slow origins #2559
  • [Origin] Added in a collections API to create mutable groupings of data #2550
  • [Client] Added --pack flag to get and put in the CLI #2549

Bugs Fixed

  • [Client] Fixed the race condition causing the failed to verify size bug #2533
  • [Client] Fix deadlock where results channel is not read after cancellation #2536
  • [Client] Fixed an issue where checksum requirements on put failed when not required #2546
  • [Client] Fix segfault in object get with pack=auto when destination directory doesn't exist #2561
  • [Client] Fixed a deadlock bug #2655
  • [Cache] Increased the monitoring eviction interval from 60ns to 60s #2662
  • [Cache]: Upgraded the containerized xrdcl-pelican dependency to v1.5.6 to fix a Cache thread deadlock in #2688

Notes for Admins

  • Bumped xrdcl-pelican version to 1.5.4
  • Bumped XRootD to v5.8.4-1.2
  • [Federation Admin] Before upgrading Registry, finish the cleanup for site name, as explained in the “for Registry admin” section in #2558

Changelog

  • c3a376bb89d145d0192d5998fe9943c5589ff9e1 Changed version to 7.19.5

Pelican v7.19.5

Pelican v7.19.5 fixes a Cache bug that results in unresponsive Caches.

Bugs Fixed

  • [Cache]: Upgraded the containerized xrdcl-pelican dependency to v1.5.6 to fix a Cache thread deadlock in #2688

Changelog

  • c3a376bb89d145d0192d5998fe9943c5589ff9e1 Changed version to 7.19.5

Pelican v7.19.3 Release Notes

Bugs Fixed

  • [Client] Fixed a deadlock bug #2655
  • [Cache] Increased the monitoring eviction interval from 60ns to 60s #2662

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.19.2...v7.19.3

Pelican v7.19.2 Release Notes

New Features and Enhancements

  • [Client] Prevent overwriting of objects on upload #2480
  • [Logging] Logs now don’t include ANSI color codes when going to non-terminals #2529
  • [Client] New error classification for slow origins #2559
  • [Origin] Added in a collections API to create mutable groupings of data #2550
  • [Client] Added --pack flag to get and put in the CLI #2549

Bugs Fixed

  • [Client] Fixed the race condition causing the failed to verify size bug #2533
  • [Client] Fix deadlock where results channel is not read after cancellation #2536
  • [Client] Fixed an issue where checksum requirements on put failed when not required #2546
  • [Client] Fix segfault in object get with pack=auto when destination directory doesn't exist #2561

Notes for Admins

  • Bumped xrdcl-pelican version to 1.5.4
  • Bumped XRootD to v5.8.4-1.2
  • [Federation Admin] Before upgrading Registry, finish the cleanup for site name, as explained in the “for Registry admin” section in #2558

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.19.1...v7.19.2

Do Not Use This Release

This version was improperly packaged due to an issue with GoReleaser. Please use the latest version in Pelican v7.19 series instead.

V7.18

New Features and Enhancements

  • [Client] Introduced client transfer statistics reporting. #2436
  • [Origin] S3 tracing can now be enabled via Logging.Origin.Oss configuration #2410
  • [Broker] Allow connection broker to be used for director-to-cache communications #2378
  • [Origin] Added S3 file cache Prometheus metrics #2433
  • [Client] Added checksum support to pelican object put #2415
  • [Client] Keyring errors are now set to the Debug log level #2420
  • [Client] Marked certain transient errors as retryable, including invalid chunk headers, unexpected EOFs, and TLS connection failures during uploads. #2442
  • [Client] Improved handling of deprecated/hidden parameter handling in pelican config get #2513
  • [Director,Origin,Cache] Enabled web client to GET authenticated files, PUT all files, and interactively list objects #2468 #2467
  • [Origin] Add support for regular expressions to the built-in token issuer's authorization templates #2462
  • [Packaging] Update OA4MP #2444
  • [Packaging] ​​Bump XRootD release to 5.8.3-1.3 #2474

Bugs Fixed

  • [Client] Fixed how tokens are deemed as acceptable #2455
  • [Client] Fixed “failed to verify size” bug #2533
  • [LocalCache] Fixed local cache cleanup bugs. #2432
  • [Director] Fix not-found logic when querying against caches and origins #2392
  • [Director] Fix link depth header for listings #2437
  • [Director] High Availability Director Bug Fixes #2400
  • [Packaging] Fix pelican-server binary panic and upgrade xrdhttp-pelican to v0.0.7 #2476
  • Bumped XRootD to v5.8.4-1.2 and upgraded xrdhttp-pelican to v0.0.7.
  • Native packages now require XRootD ≥ 5.8.2.
  • [Client] pelican object put now only fails during the checksum process if we required a checksum in the transfer
  • [CLI] pelican config tool now takes environment variables into account
  • [Cache] DropPrivileges is now ready to used in Cache
  • [Client] Client doesn't stall during shutdown
  • [Cache] Use xrdcl-pelican 1.5.4 to fix deadlocks in Cache

Bugs Fixed

  • [Cache] Use xrdcl-pelican 1.5.4 to fix deadlocks in Cache

Bugs Fixed

  • [Client] pelican object put now only fails during the checksum process if we required a checksum in the transfer
  • [CLI] pelican config tool now takes environment variables into account
  • [Cache] DropPrivileges is now ready to used in Cache
  • [Client] Client doesn't stall during shutdown

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.18.0...v7.18.1

Pelican v7.18.0 Release Notes

New Features and Enhancements

  • [Client] Introduced client transfer statistics reporting. #2436
  • [Origin] S3 tracing can now be enabled via Logging.Origin.Oss configuration #2410
  • [Broker] Allow connection broker to be used for director-to-cache communications #2378
  • [Origin] Added S3 file cache Prometheus metrics #2433
  • [Client] Added checksum support to pelican object put #2415
  • [Client] Keyring errors are now set to the Debug log level #2420
  • [Client] Marked certain transient errors as retryable, including invalid chunk headers, unexpected EOFs, and TLS connection failures during uploads. #2442
  • [Client] Improved handling of deprecated/hidden parameter handling in pelican config get #2513
  • [Director,Origin,Cache] Enabled web client to GET authenticated files, PUT all files, and interactively list objects #2468 #2467
  • [Origin] Add support for regular expressions to the built-in token issuer's authorization templates #2462
  • [Packaging] Update OA4MP #2444
  • [Packaging] ​​Bump XRootD release to 5.8.3-1.3 #2474

Bugs Fixed

  • [Client] Fixed how tokens are deemed as acceptable #2455
  • [Client] Fixed “failed to verify size” bug #2533
  • [LocalCache] Fixed local cache cleanup bugs. #2432
  • [Director] Fix not-found logic when querying against caches and origins #2392
  • [Director] Fix link depth header for listings #2437
  • [Director] High Availability Director Bug Fixes #2400
  • [Packaging] Fix pelican-server binary panic and upgrade xrdhttp-pelican to v0.0.7 #2476

Notes for Admins

  • Bumped XRootD to v5.8.4-1.2 and upgraded xrdhttp-pelican to v0.0.7.
  • Native packages now require XRootD ≥ 5.8.2.

Full Changelog: v7.17.2...v7.18.0

V7.17

New Features and Enhancements

  • [Client]: New pelican token create command for on-demand JWT generation #2314
  • [Client]: The new pelican key create command generates a private-public key pair. If a private key already exists at the specified location, the command uses it to derive the corresponding public key. This new command retires pelican generate keygen. https://github.com/PelicanPlatform/pelican/pull/2380
  • [Client]: Client uses token issuer as selection criteria during token selection process, also displays which token it chooses #2219, #2348
  • [Client]: SciTokens-profile JWTs are now accepted in addition to wlcg profile #2223
  • [Client]: Friendlier message when pelican get is run without a destination #2351
  • [Origin/Cache]: Graceful shutdown a server, waiting some time for in-flight transfers before exit #2318
  • [Origin/Cache]: Persist an Origin/Cache’s registered name #2300
  • [Origin/Cache]: Server ads now carry a status field containing degraded state when concurrency limits are exceeded #2338
  • [Origin/Cache]: Collect XRootD CPU-utilisation metric in monitoring #2225
  • [Origin/Cache]: Added XRootD logging on the open, close, and transfer events #2277
  • [Origin]: Concurrency limits can now be enforced in OSS throttle mode #2276
  • [Servers]: Login cookies (and generated tokens) now last 16 hours #2255, #2396
  • [Registry]: Automatically adds the current user as security contact when creating a new namespace #2132
  • [Registry]: Namespace list refreshes automatically after edits #2269
  • [Director]: Service-version graphs added to the Director dashboard #2402
  • [Documentation]: Origin documentation overhaul #2298
  • [Documentation]: Update FAQ section in Pelican Docs #2089, #2261
  • [All]: Startup warning if the deprecated IssuerKey file is still in use #2228
  • [Packaging]: New pelican-client-debug RPM for easier troubleshooting #2317
  • [Packaging]: Container images rebased on XRootD 5.8.3-1.2, xrdcl-pelican 1.2.3 and xrootd-s3-http 0.4.1 #2283, #2340, #2342, #2368, #2399
  • [Plugin]: Improved error propagation when servers return HTTP error codes #2344

Notes for Admins

  • Deprecated key path – A clear warning is now emitted if IssuerKey is still set; please migrate to the new directory-based key layout (put private keys into IssuerKeysDirectory, and ensure they have .pem extension) #2228
  • Name – When a new origin/cache registers, Pelican will use os.Hostname as sitename, unless the admin specifies its name by setting Xrootd_Sitename in the config. For all servers in the federation, if their name in the local config is different from the registered one, Pelican will still use the registered one and throw a warning in the logs. #2300
  • Container images – Note that Pelican are now based on XRootD 5.8.3-1.2 #2399
  • Graceful shutdown timeout – The default Pelican Origin/Cache graceful shutdown timeout is 1m (set by config param Xrootd_ShutdownTimeout). If the Origin/Cache is running in a k8s pod, the pod should be configured to have a slightly longer termination timeout than Pelican itself.

Bugs Fixed

  • [Client]: Fixed Windows client’s ability to request and save tokens #2267
  • [Client]: Fixed checksum validation and corrected downloaded-size comparisons #2259, #2275
  • [Client]: Repaired rate/progress calculations #2322
  • [Client]: Aligned runPut proxy logic with downloadHttp behaviour #2372
  • [Director]: Downtime toggles now work reliably and downtime history tests are no longer flaky #2280
  • [Director]: Stopped unintentionally extending TTL on every serverAds.Get() call #2290
  • [Director]: Geo-location widget hidden from non-admin users #2272
  • [Registry]: Properly handle trailing slashes in namespace registration and advertisement #2270
  • [Origin/Cache]: Fixed self-test failures and permission-denied errors in drop-privileges mode, though it is not ready for production due to Kubernetes limitations #2120, #2291
  • [Origin/Cache]: Restored debug logging from XRootD framework itself #2387
  • [Origin/Cache]: Uses the proper name when requesting federation tokens #2343
  • [Client]: Fix token scope for specific object operation https://github.com/PelicanPlatform/pelican/pull/2407
  • [Cache]: Trim trailing '/' from federation-derived lot paths https://github.com/PelicanPlatform/pelican/pull/2435
  • [Director]: Prevent Director's web UI crash from non-semantic versions
  • [Origin]: Actually make use of Origin.ScitokensNameMapFile in SciTokens configuration generated for XRootD

New Contributors

Bugs Fixed

  • [Director]: Prevent Director's web UI crash from non-semantic versions
  • [Origin]: Actually make use of Origin.ScitokensNameMapFile in SciTokens configuration generated for XRootD

Bugs Fixed

Pelican v7.17.0

New Features and Enhancements

  • [Client]: New pelican token create command for on-demand JWT generation #2314
  • [Client]: The new pelican key create command generates a private-public key pair. If a private key already exists at the specified location, the command uses it to derive the corresponding public key. This new command retires pelican generate keygen. https://github.com/PelicanPlatform/pelican/pull/2380
  • [Client]: Client uses token issuer as selection criteria during token selection process, also displays which token it chooses #2219, #2348
  • [Client]: SciTokens-profile JWTs are now accepted in addition to wlcg profile #2223
  • [Client]: Friendlier message when pelican get is run without a destination #2351
  • [Origin/Cache]: Graceful shutdown a server, waiting some time for in-flight transfers before exit #2318
  • [Origin/Cache]: Persist an Origin/Cache’s registered name #2300
  • [Origin/Cache]: Server ads now carry a status field containing degraded state when concurrency limits are exceeded #2338
  • [Origin/Cache]: Collect XRootD CPU-utilisation metric in monitoring #2225
  • [Origin/Cache]: Added XRootD logging on the open, close, and transfer events #2277
  • [Origin]: Concurrency limits can now be enforced in OSS throttle mode #2276
  • [Servers]: Login cookies (and generated tokens) now last 16 hours #2255, #2396
  • [Registry]: Automatically adds the current user as security contact when creating a new namespace #2132
  • [Registry]: Namespace list refreshes automatically after edits #2269
  • [Director]: Service-version graphs added to the Director dashboard #2402
  • [Documentation]: Origin documentation overhaul #2298
  • [Documentation]: Update FAQ section in Pelican Docs #2089, #2261
  • [All]: Startup warning if the deprecated IssuerKey file is still in use #2228
  • [Packaging]: New pelican-client-debug RPM for easier troubleshooting #2317
  • [Packaging]: Container images rebased on XRootD 5.8.3-1.2, xrdcl-pelican 1.2.3 and xrootd-s3-http 0.4.1 #2283, #2340, #2342, #2368, #2399
  • [Plugin]: Improved error propagation when servers return HTTP error codes #2344

Notes for Admins

  • Deprecated key path – A clear warning is now emitted if IssuerKey is still set; please migrate to the new directory-based key layout (put private keys into IssuerKeysDirectory, and ensure they have .pem extension) #2228
  • Name – When a new origin/cache registers, Pelican will use os.Hostname as sitename, unless the admin specifies its name by setting Xrootd_Sitename in the config. For all servers in the federation, if their name in the local config is different from the registered one, Pelican will still use the registered one and throw a warning in the logs. #2300
  • Container images – Note that Pelican are now based on XRootD 5.8.3-1.2 #2399
  • Graceful shutdown timeout – The default Pelican Origin/Cache graceful shutdown timeout is 1m (set by config param Xrootd_ShutdownTimeout). If the Origin/Cache is running in a k8s pod, the pod should be configured to have a slightly longer termination timeout than Pelican itself.

Bugs Fixed

  • [Client]: Fixed Windows client’s ability to request and save tokens #2267
  • [Client]: Fixed checksum validation and corrected downloaded-size comparisons #2259, #2275
  • [Client]: Repaired rate/progress calculations #2322
  • [Client]: Aligned runPut proxy logic with downloadHttp behaviour #2372
  • [Director]: Downtime toggles now work reliably and downtime history tests are no longer flaky #2280
  • [Director]: Stopped unintentionally extending TTL on every serverAds.Get() call #2290
  • [Director]: Geo-location widget hidden from non-admin users #2272
  • [Registry]: Properly handle trailing slashes in namespace registration and advertisement #2270
  • [Origin/Cache]: Fixed self-test failures and permission-denied errors in drop-privileges mode, though it is not ready for production due to Kubernetes limitations #2120, #2291
  • [Origin/Cache]: Restored debug logging from XRootD framework itself #2387
  • [Origin/Cache]: Uses the proper name when requesting federation tokens #2343

New Contributors

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.16.6...v7.17.0

V7.16

New Features and Enhancements

  • [Client]: Added end-to-end checksumming in the client for GETs in #2205
  • [Servers]: Created user interface for creating/managing API tokens in #2151
  • [Director]: Added Origin/Cache version information to the Director's web display cards in #2135
  • [Cache]: Added Cache metrics page to the Director's user interface in #1788
  • [Origin]: Added Origin metrics page to the Director's user interface in #1990
  • [Client]: Clients now perform a sanity check to verify that the size of a downloaded object is the same size as reported by the source in #2177
  • [Origin]: Gave Origin admins the ability to configure connection throttling to act as a circuit breaker between the Origin and the object store in #2186
  • [LocalCache]: Added local cache API that lets admins control some aspects of the cache's purge queue by denoting which objects should be purged first in #2118
  • [Servers]: Added flags to selectively disable Topology integration (OSDF Only) in #2189
  • [Origin]: Finished "CacheAuthz" features that let Origins specify they're only willing to send data to federation-authorized caches in #2175, #2234
  • [Registry,Origin,Cache]: Added new downtime CLI and Web UIs so administrators can configure service downtimes with full history in #2197, #2218, #2210
  • [Cache,Origin]: Added fair-share throttling to XRootD that uses token IDs to throttle individuals based on the demand they generate in #2280
  • [Client]: Added the ability for clients to log their transfer progress at a defined interval using the Logging.Client.ProgressInterval config option (default 1m)

Bugs Fixed

  • [Client]: Fixed bug that prevented schemeless Pelican URLs from working with federation info passed through -f flag in #2116
  • [Plugin]: Corrected transfer start time in the plugin's transfer classads in #2144
  • [Servers]: Standardized Prometheus metric names to follow a common pattern in #2105
  • [Cache]: Fixed processing of XRootD throttle plugin metrics in #2133
  • [Origin]: Corrected issue preventing multi-user origins from caching SciTokens issuer information in #2184
  • [LocalCache]: Fixed bug that prevented server restarts from re-binding to the cache socket in #2224
  • [Registry]: Fixed bug that prevented Registry admins from updating namespace keys via the Web UI in #2173
  • [Client]: Corrected issues generating tokens for Windows clients in #2267 https://github.com/PelicanPlatform/pelican/pull/2267
  • [Director]: Corrected an issue that prevented the Director from forgetting about some Origins/Caches in #2290
  • [Director]: Fixed a bug that impacted some types of server downtime filtering and prevented the Director from receiving the proper updates to these configurations in #2280
  • [All]: Fixed RPM packaging issues involving unavailable dependencies for some platforms in #2286
  • [Origin/Cache]: Updated XRootD to 5.8.2-1.3 to fix potential segfault related to "Close does not refer to an open file" error message.
  • [Origin/Cache]: Updated Pelican container images to use XRootD 5.8.2-1.3, which contains fixes for a potential segfault related to the "Close does not refer to an open file" error message.
  • [Client]: Corrected an issue that caused calculated download rates to tend toward 0, causing long transfers to terminate
  • [Client]: Fixed incorrect calculation in the client's EWMA download progress bars
  • [Client]: Fixed a bug that prevented Client.MinimumDownloadSpeed from taking effect unless a deprecated config option MinimumDownloadSpeed was also set
  • [Caches]: Updated containerized dependencies (xrdcl-pelican v1.2.3 and xrootd v5.8.2-1.5) to pull in fixes for additional segfaults triggered by high-concurrency cache failures.
  • [Client]: Fixed part of an issue related to pelican object puts being proxied through an HTTPs proxy by allowing the client to ignore those proxies.
  • [Cache]: Fixed a bug that prevented Caches from obtaining a valid Federation Token when the service was registered using the Xrootd.Sitename configuration parameter.
  • [Origin/Cache]: Updated the containerized XRootD version to 5.8.3-1.2, which among other things comes with additional segfault fixes.
  • [Origin]: Bumped containerized xrootd-s3-http plugin to v0.4.1, which comes with bug fixes related to data corruption coming from the Origin and deadlocks that blocked transfers from completing.
  • [Cache]: Fixed various bugs related to Lotman storage accounting in Lotman-enabled caches

Pelican v7.16.7

Bugs Fixed

  • [Cache]: Fixed various bugs related to Lotman storage accounting in Lotman-enabled caches

Pelican v7.16.6

Bugs Fixed:

  • [Client]: Fixed part of an issue related to pelican object puts being proxied through an HTTPs proxy by allowing the client to ignore those proxies.
  • [Cache]: Fixed a bug that prevented Caches from obtaining a valid Federation Token when the service was registered using the Xrootd.Sitename configuration parameter.
  • [Origin/Cache]: Updated the containerized XRootD version to 5.8.3-1.2, which among other things comes with additional segfault fixes.
  • [Origin]: Bumped containerized xrootd-s3-http plugin to v0.4.1, which comes with bug fixes related to data corruption coming from the Origin and deadlocks that blocked transfers from completing.

Pelican v7.16.5

Bugs Fixed

  • [Caches]: Updated containerized dependencies (xrdcl-pelican v1.2.3 and xrootd v5.8.2-1.5) to pull in fixes for additional segfaults triggered by high-concurrency cache failures.

Pelican v7.16.4

Bugs Fixed

  • [Client]: Corrected an issue that caused calculated download rates to tend toward 0, causing long transfers to terminate
  • [Client]: Fixed incorrect calculation in the client's EWMA download progress bars
  • [Client]: Fixed a bug that prevented Client.MinimumDownloadSpeed from taking effect unless a deprecated config option MinimumDownloadSpeed was also set

New Features and Enhancements

  • [Client]: Added the ability for clients to log their transfer progress at a defined interval using the Logging.Client.ProgressInterval config option (default 1m)

Pelican v7.16.3

Bugs Fixed

  • [Origin/Cache]: Updated Pelican container images to use XRootD 5.8.2-1.3, which contains fixes for a potential segfault related to the "Close does not refer to an open file" error message.

Pelican v7.16.2

Note: This release failed to generate container artifacts. The issue has been corrected and this bugfix will be released as v7.16.3

Bugs Fixed

  • [Origin/Cache]: Updated XRootD to 5.8.2-1.3 to fix potential segfault related to "Close does not refer to an open file" error message.

Pelican v7.16.1

Bugs Fixed

  • [Director]: Corrected an issue that prevented the Director from forgetting about some Origins/Caches in #2290
  • [Director]: Fixed a bug that impacted some types of server downtime filtering and prevented the Director from receiving the proper updates to these configurations in #2280
  • [All]: Fixed RPM packaging issues involving unavailable dependencies for some platforms in #2286

Pelican v7.16.0

New Features and Enhancements

  • [Client]: Added end-to-end checksumming in the client for GETs in #2205
  • [Servers]: Created user interface for creating/managing API tokens in #2151
  • [Director]: Added Origin/Cache version information to the Director's web display cards in #2135
  • [Cache]: Added Cache metrics page to the Director's user interface in #1788
  • [Origin]: Added Origin metrics page to the Director's user interface in #1990
  • [Client]: Clients now perform a sanity check to verify that the size of a downloaded object is the same size as reported by the source in #2177
  • [Origin]: Gave Origin admins the ability to configure connection throttling to act as a circuit breaker between the Origin and the object store in #2186
  • [LocalCache]: Added local cache API that lets admins control some aspects of the cache's purge queue by denoting which objects should be purged first in #2118
  • [Servers]: Added flags to selectively disable Topology integration (OSDF Only) in #2189
  • [Origin]: Finished "CacheAuthz" features that let Origins specify they're only willing to send data to federation-authorized caches in #2175, #2234
  • [Registry,Origin,Cache]: Added new downtime CLI and Web UIs so administrators can configure service downtimes with full history in #2197, #2218, #2210
  • [Cache,Origin]: Added fair-share throttling to XRootD that uses token IDs to throttle individuals based on the demand they generate in #2280

Bugs Fixed

  • [Client]: Fixed bug that prevented schemeless Pelican URLs from working with federation info passed through -f flag in #2116
  • [Plugin]: Corrected transfer start time in the plugin's transfer classads in #2144
  • [Servers]: Standardized Prometheus metric names to follow a common pattern in #2105
  • [Cache]: Fixed processing of XRootD throttle plugin metrics in #2133
  • [Origin]: Corrected issue preventing multi-user origins from caching SciTokens issuer information in #2184
  • [LocalCache]: Fixed bug that prevented server restarts from re-binding to the cache socket in #2224
  • [Registry]: Fixed bug that prevented Registry admins from updating namespace keys via the Web UI in #2173
  • [Client]: Corrected issues generating tokens for Windows clients in #2267 https://github.com/PelicanPlatform/pelican/pull/2267

Notes for Admins

None

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.15.2...v7.16.0

V7.15

New Features and Enhancements

  • [Origin] Added config parameter Issuer.OIDCPreferClaimsFromIDToken which causes Pelican’s user authentication to look for claims in the OIDC ID token before falling back to the OIDC Userinfo endpoint. #2100
  • [Origin] Pelican origins can now specify multiple issuers per namespace #2103
  • [Director] Director can now report cache redirection reasoning when client commands are run with -d #2054
  • [Director] Added support for a Federation to have multiple Directors #2023
  • [Origin, Cache] Integrate XRootD OSS statistics #2092
  • [Director] Add API Token Listing endpoint #2086

Bugs Fixed

  • [Client] Fixed a regression in pelican generate keygen that caused it to generate invalid public keys #2085
  • [All] When key files are deleted, the private keys are also now removed from Pelican’s in memory processes #2087
  • [Client] The client no longer panics when doing a put or a share with just the base namespace prefix #2097
  • [Client] Fixed client bug where schemaless Pelican URLs don’t work with ‘-f’ fed infor #2116
  • [All] Assorted issues that prevented using an authentication provider other than CILogon or Globus #2091 #2046
  • [Client] Now correctly reports the start time for uploads [#2144]
  • [Origin] Made Origin throttling configurable #2186
  • [Origin] Enabled multiuser origins to cache public keys for validating tokens #2184
  • [Client] Now sanity checks size for client GETs #2177
  • [Cache] XRootD now uses xldcl-pelican v1.2.0 which contains bug fixes for the cache

Changelog

  • 4d103c12a50778980332ec3d7d5aa223a9e52d52 Build and package a version of pelican that includes debugging symbols
  • e294ba6af579487109f02591240e5094670228ce Install 'delve' into the dev container image
  • d8dc5c8851c15e9bd0b9cbc719a09757faed7662 Updated version is ".goreleaser.yml" for pelican-debug
  • 2d8132bd2543461634bc3275f5e2a6c5a4f717b6 Updated version to 7.15.4

Changelog

  • 4d103c12a50778980332ec3d7d5aa223a9e52d52 Build and package a version of pelican that includes debugging symbols
  • e294ba6af579487109f02591240e5094670228ce Install 'delve' into the dev container image
  • d8dc5c8851c15e9bd0b9cbc719a09757faed7662 Updated version is ".goreleaser.yml" for pelican-debug
  • 2d8132bd2543461634bc3275f5e2a6c5a4f717b6 Updated version to 7.15.4

Pelican v7.15.3

Bugs Fixed

[Cache] Fixed throttling in the cache [Cache] Updated xldcl-pelican to 5.8.1 to prevent caches from seg faulting, this is an urgent fix for caches. [Cache/Origin] Fixed bug in metric collections

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.15.2...v7.15.3

Pelican v7.15.2

Bugs Fixed

  • [Cache] XRootD now uses xldcl-pelican v1.2.0 which contains bug fixes for the cache

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.15.1...v7.15.2

Pelican v7.15.1

Bugs Fixed

  • [Origin] Made Origin throttling configurable #2186
  • [Origin] Enabled multiuser origins to cache public keys for validating tokens #2184
  • [Client] Now sanity checks size for client GETs #2177

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.15.0...v7.15.1

Pelican v7.15.0

New Features and Enhancements

  • [Origin] Added config parameter Issuer.OIDCPreferClaimsFromIDToken which causes Pelican’s user authentication to look for claims in the OIDC ID token before falling back to the OIDC Userinfo endpoint. #2100
  • [Origin] Pelican origins can now specify multiple issuers per namespace #2103
  • [Director] Director can now report cache redirection reasoning when client commands are run with -d #2054
  • [Director] Added support for a Federation to have multiple Directors #2023
  • [Origin, Cache] Integrate XRootD OSS statistics #2092
  • [Director] Add API Token Listing endpoint #2086

Bugs Fixed

  • [Client] Fixed a regression in pelican generate keygen that caused it to generate invalid public keys #2085
  • [All] When key files are deleted, the private keys are also now removed from Pelican’s in memory processes #2087
  • [Client] The client no longer panics when doing a put or a share with just the base namespace prefix #2097
  • [Client] Fixed client bug where schemaless Pelican URLs don’t work with ‘-f’ fed infor #2116
  • [All] Assorted issues that prevented using an authentication provider other than CILogon or Globus #2091 #2046
  • [Client] Now correctly reports the start time for uploads [#2144]

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.14.2...v7.15.0

V7.14

New Features and Enhancements

  • Add /version endpoint to expose version and build info #2038
  • Log Pelican config on server startup #1970
  • Drop privileges to user pelican if requested #1850
  • Fix xrootd log level mapping bug and inherit from Pelican log level #1988
  • Allow for Shoveler messages to be understood by monitoring packet handler #1981
  • Prometheus Metric For When a Service Crashed #2015
  • Refactor & Cleanup Origin Exports code #1919
  • Deprecate Lotman.DbLocation in favor of Lotman.LotHome #2001
  • Remove tls authentication from caches #1994
  • Fix Director Capabilities Display #1980
  • Add Cache and Server Info Cards to Map #1947
  • Geoip table #1893
  • Add Server Outage information #1982
  • Add federation token endpoint in Director, and implement routine for caches to fetch one #1985
  • Grafana Token API Access #1917
  • Enable PROPFIND of Director health test API #2002
  • Support using IP address to query the geo location of a server in Director container #2059

Condor-Pelican

  • K8SPhysicalHostName Error Message #1891

Document Updates

  • Simplify Linux install instructions #1921
  • Remove jargon from the credential management page #1922
  • Document restricted characters in fed prefixes #2050
  • Update Server.XYZ docs to include all server components #1997

Operations & Bugs Fixed

  • Don't build dev images on git tags, tag image with latest-itb and sha256 #1946
  • Clean up the container image build and testing process #1972
  • Run tests in more appropriate container images #2000
  • Update xrootd release version to 1.5.purge.osg #1983
  • Pinned the goreleaser version to 2 #1991
  • Attempted speedups for GHA PR runs #2030
  • Rebase on OSG 24 (#1858) #2016
  • Install new plugin: xrdhttp-pelican v0.0.3 #2013
  • Update OS X install of xrootd-s3-http to v0.2.1 #2026
  • Reduce memory hoarding in the unit test #2028
  • Pin xrootd log levels in fed test and use typed params throughout #2019
  • Fix container building process: Properly install pelican-server into /usr/local/sbin #2053
  • Resolve the excessive use of xrootd user #2057 #2062

Operation & Bugfix

  • Fix index out of bound error in origin servers #2082
  • Update XRootD version from 5.7.2 to 5.7.3 c555c47b

Bugfix

  • Install sssd-client into both origin container images #2102

Bugfix

  • Install sssd-client into both origin container images #2102

Operation & Bugfix

  • Fix index out of bound error in origin servers #2082
  • Update XRootD version from 5.7.2 to 5.7.3 c555c47b

Pelican v7.14.0

New Features and Enhancements

Client

  • N/A

Cache, Director, Origin, Registry

  • Add /version endpoint to expose version and build info #2038
  • Log Pelican config on server startup #1970

Registry, Director

  • Drop privileges to user pelican if requested #1850

Origin, Cache

  • Fix xrootd log level mapping bug and inherit from Pelican log level #1988
  • Allow for Shoveler messages to be understood by monitoring packet handler #1981
  • Prometheus Metric For When a Service Crashed #2015

Origin

  • Refactor & Cleanup Origin Exports code #1919
  • Deprecate Lotman.DbLocation in favor of Lotman.LotHome #2001

Cache

  • Remove tls authentication from caches #1994

Director

  • Fix Director Capabilities Display #1980
  • Add Cache and Server Info Cards to Map #1947
  • Geoip table #1893
  • Add Server Outage information #1982
  • Add federation token endpoint in Director, and implement routine for caches to fetch one #1985
  • Grafana Token API Access #1917
  • Enable PROPFIND of Director health test API #2002
  • Support using IP address to query the geo location of a server in Director container #2059

Condor-Pelican

  • K8SPhysicalHostName Error Message #1891

Document Updates

  • Simplify Linux install instructions #1921
  • Remove jargon from the credential management page #1922
  • Document restricted characters in fed prefixes #2050
  • Update Server.XYZ docs to include all server components #1997

Operations & Bugs Fixed

  • Don't build dev images on git tags, tag image with latest-itb and sha256 #1946
  • Clean up the container image build and testing process #1972
  • Run tests in more appropriate container images #2000
  • Update xrootd release version to 1.5.purge.osg #1983
  • Pinned the goreleaser version to 2 #1991
  • Attempted speedups for GHA PR runs #2030
  • Rebase on OSG 24 (#1858) #2016
  • Install new plugin: xrdhttp-pelican v0.0.3 #2013
  • Update OS X install of xrootd-s3-http to v0.2.1 #2026
  • Reduce memory hoarding in the unit test #2028
  • Pin xrootd log levels in fed test and use typed params throughout #2019
  • Fix container building process: Properly install pelican-server into /usr/local/sbin #2053
  • Resolve the excessive use of xrootd user #2057 #2062

Full Changelog

Compare v7.13.0...v7.14.0

V7.13

New Features and Enhancements

  • Improved detection and retry logic for clients that start transfers while the Director is down #1890
  • Added new prestage functionality in the client #1923
  • Added pelican user and group along with some default directories for RPM installations #1856
  • Added a "restart" button in each server's web UI #1869
  • Added configuration knobs to increase server memory performance by limiting Prometheus cardinality #1887
  • Created a new config parameter, Server.HealthMonitoringPublic, to make the server's health monitoring endpoint public #1936
  • Disabled redirects for HTTP caches #1915
  • Added experimental patches that modify a cache's ability to monitor resources and keep track of purgeable objects #1926
  • Created a Prometheus metric in the Director to track rejected cache/origin advertisements #1885
  • Enabled server-side key refreshing so that origins/caches can update their keys in the Registry #1748

Bugs Fixed

  • Fixed issue with director tests failing due to the token being created using the director URL instead of the discovery URL #1936

Bug fixes

  • [Cache, Origin] - Rebuilt containers to include newer version of xrootd: xrootd-5.7.3-1.3.purge.osg24 which contains additional bug fixes.

Bug fixes

  • [Cache, Origin] - Rebuilt containers to include newer version of xrootd: xrootd-5.7.3-1.3.purge.osg24 which contains additional bug fixes.

Pelican v7.13.0

Breaking Changes

Registry Upgrade Requirement

Pelican v7.13 Director is not compatible with older versions of the Registry. The Registry must be upgraded before upgrading the Director to ensure proper functionality.

New Features and Enhancements

Client

  • Improved detection and retry logic for clients that start transfers while the Director is down #1890
  • Added new prestage functionality in the client #1923

Cache, Director, Origin, Registry

  • Added pelican user and group along with some default directories for RPM installations #1856
  • Added a "restart" button in each server's web UI #1869
  • Added configuration knobs to increase server memory performance by limiting Prometheus cardinality #1887
  • Created a new config parameter, Server.HealthMonitoringPublic, to make the server's health monitoring endpoint public #1936

Cache

  • Disabled redirects for HTTP caches #1915
  • Added experimental patches that modify a cache's ability to monitor resources and keep track of purgeable objects #1926

Director

  • Created a Prometheus metric in the Director to track rejected cache/origin advertisements #1885

Origin

  • Enabled server-side key refreshing so that origins/caches can update their keys in the Registry #1748

Bugs Fixed

Director

  • Fixed issue with director tests failing due to the token being created using the director URL instead of the discovery URL #1936

Full Changelog

Compare v7.12.4...v7.13.0

V7.12

Bugs Fixed

  • [Servers] Restored the functionality of various tickers that had been erroneously stopped, preventing XRootD servers from refreshing some files like scitokens/auth config and preventing some metrics from being scraped by Prometheus.
  • [Caches] Updated caches' underlying version of xrdcl-pelican from 0.9.4 to 1.0.2, which prevents a segfault in caches whenever a non-existent object is requested.
  • [Cache Containers] - Grabbed patched version of xrootd 5.7.2 to mitigate further segfault bugs related to caches that are handling many concurrent requests for non-existent objects.
  • [Caches] - Rebuilt containers to ingest xrdcl-pelican v1.0.5, which comes with additional segfault bugfixes.
  • [Origins] - Rebuilt containers to ingest upstream bugfixes for S3/HTTP origins.

Bugs Introduced

  • [Caches] The directorTest functionality will fail in caches due to the new xrdcl-pelican version until updates can be made in Directors. These test failures are limited in scope and should not affect the rest of the cache's functionality, but may generate excess noise in logs.

Bugs Fixed

  • [Caches] - Rebuilt containers to ingest xrdcl-pelican v1.0.5, which comes with additional segfault bugfixes.
  • [Origins] - Rebuilt containers to ingest upstream bugfixes for S3/HTTP origins.

Bugs Fixed

  • [Cache Containers] - Grabbed patched version of xrootd 5.7.2 to mitigate further segfault bugs related to caches that are handling many concurrent requests for non-existent objects.

Bugs Fixed

[Cache] - Fixed a broken dependency for our cache containers that prevented caches from fetching non-cached objects from origins. [Cache] - Updated an underlying library version to prevent caches from segfaulting when non-existent objects are requested. [Client] - Added extra handling for client PUTs to accept XRootD's new HTTP 201 response.

Known Bugs

[Director/Cache] - This release contains a known bug where the Director's test utility fails for caches. This shouldn't affect the cache's ability to function, but will result in the cache showing errors in the Director's Web interface.

NOTE

Containerized versions of this release were dead on arrival due to a broken dependency for xrdcl-pelican.

Bugs Fixed

  • [Servers] Restored the functionality of various tickers that had been erroneously stopped, preventing XRootD servers from refreshing some files like scitokens/auth config and preventing some metrics from being scraped by Prometheus.
  • [Caches] Updated caches' underlying version of xrdcl-pelican from 0.9.4 to 1.0.2, which prevents a segfault in caches whenever a non-existent object is requested.

Bugs Introduced

  • [Caches] The directorTest functionality will fail in caches due to the new xrdcl-pelican version until updates can be made in Directors. These test failures are limited in scope and should not affect the rest of the cache's functionality, but may generate excess noise in logs.

Pelican v7.12.0

Breaking Changes

[Cache] Corrected previously unsafe creation of namespace, meta and data directories in caches in #1709. This change may result in cache data invalidation, and will likely require manual intervention during upgrades. Failure to adjust the data hierarchy of existing caches could result in unintended data leaks or zombie data that can't be cleaned by the cache. If this affects you, please make sure you understand update requirements before beginning your upgrades.

New Features and Enhancements

[All] The config parameter Server.TLSCertificate has been deprecated in favor of Server.TLSCertificateChain to more clearly convey the contents of the file pointed to. The deprecated parameter should continue to function for the time being, but may removed in a future release. [All] Added a CLI tool pelican config for displaying and interacting with Pelican configuration in #1693. [All] Standardized and improved RPM packaging in #1666. [Director] Gave Directors the ability to collect Prometheus metrics for GeoIP resolution failures in #1772. [Director] Servers now advertise their version to the Director in #1753. [Director] Started displaying namespace capabilities in the Director in #1737. [Director] Improved Prometheus metrics for tracking the number of currently-advertising servers in #1684. [Director] Gave the ability for the Director to persist origin/cache downtime labels between restarts in #1673. [Origin] Gave S3 origins the ability to specify sentinel files in #1791. [Plugin] Allowed the plugin to detect when it is named pelican_plugin in addition to pelican_xfer_plugin in #1733.

Bugs Fixed

[Client] Cleaned up display of progress bars for object gets in the client in #1814. [Client] Fixed bug where object ls against namespaces that do not support listings caused nil pointer panics in #1710. [Client] Added additional client checks to prevent creation of empty objects with directory uploads in #1707. [Director] Fixed over-collection of server redirects in the Director's Prometheus database in #1743. [Director] Fixed OSDF Director startup bug that caused erroneous warnings/errors mentioning the proj label in #1708.

New Contributors

@alexandertuna made their first contribution in #1730 @mwestphall made their first contribution in #1791 @georgezzeng made their first contribution in #1727

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.11.8...v7.12.0

V7.5

For Users

  • [General] – Improved logging doesn't report as many false errors
  • [Client] – Improved file unpacking to handle more use cases
  • [Client] – The plugin now downloads files in parallel

For Admins

  • [Origin] – Pelican origins now support public namespaces
  • [General] – Federation admins can now fine tune whether origins and caches require admin approval before joining the federation
  • [Origin/Cache] – XRootD log levels are now configurable
  • [Director] – The director can now report whether a specific origin has a requested object via the new director stat feature
  • [Origin] – Fixed rate graph data loss issues in the origin
  • [Registry] – Added the ability to configure a URL for fetching the registry's institution list
  • [Origin] -- Configured public origins to use XRootD directive sec.protbind to allow backwards compatibility with legacy OSDF caches.

Changelog

  • a47a13b Fixed issue with added new line in cache authfile
  • 3cc2452 Pre Release: Update npm version
  • 1ee7db3 Add missing xrootd key to osx_install test
  • 10c963d Add virtual-style url test
  • 860f979 Allow Pelican to use path-style URLs for S3 origins
  • b22440e Empty commit to force sync
  • d260a07 Pre Release: Update npm version
  • ea65a38 Update S3 test to use path-style URL

What's new

  • [Client]: Bug fixes for pelican object put with pelican:// urls
  • [Director]: The director can now display more federation information without requiring authentication
  • [Registry]: Fixes bugs related to the registry's topology table that introduced race conditions and caused excess load on the topology endpoint.
  • [Cache]: Fixed bug that prevented debug output from XRootD processes.
  • [Cache]: Fixed bug that prevented the cache's pss.origin directive from being set correctly with pelican://-style URLs.
  • [Origin]: Origins configured to export S3 backends are no longer explicitly tied to a single bucket.
  • [Cache]: Caches now fetch new namespace ads every time they advertise with the director. This is necessary for discovering new origins if they join the federation after the cache is served.

Changelog

For Admins

  • [Origin] -- Configured public origins to use XRootD directive sec.protbind to allow backwards compatibility with legacy OSDF caches.

Changelog

  • 1ee7db3 Add missing xrootd key to osx_install test
  • 10c963d Add virtual-style url test
  • 860f979 Allow Pelican to use path-style URLs for S3 origins
  • b22440e Empty commit to force sync
  • d260a07 Pre Release: Update npm version
  • ea65a38 Update S3 test to use path-style URL

What's new

For Admins

  • [Cache]: Caches now fetch new namespace ads every time they advertise with the director. This is necessary for discovering new origins if they join the federation after the cache is served.

What's new

For Admins

  • [Origin]: Origins configured to export S3 backends are no longer explicitly tied to a single bucket.

What's new

For Admins

  • [Cache]: Fixed bug that prevented the cache's pss.origin directive from being set correctly with pelican://-style URLs.

What's new

For Admins

  • [Cache]: Fixed bug that prevented debug output from XRootD processes.

What's new

For Admins

  • [Registry]: Fixes bugs related to the registry's topology table that introduced race conditions and caused excess load on the topology endpoint.

What's new

For Users

  • [Client]: Bug fixes for pelican object put with pelican:// urls

For Admins

  • [Director]: The director can now display more federation information without requiring authentication

Changelog

  • a47a13b Fixed issue with added new line in cache authfile
  • 3cc2452 Pre Release: Update npm version

Updates

Congratulations @williamnswanson for your first contribution in #684!

Highlights

For Users

  • [General] – Improved logging doesn't report as many false errors
  • [Client] – Improved file unpacking to handle more use cases
  • [Client] – The plugin now downloads files in parallel

For Admins

  • [Origin] – Pelican origins now support public namespaces
  • [General] – Federation admins can now fine tune whether origins and caches require admin approval before joining the federation
  • [Origin/Cache] – XRootD log levels are now configurable
  • [Director] – The director can now report whether a specific origin has a requested object via the new director stat feature
  • [Origin] – Fixed rate graph data loss issues in the origin
  • [Registry] – Added the ability to configure a URL for fetching the registry's institution list

V7.4

Changelog

  • Fix the issue where Pelican won't exit main process if error group goroutine returns error, unless an error returns in the main goroutine
  • Fix the panic at the director when a cache advertise itself to the director, as we didn't have the case for cache when doing version compatibility check
  • Improve logging at program exit to be consistent.
  • Improve cache error group, remove duplicated exit handler that root.go already had.
  • Fix duplicated director file transfer tests
  • Added three Prometheus metrics recording number of test file transfers the director issues:
  • pelican_director_total_ftx_test_suite: total number of test suite the director issued, which means how many times the director started a new test go routine to an origin
  • pelican_director_active_ftx_test_suite: the number of active test suite running per origin, which should be 1, meaning only one active test go routine is running per origin. This will be a good indicator of whether the director is doing wild things.
  • pelican_director_total_ftx_test_runs: the total number of test file transfer runs per origin. i.e. How many tests in total were ran per origin. We run A test per 15s (by default) per origin, so this is also a good indicator of whether the origin is up or whether the director is going crazy. It comes with label status and report_status with value Success and Failure indicating if the transfer itself was successful and the reporting to the origin was successful.

Changelog

  • Fix duplicated director file transfer tests
  • Added three Prometheus metrics recording number of test file transfers the director issues:
    • pelican_director_total_ftx_test_suite: total number of test suite the director issued, which means how many times the director started a new test go routine to an origin
    • pelican_director_active_ftx_test_suite: the number of active test suite running per origin, which should be 1, meaning only one active test go routine is running per origin. This will be a good indicator of whether the director is doing wild things.
    • pelican_director_total_ftx_test_runs: the total number of test file transfer runs per origin. i.e. How many tests in total were ran per origin. We run A test per 15s (by default) per origin, so this is also a good indicator of whether the origin is up or whether the director is going crazy. It comes with label status and report_status with value Success and Failure indicating if the transfer itself was successful and the reporting to the origin was successful.

Changelog

  • Fix the issue where Pelican won't exit main process if error group goroutine returns error, unless an error returns in the main goroutine
  • Fix the panic at the director when a cache advertise itself to the director, as we didn't have the case for cache when doing version compatibility check
  • Improve logging at program exit to be consistent.
  • Improve cache error group, remove duplicated exit handler that root.go already had.

Highlights

For Users

  • [Client] Introduced pelican object get and pelican object put to specify a get or put rather than just running pelican object copy. #259
  • [Client] Fixed issue where the stash_plugin file-transfer plugin was not reporting proper upload bytes. #510
  • [Client] Fixed issue with progress bars not outputting properly with debug logs for recursive uploads/downloads #402
  • [Client] Fixed issue where stash_plugin was not working with environment variable-based configuration #508
  • [Client] pelican object copy now doesn’t rely on topology for PUTs, it gets the endpoint by querying the director #504

For Admins

  • [Director] Introduced GeoIP Overrides that allow a director to pin IP ranges to a particular lat/long. #496
  • [Registry] Introduced web-ui for registry that allows users to register their namespace through web interface. #264
  • [Director] Added federation discovery endpoint to director. #417
  • [General] Added topology components to Prometheus health metrics. #494
  • [Cache] Added Admin approval to caches before they can advertise to director #439
  • [Registry] Added support to use CILogon to login to registry UI #355
  • [Registry] Deprecated Federation.NamespaceUrl which will be removed in the next major release #388
  • [General] Web-ui now outputs more detailed error messages if login fails #424
  • [Origin] Basic S3 origin functionality has been restored #592
  • [Cache] Fixed issue where caches were logging sensitive information #604
  • [Origin] Boolean flag added to starting origins indicating if they are writable or not (default is true) #507

V7.3

Highlights

  • Users can now recursively upload and download files. #260, #261
  • Users can upload a local directory to a data federation as a tarball and can unpack a tarball into a directory as it downloads.
  • For improved debugging, writing a debug file into a known location is now supported. #22
  • Introduced Javascript client allowing browsers to perform file transfers. Installation of the Javascript client enables users via their browser to download a file from a data federation supported by Pelican. #344
  • [Origin] Added support for S3 file backend so that admins may expose or configure the S3 backend. #169
  • [Cache] Added authentication to cache service. #311
  • [Issuer] Introduced new OAuth2 issuer built into the origin packaging. #262
  • [General] Introduced web pages for the director and registry components. #265, #354, #355
  • New feature to allow the director to aggregate metrics from registered origins and to allow the director to periodically test origins #291 #175

What's Changed

New Contributors

Changelog

  • ef6d348 Truncate the plugin output file as soon as possible
  • 85cf48c Remove chmod on user-provided xrootd config files
  • 4e93152 Address code review feedback * Update minio package version as the existing one was removed from archive * Remove redundant code * Preserve dev environment for dev container
  • 8904cde Replace xrd-http plugin by xrdcl-pelican plugin in dev-container
  • d1e3b82 Replace xrd-http plugin by xrdcl-pelican plugin in prod container

Changelog

  • 4e93152 Address code review feedback * Update minio package version as the existing one was removed from archive * Remove redundant code * Preserve dev environment for dev container
  • 8904cde Replace xrd-http plugin by xrdcl-pelican plugin in dev-container
  • d1e3b82 Replace xrd-http plugin by xrdcl-pelican plugin in prod container

Changelog

  • 85cf48c Remove chmod on user-provided xrootd config files

Changelog

  • ef6d348 Truncate the plugin output file as soon as possible

Highlights

For Users

  • Users can now recursively upload and download files. #260, #261
  • Users can upload a local directory to a data federation as a tarball and can unpack a tarball into a directory as it downloads.
  • For improved debugging, writing a debug file into a known location is now supported. #22
  • Introduced Javascript client allowing browsers to perform file transfers. Installation of the Javascript client enables users via their browser to download a file from a data federation supported by Pelican. #344

For Admins

  • [Origin] Added support for S3 file backend so that admins may expose or configure the S3 backend. #169
  • [Cache] Added authentication to cache service. #311
  • [Issuer] Introduced new OAuth2 issuer built into the origin packaging. #262
  • [General] Introduced web pages for the director and registry components. #265, #354, #355
  • New feature to allow the director to aggregate metrics from registered origins and to allow the director to periodically test origins #291 #175

What's Changed

New Contributors

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.2.0...v7.3.0

V7.2

What's Changed

New Contributors

Changelog

  • fc84142 Fix small bugs in DiscoverFederation()
  • 8515fb4 Add cache supervisord process
  • a2938c0 Add default environment var to origin supervisord script
  • 4173537 Add supervisor script to make origins servable
  • 0c5b1a0 Fix process name

Changelog

  • 8515fb4 Add cache supervisord process
  • a2938c0 Add default environment var to origin supervisord script
  • 4173537 Add supervisor script to make origins servable
  • 0c5b1a0 Fix process name

Changelog

  • fc84142 Fix small bugs in DiscoverFederation()

What's Changed

New Contributors

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.1.0...v7.2.0

V7.1

What's Changed

New Contributors

This release modifies the creation of Pelican sub-RPMs to allow compatibility with osdf-client.

This patch fixes authentication headers in the director so that pelican object copy can determine issuer metadata for some namespaces dynamically.

What's Changed

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.1.1...v7.1.2

What's Changed

New Contributors

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.0.0...v7.1.0

V7.0

Release Notes

New Contributors

The first release of the new Pelican codebase!

Pelican currently includes the functionality of the prior osdf-client / stashcp; it emulates the CLI of those clients if the binary is renamed to stashcp or stash_plugin.

Two new components have been developed, the director and the namespace registry. The director is responsible for routing clients to an appropriate cache or origin to serve up their request (and has similar functionality to the redirector for the OSDF). The namespace registry adds an authorization layer to the data federation, associating a prefix with the public key of an origin.

This includes experimental support for launching an origin via the Pelican command line. Over the next few monthly releases, this support will mature to the point of replacing currently running origin containers in the OSDF.

Release Notes:

New Contributors

Full Changelog: https://github.com/PelicanPlatform/pelican/commits/v7.0.0