Pelican Releases

Pelican Release Notes V7.8.0: Highlights For Users [Client] Stopped file transfers are now retryable errors #1101 [Client] General improvements to error messages within the client and the plugin, ensuring more accurate errors and removing redundancies [Client] The client can bypass the cache and download directly from the origin if a directread query parameter is set #1166

For Admins [Documentation] Added documentation on how to get the public key of a running origin #1078 [Documentation] Added documentation on how to run pelican servers via docker containers #1089 [Origin] Pelican origin token creation now respects the issuer flag #1135 [Cache] Cache UI will now redirect to a login page for an unauthorized user #1103 [Origin/Cache] Improved “site name” handling #1176 [Director] Director JWKS cache no longer becomes invalid when the registry goes down #1119 [Director] Added a prometheus metric keep track of the total number of advertisements to the director #1133 [Director] Enable director login via OAuth #1150 [Cache] Allow the cache xrootd to have multiple data/meta mount locations #1191 [Director/Registry/Origin/Cache[ Read environment variables with PELICAN prefix for Pelican binaries with alias (stashcp and osdf) and deprecate alias-specific prefixes #1162

Highlights

For Users

• [Plugin] stash-plugin now supports recursive uploads and downloads #1011
  

For Admins

• [General] Fixed administrator authentication check failure for all servers #1013
  
• [Documentation] Major revisions to the documentation with new documentation for serving a cache #1028 #904 #912
  
• [General] Globus was added as the second supported OAuth provider #963
  
• [General] Formatted discovery and federation endpoints to be standard https URLs #974
  
• [Origin/Cache] Pelican Origin and Cache now support OAuth authentication for the admin website #963
  
• [Origin] Origins that export S3 can now export multiple buckets with unique access/secret keys #1045
  
• [Origin] Origin now can check a sentinel file for the mounted POSIX storage #1003
  
• [Origin] Origins now support HTTPS servers as a storage backend #958
  
• [Cache] Admin website is now available for the Cache #1039
  
• [Cache] New Prometheus metric to monitor cache access statistics #1007
  
• [Director] Director admin can now disable traffic to cache and origin servers #933
  
• [Director] HEAD request is now supported when the client request a file from the federation #1000
  

What's Changed

• Upgrade golangci GHA version and turn off caching by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/935
  
• Fix log color loss and LogLocation not working by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/939
  
• Fix pelican plugin bug no valid classads... by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/942
  
• Embed serve as default command to run the server container by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/947
  
• Improve installation docs and include running Pelican server with docker image by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/904
  
• Update documentation to serve an origin by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/912
  
• Correct the key for docker doc page by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/955
  
• Moved failing macos tests into a linux only test file so as to stop c… by @turetske in https://github.com/PelicanPlatform/pelican/pull/959
  
• Remove generate package when releasing by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/937
  
• Create tmp cinfo file for cache self test under the same folder as the final file by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/952
  
• Updating GitHub Actions to deal with size issues by @turetske in https://github.com/PelicanPlatform/pelican/pull/950
  
• Fix bug with version being "dev" by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/962
  
• API support for registry admin to delete a namespace registration by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/915
  
• Allow RSA key for TLS by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/965
  
• Revert changes that remove generate package by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/967
  
• Airplane mode by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/926
  
• Update Pre Release by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/982
  
• Consolidate two namespaceExists DB function by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/924
  
• Allow admin to configure filtering servers for redirecting traffic at the director by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/933
  
• Director based cache health tests by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/960
  
• Import parameters.json into website by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/972
  
• Fix director default response and add HEAD as allowed verb by @jhiemstrawisc in https://github.com/PelicanPlatform/pelican/pull/1000
  
• Pelican client advertises timeout to cache by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/1002
  
• Overhauling discovery endpoint by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/974
  
• Unit test for statHttp within Pelican Client by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/890
  
• Redirect logging location reminder to stderr by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/989
  
• Fix bug where project name not set in user-agent by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/977
  
• Authenticated self-tests for cache by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/978
  
• More verbose error message at director registration failure by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/992
  
• Bump github.com/lestrrat-go/jwx/v2 from 2.0.20 to 2.0.21 by @dependabot in https://github.com/PelicanPlatform/pelican/pull/1014
  
• Periodic cleanup of origin test files by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/987
  
• Bump google.golang.org/protobuf from 1.31.0 to 1.33.0 by @dependabot in https://github.com/PelicanPlatform/pelican/pull/1015
  
• Output pelican config to stderr at server start by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/994
  
• More detailed error message at WaitUntilWorking not working by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1016
  
• Update Prometheus doc to introduce counting server number by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1009
  
• Cache hit/miss bytes Prometheus metric by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1007
  
• Fix invalid error message type as server response by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1023
  
• Pin external c++ dependencies at a specific commit by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1022
  
• Add LotMan configuration to the cache by @jhiemstrawisc in https://github.com/PelicanPlatform/pelican/pull/889
  
• getNamespaceByPrefix now checks topology for namespace existence by @turetske in https://github.com/PelicanPlatform/pelican/pull/1021
  
• List namespace topology through UI by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1031
  
• Remove Testing Code by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1036
  
• Create serving_a_cache.mdx by @turetske in https://github.com/PelicanPlatform/pelican/pull/1028
  
• Add server urls to error messages for WaitUntilWorking by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1034
  
• Print pelican version at server startup to stderr by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1027
  
• Remove use of global fed metadata within client by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/878
  
• Add the option for a sentinel file for the origin by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1003
  
• Do not log "director test timeout" warning if director can't be reached out by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/991
  
• Improved plugin error messaging by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/999
  
• Add "login required" endpoints behind "admin" check and fix bug in admin auth handler by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1013
  
• Fixes to pelican plugin staging by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/882
  
• Allow configurable cache watermarks by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/973
  
• Add Cache Webpage by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1039
  
• Add support for director/federation contact info as a configuration by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1012
  
• Protocol version added to plugin -classad output by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/1051
  
• Fix error responses for local cache by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/1041
  
• Removed the setting of an AuthURL for Pelican serverAds by @turetske in https://github.com/PelicanPlatform/pelican/pull/1043
  
• Refactor registry namespace check to be compatible with OSDF topology by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1038
  
• More error tolerant web ui check by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1040
  
• Add support for custom xrootd config file by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/968
  
• Config fixups by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1057
  
• Add topology timeout by @turetske in https://github.com/PelicanPlatform/pelican/pull/1058
  
• Plug S3 plugin changes into multi-export S3 origins by @jhiemstrawisc in https://github.com/PelicanPlatform/pelican/pull/1045
  
• Bump the revision of xrdcl-pelican for libcurl testing by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/1065
  
• Correctly return the transfer attempt information by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/1063
  
• Update Director by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1068
  
• Move default S3 url style from defaults.yaml to config.go by @jhiemstrawisc in https://github.com/PelicanPlatform/pelican/pull/1070
  
• Update Origin UI to use API for the new Origin.Exports scheme by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1042
  
• Allow recursive uploads/downloads in plugin by @joereuss12 in https://github.com/PelicanPlatform/pelican/pull/1011
  
• Make the local cache location unique for each test run by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/1066
  
• Enabling OAuth for the origin and cache, adding support for Globus as another auth server by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/963
  
• Update login UI by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1030
  
• Only push the production container when the release was cut by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1077
  
• Cache Updates by @CannonLock in https://github.com/PelicanPlatform/pelican/pull/1083
  
• Fix broken URLs of namespaces from the topology at the director by @haoming29 in https://github.com/PelicanPlatform/pelican/pull/1071
  
• Add support for HTTP backend by @bbockelm in https://github.com/PelicanPlatform/pelican/pull/958
  

Full Changelog: https://github.com/PelicanPlatform/pelican/compare/v7.6.5...v7.7.0

Highlights

For Users

• [Breaking Change] – Removed the deprecated Federation.NamespaceURL parameter. This will cause failures if a user is still using that value as opposed to Federation.RegistryURL
  
• [Experimental] – Adds a local whole-file-only cache which works over a Unix domain socket
  
• [Client] Multiple uploads and downloads are always done in parallel
  
• Federation discovery download failures are marked as retryable
  

For Admins

• [Breaking Change] – Removed the redacted Federation.NamespaceURL parameter. This will cause failures if a user is still using that value as opposed to Federation.RegistryURL
  
• [Documentation] – Added documentation for serving a registry and director to the website
  
• [Documentation] – Added documentation for setting up Grafana to visualize metrics collected on Pelican servers
  
• [General] – We now build separate images for each Pelican server module
  
• [Origin] – S3 origins no longer require a bucket to be specified
  
• [General] – Fixed a bug where Prometheus token didn’t renew and caused 403 errors at the origin and cache servers
  
• [General] – We now allow edits of Pelican server configurations via the web UI
  
• [Registry] – Improved web UI for namespace registration and management
  
• [Cache] – Added a Cache.Concurrency parameter to limit the number of connections to a cache
  
• [General] – Fixed a bug where XrootD process leaks when Pelican server encounters error at its start
  
• [Cache] – Cache’s can now start with a list a accepted namespaces they will accept origin data from
  
• [Origin] – Origins now support multiple exports with different authorization configurations
  

Updates

Congratulations @williamnswanson for your first contribution in #684!

Highlights

For Users

• [General] – Improved logging doesn't report as many false errors
  
• [Client] – Improved file unpacking to handle more use cases
  
• [Client] – The plugin now downloads files in parallel
  

For Admins

• [Origin] – Pelican origins now support public namespaces
  
• [General] – Federation admins can now fine tune whether origins and caches require admin approval before joining the federation
  
• [Origin/Cache] – XRootD log levels are now configurable
  
• [Director] – The director can now report whether a specific origin has a requested object via the new director stat feature
  
• [Origin] – Fixed rate graph data loss issues in the origin
  
• [Registry] – Added the ability to configure a URL for fetching the registry's institution list
  

Highlights

For Users

• [Client] Introduced pelican object get and pelican object put to specify a get or put rather than just running pelican object copy. #259
  
• [Client] Fixed issue where the stash_plugin file-transfer plugin was not reporting proper upload bytes. #510
  
• [Client] Fixed issue with progress bars not outputting properly with debug logs for recursive uploads/downloads #402
  
• [Client] Fixed issue where stash_plugin was not working with environment variable-based configuration #508
  
• [Client] pelican object copy now doesn’t rely on topology for PUTs, it gets the endpoint by querying the director #504
  

For Admins

• [Director] Introduced GeoIP Overrides that allow a director to pin IP ranges to a particular lat/long. #496
  
• [Registry] Introduced web-ui for registry that allows users to register their namespace through web interface. #264
  
• [Director] Added federation discovery endpoint to director. #417
  
• [General] Added topology components to Prometheus health metrics. #494
  
• [Cache] Added Admin approval to caches before they can advertise to director #439
  
• [Registry] Added support to use CILogon to login to registry UI #355
  
• [Registry] Deprecated Federation.NamespaceUrl which will be removed in the next major release #388
  
• [General] Web-ui now outputs more detailed error messages if login fails #424
  
• [Origin] Basic S3 origin functionality has been restored #592
  
• [Cache] Fixed issue where caches were logging sensitive information #604
  
• [Origin] Boolean flag added to starting origins indicating if they are writable or not (default is true) #507