UK Retail Bank – OpenShift on Z Logging
Client Summary
The client is one of the UK’s largest financial services groups, serving around 30 million customers through a portfolio of well-known retail and commercial banking brands. With roots stretching back over 300 years and a workforce of more than 60,000 people, the group provides a full range of services including current accounts, savings, mortgages, insurance, pensions, and lending to individuals, small businesses, and large corporates. As a structurally important, FTSE 100-listed institution with the UK’s largest digital banking customer base, it operates under intense regulatory scrutiny while pursuing a major ongoing investment in IT and digital modernisation.
The Challenge
The modernisation of the digital services transaction router application from IBM pSeries to OpenShift on IBM Z required considerable refactoring of the application as well as re-platforming. Technical logging performed by the application is hugely important to the client to be able to provide business support to individual customers as well as to provide a financial services audit trail of customer account interactions.
The problem statement presented by the client focused on the large volume of log messages that were being lost in the performance testing environment, which would prevent the solution moving forwards into production. Following investigations, there were additional challenges around very high CPU consumption in the logging infrastructure and scalability challenges.
Objectives
The principal objectives for this work were to modify the service to ensure 100% of the application log messages reach the Splunk data store, and to reduce the CPU cost of logging to a level where the service can scale to meet the demands of the business.
Technical Context
OpenShift has a Logging Operator which operates as part of the infrastructure. This captures all of the stdout and stderr output from each executing container and saves it in a file structure on the worker node where these run. To protect the service, this file store is finite – by default 3 x 10MB files per executing container – and the log forwarder (implemented by Vector) has to keep up with the job of reading these and forwarding them to the target store – Splunk, in this case. The log capture is simple and wraps around the set of three files. This means that if the records captured are not forwarded promptly and there is a high enough application load, it is possible for unforwarded records to be overwritten.
As well as reading and forwarding the application and infrastructure log data, Vector decorates each record with metadata retrieved from Kubernetes to provide additional context for the source.
This metadata adds between 2KB and 2.5KB to each log record, which is significant when the application log records are 80 – 100 bytes each.
Our Approach
Detailed investigation and performance metrics supported the diagnosis of the root cause of the problems that the client was facing:
- The volume of application log records (c. 24K log events/second at peak loading) far exceeded the capabilities of the OpenShift log forwarder (c. 4K events/per second).
- The level of meta data decoration was not configurable, which resulted in application log records (80 – 100 bytes each) being forwarded with an additional 2-2.5KB of metadata added to them
- The F5 network layer between OpenShift on Z and the Splunk forwarders had insufficient licensed capacity to handle the load. This exacerbated the slowdown in the log forwarding infrastructure, and resulted in captured log records being overwritten before they could be offloaded.
In conjunction with a reduction in the number of logging events implemented by the application team, we proposed moving the application log forwarding into a custom Splunk Universal Forwarder sidecar, which would run alongside each of the application container images.
Outcome and Benefits
The primary objective of ensuring that 100% of application log messages reach Splunk was successfully achieved.
The CPU footprint for the whole performance test OpenShift cluster was dramatically reduced from 24 IFLs (IBM Z series) processors to 15 during a representative peak load test. This is a 37.5% reduction in capacity and licensing footprint.
Other notable benefit that came from this work:
- Direct customer contact with the Red Hat Observability development team
- A clear view of the network topology between the OCP on Z clusters and Splunk Cloud
- Some clarification on the capabilities of the Cluster Log Forwarder for future implementation planning
- Removal of application logging scalability constraints
Lessons Learned
The client was an early adopter of OpenShift on Z, and this meant that we unearthed some of the challenges and limitations for the first time, with help from the vendors. Takeaways from this work would include:
- The Cluster Log Forwarder (Vector) in OpenShift is tested to 7K log events per second by the development team. However, the Level 3 support team documentation speaks to a 4K log events per second upper limit.
- The CPU cost in Vector of gathering the metadata to decorate log records with is significant at the higher logging levels.
- OpenShift infrastructure components produce log messages at a surprisingly high level of detail by default.
Follow Up
The application has cleared performance test with strong confidence to deliver into production and scale up to full capacity. Triton Consulting continues to support the customer on OpenShift on Z solution delivery, performance tuning, architecture and data.
Looking to modernize your mission-critical applications with confidence? Whether you’re planning an OpenShift on IBM Z deployment, optimising application performance, or preparing to scale for production, Triton Consulting has the expertise to help. Get in touch with our team to discuss how we can support you.