Triton Consulting

IBM Optim provides functionality for secure archiving, as well as the functionality to compare data, restore data, and browse data. The Optim solution can handle many different database management systems singularly and also works well cross-platform, this includes DB2, Oracle, SQL Server, Sybase and also covers legacy systems such as IMS. The following will provide a brief outline of the Optim product and its capabilities.

Installing Optim

This fairly simple, the following covers a simplistic Windows installation, using a gui to install. The first step requires setting up the first workstation, this is the machine that will have the connectivity to the databases/s, you can also install on a file server. Once the installation is complete, run through the configurations for the first workstations (be aware that not all the configurations have to be completed, you can come back to them through optim configuration) and any further workstations. A registry file can be used to configure further workstations by importing the previously exported registry file during the configuring of the first workstation. Other configurations include personal options, security, file locations etc.

The security aspect is covered by three areas: functional, object and archival security. Archive file security refers to the control over an archive file, specifically the access to the data within the archive files, which is associated with a file access definition (FAD). The FAD will list the tables, columns etc. and specify the privileges for them and the specific roles involved. Functional security restricts or allows the user’s access to the Optim objects, through the use of ACD access control domain, which is automatically created when you initialize the security parameter. Lastly object security, using an ACL access control list an administrator can control the access to specific Optim objects within the Optim directory. Suffice to say security is tight and all aspects are interrelated and work together to provide the required level.

This first workstation can also be used as the server, which can be configured using the same process as the first workstation, but instead selecting the check box for server set-up, once completed you can configure further workstations or servers. The Optim Directory is home to sets of tables, which store Optim related object attributes that have been created. Installations can be undertaken on a file server, however this installation has its drawbacks, mostly performance related. An advantage is the administration of a single file server is much easier than several workstations.  This is the simplistic view of the Optim install and configuration, a more detailed overview can be located within the Optim installation and configuration documentation.

The user configuring Optim will need to have the necessary priviliges which allow the user to create tables, procedures etc. All database management systems will have their own related permissions. DB2 UDB users at a basic level require CONNECT, CREATETAB, IMPLICIT_SCHEMA, BINDADD and CREATE_NOT_FENCED_ROUTINE.

Your license agreement determines which components you will have access to. The crux of the software is to provide mechanisms to handle data growth and therefore the following components are integral. As mentioned in my first post, archive is the process of copying data from one location to another usually to a file, nominated as the archive file. The aim is to remove aged data or data that is used infrequently out of an environment where data growth is substantial. This data will then be available for a period of time for searching and even restoration using the browse utility. The process is not very involved and requires an archive request, which governs the data to be archived through the access definition. The archive request paramitises the archive file, index file, access definition etc. and the the access definition details the start tables, relationships, archive indexes, archive actions covers features like the archive process. The deletion process can be invoked after the archive or deferred till a later time. If using a deffered delete the data can be browsed and confirm the data archived is correct. A further utility provides the restoration of archived data to an existing database or to a new database. As with nearly all tasks associated with Optim, the parameters for the utilities are set within request editors, whether this is the delete process request editor, restore process request editor etc. 

The concept of the move facility is to extract and insert data that can be complex or simplistic, the selection criteria will specify what is required to be moved and saved as an extract file. The extract file can then be used to populate tables, that may reside on a development database or a production database, the file can also be sued to create the tables if required. Using move to create a test region by way of moving production data into a test environment can benefit the user by providing live data to test against. The data extracted can be selected through several methods and can be transformed, aged and migrated.

Opinions are split when talking about IT certification and its importance in the industry. It is easy enough to pick up a book and study chapter after chapter, running through sample questions. Once confident enough you sit the exam and pass and then move on to the next one. Currently I am working my way through the IBM Optim Certification. The simple “read the manual” approach  does not apply to this particular certification. It is important to study but also to apply what you are studying and that is what I am currently doing with Optim. Get yourself a working demo and then work through the various revision notes and create actual archive files, extract files, access definitions etc. You may be wondering what Optim is and what an archive file has to do with anything so I have decided to share my learnings with you as I go along

Optim’s main function is to manage data growth and to archive legacy data, but this simple explanation would not be doing it justice.  In later posts I will talk about the full range of Optim’s capabilities.  In this post I will concentrate on how Optim can be used to manage data growth.

Data has been growing at an alarming rate over recent years. A lot of organizations experiencing this growth are not  managing it effectively. This can cause significant issues with various database utilities, specifically slow running transactions, backup and recovery problems and storage issues to name but a few.

The option of archiving data has been available to organizations for a long time but has been complicated rather than simplistic and has not been a process which delivers security, privacy, data integrity, storage media and the ability to keep up service levels. Previously, data that had been archived may not have been easily accessible or may have been deleted. Meeting service level agrrements and how data is decommissioned requires some planning, ensuring the data is archived in stages and only deleted once the agreed timescales have been met. This means that everyone is aware how long the data will be kept and its availability throughout each stage of its decommissioning. Optim has the capability to not only archive an order line but everything related to a particular order, for example a purchase made for a printer is not archived on its own, all related rows that make up the whole order are also archived, such as the customer who placed the order, the address where the order will be delivered, all related rows, the order as a whole is archived so that the data is relationally intact.

In the next blog I’ll look at installing Optim.

We’re delighted to announce that our event “Data Governance on System z” is confirmed for the following dates:

10th November 2009 – London, IBM Southbank
11th November 2009 – Edinburgh, Apex Waterloo Hotel

Places are complimentary but limited so book today!

Data Governance is a hot industry buzz word at the moment. In this upcoming event we take a look behind the jargon and explore the key areas of governance that are affecting organisations and how, by utilising their existing system z infrastructure, they can meet these challenges.

We’re proud to bring you a top list of industry expert speakers who will be covering topics including Data Privacy, Data Security and Compliance on System z. We’ll round off the day with an open forum where our panel of industry experts will help you build a blueprint for establishing security standards on system z.

On the panel includes:

Julian Stuhler – Triton Consulting

Paul Garstang – IBM

Mark Wilson – RSM Partners

It’s your last chance to book your place on the upcoming “Managing the financial services data explosion” webinar on 9th July.

This free webinar will be run by IBM Data Champion and Gold Consultant, Julian Stuhler and will feature a Q&A session with Andrew Crouch of AVIVA on his prespective on implmenting a data archiving solution.

Post your questions for Andrew and Julian and get your questions answered…

Don’t miss your chance to hear from the experts on how to cure your data growth issues and deliver significant and ongoing cost savings.

Join IBM Data Champion Julian Stuhler and industry insider Andrew Crouch of AVIVA on our upcoming webinar to hear their tips on coping with the data explosion.

To register for this free event visit http://www.triton.co.uk/eventregistration.php

Event details:

Date: 9th July 09
Time: 11am GMT

This event is design specifically for senior IT professionals working in the financial services sector who are experiencing issues related to growing data volumes.

We chat to Andrew Crouch from AVIVA about his experiences of implementing a data archiving solution:

Can you start by telling us a little about how the data archiving project first came about?

Back in 2000 the NUI (Norwich Union Insurance) claims and workflow systems had been running on the current platform for 7 years and policy for 2 years.  The Capacity Management team began to have concerns about the levels of data growth and the cost of a serious system failure to a rapidly increasing user base.  This kick-started the project with problem definition and research of options being undertaken.

You say that the levels of data within NUI were growing at that time, what were the main drivers for that growth?

Data from legacy systems was converting to the new platform, this was also the period that  Norwich Union was merging with CGU.  The user base was increasing and with it the need for greater availability. The whole business was starting to move to where it is today.

What sort of pressures were IT facing from around the business with regards to the growing data volumes?

The business weren’t experiencing any issues as a result of the increased data at that time. The Capacity Management team could see problems on the horizon so the pressure was convincing the business that action was required. We were a bit ahead of the game. 

Excellent, so you weren’t facing pressure from the business, this must have made your lives easier whilst you were researching the solutions.

Not really because the pressure was reversed in a way.  As an IT department we needed a business case to justify the need to run the project.  The fact that the business were not experiencing any issues relating to data growth at that time meant that we faced some resistance.

So how were you able to demonstrate to the business that a data archiving strategy was needed?

The two main drivers were the Data Protection Act and system recovery times. 

The DPA convinced the business that we needed to implement an archiving strategy.  The business was fully aware of the need to comply and potential costs and damage to the brand if Norwich Union were to be found in breach.  DPA compliance was high profile and resulted in a separate group wide project being set up to deal with the issues.

The other major factor was system recovery times.  Due to the large amounts of data we were dealing with we concluded that should we suffer a major system fail we would not be able to keep within our SLA for recovery.  Given the increased user base mentioned earlier, any outage would mean the costs incurred would be far greater than previously.

When presented with these two strong arguments the business agreed and the project began in earnest.

How did you go about choosing which tool you would use?

We worked in conjunction with our partner CSC.  They did a lot of the initial research into the market for us and came back with 3 options- IBM, Compuware and Princeton Softech.  (This was prior to IBM’s acquisition of Princeton Softech).

Norwich Union used a standard checklist for software acquisition which enabled us to list functional and non functional requirements, this helped us to measure the products against one another.  It was clear very early on that the Optim (formerly Princeton Softech) solution was far superior in terms of functionality.  The selection process only took around 6 weeks.

So, at this stage were you ready to begin implementation?

We should have been but we came across some resistance during this process as the question of “build vs buy” was raised.  A number of alternative ideas and theories for internal solutions were put forward but this was from a support area with limited resource and not a development area, it was impractical to have them tied up in a build project like this which would take up a considerable amount of their time.

It took a significant amount of time to come to an agreement that the best solution would be to buy Optim rather than try and build something bespoke.

Once you had reached an agreement on this how did you find the process of implementing Optim?

We had a big advantage in that we held existing ERD’s (Entity Relationship Diagrams) for the data. This enabled us to build the relationship model in Optim in just a couple of days.  The business rules in some cases were quite complicated though, particularly around retention of data for claims with injury.

The solution was implemented into production in 2002 and we began with simple criteria to get us up and running.  We then moved onto more complicated business rules which needed supporting batch suites built to handle the diverse criteria held across multiple DB2 tables and objects. The batch suites did all the work in identifying cases that qualified for archiving and Optim did the rest.

You mention the business rules; I guess you had to involve quite a lot of other departments in this project?

Indeed, this wasn’t a project which sat just within IT.  We had to consult with Legal, Business Users, Data Owners and users and various teams within the IT department.

To give you an idea of the scale, we had sign offs from over 50 impacted areas who had an interest in the data (or copies of the data)

A few years down the line, what have been the main outcomes?

There have been three main outcomes. 

1. Recovery times – this had been one of our key objectives.  We’re now confident that recovery times will be kept within SLAs now that we have less data in production.
2. Data Protection Act – The implementation of data archiving means we’re now happy that we’re meeting DPA regulations.
3. Performance – We’re now comfortably managing system performance.  Although, as with most things there is always room for improvement!

What would your “top tips” be for anyone embarking on a data archiving project?

My top tips would be:  

1. Make sure you have buy in from everyone with an interest, the “build vs buy” issue we hit caused delay for many months.
2. Know the data model & database, with the up to date ERD’s and some knowledge of the database we built the model in Optim in a couple of days.  If you don’t have this you could be looking at a significant amount of extra development time although I understand there are moves to automate this in Optim.
3. The business rules for what you can archive may be more complex than you think, we have to code in black and white but the business may present a lot of grey.
4. You need to know who is querying the data or extracting the data or working on copies of the data. They will all want a say on what gets archived.

As BT famously pronounced “It’s good to talk” – and we’ve been doing a lot of that recently!  In particular we’ve been focussing our “chatty” attention to LinkedIn.  Because we enjoy talking we thought that it is quite likely that some of our customers do too so we started our very own group especially for Z series users.  

The idea of the group is that we get as many z/OS people together as we can to discuss industry trends, best practice and to share knowledge.  We’ve currently got discussions running about managing data growth and data security.

There’s quite a gaggle of us now but the more the merrier!  So take a look and join in the discussion!