Coming through an Oracle license review... smelling of roses

A license review can be stressful at the best of times. Licensing is such a complicated topic and non-compliance can be so costly... 

At one of my previous employers, we had an Oracle Siebel CRM application, bought when Siebel was still Siebel (and not Oracle). Over the years, Siebel changed its architecture, newer versions came out, and from an initial deployment somewhere in the Siebel 6.x series, the organization upgraded to 7.5 and then to 7.8. Today they are at a higher version, but this story is from the 7.8 times.

As the CRM team manager within the IT department, I once got a list handed to me for verification. It was the annual Siebel Support contract, listing out the modules / components / whatever we had (supposedly) bought from Siebel / Oracle and for which we paid annual support costs. Normally, the support was always paid directly after approval from the service manager, but due to changed procedures within the company, the CRM manager (me) was asked to rubber stamp the CRM support contract.

Only problem was - when I read through the list, it made no sense to me at all. Though a manager, I was pretty hands on and knowing exactly what our Siebel implementation did for us, I was pretty sure that the items listed in the list were outdated by a few years at least. My top Siebel expert verified my doubts when I showed him the list - This "component XYZ" doesn't even exist in the current version of Siebel, he said.

Anyways, not one to let sleeping dogs lie, after agreement with the IT head and the Service Manager, I asked our Oracle Account Executive to come over for a cup of coffee. I then handed over the list to him, and asked him to explain. He couldn't, and promised to get us more information.

Two days later, a letter dropped onto the desk of the IT head, reading which caused him to use words which would be highly unprofessional to quote here. It was a letter from the Oracle licensing department, informing us that we would be audited for license compliance, and naturally a lack of compliance would have financial consequences. They promised to get in touch within a week.

Panic ensued, to put it mildly. We went desperately in search for old contracts archived long ago, to see what exactly we owned licenses for. We found a few old contracts, and couldn't trace some others but whatever we found made no sense either. The names of the purchased modules were cryptic and un-traceable to modules we recognized from today's Siebel, as we owned it.

Calls to the Account executive got only sworn promises of having had nothing to do with the audit landing on our heads. By coincidence, the day after we asked him information on why our support costs included items we could not place at all,someone took a decision to audit us. Well, life can be strange sometimes and coincidences do happen :-)

A lady from Oracle licensing contacted us within a week, as promised. She outlined very clearly and unemotionally, the procedure to be followed in the next few weeks:

• She would coordinate from NL, but the actual audit compliance would be checked in the US
• The US team would arrange a session with us, to explain exactly what information they would collect, and how they matched that to various modules within Siebel.
• We would have 2 weeks time to "clean  up" our production, since they recognized that unintended mistakes could happen (wink wink).
• Oracle would run scripts on our Production environment, some of them continuously for a 3 week period, which would gather all required information directly. This information would be evaluated by the US team.
• A report would be generated, which would give details of what we we were entitled to, versus what we were using. No emotions, just facts. We would have 1 week time to evaluate the report and come up with arguments and objections to the findings. Then the report would be final and handed over to the Account Executive.
• Then it would be time to pay for our sins .... :-(

We tried to bluster it out -

• "And what happens if we are compliant", we asked. "Nothing, life goes on as usual", she said.
• "And what if we are in reality using less than what we are paying for. I hope we will get some money back", we blustered. "That has never happened in all my years as a license audit manager", she smiled.

A small point of appreciation here for Oracle. Even though we felt betrayed, the way the licensing team at Oracle approached this audit was extremely professional. The lady in question (who was coordinating from NL) tried to get answers to every query we had, and seemed genuinely eager to assist us where possible. She was very professional, but human. She understood we were pissed as hell, but promised to do her job fairly and honestly. Take a bow, Caroline Struyck.

The next few weeks were a haze. We had the sessions with the US team, and understood how the business objects within Siebel could be mapped to the modules which formed the basis of the current licensing structure. Further, mapping user's responsibilities within Siebel to the views enabled us to come up with our own matrix of what business objects we were using in production, and put a number to it by counting number of users having those responsibilities.

Without going into low level details, we understood that optimizing the structure of responsibilities in Siebel would help us the most. Over the years, our Siebel implementation had grown from a handful of responsibilities to a total of 28 different responsibilities (which determines what users having those responsibilities can see within Siebel), with a lot of redundancy. Within 3 weeks, often working at midnight to avoid disturbing production users, we merged responsibilities and cut and fine tuned views within each responsibility, till we ended up with a final list of 9 responsibilities which we thought would cover every potential use of our Siebel application by our business users.

It was always possible be that we cut and fine tuned too much, and made necessary functionality disappear. To prevent chaos on a large scale, we always applied the changed settings to two of our most fanatical (and critical) users first. If 2 days passed without furious phone calls to the service desk, we applied the changes to everyone else. It speaks volumes of our knowledge of how our users used our Siebel application, that the entire exercise ended with 0 complaints.

We also took a chopper to the number of users "active" in Siebel. We were informed by the US audit team that it wasn't the "Status" field which decided which user was active or not, but whether the user had any responsibilities attached to him. All of our expired users (people who had left over the years, for example) still had their responsibilities attached. Boom ... cleaned them all up in one night. Then users who hadn't logged into Siebel for 3 months or more were the next to go. Often, out of service users just weren't reported as gone, and continued to exist in the system. Boom ... cleaned them up too. Made some enemies in this process as some "still in service" users who had logins but never used the application wanted to hold onto their logins (why ... didn't make sense at all). Sorry, you can hate me if you want, but your account is gone too. BOOM!

The day came when we called up Oracle that we were all cleaned up, and ready for their audit scripts to be run. Since the Oracle scripts and evaluation method was a black box for us (even after the process had been described to us), all we could do was make an educated guess and cross our fingers that the damage wouldn't be too much.

The scripts ran for 3 weeks in Production, and all the audit logs were uploaded to the Oracle FTP site. A painstaking wait for almost a month, before a draft copy of the report was emailed to us.

As I flipped quickly through the report, I felt like a million bucks. The audit findings were so close to our own matrix of findings, that it was scary. Here, I would like to give full credit to our Siebel expert who painstakingly mapped every single business object in our Siebel Repository to the module structure explained by the US audit team. Take a bow, Remco van der Heijden.

We weren't completely in the clear, though. We were compliant in terms of modules used, however a little short in the number of named users. We had to make a one-time purchase of additional named users to fulfill this shortfall. It wasn't a big amount at all.

The icing on the cake was that the audit report proved indeed that the list of items in the annual support costs contract was indeed way out of line. After hard discussions with a very confused Oracle Account Executive - as said earlier, this had never happened before in their experience - we managed to get a 25% cut in the annual support cost fees. "And what about all the years we have been paying 25% more, then", we asked. "Pay us back"!

Getting money back wouldn't happen in a million years, and we knew it. It was just the euphoria of having walked on the edge of a black chasm - a threat of having to pay potentially huge damages for non-compliance - and now basking in the glory of a well earned victory.

 This entire exercise taught me some very valuable lessons:

• Guts and glory is a valid leadership style: If you have confidence that something doesn't make sense, have the guts to bring it out. You could let sleeping dogs lie, but you will never influence a change that way.
• Work for people who back you: The entire thing started because I insisted on Oracle giving us a clarification for the money we were paying towards support costs. This insistence could have ended very badly for my employer in terms of financial consequences. Till the day the report came out and we were in the clear, my boss never said to me even once - "Why the hell did you create this mess?". Take a bow, Matthijs de Zoeten.
• Know your application AND also your business users: It boggles me to think that normally we would make every small change in production after taking it through various environments (test / acceptance etc.) and getting it tested at various levels. During the clean up operation just completed, we made plenty of changes directly in Production, always at night, and always by testing first against our most critical users. That was the only insurance we had, that any screw ups would be reported asap. Our knowledge of our business users turned out to be impeccable. Everything we weeded out as not required was .... not required.
• Lead from the front, always: It sounds corny but true. Being the hands on guy that I am, I was in the middle of every activity during this license review - from nightly clean ups of the production environment, to pruning down the responsibility-view matrix list from 28 to 9, to engaging in arguments with still-in-service users whose accounts I killed because they just didn't need them (but still wanted to keep them), to every single meeting with Oracle License department, to reviewing in detail the business objects matrix created by my Siebel expert (no, he wasn't happy at this proactive meddling, in case you were wondering). It could have ended badly, but at least I would have the happiness of knowing that I tried my best.

A professional career gives you many experiences - good and bad - and this is one I will always cherish: Coming through an Oracle license review ... smelling of roses!