How To Make IBM Cognos Work

I could finish this article in one sentence by telling you what IBM cannot say because of the risk of offending customers.

Fix your database design using a known data modeling strategy so that Cognos can understand what you did!

Cognos is not the problem. It's the lack of proper design for the data model to which Cognos is connected. Less than one percent of companies and governments employ a Data Modeling contractor before their software is written. Additionally, most people in IT management don't seem to know that the database must be complete BEFORE the software is written. Most continue to deploy the failing strategy of having code developers create databases AFTER writing code. Since the data model is a capture of BUSINESS RULES AND BUSINESS LOGIC DERIVED FROM DOCUMENTED BUSINESS REQUIREMENTS, via its structure, creating the data model after the code is already written, based on the code instead of business logic, is not logical and never works. If you are wondering why I put the business rules and logic in all caps and bold letters, it's because their omission in the analysis and design phases is the primary cause of all IT project failures, which fail at 93 percent or higher via budget, business requirements, security, interoperability, or performance failures.

To succeed, one must make the code conform to the data model which locks in business rules and business logic, which includes security logic, so that those rules and logic cannot be violated by anyone or anything that connects to the databases. Not following that one Data Governance rule will ensure that one's system can be easily hacked by compromising weak programming. It will also ensure that one's project will not meet business requirements nor budget requirements by the end of the project. If one has a project plan that does not include data modeling by an actual data modeler, one's project will fail. If the person claiming to be one's data modeler does not have the minimum 30 skill sets required to become a data modeler, he or she is a liar and not a real data modeler.

Additionally, one needs a functioning ERP system with a functioning master data database to even begin to consider business intelligence with Cognos. Therefore, the data modeler is the one role in IT that cannot be faked. You need the "real deal" to succeed. There is no college student nor H1B from a third world country who can do the job required by at least ten years of experience combined with natural ability. Additionally, most companies spend an average of six million dollars on third party analysis and project failures to find out that they need a "real" data modeler. However, they then seek to hire a data modeler as if he or she is a "normal" five or six figure income employee. Realistically, most data models require two hours of labor at $500 / hour or higher for each individual table and its relationships. Therefore, your data modeler should be a contractor or officer with a seven figure income. If you allow the data model to be done wrong, it's like allowing the foundation of a 150 story, multi-billion dollar sky scraper to be done wrong.

The usual minimum size of an ERP data model, which most companies desperately need, is eight hundred tables. The minimum cost of an ERP data model is $800,000 USD. It will usually only include HR, payroll, accounting, product database, and customer database. ERP data models with specialized sections such as supply chain management, fraud detection, change data capture, encryption, row level security, asset management, Project and Portfolio Management, staffing profiles, Business Information Management, Digital Agenda, Document management and data lake, and other advanced features can reach 6,000 to 10,000 tables and can take up to 10 years to complete.

The minimum cost for an all-encompassing data model is $6,000,000 USD. That's why it's hard to "hire" a real data modeler who knows the value of data models. I have gotten thousands of emails and calls offering jobs that pay as little as $60,000 USD / year to create data models worth over $800,000 USD to over $60,000,000 USD. Additionally, having the model would save most companies over 14 million dollars per year for 30 to 50+ years, bringing the actual value of data modeling to over $700,000,000 in bad data cost savings alone. Data modeling also increases profit by increasing efficiency and productivity, while also reducing fraud. Most of the time, the recruiters don't even know what a data model is nor what a data modeler does. Many say, "Sure. I know what it is. Data modelers model data." In many cases, nowadays, even CIOs do not know what a data modeler is nor what one does. Ironically, most data modelers have experience far beyond that of a CIO. Once upon a time, a CIO was a veteran data modeler who earned an MBA in night school. Understanding data and data modeling was once a requirement to become a CIO just one generation ago.

A proper data model can streamline a company's collaboration, HR, payroll, accounting, and anti-fraud systems, change data capture systems, data entry systems, supply chain, and other functions, and reduce costs by billions in a single year. Over 99 percent of ERP projects fail because the analysis and design for the data model is skipped due to the required time to complete it. For a major corporation, an ERP cannot be made from scratch in one year or even five years for most large companies. A real ERP project is a long-term five to ten-year or longer project with real resources paid at real rates with real contracts. The only way to reduce that time is to buy a pre-made data model that conforms to industry standards which is also at least 80 percent in conformance with a company's business logic. The model can then be modified and documented to adapt to a specific company. Such a model would also require a digital data dictionary to prevent project failure due to a lack of analysis and design. Such pre-made industry standard models are only made by IBM or very small firms or individual data modelers. Since professional data modeling software, such as Erwin and E/R Studio, costs $3500 to over $5000, most people do not have their own data models even if they can create them. Free software simply does not meet enterprise standards and speed requirements. Additionally, a professional data modeling station can cost well over $300,000 if it has the capacity to create an ERP data model.

The failure of an ERP project starts with inexperienced managers who do not plan the project as a long-term project and gives a team an arbitrary deadline that is far below the time required to complete the project as if it is a manufacturing project instead of a technology development project. Additionally, most managers do not know the specifications for professional data modeling equipment. The vast majority of managers have a manufacturing mentality that mandates cost cutting, usually labor and equipment before the project even begins. When the time estimate on a manufacturing project is three months, one can cut that time in half by overworking employees without paying overtime or adding additional resources. However, a technology project is totally opposite. The more resources one adds to the project beyond a certain point of a functioning base team, the slower the project will go due to collaboration and advanced conceptual thought requirements. The more IT workers are overworked without work and life balance, the lower their productivity will become. IT project time requirements are rarely alterable and an arbitrary time alteration usually causes a cascading project failure. When the ERP fails, the BI, Business Intelligence, also fails. There is no way to collect business intelligence without a functioning ERP system. Therefore, most companies do not have real, actual, real-time business intelligence, business information modeling, nor predictive analytics. Most CEOs literally fly by the seat of their pants because they don't have proper access to their data. Consequently, the CEO then has to become "better" and far more resilient and usually more expensive.

Unfortunately, most IT managers and even executives as high as CIO are unfamiliar with advanced ERP data modeling and may have no idea what is required to create one. I have repeatedly been hired to create an ERP data model using a 2GB to 8GB RAM laptop, a docking station, and two 1080p monitors, but I cannot say, "Are you crazy," in response. Many don't even know that it takes 16GB of RAM just to compile the EJBs for an ERP. Most importantly, IT managers, who are really trained for manufacturing, may not know that the data modeler is the source of the base code for most advanced computer software systems because the base code must be generated from a complete data model by a professional data modeler using one of two major languages, Java or C++, the only cluster capable and multi-platform capable languages that can scale to enterprise levels with tens of millions to billions of end-users on a single system. Most code developers are trained by creating one program from scratch, Hello World, then trained with base code that was already in existence because that is how real-world projects are run.

A fake data modeler, who is really probably a data analyst, code developer, or maybe a DBA, can create a fake data model, put actual data into it, get paid, and then be gone or promoted by the time anyone realizes the database has no real design strategy. Fake data modelers usually cannot write Java nor C++ and may try to insert easier languages, legacy systems, and systems that cannot perform at an enterprise level such as Python, .Net, and Hadoop. Many fake data modelers will not generate any base code at all, especially if they use legacy systems, which often have new names, leaving 100 percent of coding to code developers, which ensures that it will not conform to business requirements inside the data model. Legacy systems are not compatible with modern code generators and require manual code. The base code in Java or C++ can be generated for an ERP in less than one minute if the data model is correct. Manual code for an ERP will take decades; decades plural. It would be like rewriting SAP from scratch. Staffing agencies often insert legacy systems into projects to extend the project and their own profit.

Most importantly, legacy systems cannot be secured against modern computers because the speed and sophistication of modern computers did not exist when those legacy technologies were created. The average fifty-dollar smartphone has more computing power than NASA had when they launched the first spacecraft into orbit and to the moon. Yet, companies are still implementing file system databases, such as Hadoop, which is technology from the 1960's to 1980's.

So many companies are going backwards in technology decisions due to a lack of skill within the company which is due to an unwillingness to pay for highly skilled workers. IBM has bolted on legacy tools to Cognos to allow the import of data from tabular databases, such as Microsoft Excel. However, since the legacy databases are not compatible with modern technology, most of the automated Cognos features "mysteriously" do not work when the "bolt-on" legacy support system is used.

The realization of the need for a real data modeler will come, usually, after a severe data breach and/or severe data integrity, performance, and interoperability issues. Once the breach happens, fake data molders and advocates of legacy systems with new cool-sounding names, such as Hadoop, No SQL, Python, tabular databases, and .Net, are likely to blame third parties, such as Amazon AWS, or other teams, and they are likely to hide the real cause of the breach so well, that only a real professional will understand what really happened. Fake data modelers are also likely to throw everything into a few big tables based on spreadsheets with no design nor relationships and call it a data model. That said, most businesses have not documented their business rules and requirements in an actual approved document, usually due to a lack of patience and long-term planning. Don't believe it? Start asking for the complete business requirements document where you work.

The Business Requirements Document should have existed before the business was started as part of the original business plan. Over the years, as the business adapts, the document should be updated. Ironically, as I have found over the years, most companies lay off the middle to upper managers who usually update that document as part of a short-term plan to boost stock prices. If the document ever existed, it may be outdated due to neglect. Also, ironically, the lack of business documentation is what leads executive staff members to Business Intelligence tools, such as Cognos, to try to gain an understanding of how the business is working in real-time so they can make proactive instead of reactive decisions. It's the proverbial chicken and the egg. Which comes first; functioning BI tools, or the Business Requirements Document required to make the tools function?

Additionally, most people do not have the patience to learn data modeling because it takes at least three years to learn the basics of conceptual, logical, and physical data modeling. It takes approximately five to ten-plus years to master physical data modeling, which is also enough time to get a master's degree or PhD. A master data modeler with fifteen or more years of experience is far beyond the PhD in skill. Only a few can turn a document of business requirements into a real database design. It's not something one can learn on YouTube overnight. Data modelers are not simply trained, they are born with a specific type of logical mind and nearly unlimited patience. Additionally, very few people have the patience to draw thousands of boxes and hundreds of thousands of lines that represent business objects, business logic, and business rules on a computer screen. Even fewer have the patience to do it every day for over ten years.

Again, IBM Cognos and other BI tools will only work if they can recognize a known strategy in one's database design. That means that someone has to be patient enough to create the data model with strict conformance to strategies and standards. The data modeler must be bold enough to tell the truth especially when no one wants to hear it in order to ensure that the database design is done correctly. That also means that one's data models must conform to either the Ralph Kimball strategy with full multi-dimensional design or the William Inmon strategy of Master Data Management, MDM, in normalized format, BUT with the BI tools, such as Cognos, and applications ONLY connected to Data Marts which must be in multi-dimensional format and fed by a repeatable ETL process with data from the MDM system. Additionally, unstructured data should be stored in a strategic data lake with a document management database, another area of known data modeling strategy.

Once Cognos has the strategy-conforming data model, it can be transformed into a Java Object Model in the Cognos Framework Manager. The Java Object Model can then be used to create reports, charts, graphs, dashboards, etc. The entire system can also run in real time. Every real data modeler knows that data modeling evolved from object-oriented code standards. Therefore the data model strategy must conform to those known standards.

In the first image below, the interlinkage via foreign keys between stars in an extended star schema is shown. In the second image, the green lined image on the left is a java object model generated from the data model on the right which is in a strict extended star schema design. A single incorrect relationship in the linkage will cause Cognos or any other BI tool to assume the data model is custom and non-conforming to known designs. Consequently, most of the automated features of the BI tools will be disabled and the tools will assume you intend to write 100% of your code manually, which is just as intensive as creating your own BI tool from scratch. The third image shows a functioning Cognos dashboard with automated charting and drill up and down functions active. If the data model is incorrect, most of the functionality simply disappears with no explanation.

The biggest problem with data is the corporate culture that mandates the following practices. "Tell me what I want to hear even if it's a lie or you're fired!" Most people are not able to tell the truth about where they work because of a lack of maturity in management. In life, 99% of what happens will not be what one emotionally wants to happen, but few managers can take the emotion out of management, face reality immediately, and make a logical decision based on actual facts. Maturity often separates good managers and bad managers from great managers. Again, most managers employ people who tell them what they want to hear, and usually, that means staffing the data modeling position with code developers who also tell managers what they want to hear even if it's a lie.

In 29 years of IT experience, I have never met a single code developer, who only develops code, who even knew what is multi-dimensional data modeling. Because most companies blindly cut costs without actually knowing what they are cutting, less than one percent of corporate and government databases have the correct format for compatibility with anything, including Cognos, most third-party software, nor object-oriented programming languages, written in actual object mode, such as Java and C++. Cutting data modeling costs is like owning a trucking and shipping company and cutting 100% of the fuel budget. The trucks will still look great, one will still be able to load them, just as one can load a bad database, but they will not perform.

Most code developers do not understand objects and object code but pass off legacy code, often written in an object-capable language, as enterprise-level code. However, it is simply weak legacy code using a language that supports legacy code modes for reverse compatibility for software written in the 1970's.

I have seen companies waste over thirty million dollars to over one billion dollars on projects run without a data modeler designed and approved database design at the beginning of the project. A contract data modeler could design or fix most databases in eight to twelve weeks for less than $800,000, then generate the base source code for developers to use, which also ensures object-compatible coding standards are enforced. Why does one need object-compatible code? That's the only type of code that is compatible with BI tools, multi-dimensional databases, application servers, and most third-party enterprise-level software. Most of all, that is the only kind of code that can scale with high availability for millions of concurrent end-users connected through a private network or the Internet. If your developers say, "We don't write object code," you need to retrain or fire all of them. You're not going to make it in the modern age without object-oriented code.

Again, most software is connected to databases that do not conform to any known standards. Since the databases do not conform to standards, the software does not know what to do. How could it know? Subsequently, users start to fail over to stop-gap measures such as connecting Microsoft Excel to Cognos, manual queries, and endless rigging. If all databases conformed to known standards, most software would be automatically interoperable using a canonical data model to translate data between applications. On a side note, the lack of interoperability in health care applications is literally killing patients worldwide.

One cannot make a database any way one wants with no conformance to standard strategies, and then expect software written by a third party to understand what one did.

That's why it's not working.

#ibmcognos #businessintelligence #datamodeling #datamodeler #baddata #bigdata #data #java #c++