Business intelligence systems combine operational data with analytical tools to present complex and competitive information to planners and decision makers. The objective is to improve the timeliness and quality of inputs to the decision process. Business Intelligence is used to understand the capabilities available in the firm; the state of the art, trends, and future directions in the markets, the technologies, and the regulatory environment in which the firm competes; and the actions of competitors and the implications of these actions. By Solomon Negash – Computer Science and Information Systems Department – Kennesaw State University.
Demand for Business Intelligence (BI) applications continues to grow even at a time when demand for most information technology (IT) products is soft [Soejarto, 2003; Whiting, 2003]. Yet, information systems (IS) research in this field is, to put it charitably, sparse. While the term Business Intelligence is relatively new, computer-based business intelligence systems appeared, in one guise or other, close to forty years ago. BI as a term replaced decision support, executive information systems, and management information systems [Thomsen, 2003]. With each new iteration, capabilities increased as enterprises grew ever-more sophisticated in their computational and analytical needs and as computer hardware and software matured. In this post, BI systems are defined as follows:
- BI systems combine data gathering, data storage, and knowledge management with analytical tools to present complex internal and competitive information to planners and decision makers.
Implicit in this definition is the idea (perhaps the ideal) that business intelligence systems provide actionable information delivered at the right time, at the right location, and in the right form to assist decision makers. The objective is to improve the timeliness and quality of inputs to the decision process, hence facilitating managerial work.
Sometimes business intelligence refers to on-line decision making, that is, instant response. Most of the time, it refers to shrinking the time frame so that the intelligence is still useful to the decision maker when the decision time comes. In all cases, use of business intelligence is viewed as being proactive. Essential components of proactive BI are [Langseth and Vivatrat, 2003]:
- real-time data warehousing,
- data mining,
- automated anomaly and exception detection,
- proactive alerting with automatic recipient determination,
- seamless follow-through workflow,
- automatic learning and refinement,
- geographic information systems (Appendix I)
- data visualization (Appendix II)
Figure 1 shows the variety of information inputs available to provide the intelligence needed in decision making.
where OLAP = On-Line Analytic Processing, DW=Data Warehouse, DM=Data Mining, EIS =
Executive Information Systems, and ERP = Enterprise Requirement Planning.
WHAT DOES BI DO?
BI assists in strategic and operational decision making. A Gartner survey ranked the strategic use of BI in the following order [Willen, 2002]:
- Corporate performance management
- Optimizing customer relations, monitoring business activity, and traditional decision support
- Packaged standalone BI applications for specific operations or strategies
- Management reporting of business intelligence
One implication of this ranking is that merely reporting the performance of a firm and its competitors, which is the strength of many existing software packages, is not enough. A second implication is that too many firms still view business intelligence (like DSS and EIS before it) as an inward looking function.
Business intelligence is a natural outgrowth of a series of previous systems designed to support decision making. The emergence of the data warehouse as a repository, the advances in data cleansing that lead to a single truth, the greater capabilities of hardware and software, and the boom of Internet technologies that provided the prevalent user interface all combine to create a richer business intelligence environment than was available previously. BI pulls information from many other systems. Figure 2 depicts some of the information systems that are used by BI.
where: OLAP = on-line data processing, CRM=customer relationship management,
DSS= decision support systems, GIS = geographic information systems.
BI converts data into useful information and, through human analysis, into knowledge. Some of the tasks performed by BI are:
- Creating forecasts based on historical data, past and current performance, and estimates of the direction in which the future will go.
- “What if” analysis of the impacts of changes and alternative scenarios.
- Ad hoc access to the data to answer specific, non-routine questions.
- Strategic insight (e.g., item 3 in Appendix III)
II. A DATA FRAMEWORK FOR BI
STRUCTURED VS. SEMI-STRUCTURED DATA
BI requires analysts to deal with both structured and semi-structured data [Rudin and Cressy, 2003; Moss, 2003]. The term semi-structured data is used for all data that does not fit neatly into relational or flat files, which is called structured data. We use the term semi-structured (rather than the more common unstructured) to recognize that most data has some structure to it. For example, e-mail is divided into messages and messages are accumulated into file folders.
A survey indicated that 60% of CIOs and CTOs consider semi-structured data as critical for improving operations and creating new business opportunities [Blumberg and Atre, 2003b].
“We have between 50,000 and 100,000 conversations with our customers daily, and I don’t know what was discussed. I can see only the end point – for example, they changed their calling plan. I’m blind to the content of the conversations.” Executive at Fortune 500 telecommunciations provider [Blumberg and Atre, 2003b].
Semi-structured data is not easily searched using existing tools for conventional data bases [Blumberg and Atre, 2003a]. Yet, analysis and decision making involves using a variety of semi-structured data such as is shown in Table 1.
Gartner group estimates that 30-40% of white-collar workers time is being spent on managing semi-structured data in 2003, up from 20% in 1997 [Blumberg and Atre, 2003b]. Merrill Lynch, for example, estimates that more than 85% of all business information exists as semi-structured data [Blumberg and Atre, 2003b]. Furthermore, roughly 15% of the structured data are commonly captured in spreadsheets, which are not included in structured data base
architectures.[Blumberg and Atre, 2003b].
While data warehouses, ERP, CRM, and databases mostly deal with structured data from data bases, the voluminous semi-structured data within organizations is left behind. Blumberg and Atre [2003b] posit that managing semi-structured data persists as one of the major unsolved problems in the IT industry despite the extensive vendor efforts to create increasingly sophisticated document management software.
Figure 3 shows a framework that integrates the structured and semi-structured data required for Business Intelligence.
One implication of the BI framework is that semi-structured data are equally important, if not more, as structured data for taking action by planners and decision makers. A second implication is that the process of acquisition, cleanup, and integration applies for both structured and semi-structured data.
To create business intelligence information, the integrated data are searched, analyzed, and delivered to the decision maker. In the case of structured data, analysts use Enterprise Resource Planning (ERP) systems, extract-transform-load (ETL) tools, data warehouses (DW), data-mining tools, and on-line analytical processing tools (OLAP). But a different and less sophisticated set of analytic tools is currently required to deal with semi-structured data.
DATA TYPE/SOURCE MATRIX
Structured and semi-structured data types can be further segmented by looking at the internal and external data sources of the organization. These two dimensions – data type and data source – are illustrated in Figure 4.
The transition between structured and semi-structured data types and between internal and external data sources is not defined sharply. For example, semi-structured data from e-mail and Web sites deal with both internal and external data sources— intranets and extranets for Web sites. Nevertheless, this matrix is useful to guide research and to view the available analytic tools for BI. For example, ERP systems capture operational (internal) data in a structured format, whereas, CRM focuses on customer (external) information. On the other hand, semi-structured data is captured in business processes and news items, among other documents. For the purpose of this paper, business processes and news items are used to represent internal and external data sources, respectively.
III. DATA SOURCES AND ARCHITECTURE
BI FOR THE MASSES
Established analytic practice for BI typically involves a solitary user exploring data in what is usually a one-off experience [Russom, 2003]. Specialists performing analyses in a staff position for senior management can, and often do, create a sub-optimized BI solution. Because decisions are made at many organizational levels, not just the executive level, a new class of analytic tools is emerging that serves a much broader population within the firm. These new tools are referred to as “BI for the masses”. BI for the masses is about providing reporting and analysis capability at all levels of the organization. For example, firms are rolling out tools such as data mining designed for use by non-specialists [McNight 2003].
The challenges of accomplishing BI for the masses are:
- easy creation and consumption of reports,
- secure delivery of the information, and
- friendly user interface, such as Internet browsers
Deployment of BI tools to many staff members indicates that organizations are ready to expand BI to all levels. For example, BusinessObjects deployed its BI tools to 70,000 users at France Telecom, 50,000 users at US Military Health System, and to several other firms at the 20,000 user level range [Schauer, 2003].
DATA VOLUME CONSIDERATIONS
By the end of 2001, the public Internet was the source of fully half the information used by workers – in excess of 3 billion documents, 80% of which is semi-structured data [Blumberg and Atre, 2003a]. Google.com estimates the Net is doubling in size every eight months. IDC, a marketing research firm, reported that 31 billion e-mail messages were sent worldwide during 2002, with a prediction to double by 2006, exceeding 60 billion messages [Blumberg and Atre, 2003a].
More than 2 billion new Web pages were created since 1995, with an additional 200 million new pages being added every month [IDC, as reported in Blumberg and Atre, 2003b]. BI analysts who fail to integrate semi-structured data do so at their own peril. The sheer volume of semi-structured data is daunting, “The only thing worse than having too little data is having too much of it” [Darrow, 2003].
Since it must deal with both structured and semi-structured data simultaneously, BI’s data architecture is business rather than technically oriented. While technical data architectures focus on hardware, middleware, and DBMSs, BI data architecture focuses on standards, metadata, business rules, and policies [Moss, 2003]. An example of structured and unstructured metadata is shown in Table 2.
ARCHITECTURE FOR STRUCTURED DATA
Typical BI architecture for structured data centers on a data warehouse. The data are extracted from operational systems and distributed using Internet browser technologies (Figure 5). The specific data needed for BI are downloaded to a data mart used by planners and executives. Outputs are acquired from routine push of data from the data mart and from response to inquiries from Web users and OLAP analysts. The outputs can take several forms including exception reports, routine reports, and responses to specific request. The outputs are sent whenever parameters are outside pre-specified bounds.
ARCHITECTURE FOR SEMI-STRUCTURED DATA
BI architecture for semi-structured data (Figure 6) includes business function model, business process model, business data model, application inventory, and meta data repository [Moss, 2003].
IV. RETURN ON INVESTMENT
BI projects are not exempt from the increasing pressure in firms to justify return on IT investments. Surveys show that Return on Investment (ROI) for BI installations can be substantial. An IDC study on the financial impact of business analytics, using 43 North American and European organizations indicated a median five-year ROI of 112% from an investment of $2 million [Morris, 2003]. Return ranged from 17% to 2000% with an average ROI of 457%.
However, BI budget and ROI were not found to be correlated. [Morris, 2003; Darrow, 2003]. The challenge comes in trying to assess ROI prior to installation. Computing anticipated return on investment for business intelligence is a difficult problem. Like most information systems, BI up-front costs are high as is upkeep. Unfortunately, although reductions in information systems costs from efficiencies3 can be forecasted, the efficiency savings are only a small portion of the payoff (Appendix III). It would be rare for a BI system to pay for itself strictly through cost reductions.
Most firms today do use some form of business intelligence, although only a few operate complete BI systems. To simplify the cost discussion, consider a firm starting from scratch. Putting a BI system in place includes:
- Hardware costs. These costs depend on what is already installed. If a data warehouse is in use, then the principal hardware needed is a data mart specifically for BI and, perhaps, an upgrade for the data warehouse. However, other hardware may be required such as an intranet (and extranet) to transmit data to the user community.
- Software costs. Typical BI packages can cost $60,000. Subscriptions to various data services also need to be taken into account. For example, firms in the retail industry buy scanner data to ascertain how demand for their products and competing products responds to special offers, new introductions, and other day-to-day changes in the marketplace (Appendix IV).
- Implementation costs. Once the hardware and software are acquired, a large one-time expense is implementation, including initial training. Training is also an ongoing cost as new people are brought in to use the system and as the system is upgraded. In addition, annual software maintenance contracts typically run 15% of the purchase costs.
- Personnel costs. Personnel costs for people assigned to perform BI and for IT support personnel, need to be fully considered to take into account salary and overhead, space, computing equipment, and other infrastructure for individuals. A sophisticated cost analysis also takes into account the time spent reading BI output and the time spent searching the Internet and other sources for BI4 .
Most BI benefits are intangible before the fact. An empirical study for 50 Finnish companies found most companies do not consider cost or time savings as primary benefit when investing in BI systems [Hannula and Pirttimaki, 2003]. The hope is that a good BI system will lead to a big bang return at some time in the future. However, it is not possible to forecast big bangs because they are serendipitous and infrequent.
V. COMPETITIVE ANALYSIS
“Next to knowing all about your own business, the best thing to know about is the other fellow’s business.” John D. Rockefeller[Amazon, 2003].
Competitive intelligence (CI) is a specialized branch of Business Intelligence. It is “no more sinister than keeping your eye on the other guy albeit secretly” [Imhoff, 2003]. The Society of Competitive Intelligence Professionals (SCIP) defines CI as follows [SCIP, 2003]:
Competitive Intelligence is a systematic and ethical program for gathering, analyzing and managing external information that can affect your company’s plans, decisions and operations.
In other words, CI is the process of ensuring your competitiveness in the marketplace through a greater understanding of your competitors and the overall competitive environment. “You can use whatever you find in the public domain to ensure that you will not be surprised by your competitors.” [Imhoff, 2003].
CI is not as difficult as it sounds. Much of what is obtained comes from sources available to everyone, including [Imhoff, 2003]:
- government websites and reports
- online databases, interviews or surveys,
- special interest groups (such as academics, trade associations, and consumer groups),
- private sector sources (such as competitors, suppliers, distributors, customer) or
- media (journals, wire services, newspapers, and financial reports).
The challenge with CI is not the lack of information, but the ability to differentiate useful CI from chatter or even disinformation. Of course, once a firm starts practicing competitive intelligence, the next stage is to introduce countermeasures to protect itself from the CI of competitor firms. The game of measure, countermeasure, counter-countermeasure, and so on to counter to the nth measure is played in industry just as it is in politics and in international competition. Appendix IV presents examples of competitive analysis.
VI. MARKETS, CUSTOMERS AND VENDORS
The size of the business intelligence market can be seen from the published forecasts. For example, AMR research estimates the current BI market at $6 billion with projection to reach $12 billion by 2006 [Darrow, 2003].
Customers are asking their BI vendors for “the ability to support improved operational decision making – not just deliver analytic and reporting bells and whistles” [Kestelyn, 2003]. The business of analytic vendors is to get customers out of the business of searching for data and move them into doing more analysis [Blumberg and Atre, 2003a].
The traditional custom design, build, and integrate model for BI systems is lengthy (at least six months) and costly ($2-3 million) [Rudin and Cressy, 2003]. Therefore, many firms opt for pre-built analytic applications to achieve lower total cost of ownership, quicker implementation, rapid return on investment, while still obtaining underlying structure for performance, scalability, and flexibility [Rudin and Cressy, 2003].
Gartner research found the number of firms that plan to manage their BI integration internally dropping from 49% in 2001 to 37% in 2002 [Soejarto, 2003]. A study by IDC on OLAP investment over 5-years indicated a $2.1 million investment in building OLAP solutions in-house resulted in 104% ROI [Morris, 2003]. The same study indicated a $1.8 million investment in buying pre-built OLAP solutions resulted in 140% ROI, implying that BI solutions cost less and bring higher ROI [Morris, 2003].
Intelligent Enterprise’s study for the 2003 editor’s choice award for BI identified 12 firms to watch in 2003 including Adaytum, Brio Software, Cognos, Crystal Decisions, E.Intelligence, Fair Issac & Co., Hyperion Solutions, Information Builders, MicroStrategy, ProClarity, Siebel Systems, and Spotfire [Kestelyn, 2003]. The study also named 12 vendors as most influential in the overall category including Teradata, SAS, IBM, OutlookSoft, Business Objects, Microsoft, Manhattan
Associates, PeopleSoft, Oracle, Ilog, Insight Software, and Open Source/Linux [Stodder, 2003].
VII. MANAGERIAL QUESTIONS
1. Is business intelligence an oxymoron? A shorthand for cloak and dagger spying on competitors and government? An important, legitimate activity?
Despite its name, business intelligence is about trying to understand your own position, your customers, and your competitors. While it is neither ethical nor legal to spy on competitors, competitive intelligence is an important part of a firm’s planning and operational decision making to understand its competition.
2. What is new about today’s business intelligence compared to previous systems?
Business intelligence is a natural outgrowth of a series of previous systems designed to support decision making. The emergence of the data warehouse as a repository, the advances in data cleansing that lead to a single truth, the greater capabilities of hardware and software, and the boom of Internet technologies that provided the prevalent user interface all combine to create a richer business intelligence environment than was available previously.
3. What types of business intelligence are there?
Business Intelligence is used to understand the capabilities available in the firm: the state of the art, trends, and future directions in the markets, the technologies, and the regulatory environment in which the firm competes; and the actions of competitors and the implications of these actions.
4. What will you be able to do if you invest in BI?
Business Intelligence systems present complex corporate and competitive information to planners and decision makers. The objective is to improve the timeliness and quality of the input to the decision process.
5. Who uses BI?
Business intelligence is used by decision makers throughout the firm. At senior managerial levels, it is the input to strategic and tactical decisions. At lower managerial levels, it helps individuals to do their day-to-day job. In some firms, business intelligence capabilities are rolled out to most of its professionals (i.e., ‘BI for the masses’).
6. How do you gather and transfer BI?
Business intelligence is a form of knowledge. The techniques used in knowledge management for generating and transferring knowledge [Davenport and Prusak, 1998] apply. Some knowledge is bought (e.g., scanner data in the food industry) while other knowledge is created by analysis of internal and public data. Knowledge transfer often involves disseminating intelligence information to many people in the firm. For example, salespeople need to know market conditions, competitor offerings, and special offerings.
7. Do you need a separate organization for it?
Most medium and large firms assign people, often full time, to plan and to monitor competitor action. These people are the ones who form the core groups for business intelligence initiatives. Whether they are centralized or scattered through strategic business units (SBUs) is a matter of organizational style.
8. What technologies are available?
Most of the technologies needed for business intelligence serve multiple purposes. For example, the World Wide Web is used for both knowledge generation and knowledge transfer. However, specialized software for doing analysis is the heart of business intelligence. This software is an outgrowth of the software used for decision support and executive information systems in the past.
The term Business Intelligence may turn out to be a fad. However, the underlying concepts, using information technology to deliver actionable information for decision makers, are essential for managing today’s global businesses. BI uses both structured and semi-structured data. The former is much easier to search but the latter contains the information needed for analysis and decision making.
For structured data, many BI tools exist for acquisition, integration, cleanup, search, analysis, and delivery. Further work is needed, however, to integrate these tools and to provide actionable information. BI tools for semi-structured data, on the other hand, are not yet mature. However, significant work is being done in industry to deal with semi-structured data (Kontzer, 2004; MacIntyer, 2004). Current trends and topics about unstructured information management can be
found at http://www.unstruct.org, among others.
This paper develops a BI framework and identifies potential research areas. The BI framework highlights the importance of semi-structured data to support informed action by decision makers. In addition this paper explored a matrix for BI data types (structured vs. semi-structured) and data sources (internal vs. external) to guide research.
The development of analytical tools to integrate structured and semi-structured data can benefit from attention by researchers. The BI market is growing, and the proportion of semi-structured data used in daily decisions is growing. Exploring the underlying issues and the development of information technology that provide intelligence to business therefore is a fertile area for research.
APPENDIX I. A TECHNOLOGY FOR BUSINESS INTELLIGENCE: GEOGRAPHIC INFORMATION SYSTEMS (GIS)
In the narrow sense, a geographic information system (GIS) is a software package that links databases and electronic maps. At a more general level, the term GIS refers to the ability to analyze spatial phenomena. These systems are an important business intelligence tool for exploiting and presenting the increasing amount of two (and more) dimensional data available in a form that can be understood by analysts and managers.
In addition to collecting, storing, and retrieving spatial location data, a GIS is used to identify locations which meet specified criteria (e.g., for new store location), exploring relations among data sets, assessing alternatives and aiding in decision making, and displaying selected environments both visually and numerically. In practice, a GIS consists of a series of layers, each presenting a particular two-dimensional feature, which can be superimposed accurately on top of one another. Some examples:
- a marketing group overlays customer locations, school locations, distribution centers, and existing retailers selling their own and/or their competitors products.
- A telecommunications company selects the number and location of switching centers and routes in a communication network. The system displays such quantities as traffic, costs, and transmission times. Users can redefine the network on the screen, can create multiple views, see the effect of ‘what if’ changes and new data because the system re-computes for each change, take constraints into account, and shows where the proposed solution fails to meet criteria.
APPENDIX II. A TECHNOLOGY FOR BUSINESS INTELLIGENCE: VISUALIZATION
With the flood of data available from information systems, business intelligence analysts and decision-makers need to make sense out of the knowledge it contains. Visualization is the process of representing data with graphical images. Unlike geographic information systems which typically deal with physical spaces, the underlying data could, for example, represent abstract objects, such as profit, sales, or cost. If the data is abstract, then a visual analog must be created.
Visual analogs today go far beyond the pie chart and the bar chart [Tegarten, 1999]. Visualization is used to create advanced dashboard in which large amounts of information are presented on a single screen. Today’s results are far advanced over early dashboards such as those shown by Houdeschel and Watson in 1987 [Houdeshel and Watson, 1987].
Visualization exploits the human visual system to extract information from data and provides an overview of complex data sets. It identifies structure, patterns, trends, anomalies, and relationships in data to assists in identifying the areas of “interest”. That is, visualization allows BI analysts to use their natural spatial/visual abilities to determine where further exploration should be done and where action is required.
Visualization technologies are deployed in finance, litigation, marketing, manufacturing, training, and organizational modeling.
APPENDIX III. EXAMPLES OF BI APPLICATIONS
The following are some examples of BI applications:
- A company that provides natural gas to homes created a dashboard that supports operational performance metric management and allows real time decision making. In one application of the dashboard, the number of repeat repair calls was reduced, resulting in a saving of $1.3 million
- At a large member-owned distributor to hardware stores, use of a dashboard reduced the amount of inventory that must be liquidated or sold as a loss leader from $60 million to $10 million. Their BI system also allowed their member stores to see their own performance relative to similar stores.
- The Dallas Teachers Credit Union wanted to expand. They asked two questions of the BI system: (1) Who are our most profitable customers? 2. How far will they drive to reach our outlets? Once they found the answer to these two questions, they were able to select branch sites that were within the buying radius of the preferred customers.
APPENDIX IV. EXAMPLES OF COMPETITIVE ANALYSIS
The following are examples of competitive analyses:
- Texas Instruments made a $100 million acquisition based on their analysis of a competitors potential bid, [Lavelle, 2001].
- Merck & Company developed a counter strategy to its competitor’s forth- coming product based on competitive intelligence reports, and generated a savings of $200 million, [Imhoff, 2003].
- Illuminet, a company that delivers advanced network, database, and billing services, stayed a step ahead by using a vendor (QL2 Software) to retrieve information posted on their competitors web sites [Moores, 2002].
- Most grocery chains sell their bar code scanner data to organizations such as Information Resources Inc. (IRI) who, in turn collates the data and sells it to grocery wholesalers and vendors. Individual firms want to find out how well their (and their competitors) special offers, such as a 20 cent off coupon, worked in the marketplace. By examining how well the offer worked previously, how well it worked in the current situation, and forecasting the future effects of the promotion, a firm can decide whether to continue the offer or change it. If it is a competitor’s offer, the forecast is used to decide whether to match or exceed the competitor. Thus, the forecasts based on the data are converted into policy at the tactical level.