Data warehouses allow us to store large amount of data and then data mine against it. Data mining helps provide insight towards customer behavior or business strategy by exploring patterns that are relevant to business success. These are all part of business intelligence that allows data to be transformed into actionable insights.
The first use of the term dates back to the 1980s, and during the 1990s data warehousing emerged as its own research area. Today’s data warehouses are at the very core of modern decision making. Data warehouses store data from different sources making it available in a multidimensional form or aggregate form for creating knowledge to drive decision making. It can also be used to analyze patterns and trends. Data warehouses store detailed historical data for use in decision support systems. At a high level, data warehousing systems are designed to support OLAP which helps with data analysis and visualization. Data warehouses receive much of their value through data mining techniques that produce knowledge.
Data mining can occur on many different data sets including the data warehouse. At a simple level, data mining is just applying mathematical techniques to extract patterns and trends to provide information. With data sets growing exponentially over the last few years, there is more demand to turn raw data into knowledge. In artificial intelligence and machine learning, data mining is a rapidly growing field.
There is a general process for turning raw data into knowledge. First, the raw data must be selected and then pre-processing can occur to detect outliers and other trends. In some cases, algorithms do not work well with outliers, and they may need to be removed, or they may indicate a data quality issue. Also, missing values can be detected. Next, the data is transformed. This also helps normalize the data. Correlated variables can be better understood at this point. If the variables are uncorrelated, more valuable information may be provided. Finally, the actual data mining occurs from this transformed data. Algorithms can be applied to find trends and patterns, like the decision tree application. Then when the data is accurately divided into patterns, then the user looks at the patterns to apply an interpretation.
There are many data mining functionalities that include but are not limited to classification and prediction, cluster analysis and trend and evolution analysis. In data mining, it is essential to be able to identify which of the patterns is most useful to achieve the goal. Data mining is about information discovery which provides support for better decision making. Generally, patterns are attractive if they are understood and validate a hypothesis that the user wants to confirm. Potential applications for data mining include targeting marketing, cross-selling, market segmentation, risk analysis, competitive analysis and text mining.
There are some common issues in data mining. There can be challenges in the methodology, in the handing noise and incomplete data, as well as the integration of discovered knowledge with existing data. Also, the user interactions can be a challenge with ad-hoc mining, visualization of data mining results and interactive minding of knowledge at different levels of abstraction.
Business intelligence can help organizations improve efficiency. There is no good in storing data if you cannot turn it into actionable information. There are many analytical support tools today for data mining that can help you identify key trends and patterns in your organization’s data to help you move your business forward. We live in the age of a the digital revolution where information is power.
#DigitalRevolution #DataWarehouse #DataMining #BusinessIntelligence