Big Data Mining and Analysis

Big data mining and analysis Data mining is a technique that analyzes each data and finds its rules from a large amount of data, which mainly consists of three steps: data preparation, pattern search, and pattern representation. The tasks of data mining include association analysis, cluster analysis, classification analysis, anomaly analysis, specific group analysis, and evolution analysis. Data mining is a technique that analyzes each data and finds its rules from a large amount of data, which mainly consists of three steps: data preparation, pattern finding and pattern representation. Data preparation is the selection of required data from relevant data sources and integration into datasets for data mining. Pattern search is to find out the rules contained in the dataset in a certain way; Pattern representation is the representation of the identified pattern in a way that is understandable to the user as much as possible, such as a visualization. The complete steps of data mining are as follows: (1) Understand the data and the source of the data. (2) Acquisition of relevant knowledge and technology. (3) Integration and checking. (4) Removal of erroneous or inconsistent data (data cleaning). (5) Model and hypothesis development. (6) Actual data mining. (7) Testing and verification. (8) Interpretation and use. A common method used in data mining (1) Classification. Classification is to find out the common characteristics of a group of data objects in the database and divide them into different classes according to the classification pattern, and its purpose is to map the data items in the database to a given category through the classification model. It can be applied to application classification and trend prediction, such as Taobao stores divide users' purchases into different categories over a period of time, and recommend related products to users according to the situation, so as to increase the sales volume of stores. (2) Regression analysis. Regression analysis reflects the characteristics of the attribute values of the data in the database, and finds the dependencies between the attribute values by expressing the relationship between the data mappings through functions. It can be applied to the study of prediction and correlation of data series. In marketing, regression analysis can be applied to various aspects. For example, through the regression analysis of the current quarter's sales, the sales trend of the next quarter can be predicted and targeted marketing changes can be made. (3) Clustering. Clustering is similar to classification, but unlike classification, the purpose of classification is to divide a set of data into several categories based on the similarity and difference of the data. The similarity between data belonging to the same category is large, but the similarity between data between different categories is small, and the correlation between data across classes is low. (4) Association rules. Association rules are associations or interrelationships hidden between data items, that is, the occurrence of other data items can be deduced from the appearance of one data item. The mining process of association rules mainly includes two stages: the first stage is to find out all high-frequency project groups from massive raw data; The second extreme is the generation of association rules from these high-frequency project groups. Correlation rule mining technology has been widely used in the financial industry to predict customer needs, and banks are improving their marketing by bundling information that customers may be interested in and obtaining from users at their ATMs. (5) Neural network methods. As an advanced artificial intelligence technology, neural network is very suitable for dealing with nonlinear and those processing problems characterized by fuzzy, incomplete and imprecise knowledge or data due to its self-processing, distributed storage and high fault tolerance, and it is very suitable for solving data mining problems. Typical neural network models are mainly divided into three categories: the first type is a feedforward neural network model for classification prediction and pattern recognition, which is mainly represented by functional networks and perceptrons; The second type is feedback neural network models for associative memory and optimization algorithms, represented by Hopfield's discrete and continuous models. The third category is the self-organizing mapping method for clustering, represented by the ART model. Although there are many models and algorithms for neural networks, there are no uniform rules for which models and algorithms to use in data mining in a specific domain, and it is difficult for people to understand the learning and decision-making process of the network. (6) Web Data Mining. Web data mining is a comprehensive technology that refers to the discovery of implicit patterns P by the Web from the document structure and the set C used, if C is regarded as an input and P as an output, then the Web mining process can be regarded as a mapping process from input to output. At present, more and more web data is in the form of data streams, so it is of great significance for web data stream mining. At present, the commonly used web data mining algorithms are: PageRank algorithm, HITS algorithm and LOGSOM algorithm. The users mentioned in these three algorithms are general users and do not distinguish between individual users. At present, Web data mining is faced with some problems, including: the problem of user classification, the timeliness of website content, the time of users on the page, and the number of links in and out of the page. In today's rapid development of Web technology, these problems are still worth studying and solving.