Data Scientists are in High Demand

Data scientists are in high demand From a technical point of view, the decline in the price of hard disks and the emergence of technologies such as NoSQL databases have made it possible to store large amounts of data in a cheap and efficient way compared to the past. In addition, the advent of distributed processing technologies like Hadoop that can work on general-purpose servers has made statistical processing of large amounts of unstructured data faster and cheaper than ever before. However, no matter how perfect the tool is, it is impossible to make data valuable on its own. Next, we need people who can use these tools to find gold mines in mountains of data and communicate the value of that data to decision-makers in an understandable format that can ultimately be realized in the business. The talent with these skills is the "data scientist" who is currently hard to find in the United States, where the wave of big data is in full swing. The focus on data scientists stems from the growing recognition that there is such a group of professionals behind the success of companies such as Google, Amazon, and Facebook. These web companies don't just store large amounts of data, they turn it into a valuable gold mine – for example, search results, targeted advertising, accurate product recommendations, lists of friends you might know, and so on. Data science is a term that has been around for a long time, but data scientist is a new word that popped up a few years ago. There are many theories about the origin of the term, but in the book "Beautiful Data" (Toby Segaran, Jeff Hammerbacher, published by O'Reilly), there is a description of Facebook's data scientists. "At Facebook, we've found that traditional titles like business analyst, statistician, engineer, and research scientist don't exactly define our team's role. The job in this role is varied: on any given day, a member of the team can implement a multi-stage processing pipeline flow in Python, design hypothesis testing, perform regression testing on data samples with tool R, design and implement algorithms for data-intensive products or services on Hadoop, or present the results of our analysis to the rest of the organization in a clear and concise manner. In order to master the skills needed to accomplish this multifaceted task, we created the role of 'data scientist'. ” Just a few years ago, data scientist was not a formal profession, but in the blink of an eye, the profession has been hailed as "the most important talent in the IT industry in the next decade". Mr. Hal Varian (1947~), chief economist at Google and professor at the University of California, Berkeley, spoke the following in a conversation with McKinsey Director James Manyika in October 2008 (the Chinese version is excerpted from the official Chinese version of McKinsey Quarterly). "I always say that in the next 10 years, the most interesting job will be that of a statistician. People think I'm joking. But who would have thought in the past that computer engineering would become the most interesting job in the 90s of the last century? In the next 10 years, the ability to access data – to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it – will be an extremely important skill, not only at the professional level, but also at the educational level, including for school, high school and university students. Since we truly have essentially free and ubiquitous data today, a complementary scarcity is the ability to understand and extract value from that data. ” Professor Van Rien used the term "statisticians" in the original conversation, and although he didn't use the term "data scientist" at the time, he was referring to the data scientist we are talking about today. Skills required of a data scientist There is no fixed definition of the profession of data scientist, but it generally refers to such talents. "A data scientist is a person who uses statistical analysis, machine learning, distributed processing and other technologies to extract business-meaningful information from a large amount of data, convey it to decision-makers in an easy-to-understand form, and create new data application services." The skills required for a data scientist are as follows. (1) Computer Science Generally speaking, most data scientists are required to have a professional background in programming and computer science. To put it simply, it is the skills related to large-scale parallel processing technologies such as Hadoop and Mahout, which are necessary for processing big data, and machine learning. (2) Mathematics, statistics, data mining, etc In addition to mathematical and statistical literacy, it is also necessary to have the skills to use mainstream statistical analysis software such as SPSS and SAS. Among them, the open-source programming language for statistical analysis and its runtime environment "R" have recently attracted attention. R's strength lies not only in the rich library of statistical analysis, but also in the ability to generate high-quality graphs that can be visualized and run with simple commands. In addition, it has a package extension mechanism called CRAN (The Comprehensive R Archive Network), which allows you to import an extension package to use functions and datasets that are not supported in the standard state. (3) Data Visualization The quality of the information depends largely on the way it is expressed. One of the most important skills for a data scientist is to analyze the meaning contained in the data composed of numbers, develop web prototypes, and use external APIs to unify other services such as charts, maps, and dashboards to visualize the results of the analysis. For reference, here are excerpts from Facebook and Twitter's data scientist job postings. It should be possible to provide you with some more practical experience of what kind of skills are needed in real enterprises. Facebook job postings for data scientists Facebook plans to hire data scientists for the data science team. Applicants for this position will work as software engineers and quantitative researchers. The ideal candidate should have a strong interest in the study of online social networks, be able to identify the issues that arise in creating the best products, and have a passion for solving them. Job description ? Identify important product issues and work closely with the product engineering team to find solutions ? Solve problems by applying appropriate statistical techniques to data ? Communicate conclusions to product managers and engineers ? Promote the collection of new data and the improvement of existing data sources ? Analyze and interpret the experimental results of the product ? Find best practices for measurement, experimentation, and communicate them to the product engineering team prerequisite ? Master's or Ph.D. degree in related technical field, or at least 4 years of relevant work experience ? Extensive experience in solving analytical issues using quantitative methods ? Ability to easily manipulate and analyze complex and large amounts of multidimensional data from all sides ? Passionate about empirical research and solving data-related challenges ? Flexible analysis methods can be used for results in various accuracy levels ? Ability to communicate complex quantitative analyses in a practical, accurate, and feasible manner ? Proficiency in at least one scripting language, such as Python, PHP, etc ? Proficient in relational database and SQL ? Expertise in R, MATLAB, SAS and other analytical tools ? Experience in working with large datasets, as well as experience in using distributed computing tools such as MapReduce, Hadoop, Hive, etc Source: Facebook (Chinese translation by translator) Twitter's job postings for data scientists who are responsible for increasing the number of users About business content Twitter plans to recruit people who can provide information and direction to increase the number of Twitter users, and have the ability to act and be highly skilled. Candidates should have a professional background in statistics and modeling, as well as extensive experience in large-scale dataset processing. We expect the judgment of candidates to determine the direction of the Twitter product group on multiple levels. Duty ? Use Hadoop and Pig to write data analysis in MapReduce format ? Ability to write complex SQL queries for ad-hoc data mining processes and standard data mining processes ? Able to write code using SQL, Pig, scripting language, statistical software package ? Summarize and report the results of the analysis, both orally and in writing ? Process large-scale structured and unstructured data with terabytes and more than 1 billion transactions every day prerequisite ? Master's degree in Computer Science, Mathematics, Statistics or equivalent experience ? More than 2 years of experience in data analysis ? Experience with large-scale datasets and MapReduce architectures such as Hadoop ? Experience in scripting languages and regular expressions ? Interest in discrete mathematics, statistics, probability ? Experience in mapping business requirements to engineering systems Source: Twitter (Chinese translation by translator) Qualities required of a data scientist This section overlaps with the skills section, and the qualities required of a data scientist are as follows. (1) Communication skills Even if useful information is obtained from big data, its value will be greatly diminished if it cannot be realized in the business. To this end, it is important to have the ability to effectively communicate the results of data analysis to employees of business units and management who lack knowledge of data analysis. (2) Entrepreneurship (entrepreneuership) The entrepreneurial spirit of creating new services centered on data that does not yet exist in the world is also an important quality necessary for data scientists. Companies such as Google, Amazon, and Facebook, which have spawned new types of services through data, have succeeded by painstakingly exploring what kind of services can be created by huge amounts of data. (3) Curiosity It takes a lot of curiosity to find out what lies behind the huge amount of data. In addition to this, successful data scientists all have one thing in common: they are not limited to specific fields such as art, technology, medicine, natural sciences, etc., but have a strong curiosity about each field. By integrating and analyzing data from different domains, it is possible to uncover valuable insights that have never been discovered before. Most data scientists in the U.S. have extensive experience, such as experimental physicists, computational chemists, oceanographers, and even neurosurgeons. Some people may think that this is a phenomenon unique to the United States, where there is a high level of human mobility, but in fact, as we introduced in Chapter 4, there are also companies in Japan that are actively recruiting people from different professional backgrounds, and this situation is not far away. There is a serious shortage of talent Data scientists need to possess a wide range of skills and qualities, so it is expected that the position will be in short supply, i.e., understaffed. For example, the McKinsey Global Institute (MGI) published a May 2011 report titled "Big data: The next frontier for innovation, competition and productivity", which pointed out that the supply of highly analytical skills in the United States (students studying statistics and machine learning at universities and graduate schools) It was 150,000 in 2008 and is expected to double to 300,000 by 2018. However, it is expected that the demand for such talents will exceed the supply by then, reaching a scale of 440,000~490,000 people, which means that there will be a talent gap of 140,000~190,000. Just four or five years ago, the demand for data scientists was limited to web-based companies like Google and Amazon. Recently, however, companies that value data analytics, regardless of industry, are actively recruiting data scientists, which has exacerbated the shortage. The December 2011 EMC Data Science Study, a survey of data scientists by a major IT vendor, provided some interesting insights. The survey surveyed 462 people from data scientists in the United States, the United Kingdom, France, Germany, India, and China, as well as decision-makers in IT departments such as business intelligence experts. In addition, EMC invited 35 people to participate in the survey from attendees of the Data Scientist Summit in Las Vegas in May 2011, as well as from Kaggle, an online community of data scientists. The main points of the survey results are as follows. First, two-thirds of the participants said there was a shortage of data scientists. This is the same as the aforementioned McKinsey report. When it comes to new sources of data scientists, onethird expect "computer science students" to rank first, compared to a surprising 12% of existing business intelligence experts (Figures 8-6). That said, most people believe that today's business intelligence experts can't keep up with the need for data scientists. The difference between a data scientist and a business intelligence expert is that most data scientists go through the entire lifecycle of data, from the data acquisition stage, including data outside the company, to the final business decision based on the data. This process also includes filtering, systematizing, and visualizing the data Establishment of the Graduate School With the high demand for big data analysis, there will inevitably be a serious shortage of data scientists in the future, and in order to solve this problem, some universities in the United States have begun to establish graduate schools for analytics. Northwestern University, a prestigious private university in the United States located in Evanston, just outside Chicago, Illinois, is one of them. Northwestern University decided to establish a graduate school of analytics with a focus on big data analytics under its School of Engineering in September 2012 and began recruiting students. Northwestern University explains the establishment of the graduate school as follows: "While it is easy to find a job with a few basic knowledge of Hadoop and Cassandra, there is a shortage of people with in-depth knowledge. ” In addition, the graduate school's curriculum program aims to "teach and mentor the skills that lead business to success and develop excellent analysts who can lead project teams," and the curriculum content combines cutting-edge computer engineering and data analysis in addition to mathematics and statistics. The course is expected to cover the three main data analysis methods in the field of analytics: predictive analytics, descriptive analytics (business intelligence and data mining), and prescriptive analytics (optimization and simulation), as follows. (1) Autumn semester Statistical methods related to data mining (multivariate logistic regression analysis, nonlinear regression analysis, discriminant analysis, etc.) Quantitative methods (timeline analysis, probabilistic modeling, optimization) Decision analysis (multi-purpose decision analysis, decision tree, impact map, sensitivity analysis) ? Competitive Advantage Analysis (learning basic analytical concepts through projects and success stories) (2) Winter semester ? Introduction to Databases (Data Model, Database Design) Predictive analytics (timeline analysis, principal component analysis, nonparametric regression, statistical process control) ? Data management (ETL (Extract, Transform, Load), data governance, management responsibility, metadata) ? Optimization and heuristics (integer programming, nonlinear programming, local exploration, superheuristics (simulated annealing, genetic algorithm)) (3) Spring semester Big data analysis (learning of unstructured data concepts, MapReduce technology, big data analysis methods) Data Mining (Clustering (K-Means, Segmentation), Correlation Rules, Factor Analysis, Time-toLive Analysis) ? Others, choose from two of the following (Social Networking, Text Analytics, Web Analytics, Financial Analytics, Analytics in the Service Industry, Energy, Healthcare, Supply Chain Management, Probabilistic Models in Integrated Marketing Communication) (4) Autumn semester ? Computer simulation of risk analysis and operational analysis Software-level analytics (organization-level analysis issues, IT and business users, change management, data issues, presentation and communication of results) ? Graduation project In addition to the establishment of graduate schools, IT vendors are also actively collaborating. IBM not only sponsored $40,000, but also issued a statement promising to provide its predictive analytics software SPSS for free and offer discounts on hardware purchases. As a competitor to SPSS, SAS, which also develops predictive analytics software, and Teradata, a data warehouse vendor, will also provide sponsorship and support for the necessary products, as well as training on how to use them. Since Virginia Rometty (1958~), IBM's first female CEO, who took office in January 2012, received a bachelor's degree in computer science from Northwestern University and is currently a trustee of the university, IBM is very committed to working with Northwestern University, and in addition to hardware and software, it has also stated that it will help in the development of curriculum and case studies and other teaching materials. Considering IBM's recent efforts in big data and analytics, this move is undoubtedly hoped that Northwestern University will become a source of supply of analytics talent in the future. Japan has also begun a scramble for data scientists In Japan, as the level of use of big data gradually increases, there will be a shortage of human resources just like in the United States. Therefore, it is necessary to take medium- to long-term measures, such as replanning higher education, led by the government. However, it is often said that "big data is the next step in cloud computing", and for system integrators who want to establish big data-related business as soon as possible, even if they do not reach the level of "data scientists", they must acquire professionals with data analysis capabilities as soon as possible. Since 2012, this trend has become even more active. For example, in February 2012, NEC announced the strengthening of its big data business: "To cultivate or cooperate with the development of domain experts and data analysis experts who are proficient in the customer's business and can propose solutions within the company." The company plans to expand the team to 200 people over the next three years. ” Just two days after NEC's announcement, NTT Data announced the acquisition of Mathematical Systems, Inc., an analytical consulting company with approximately 70 business analytics professionals, causing a strong stir in the IT industry. With this acquisition, NTT Data brings its "business analytics professionals to 90" and announced that it plans to expand its workforce by 10 to 100 by mid-2012. These actions by system integrators are undoubtedly aimed at the business opportunity of "supporting user companies that have difficulty obtaining data analytics expertise". On the other hand, some of the leading Japanese companies that have already started to use big data have begun to expand their tentacles overseas in order to acquire highly skilled human resources. For example, Rakuten, which has been working on big data analysis for a long time, has established a new "big data department" within the company, and has begun to recruit people all over the world. Mr. Masaya Mori, the company's executive director and director of Lotte R&D Center, explained the reason: "It is difficult to obtain such highly skilled human resources as being able to implement machine learning algorithms on Hadoop clusters in Japan, and as a result, 90% of the human resources we recruit are foreigners. ” Rakuten, which manages more than 90 million products and has about 70 million members, has hundreds of terabytes of data such as clickstream data and search logs, in addition to product and member attribute data, member purchase history, product reviews, ranking information, and other data. In particular, many small and medium-sized enterprises around the world have opened stores on Rakuten, and "there is a lot of unstructured data such as product information in various regions" (Masaya Mori). As a result, Rakuten has been developing its own technologies such as Hadoop and NoSQL databases for a long time, and even American Web companies have been surprised. That's why it's no surprise that Rakuten is already in a far ahead position compared to system integrators who are only now considering training data analytics specialists. In recent years, Rakuten has been actively promoting global business in Taiwan, Indonesia, Thailand, France, and the United States. Come to think of it, there is no reason for companies that are committed to globalization to limit their recruitment to Japan alone. For companies that are truly pursuing globalization with big data as a weapon, in order to solve the problem of lack of data scientists in the short term, perhaps more companies will take measures to introduce overseas talents. The final question: organizational structure and corporate culture With high-quality data and an IT infrastructure that processes it efficiently, coupled with good data scientists, it all comes down to organizational structure and corporate culture. Even if you can get good analytical results, it will be all in vain if you don't have an organization that can make accurate decisions and act quickly. When the results of the analysis are contrary to empirical intuition, data analysis becomes meaningless if the organizational culture values empirical intuition unconditionally. In the preface to the famous book "Competing on Analytics" (by Thomas H. Davenport and Jeanne G. Harris, translated by Akiko Murai and published by Nikkei BP), which explains the usefulness of data analysis, I introduce the words of Mr. Gary Loveman, chairman, president, and CEO of Harrah's Entertainment, mentioned in Chapter 1 of this book. The management of analytics here has absolutely nothing to do with the methodology, the availability of data, or the technology. (omitted)...... There are some common problems with analytics management, such as: ? Prevailing traditional ideas are often associated with powerful people, and these ideas are often perpetuated without passing important tests. Decision-making, especially at the C-suite level, is not only a failure to conduct rigorous, unbiased analysis, but also a prevalence of unanalytical practices because it is perceived as a rare talent for the C-suite, which is thought to be what sets business leaders and visionary leaders apart from those who are smart but uninspired. ? There is a lack of people in the enterprise who are diligent and proficient in data analysis. Data analysis is often seen as a last resort, and often the people involved in the analysis are not familiar with the proper analytical methods. ? It is often the person who determines the thought, not the mind that controls the person. What do you think? Did Mr. Loveman's questions hit the nail on the head of many companies? While it's important to recruit good data scientists, it's important to educate managers and executives so that they can make accurate decisions and actions based on the information they receive. The goal: to become a data-driven enterprise Companies that can reflect the results of data analysis and the information obtained from them in a timely manner to establish a competitive position in the business are called "data driven enterprises" in Europe and the United States. In order to become a data-driven business, you need to cultivate a culture that always makes decisions based on data. When it is time to make a judgment, it is important to create an atmosphere that always asks for data to corroborate. At Google, which is the representative of a data-driven enterprise, every time a new idea is proposed, it is bound to be asked questions such as "has it been tested and has data to back it up?" Of course, the path to a data-driven enterprise is not easy, and there are many points to explore, such as creating a data environment that is accessible to not just a subset of analytics teams and management, but to any employee, and an organizational structure that can quickly bring the results of analytics to business. It has been repeatedly emphasized in this book that the so-called "big data" does not simply refer to the increase in the amount of data. Although a large amount of data may be generated as a result of the appropriate collection of a variety of data that has not been used or cannot be used in the past, it does not mean that the amount of data must reach terabytes or even petabytes to be considered big data, and the most important thing is to discover the value of data that has been neglected.