The concept of big data itself is relatively new, the origins of large data sets go back to the 1960s and '70s when the world of data was just getting started with the first data centers and the development of the relational database. In the year 2005, people began to realize the data which are generated through Facebook, YouTube, and other online services. Hadoop (an open-source framework created specifically to store and analyze big data sets) was developed that same year. Not only SQL also began to trend during this time. The open-source frameworks were developed such as Hadoop (and more recently, Spark) was essential for the magnification of big data because they make big data easier to work with and affordable to store. In frequent years since then, the volume of big data has proliferated. Users are still generating tremendous amounts of data—but it’s not just humans who are doing it. With the advent of the Internet of Things (IoT), more objects and devices are connected to the internet, gathering data on customer usage patterns and product performance. The emergence of machine learning has produced still more data. The usefulness of big data is only just beginning. The possibilities of big data had expanded even further by cloud computing. Today, big data has become capital. A large part of the value they offer comes from their data, which they’re constantly analyzing to produce more efficiency and develop new products. Recent technological breakthroughs have exponentially reduced the cost of data storage and compute, making it easier and less expensive to store more data than ever before. The big data now cheaper and more accessible with an increased volume, and it is more accurate and precise business decisions. Before we delve into the most common big data challenges, we should first define 'big data'. Data stores are constantly growing, so what seems like a lot of data right now may seem like a perfectly normal amount in a year or two. In addition, every organization is different, so the amount of data that seems challenging for a small retail store may not seem like a lot to a large financial services company. Most experts define big data in terms of the three characteristics:
- Volume. Big data is any set of data that is so large that the organization that owns it faces challenges related to storing or processing it. In reality, trends like ecommerce, mobility, social media and the Internet of Things (IoT) are generating so much information, that nearly every organization probably meets this criterion.
- Velocity. If your organization is generating new data at a rapid pace and needs to respond in real time, you have the velocity associated with big data. Most organizations that are involved in ecommerce, social media or IoT satisfy this criterion for big data.
- Variety. If your data resides in many different formats, it has the variety associated with big data. For example, big data stores typically include email messages, word processing documents, images, video and presentations, as well as data that resides in structured relational database management systems.
Legal and Ethical Challenges
- Consumer privacy. The legal risks of big data begin with consumer privacy. Most websites, online services, and mobile apps have a privacy policy. Implementing and enforcing a privacy policy is not only a good business practice, but it may also be required by law or by third party services that collect information through a website. The terms of service (TOS) ought to be sporadically reviewed to work out whether or not they accurately mirror business practices, particularly with respect to the collection, use, and sharing of personal information. A user may accept the TOS and privacy policy by affirmatively checking a box or by simply continuing to use the service
- Control over data. Ownership rights to big data can provide a competitive business advantage since the data owner controls how the data may be used and shared. For example, Twitter’s data licensing business is its fastest growing revenue. Twitter sells its ‘firehose’ of over 500 million daily tweets to various companies that try to turn the tweets into actionable information. Most of the business price in massive knowledge comes from combining knowledge from totally different sources. Ownership of information ensuing from the information analytics is additionally vital. Right to data are usually allocated in the privacy policy and TOS for websites, online services and mobile apps. Traditional signed agreements may be used in business-to-business transactions. For example, a signed agreement might be used between an IoT provider and its farm customers in a smart agriculture application. Joint ownership is a middle ground for ownership allocations in some business-to-business transactions.
- Terms of service agreement. A TOS is a legal agreement that establishes the obligations and restrictions for using a website, mobile app or online service. There may be liability exposure if the data analytics software provides erroneous or no actionable information. Such liability is limited in the TOS primarily by limited warranty, disclaimers of warranties and limitation of liability provisions in the same way as for other contracts. The TOS may also cover scope of permitted use, restrictions on activities, disclaimers regarding content, indemnification, term and termination, copyright and other intellectual property rights, governing law, jurisdiction, dispute resolution and other issues.
- Shortage of skilled people. There is a definite shortage of skilled big data professionals available at this time. As this has been mentioned by many enterprises seeking to better utilize big data and build more effective data analysis systems. There is a scarcity knowledgeable individuals and licensed knowledge scientists or knowledge analysts obtainable at this time, which makes the ‘number crunching’ difficult, and insight building slow.
Results and Discussions
Big data is generated by everything around us at all times and includes both personal information and non-personal information. Data analytics is used to convert big data into actionable information that can provide value in a wide range of both consumer transactions as well as business-to-business transactions. Some tools and applications which can be applied to tackle the challenges of big data. Companies need to invest money to ensure security because the bigger the data, the bigger the target it presents to steal and sell it. Many companies are now considering such options as data lakes, which can allow them to collect and store massive quantities of unstructured data in its native format. Now a days NoSQL databases seem to be ultimate trend among the mainstream enterprises. NoSQL means Not Only SQL, implying that they still can support SQL-like query languages, but basically, NoSQL databases concentrate on processing unstructured data. For data representation Google Charts, Tableau, D3, Canvas tools can be used to represents and analyze those big data in graphical representation. Big data analytics technologies are maturing. The selection of appropriate tools becomes vital when it comes to tackling the challenges of big data. The top five open-source big data analysis platforms and tools are Apache Hadoop, Hadoop MapReduce, Apache Storm, Grind-Grain, HPCC Systems.
Save your time!
We can take care of your essay
- Proper editing and formatting
- Free revision, title page, and bibliography
- Flexible prices and money-back guarantee
Conclusion
We have entered an era of big data. Many sectors of our economy are now moving to a data-driven decision-making model where the core business relies on analysis of large and diverse volumes of data that are continually being produced. This data-driven world has the potential to improve the efficiencies of enterprises and improve the quality of our lives. The amount of big data is already massive, but it is expected to grow exponentially as new technologies such as the more pervasive IoT devices, drones and wearables will jump into the fray. Ninety percent of the big data in the world today has been generated in the last two years, and the recent advancements in deep learning are playing a key role in helping businesses decrypt this precious goldmine of information. Big data and business analytics solutions are now a mainstream technology, and together with AI and automation, they represent the foundation upon which the digital transformation process is built.
Stuck on your essay?
