Ugwu Okechukwu Emmanuel
Abstract
The rapid rise of the Internet and the digital economy has fueled an exponential growth in demand for data storage and analytics, and IT department are facing tremendous challenge in protecting and analyzing these increased volumes of information. The reason organizations are collecting and storing more data than ever before is because their business depends on it. The type of information being created is no more traditional database-driven data referred to as structured data rather it is data that include documents, images, audio, video, and social media contents known as unstructured data or Big Data. Big Data Analytics is a way of extracting value from these huge volumes of information, and it drives new market opportunities and maximizes customer retention. This research primarily focuses on discussing the various technologies that work together as a Big Data Analytics system that can help predict future volumes, gain insights, take proactive actions, and give way to better strategic decision-making.
INTRODUCTION
Big Data is an important concept, which is applied to data, which does not conform to the normal structure of the traditional database. Big Data consists of different types of key technologies like Hadoop, HDFS, NoSQL, MapReduce, MongoDB, Cassandra, PIG, HIVE, and HBASE that work together to achieve the end goal like extracting value from data that would be previously considered dead. According to a recent market report published by Transparency Market Research, the total value of big data was estimated at $6.3 billion as of 2012, but by 2018, it’s expected to reach the staggering level of $48.3 billion that’s almost a 700 percent increase (Li and Lu, 2014). Forrester Research estimates that organizations effectively utilize less than 5 percent of their available data. This is because the rest is simply too expensive to deal with. Big Data is derived from multiple sources. It involves not just traditional relational data, but all paradigms of unstructured data sources that are growing at a significant rate. For instance, machine-derived data multiplies quickly and contains rich, diverse content that needs to be discovered. Another example, human-derived data from social media is more textual, but the valuable insights are often overloaded with many possible meanings.
Big Data Analytics reflect the challenges of data that are too vast, too unstructured, and too fast moving to be managed by traditional methods. From businesses and research institutions to governments, organizations now routinely generate data of unprecedented scope and complexity. Gleaning meaningful information and competitive advantages from massive amounts of data has become increasingly important to organizations globally. Trying to efficiently extract the meaningful insights from such data sources quickly and easily is challenging. Thus, analytics has become inextricably vital to realize the full value of Big Data to improve their business performance and increase their market share. The tools available to handle the volume, velocity, and variety of big data have improved greatly in recent years. In general, these technologies are not prohibitively expensive, and much of the software is open source. Hadoop, the most commonly used framework, combines commodity hardware with opensource software. It takes incoming streams of data and distributes them onto cheap disks; it also provides tools for analyzing the data. However, these technologies do require a skill set that is new to most IT departments, which will need to work hard to integrate all the relevant internal and external sources of data. Although attention to technology isn’t sufficient, it is always a necessary component of a big data strategy. With the digitization of most of the processes, emergence of different social network platforms, blogs, deployment of different kind of sensors, adoption of hand-held digital devices, wearable devices and explosion in the usage of Internet, huge amount of data are being generated on continuous basis. No one can deny that Internet has changed the way businesses operate, functioning of the government, education and lifestyle of people around the world. Today, this trend is in a transformative stage, where the rate of data generation is very high and the type of data being generated surpasses the capability of existing data storage techniques. It cannot be denied that these data carry a lot more information than ever before due to the emergence and adoption of Internet. Over the past two decades, there is a tremendous growth in data. This trend can be observed in almost every field. According to a report by International Data Corporation (IDC), a research company claims that between 2012 and 2020, the amount of information in the digital universe will grow by 35 trillion gigabytes (1 gigabyte equivalent to 40 (four-drawer) file cabinets of text, or two music CDs). That’s on par with the number of stars in the physical universe! (Forsyth, 2012).
LITERATURE REVIEW
Big Data is a data analysis methodology enabled by recent advances in technologies that support high-velocity data capture, storage and analysis. Storage and retrieval of vast amount of structured as well as unstructured data at a desirable time lag is a challenge. Some of these limitations to handle and process vast amount of data with the traditional storage techniques led to the emergence of the term Big Data. Though big data has gained attention due to the emergence of the Internet, but it cannot be compared with it. It is beyond the Internet, though, Web makes it easier to collect and share knowledge as well data in raw form. Big Data is about how these data can be stored, processed, and comprehended such that it can be used for predicting the future course of action with a great precision and acceptable time delay. Marketers focus on target marketing, insurance providers focus on providing personalized insurances to their customers, and healthcare providers focus on providing quality and low-cost treatment to patients. Despite the advancements in data storage, collection, analysis and algorithms related to predicting human behavior; it is important to understand the underlying driving as well as the regulating factors (market, law, social norms and architecture), which can help in developing robust models that can handle big data and yet yield high prediction accuracy (Boyd and Crawford, 2011). The current and emerging focus of big data analytics is to explore traditional techniques such as rule-based systems, pattern mining, decision trees and other data mining techniques to develop business rules even on the large data sets efficiently. It can be achieved by either developing algorithms that uses distributed data storage, in-memory computation or by using cluster computing for parallel computation. Earlier these processes were carried out using grid computing, which was overtaken by cloud computing in recent days.
Big Data Analytics is all about processing unstructured information from call logs, mobile-banking transactions, online user generated content such as blog posts and tweets, online searches, and images which can be transformed into valuable business information using computational techniques to unveil trends and patterns between datasets. Another dimension of the Big Data definition involves technology. Big Data is not only large and complex, but it requires innovative technology to analyze and process. In 2013, the National Institute of Standard and Technology (NIST) Big Data workgroup proposed the following definition of Big Data that emphasizes application of new technology; Big Data exceed the capacity or capability of current or conventional methods and systems, and enable novel approaches to frontier questions previously inaccessible or impractical using current or conventional methods. Business challenges rarely show up in the appearance of a perfect data problem, and even when data are abundant, practitioners have difficulties to incorporate it into their complex decision-making that adds business value. In 2012, McKinsey & Company conducted a survey of 1,469 executives across various regions, industries and company sizes, in which 49 percent of respondents said that their companies are focusing big data efforts on customer insights, segmentation and targeting to improve overall performance (Li and Lu, 2014). An even higher number of respondents 60 percent said their companies should focus efforts on using data and analytics to generate these insights. Yet, just one-fifth said that their organizations have fully deployed data and analytics to generate insights in one business unit or function, and only 13 percent use data to generate insights across the company. As these survey results show, the question is no longer whether big data can help business, but how can business derive maximum results from big data. Predictive Analytics Predictive Analytics is the use of historical data to forecast on consumer behavior and trends (Chen et al., 2014). It is the use of past/historical data to predict future trends. This analysis makes use of the statistical models and machine learning algorithms to identify patterns and learn from historical data (Li and Lu, 2014). Predictive Analysis can also be defined as a process that uses machine learning to analyze data and make predictions (Chen et al., 2014).
Sixty seven percent of businesses aim at using predictive analytics to create more strategic marketing campaign in future, and 68% sight competitive advantage as the prime benefit of predictive analysis (Sun and Heller, 2012). Broadly speaking, predictive analysis can be applied in ecommerce for product recommendation, price management, and predictive search. Typically a large e-commerce site offers thousands of product and services for sale. Navigating and searching for a product out of thousands on a website could be a major setback to consumers. However, with the invention of recommender system, an E-Commerce site/application can quickly identify/predict products that closely suit the consumer’s taste (Sun and Heller, 2012). Using a technology called Collaborative Filtering a database of historical user preferences is created. When a new customer access the ecommerce site, the customer is matched with the database of preferences, in order to discover a preference class that closely matches the customer taste. These products are then recommended to the customer (Gandomi and Haider, 2015). Another technology that is used in ecommerce is the clustering algorithm. Clustering algorithm works by identifying groups of users that have similar preferences. These users are then clustered into a single group and are given a unique identifier.
New customers cluster are predicted by calculating the average similarities of the individual members in that cluster. Hence a user could be a partial member of more than one cluster depending of the weight of the user’s average opinion (Gandomi and Haider, 2015). Advanced analytics is defined as the scientific process of transforming data into insight for making better decisions. As a formal discipline, advanced analytics have grown under the Operational Research domain. There are some fields that have considerable overlap with analytics, and also different accepted classifications for the types of analytics (Gandomi and Haider, 2015).
Dimensions of Big Data
Initially, big data was characterized by the following dimensions, which were, often, referred as 3V model:
Volume refers to the magnitude of the data that is being generated and collected. It is increasing at a faster rate from terabytes to petabytes (1024 terabytes) (Zikopoulos et al., 2012; Singh and Singh, 2012). With increase in storage capacities, what cannot be captured and stored now will be possible in future. The classification of big data on the basis of volume is relative with respect to the type of data generated and time. In addition, the type of data, which is often referred as Variety, defines “big” data. Two types of data, for instance, text and video of same volume may require different data management technologies (Gandomi and Haider, 2015).
Velocity refers to the rate of generation of data. Traditional data analytics is based on periodic updates- daily, weekly or monthly. With the increasing rate of data generation, big data should be processed and analyzed in real- or near real-time to make informed decisions. The role of time is very critical here (Singh and Singh, 2012; Gandomi and Haider, 2015). Few domains including Retail, Telecommunications and Finance generate high-frequency data. The data generated through Mobile apps, for instance, demographics, geographical location, and transaction history, can be used in real-time to offer personalized services to the customers. This would help to retain the customers as well as increase the service level.
Variety refers to different types of data that are being generated and captured. They extend beyond structured data and fall under the categories of semi-structured and unstructured data (Zikopoulos et al., 2012; Singh and Singh, 2012; Gandomi and Haider, 2015). The data that can be organized using a pre-defined data model are known as structured data. The tabular data in relational databases and Excel are examples of structured data and they constitute only 5% of all existing data (Cukier, 2010). Unstructured data cannot be organized using these pre-defined model and examples include video, text, and audio. Semi-structured data that fall between the categories of structured and unstructured data. Extensible Markup Language (XML) falls under this category.
Coined by IBM, veracity refers to the unreliability associated with the data sources (Gandomi and Haider, 2015). For instance, sentiment analysis using social media data (Twitter, Facebook, etc.) is subject to uncertainty. There is a need to differentiate the reliable data from uncertain and imprecise data and manage the uncertainty associated with the data.
Variability and Complexity were added as additional dimensions by SAS. Often, inconsistency in the big data velocity leads to variation in flow rate of data, which is referred to as variability (Gandomi and Haider, 2015). Data are generated from various sources and there is an increasing complexity in managing data ranging from transactional data to big data. Data generated from different geographical locations have different semantics (Zikopoulos et al., 2012; Forsyth, 2012). f) Low-Value density: Data in its original form is unusable. Data is analyzed to discover very high value (Sun and Heller, 2012). For example, logs from the website cannot be used in its initial form to obtain business value. It must be analyzed to predict the customer behavior.
BIG DATA VALUE CHAIN
Data value chain refers to the framework that deals with a set of activities to create value from available data. It can be divided into seven phases: data generation, data collection, data transmission, data pre-processing, data storage, data analysis and decision making.
The first and foremost step the big data value chain is the generation of data. As discussed in the previous section, data is generated from various sources that include data from Call Detail Records (CDR), blogs, Tweets and Facebook Page.
In this phase, the data is obtained from all possible data sources (Miller and Mork, 2013; Chen et al., 2014). For instance, in order to predict the customer churn in Telecom, data can be obtained from CDRs and opinions/complaints of the customers on Social Networking Sites such as Twitter (in the form of tweets) and Facebook (opinions shared on the company's Facebook page). The most commonly used methods are log files, sensors, web crawlers and network monitoring software (Chen et al., 2014).
Once the data is collected, it is transferred to a data storage and processing infrastructure for further processing and analysis. It can be carried out in two phases: Inter-Dynamic Circuit Network (DCN) transmission and Intra-DCN transmissions. Inter-DCN transmission deals with the transfer of data from the data source to the data center while the latter helps in the transfer within the data center. Apart from storage of data, data center helps in collecting, organizing and managing data.
The data collected from various data sources may be redundant, noisy and inconsistent, hence, in this phase; the data is pre-processed to improve the data quality required for analysis. This also helps to improve the accuracy of the analysis and reduce the storage expenses.
The big data storage systems should provide reliable storage space and powerful access to the data. The distributed storage systems for big data should consider factors like consistency (C), availability (A) and partition tolerance (P). According to the CAP theory proposed by Brewer (2000), the distributed storage systems could meet two requirements simultaneously, that is, either consistency and availability or availability and partition tolerance or consistency and partition tolerance but not all requirements simultaneously (Gilbert and Lynch, 2002). Considerable research is still going on in the area of big data storage mechanism. Little advancement in this respect is Google File System (GFS), Dynamo, BigTable, Cassandra, CouchDB, and Dryad.
Once the data is collected, transformed and stored, the next process is data exploitation or data analysis, which is enumerated using the following steps:
a) Define Metrics: Based on the collected and transformed data, a set of metrics is defined for a particular problem. For instance, to identify a potential customer who is going to churn out, a number of times he/she contacted (be it through a voice call, tweets or complaints on Facebook page) can be considered. (Miller and Mork, 2013).
b) Select architecture based on analysis type: Based on the timeliness of analysis to be carried out, suitable architecture is selected. Real-time analysis is used in the domain where the data keeps on changing constantly and there is a need for rapid analysis to take actions.
c. Selection of appropriate algorithms and tools: One of the most important steps of data analysis is selection of appropriate techniques for data analysis. Few traditional data analysis techniques like cluster analysis, regression analysis and data mining algorithms, still hold good for big data analytics.
d) Data Visualization: The need for inspecting details at multiple scales and minute details gave rise to data visualization. Visual interfaces along with statistical analyzes and related context help to identify patterns in large data over time (Fisher et al., 2012).
Based on the analysis and the visualized results, the decision makers can decide whether and how to reward a positive behavior and change a negative one. The details of a particular problem can be analyzed to understand the causes of the problems take informed decisions and plan for necessary actions (Miller and Mork, 2013).
SOFTWARE TOOLS FOR HANDLING BIG DATA
Apache Flume
Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. Flume deploys as one or more agents, each contained within its own instance of the Java Virtual Machine (JVM). Agents consist of three pluggable components: sources, sinks, and channels. Flume agents ingest incoming streaming data from one or more sources. Data ingested by a Flume agent is passed to a sink, which is most commonly a distributed file system like Hadoop. Multiple Flume agents can be connected together for more complex workflows by configuring the source of one agent to be the sink of another. Flume sources listen and consume events. Events can range from newline-terminated strings in stdout to HTTP POSTs and RPC calls — it all depends on what sources the agent is configured to use. Flume agents may have more than one source, but at the minimum they require one. Sources require a name and a type; the type then dictates additional configuration parameters. Channels are the mechanism by which Flume agents transfer events from their sources to their sinks. Events written to the channel by a source are not removed from the channel until a sink removes that event in a transaction. This allows Flume sinks to retry writes in the event of a failure in the external repository (such as HDFS or an outgoing network connection).
Apache Sqoop
Apache Sqoop is a CLI tool designed to transfer data between Hadoop and relational databases. Sqoop can import data from an RDBMS such as MySQL or Oracle Database into HDFS and then export the data back after data has been transformed using MapReduce. Sqoop also has the ability to import data into HBase and Hive. Sqoop connects to an RDBMS through its JDBC connector and relies on the RDBMS to describe the database schema for data to be imported. Both import and export utilize MapReduce, which provides parallel operation as well as fault tolerance. During import, Sqoop reads the table, row by row, into HDFS. Because import is performed in parallel, the output in HDFS is multiple files.
Apache Pig
Apache’s Pig is a major project, which is lying on top of Hadoop, and provides higher-level language to use Hadoop’s MapReduce library. Pig provides the scripting language to describe operations like the reading, filtering and transforming, joining, and writing data which are exactly the same operations that MapReduce was originally designed for. Instead of expressing these operations in thousands of lines of Java code which uses MapReduce directly, Apache Pig lets the users express them in a language that is not unlike a bash or Perl script. Pig was initially developed at Yahoo Research around 2006 but moved into the Apache Software Foundation in 2007. Unlike SQL, Pig does not require that the data must have a schema, so it is well suited to process the unstructured data. But, Pig can still leverage the value of a schema if you want to supply one. PigLatin is relationally complete like SQL, which means it is at least as powerful as a relational algebra. Turing completeness requires conditional constructs, an infinite memory model, and looping constructs.
Apache Hive
Hive is a technology developed by Facebook that turns Hadoop into a data warehouse complete with a dialect of SQL for querying. Being a SQL dialect, HIVEQL is a declarative language. In PigLatin, you specify the data flow, but in Hive we describe the result we want and hive figures out how to build a data flow to achieve that result. Unlike Pig, in Hive a schema is required, but you are not limited to only one schema. Like PigLatin and SQL, HiveQL itself is a relationally complete language but it is not a Turing complete language.
Apache ZooKeeper
Apache Zoo Keeper is an effort to develop and maintain an open-source server, which enables highly reliable distributed coordination. It provides a distributed configuration service, a synchronization service and a naming registry for distributed systems. Distributed applications use ZooKeeper to store and mediate updates to import configuration information. ZooKeeper is especially fast with workloads where reads to the data are more common than writes. The ideal read/write ratio is about 10:1. ZooKeeper is replicated over a set of hosts (called an ensemble) and the servers are aware of each other and there is no single point of failure.
MongoDB
MongoDB is an open source, document-oriented NoSQL database that has lately attained some space in the data industry. It is considered as one of the most popular NoSQL databases, competing today and favors master-slave replication. The role of master is to perform reads and writes whereas the slave confines to copy the data received from master, to perform the read operation, and backup the data. The slaves do not participate in write operations but may select an alternate master in case of the current master failure. MongoDB uses binary format of JSON-like documents underneath and believes in dynamic schemas, unlike the traditional relational databases. The query system of MongoDB can return particular fields and query set compass search by fields, range queries, regular expression search, etc. and may include the user-defined complex JavaScript functions. As hinted already, MongoDB practice flexible schema and the document structure in a grouping, called Collection, may vary and common fields of various documents in a collection can have disparate types of the data. The MongoDB is equipped with the suitable drivers for most of the programming languages, which are used to develop the customized systems that use MongoDB as their backend player. There is an increasingly demand of using MongoDB as pure in-memory database; in such cases, the application dataset will always be small. Though, it is probably are easy for maintenance and can make a database developer happier; this can be a bottle neck for complex applications that require tremendous database management capabilities.
Apache Cassandra
Apache Cassandra is the yet another open source NoSQL database solution that has gained industrial reputation which is able to handle big data requirements. It is a highly scalable and high-performance distributed database management system that can handle real-time big data applications that drive key systems for modern and successful businesses. It has a built-for-scale architecture that can handle petabytes of information and thousands of concurrent users/operations per second as easily as it can manage much smaller amount of data and user traffic. It has a peer to peer design that offers no single point of failure for any database process or function, in addition to the location independence capabilities that equate to a true network-independent method of storing and accessing data, data can be read and written anywhere. Apache Cassandra is also equipped with flexible/dynamic schema design that accommodates all formats of big data applications, including structured, semi-structured, and unstructured data. Data is represented in Cassandra via column families that are dynamic in nature and accommodate all modifications online.
Apache Hadoop
The Apache Hadoop software library is a framework that enables the distributed processing of large data sets across clusters of computers. It is designed to scale up from single servers to thousands of machines, with each offering local computation and storage. The basic notion is to allow a single query to find and collect results from all the cluster members, and this model is clearly suitable for Google's model of search support. One of the largest technological challenges in software systems research today is to provide mechanisms for storage, manipulation, and information retrieval on large amount of data. Web services and social media produce together an impressive amount of data, reaching the scale of petabytes daily (Facebook, 2012). These data may contain valuable information, which sometimes is not properly explored by existing systems. Most of this data is stored in a nonstructured manner, using different languages and format, which, in many cases, are in compatible. Parallel and distributed computing currently has a fundamental role in data processing and information extraction of large datasets. Over the last years, commodity hardware became part of clusters, since the x86 platform cope with the need of having an overall better cost/performance ratio, while decreasing maintenance cost. Apache Hadoop is a framework developed to take advantage of this approach, using such commodity clusters for storage, processing and manipulation of large amount of data. The framework was designed over the MapReduce paradigm and uses the HDFS as a storage file system. Hadoop presents key characteristics when performing parallel and distributed computing, such as data integrity, availability, scalability, exception handling, and failure recovery. Hadoop is a popular choice when you need to filter, sort, or pre-process large amounts of new data in place and distill it to generate denser data that theoretically contains more information. Pre-processing involves filtering new data sources to make them suitable for additional analysis in a data warehouse. Hadoop is a top-level open source project of the Apache Software Foundation. Several suppliers, including Intel, offer their own commercial Hadoop distributions, packaging the basic software stack with other Hadoop software projects such as Apache Hive, Apache Pig, and Apache Sqoop. These distributions must integrate with data warehouses, databases, and other data management products so data can move among Hadoop clusters and other environments to expand the data pool to process or query.
MapReduce
MapReduce is the model of distributed data processing introduced by Google in 2004. The fundamental concept of MapReduce is to divide problems into two parts: a map function that processes source data into sufficient statistics and a reduce function that merges all sufficient statistics into a final answer. By definition, any number of concurrent map functions can be run at the same time without intercommunication. Once all the data has had the map function applied to it, the reduce function can be run to combine the results of the map phases. For large scale batch processing and high speed data retrieval, common in Web search scenarios, MapReduce provides the fastest, most cost-effective and most scalable mechanism for returning results. Today, most of the leading technologies for managing "big data" are developed on MapReduce. With MapReduce there are few scalability limitations, but leveraging it directly does require writing and maintaining a lot of code.
Apache Splunk
Splunk is a general-purpose search, analysis and reporting engine for time-series text data, typically machine data. Splunk software is deployed to address one or more core IT functions: application management, security, compliance, IT operations management and providing analytics for the business. The Splunk engine is optimized for quickly indexing and persisting unstructured data loaded into the system. Specifically, Splunk uses a minimal schema for persisted data – events consist only of the raw event text, implied timestamp, source (typically the filename for file based inputs), source type (an indication of the general type of data) and host (where the data originated).
Once data enters the Splunk system, it quickly proceeds through processing, is persisted in its raw form and is indexed by the above fields along with all the keywords in the raw event text. Indexing is an essential element of the canonical “super-grep” use case for Splunk, but it also makes most retrieval tasks faster. Any more sophisticated processing on these raw events is deferred until search time. This serves four important goals: indexing speed is increased as minimal processing is performed, bringing new data into the system is a relatively low effort exercise as no schema planning is needed, the original data is persisted for easy inspection and the system is resilient to change as data parsing problems do not require reloading or re-indexing the data.
BIG DATA FRAMEWORK
Apache Spark
Apache Spark an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMP Lab, and open sourced in 2010 as an Apache project. Hadoop as a big data processing technology has been around for ten years and has proven to be the solution of choice for processing large data sets. MapReduce is a great solution for one-pass computations, but not very efficient for use cases that require multi-pass computations and algorithms. Each step in the data processing workflow has one Map phase and one Reduce phase and you'll need to convert any use case into MapReduce pattern to leverage this solution. Spark takes MapReduce to the next level with less expensive shuffles in the data processing. With capabilities like in-memory data storage and near real-time processing, the performance can be several times faster than other big data technologies. Spark also supports lazy evaluation of big data queries, which helps with optimization of the steps in data processing workflows. It provides a higher-level API to improve developer productivity and a consistent architect model for big data solutions. Spark holds intermediate results in memory rather than writing them to disk, which is very useful especially when you need to work on the same dataset multiple times. It’s designed to be an execution engine that works both in-memory and on-disk. Spark operators perform external operations when data does not fit in memory. Spark can be used for processing datasets that larger than the aggregate memory in a cluster. Spark will attempt to store as much as data in memory and then will spill to disk. It can store part of a data set in memory and the remaining data on the disk. You have to look at your data and use cases to assess the memory requirements. With this in-memory data storage, Spark comes with a great performance advantage. Spark is written in Scala Programing Language and runs on the Java Virtual machine. It currently supports programming languages like Scala, java, python, Clojure and R. Other than Spark Core API, there are additional libraries that are part of the Spark ecosystem and provide additional capabilities in Big Data analytics.
APPLICATION OF BIG DATA ANALYTICS
The concept of big data analytics has left no sector untouched. Few sectors like Telecommunication, Retail and Finance have been early adopters of big data analytics, followed by other sectors (Villars et al., 2011). The application of big data analytics in various sectors is discussed as follows:
Healthcare
Data analysts obtain and analyze information from multiple sources to gain insights. The multiple sources are electronic patient record; clinical decision support system including medical imaging, physician's written notes and prescription, pharmacy and laboratories; clinical data; and machine generated sensor data (Raghupathi and Raghupathi, 2014). The integration of clinical, publichealth and behavioural data helps to develop a robust treatment system, which can reduce the cost and at the same time, improve the quality of treatment (Brown et al., 2011). Rizzoli Orthopedic Institute in Bologna, Italy analyzed the symptoms of individual patients to understand the clinical variations in a family. This helped to reduce the number of imaging and hospitalizations by 60% and 30%, respectively (Raghupathi and Raghupathi, 2014).
Telecommunication
Low adoption of mobile services and churn management are few of the most common problems faced by the mobile service providers (MSPs). The cost of acquiring new customer is higher than retaining the existing ones. Customer experience is correlated with customer loyalty and revenue (Soares, 2012a,b). In order to improve the customer experience, MSPs analyze a number of factors such as demographic data (gender, age, marital status, and language preferences), customer preferences, household structure and usage details (CDR, internet usage, value-added services (VAS)) to model the customer preferences and offer a relevant personalized service to them. This is known as targeted marketing, which improves the adoption of mobile services, reduces churn, thus, increasing the revenue of MSPs. Ufone, a Pakistan-based MSP, reduced the churn rate by precisely marketing the customized offers to their customers (Utsler, 2013).
Financial Firms
Currently, capital firms are using advanced technology to store huge volumes of data. But increasing data sources like Internet and Social media require them to adopt big data storage systems. Capital markets are using big data in preparation for regulations like EMIR, Solvency II, Basel II etc, anti-money laundering, fraud mitigation, pre-trade decision-support analytics including sentiment analysis, predictive analytics and data tagging to identify trades (Verma and Mani, 2012). The timeliness of finding value plays an important role in both investment banking and capital markets, hence, there is a need for real-time processing of data.
Retail
Evolution of e-commerce, online purchasing, social-network conversations and recently location specific smartphone interactions contribute to the volume and the quality of data for data-driven customization in retailing (Brown et al., 2011). Major retail stores might place CCTV not only to observe the instances of theft but also to track the flow of customers (Villars et al., 2011). It helps to observe the age group, gender and purchasing patterns of the customers during weekdays and weekends. Based on the purchasing patterns of the customers, retailers group their items using a well-known data mining technique called Market Basket Analysis (proposed by (Agrawal and Srikant, 1994)), so that a customer buying bread and milk might purchase jam as well. This helps to decide on the placement of objects and decide on the prices (Brown et al., 2011; Villars et al., 2011). Nowadays, e-commerce firms use market basket analysis and recommender systems to segment and target the customers. They collect the click stream data, observe behavior and recommend products in the real time.
Marketing
Marketing analytics helps the organizations to evaluate their marketing performance, to analyze the consumer behavior and their purchasing patterns, to analyze the marketing trends which would aid in modifying the marketing strategies like the positioning of advertisements in a webpage, implementation of dynamic pricing and offering personalized products (Soares, 2012a).
New Product Development
There is a huge risk associated with new product development. Enterprises can integrate both external sources, i.e., twitter and Facebook page and internal data sources, i.e., customer relationship management (CRM) systems to understand the customers' requirement for a new product, to gather ideas for new product and to understand the added feature included in a competitor's product. Proper analysis and planning during the development stage can minimize the risk associated with the product, increase the customer lifetime value and promote brand engagement (Anastasia, 2015). Ribbon UI in Microsoft 2007 was created by analyzing the customer data from previous releases of the product to identify the commonly used features and making intelligent decisions (Fisher et al., 2012).
Banking
The investment worthiness of the customers can be analyzed using demographic details, behavioral data, and financial employment. The concept of cross-selling can be used here to target specific customer segments based on past buying behavior, demographic details, sentiment analysis along with CRM data (Forsyth, 2012; Coumaros et al., 2014).
Energy and Utilities
Consumption of water, gas and electricity can be measured using smart meters at regular intervals of one hour. During this interval, a huge amount of data is generated and analyzed to change the patterns of power usage (Brown et al., 2011). The real-time analysis reveals energy consumption pattern, instances of electricity thefts and price fluctuations.
TECHNOLOGICAL GROWTH AND TECHNOLOGICAL LIMITATIONS
Advantages and applications of big data analytics are being realized in various sectors. The development of distributed file systems (eg., HDFS), Cloud computing (eg., Amazon EC2), inmemory cluster computing (eg., Spark), parallel computing (eg., Pig), emergence of NoSQL frameworks, advancement in machine learning algorithms (eg., Support Vector Machines, Deep Learning, Auto-Encoders, Random Forest) have brought big data processing a reality. Despite the growth in these technologies and algorithms to handle big data, there are there are few limitations, which includes.
1. Scalability and Storage Issues: The rate of increase in data is much faster than the existing processing systems. The storage systems are not capable enough to store these data (Chen et al., 2014; Li and Lu, 2014; Kaisler et al., 2013; Assunção et al., 2015). There is a need to develop a processing system that not only caters to today's needs but also future needs.
2. Timeliness of Analysis: The value of the data decreases over time. Most of the applications like fraud detection in telecom, insurance and banking, require real time or near real time analysis of the transactional data (Chen et al., 2014; Li and Lu, 2014).
3. Representation of Heterogeneous Data: Data obtained from various sources are heterogeneous in nature. Unstructured data like Images, videos and social media data cannot be stored and processed using traditional tools like SQL. Smartphones now record and share images, audios and videos at an incredibly increasing rate, forcing our brains to process more. However, the process for representing images, audios and videos lacks efficient storage and processing (Chen et al., 2014; Li and Lu, 2014; Cuzzocrea et al., 2011).
4. Data Analytics System: Traditional RDBMS are suitable only for structured data and they lack scalability and expandability. Though non-relational databases are used for processing unstructured data, but there exist problems with their performances.
CONCLUSION
Today’s technology landscape is changing fast. Organizations of all shapes and sizes are being pressured to be data driven and to do more with less. Even though big data technologies are still in a nascent stage, relatively speaking, the impact of the 3V’s of big data, which now is 5v’s cannot be ignored. The time is now for organizations to begin planning for and building out their Hadoop-based data lake. Organizations with the right infrastructures, talent and vision in place are well equipped to take their big data strategies to the next level and transform their businesses. They can use big data to unveil new patterns and trends, gain additional insights and begin to find answers to pressing business issues. The deeper organizations dig into big data and the more equipped they are to act upon what’s learned, the more likely they are to reveal answers that can add value to the top line of the business. This is where the returns on big data investments multiply and the transformation begins. Harnessing big data insight delivers more than cost cutting or productivity improvement but it definitely reveals new business opportunities. Data-driven decisions always tend to be better decisions.
REFERENCES
Agneeswaran, V., 2012. Big-data - Theoretical, engineering and analytics perspective. In: Srinivasa, S., Bhatnagar, V. (Eds.), Big Data Analytics. Vol. 7678 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 8-15.
Agrawal, R., Srikant, R., 1994. Fast algorithms for mining association rules in large databases. In: Proceedings of the 20th International Conference on Very Large Data Bases. (VLDB „94). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, pp. 487-499.
Anastasia, February 2015. Big data and new product development. Entrepreneurial Insights http://www.entrepreneurial-insights.com/big-data-new-product-development/, Accessed on June 15, 2015.
Assunção, M. D., Calheiros, R. N., Bianchi, S., Netto, M.A.S. , Buyya, R., 2015. Big Data computing and clouds: Trends and future directions, Journal of Parallel and Distributed Computing, Volumes 79–80, May, Pages 3-15.
Bakshi, K., 2012. Considerations for big data: Architecture and approach. In: 2012 IEEE Aerospace Conference, Big Sky Montana. pp. 1-7. Beck, J., Mostow, J., 2008. How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students. In: Proceedings of the 9th International Conference on Intelligent Tutoring Systems. pp. 353-362.
Benjamins, V. R., 2014. Big data: From hype to reality? In: Proceedings of the 4th International Conference on Web Intelligence, Mining and Semantics (WIMS14). ACM, New York, NY, USA, pp. 2:1-2:2.
Bhadani, A., 2011. Cloud Computing and Virtualization. 1,Saarbrucken: VDM Verlag Dr. Muller Aktiengesellschaft & Co. KG. 116 s. ISBN 9783639347777
Bhadani, A., and Chaudhary, S., 2010. Performance evaluation of web servers using central load balancing policy over virtual machines on cloud, Proceedings of the Third Annual ACM Conference, Bangalore, ACM
Blount, M., Ebling, M., Eklund, J., James, A., McGregor, C., Percival, N., Smith, K., Sow, D., 2010. Real-time analysis for intensive care: Development and deployment of the Artemis analytic system. IEEE Engineering in Medicine and Biology Magazine 29 (2), 110-118.
Boyd, D., Crawford, K., 2011. Six provocations for big data. In: A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society.
Boyd, D., Crawford, K., 2012. Critical questions for big data. Information, Communication & Society 15:5, 662-679.
Brewer, E. A., 2000. Towards robust distributed systems (abstract). In: Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing (PODC '00). ACM, New York, NY, USA, p.7.
Brown, B., Chui, M., Manyika, J., October 2011. Are you ready for the era of big data? McKinsey Quarterly, http://www.mckinsey.com/insights/strategy/are you ready for the era of big data, Accessed on: Jan 20, 2015.
Chen, M., Mao, S., Liu, Y., 2014. Big data: A survey. Mobile Networks and Applications 19 (2), 171-209.
Coumaros, J., de Roys, S., Chretien, L., Buvat, J., KVJ, S., Clerk, V., Auliard, O., 2014. Big data alchemy: How can banks maximize the value of their customer data? Capgemini Consulting White Paper https://www.capgemini.com/resources/big-data-customer-analytics-in-banks, Accessed on: March 20, 2015.
Cukier, K., 2010. Data, data everywhere: A special report on managing information. The Economist http://www.economist.com/node/15557443, Accessed on: June 15, 2015.
Cuzzocrea, A., Song, I.-Y., Davis, K. C., 2011. Analytics over large-scale multidimensional data: the big data revolution! In: Proceedings of the ACM 14th International workshop on Data Warehousing and OLAP (DOLAP '11). ACM, New York, NY, USA, pp. 101-104. Dhar V., 2013. Data science and prediction. Communications of ACM 56(12), 64–73. Facebook, 2015. http://www.statisticbrain.com/facebook-statistics/, accessed on February 20, 2016.
Fisher, D., DeLine, R., Czerwinski, M., Drucker, S., 2012. Interactions with big data analytics. Interactions 19 (3), 50-59. Forsyth, C., January 2012. For big data analytics there‟s no such thing as too big. Cisco White paper http://www.cisco.com/en/US/solutions/ns340/ns517/ns224/big data wp.pdf, Accessed on Feb 20, 2015.
Foster, I., Zhao, Y., Raicu, I., Lu, S., 2008. Cloud computing and grid computing 360-degree compared. In: Grid Computing Environments Workshop, 2008 (GCE '08). pp. 1-10.
Gandomi, A., Haider, M., 2015. Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management 35 (2), 137-144.
Gantz, J., Reinsel, D., 2011. Extracting value from chaos. Tech. rep., IDC. Garlasu, D., Sandulescu, V., Halcu, I., Neculoiu, G., Grigoriu, O., Marinescu, M., Marinescu, V., 2013. A big data implementation based on grid computing. In: 11th Roedunet International Conference (RoEduNet), Sinaia. pp. 1-4.
Gilbert, S., Lynch, N., 2002. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33 (2), 51-59.
Jin, X., Wah, B. W., Cheng, X., Wang, Y., 2015. Significance and challenges of big data research. Big Data Research 2 (2), 59-64.
Jothimani, D., Bhadani, A.K., Shankar, R., 2015. Towards Understanding the Cynicism of Social Networking Sites: An Operations Management Perspective, Procedia - Social and Behavioral Sciences, 189, 117–132
Kaisler, S., Armour, F., Espinosa, J., Money, W., 2013. Big data: Issues and challenges moving forward. In: 46th Hawaii International Conference on System Sciences (HICSS), Hawaii. pp. 995-1004.
Khan, N., Yaqoob, I., Hashem, I.A.T. et al., 2014. Big Data: Survey, Technologies, Opportunities, and Challenges. The Scientific World Journal, vol. 2014, Article ID 712826, 18 pages
Laney, D., 2001. 3D data management: Controlling data volume, velocity and variety. Tech. Rep. 949, META Group.
Li, H., Lu, X., 2014. Challenges and trends of big data analytics. In: Ninth International, Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), Guangzhou China. pp. 566-567.
Maletic, J., Marcus, A., 2000. Data cleansing: Beyond integrity analysis. In: Proceedings of the Conference on Information Quality (IQ2000).
Manovich, L., 2011. Trending: the promises and the challenges of big social data. In: Gold, M. K. (Ed.), Debates in the Digital Humanities. The University of Minnesota Press, Minneapolis, MN. Available at: http://www.manovich.net/DOCS/Manovich_trending_paper.pdf, Accessed on: 15 July 2015
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Byers, A. H., 2011. Big data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.
Meek, T., February 2015. Big data in retail: How to win with predictive analytics. Forbes http://www.forbes.com/sites/netapp/2015/02/18/big-data-in-retail/, Accessed on: May 27, 2015.
Meijer, E., 2011. The world according to LINQ. ACM Communications 54 (10), 45-51.
Miller, H. G., Mork, P., 2013. From data to decisions: A value chain for big data. IT Professional 15 (1), 57-59.
Minelli, M., Chambers, M., Dhiraj, A., 2013. Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today‟s Businesses, 1st Edition. Wiley Publishing, Wiley CIO Series.
Mor, Y., 2014. Big data and law enforcement: Was „minority report‟ right? Wired, http://www.wired.com/2014/03/big-data-law-enforcement-minority-report-right/, Accessed on: June 20, 2015
Mukherjee, A., Pal, A., Misra, P., 2012. Data analytics in ubiquitous sensor-based health information systems. In: 6th International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST). pp. 193-198.
ParAccel, 2012. Hadoops limitations for big data analytics. ParAccel White Paper. http://www.paraccel.com/resources/Whitepapers/Hadoop-Limitations-for-Big-Data-ParAccelWhitepaper.pdf
Porter, M., 1980. Competitive Strategy: Techniques for Analyzing Industries and Competitors. The Free Press, New York.