Abstract ground-breaking solutions. Introduction The development of big data


The application of big data technology
in the various disciplines of work is progressively gaining global attention. Over
the years, many companies have struggled managing complex and different volumes
of data due to the lack of robust technology system in place. This has affected
operations and decision making. However, the advent of big data analytics paved
way for a much simplified technique of analyzing large and multifaceted data. As
an analytical tool, big data technology has afforded diverse methods of
processing and analyzing data. More significantly, the scope of the paper will
include the present state of the art of big data while discussing the
effectiveness and limitations of big data. Also, the primary focus of this
paper will be assessing the impact of big data amidst rising challenges. Although
efficient and effective in its applications, big data still needs improvement
in order to survive certain challenges. Going forward, more lights would be
shed on these issues to help design ground-breaking solutions. 

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!

order now



The development of big data technology
dates back to the mid 1990’s. Also known as business analytics, John Mashey, a
retired scientist among many others was the first to employ such technology
(Kitchin & McArdle, 2016, p. 1). Big data can be defined as a method of
collecting huge and varying sets of data, both structured and unstructured from
different sources for processes and analysis to help identify patterns of meaningful
information to enhance decision making. In the past, many companies who dealt
with volumes and varieties of information faced problems organizing,
processing, and analyzing them. This was partially due to the absence of a
sophisticated data analytic system in place. Big data, since its establishment,
has changed the way that information is collected, processed and analyzed.  From complex to simple, the impact of big data
is evident in many areas, including healthcare, education, finance, retail, and
many others.

According to a business article, big
data if prudently applied, has great value (McGuire,  Manyika, & Chui, 2012).
Also, it is reported that big data if resourcefully used to improve the quality
of care in the healthcare business, could yield annual value of $300 billion (McGuire, Manyika, & Chui, 2012). The attention
gained recently by big data is quite enormous. In reference to such awareness,
IEEE Big Data Service, as part of its objectives for the upcoming International
conference in 2018 is determined to promote more research programs and create a
platform that will motivate the exchange of innovative ideas (IEEE Computer
Society, 2017). Living in a world that is increasingly
driven by data, the demand for big data analytics will remain significant but
notwithstanding, challenges that undermine the effective and efficient use of
big data is steadily growing and needs to be addressed. Giving these issues
attention however, will create opportunities to advance big data applications.
While reflecting on some of the positive impact of big data, technical issues
involving size of data, inadequate storage capacity, security and data
disintegration would be thoroughly discussed along with proposed solutions
throughout the paper. 


of Big Data

impact of big data on major institutions and industries to begin with, is
noteworthy. For instance, big data has contributed to the preservation of
numerous business reputations through the process of identifying patterns of
errors in a data and resolving them. Sometimes traces of errors can affect
sound decision making and profit margin of a business. The capability of big
data in tracking these errors could enhance decision making and also increase
revenue. In addition, big data is very useful in identifying fraud during
analysis; allow immediate and right measures to be effected in real time.  Further, having high volumes of data could
make it vulnerable for fraudulent use, especially without employing the right
system to speed the process. Data could easily be manipulated and used for the
wrong reasons. With the business field constantly growing competitive, big data
analysis helps in monitoring competition in the market and facilitates
decisions that could leverage such competition. Also, big data helps provide “a
detailed picture of competitors, such as launching a new product, lowering or increasing
prices for a particular duration or focusing on users from a specific location”
(Pal, 2017).

Furthermore, big data is an
effective technological system that is cost effective. Enterprise and
individual owned organizations that use big data have been able to manage cost
while improving revenue. For instance, using big data, large number and
varieties of data could be analyzed at faster and steady rates while
controlling cost. Nevertheless, other technological systems that perform
similar functions are very expensive. Big data application in businesses helps
save cost on labor. For example, employing big data technology could reduce
pressure and demand for IT personnel. This allows companies to utilize extra resources
on areas in the company where necessary. From a business standpoint, big data
when utilized at its maximum capability could save cost. Cost savings on big
data technology however, could as well be invested to improve operations of a
company. Additionally, cost reduction in big data is when the system is
utilized well to prevent anomalies and also to correct errors. This however,
will inform better decision making and improve profits.  

In the area of healthcare business
for example, big data has enhanced analytics in identifying people at risk. The
application of big data in healthcare helps to design the right intervention
measures and makes it accessible. For example, in hospitals, big data
technology makes it possible for information to be made available and accessible
on wireless devices and makes it easy to monitor (Berg, 2015). The impact of
big data in delivering correct intervention in the health business helps people
to understand their own risks; and is able to share important information with
their primary care providers (Berg, 2015). Big data also provides system
coordination in healthcare. This way, physicians, nurses, and patients will be
able to work together without any difficulties (Berg, 2015). Through big data,
healthcare analytics has improved programs and the opportunity to develop new
ones. Additionally, the healthcare industry has experienced a remarkable
reduction in cost using big data.

Big data is essential for data
integration. Data integration is important in any organization, especially when
dealing with huge amounts and varieties of information both internally and
externally. Sometimes accessing variable amount of data that is disorganized
and unstructured could be challenging. Analyzing and transferring becomes an
issue when data is not arranged in the order that it is supposed to. Many
companies have lost vital information due to the influx of high volumes of
disorganized data. Nevertheless, the implementation of big data allows for unstructured
data to be simplified and integrated. The integration that big data offers
makes data available and easily accessible. It also ensures flexibility in
transferring data either internal or external. Moreover, the structure that big
data affords data analysis eliminates delays while facilitating easy and smooth
transmission of data (The Benefits of Big Data, 2013).

Challenges and Solutions

On the Contrary, while the impact
of big data technology is real and evident, some technical challenges that
interfere with efficient application of big data needs attention. For instance,
the stages involved in big data analysis are very distinct and diverse.
However, each of the stage has its own intricacies. For example, there is a
growing concern over the amount of effort and time spent on analysis or
modeling stage while giving less attention to the subsequent phases of data
analysis although, they are important too (Agrawal et al). The problem here is
that certain elaborate issues in the “context of multi-tenanted clusters where
several users’ programs run concurrently” (Agrawal et al, 2011) are
inadequately understood.

Regarding this problem, big data
needs to be managed under a condition that is very noisy and variegated as it
makes it possible to identify errors and also deal with any uncertainties
(Agrawal et al, 2011). Again, considering the different phases during big data
analysis, it would be important to understand and address critical oversights least
discussed yet, significant for success. Additionally, adequately understanding
the processes well could help minimize errors in data analysis and facilitate system

Another compelling challenge
surrounding big data technology is inadequate storage capacity. Storage is a
major conundrum for companies using big data analysis because of the size and
types of data involved. Research indicates that most companies still rely on
the traditional way of employing hard disk drive in storing data (slack, 2017).
Although good and cost effective, the process tends to be slow and time
consuming which may affect company’s operations. In contrast, considering an
alternative means of obtaining much improved data storage seems expensive for
some companies’ especially individual operating firms. Storage is very
important during big data analysis as all data needs to be securely stored to
keep data safe from both internal and external threats.

Resolving problems associated with
storage in big data is necessary. As the expansion of data sets (structured or
unstructured) grows continuously diverse, the less capable existing storage systems
can manage (Slack, 2012). Considering this problem, big data storage should
have the ability to scale and augment the capacity in modules without causing
any system dysfunction. Interestingly, scale-out storage is progressively
becoming alternate source of storage for big data (Slack, 2012). For example, scale-out
clustered architecture allows range of storage capacity with enhanced processing
power and connectivity that can function easily without incorporating the
traditional storage systems (Slack, 2012). Another storage system that can
offer big data the flexibility to increase file numbers to billions without
facing similar challenges as the traditional storage system is object-based storage
architecture (Slack, 2012). Object-based storage system is capable of scaling geographically,
while allowing widespread of infrastructures to numerous destinations (Slack,

Furthermore, dealing with
different kinds of data often times create difficulty integrating separate data
sources. This problem is mostly encountered in big data analysis because data
comes from different sources such as “enterprise applications, social media
streams, email systems, employee-created” and many others (Harvey, 2017). The
challenge associated with consolidating data tends to affect big data analysis.
 Without data consolidation, it is
impossible to generate important results to affect good decision. Many
enterprise companies are turning away from big data and considering a new technology
that has high data integrated capacity (Harvey, 2017). There is no doubt that a
highly disorganized data does not only make quantification difficult, but also makes
data vulnerable to internal and external malicious attacks.  

The constant patronage of big data
technology by organizations is increasing because of its diverse impact on
businesses. In spite of the benefits, big data still needs improvement in data
integration. This has been a conundrum for companies using the system because
of the increase in information. With businesses seeking to improve operations,
customer retention, product extension, and minimize big data analytics
problems, new integration methods ought to be adopted rather than the
traditional data integration approach (Data Virtualization, 2017). Cisco data
virtualization platform for instance, offers several approaches that are robust
and enhance high functioning data integration in big data (Data Virtualization,
2017). For example, massively parallel processing based appliances such as EMC
Greenplum, HP vertica, IBM Netezza, SAP Sybase IQ and many others provides
efficient data consolidation capability (Data Virtualization, 2017). According
to an IDG report, almost eighty-nine percent of companies are planning on
investing in a refined big data tools to enhance its integrating capability
(Harvey, 2017).

Additionally, security is also a
major worry for companies that use big data. Big data storage is increasingly
becoming the target for hackers as hackers continue to hunt big companies in
the area of security to make sure their data remains vulnerable. Other
malicious attacks is also on the rise although, many big data managers are
convinced about the level of security they provide for data. For example, highly
disintegrated information from varying sources is sometimes difficult to keep track,
allowing several possibilities for infiltration by hackers (Harvey, 2017). Further,
displeased employees could maliciously compromise the security of big data purposely
without any remorse or being held accountable for it. Privacy invasion is
another security concern in big data as it undermines confidentiality of stored

Security issues concerning big
data could be resolved using different mechanisms. For example, companies using
big data technology could implement security mechanisms such as Mandatory
Access Control (MAC) to restrict employees’ access to a defined set of tasks. Implementing
that would help control employees access levels, especially to classified
information (Parms, 2017). Mandatory access control “adds  labels to all file system objects defining
the appropriate access for each object, and users appropriately defined access”
(Parms, 2017). This will however, strengthen security and privacy of data and
alleviate data vulnerability to internal and external threats. Another important
security measure is to ensure that data is encrypted at every stage. This must
apply to data in transit and data at rest to make sure that information is
completely protected (Parms, 2017). Constant monitoring of big data activities
is also essential to data security. It could be done by continuously checking
logs and making sure there are no contradictions or anomalies. This type of
security action could help identify any type of threat that may hinder
efficient use of big data.


As the demand for big data
increases, so as it’s compounding problems. There will always be problems with
information, especially regarding the size of data produced by organizations
globally. The problem is not necessarily the volume of data, but is mostly
driven by the nature and complexity of organization managing it. Understanding
how big data applies in every aspect of analysis is fundamental to efficient
use of the technology. Using big data effectively eliminates all doubts
concerning the technical capability of this technology.  

 Users of big data technology have done little
to understand the full potential of the system, therefore, are unable to
implement it well enough to achieve desired goals and objectives. For example,
one purpose for employing big data is to help improve decision making. However,
decision making driven by big data could either progress or impede an
organization’s missions. This also means an organization using big data could
positively or negatively impact revenue depending on how it is applied. A data
whether structured or unstructured could be simplified using big data. Big data
through different stage processes is able to monitor and identify oversights
which as a result influence smart decision making. As many organizations seek
to increase its dependence on big data, it is also important to search for proven
researched innovations in order to improve on the resourcefulness of big data.


Data continues to grow almost
everywhere in the world because companies and organizations keeps creating
them; thus making the demand for big data essential. The dynamics of big data in
organizations are significant as the benefits are numerous. Capable of
analyzing structured and unstructured data, big data has been valuable in cost reduction,
data integration, identifying anomalies, and contributes to smart decision making
while improving revenues of an organization. Again, big data despite its
relevance in the world of business is faced with challenges and limitations
which seem to devalue its importance and reliance. Limitations such as data insecurity,
insufficient storage space, disintegrated data, and inadequate understanding of
the process, encumber the progress of big data as the intricacies of big data
is driven by the nature of organization managing it.  

As these problems seem to create
doubts about the dependability of big data, well thought solutions of variable importance
was also shared to help maintain the value of big data. A solution such as
obtaining a better understanding of how big data operates was vital to effective
application of the system. In addition, data encryption was emphasized as an
important security measure using big data. As the size of data keeps
increasing, encryption was the best means of ensuring data security and also
minimizing data vulnerabilities. Again, with the growing size of data, data
integration became critical to organizational information security. In spite of
the benefits and challenges of big data, the prospects of this technology in
the future is great because more research platforms are created to motivate
diverse innovative ideas to improve and maintain this technology.