AbstractThe application of big data technologyin the various disciplines of work is progressively gaining global attention. Overthe years, many companies have struggled managing complex and different volumesof data due to the lack of robust technology system in place. This has affectedoperations and decision making. However, the advent of big data analytics pavedway for a much simplified technique of analyzing large and multifaceted data. Asan analytical tool, big data technology has afforded diverse methods ofprocessing and analyzing data.
More significantly, the scope of the paper willinclude the present state of the art of big data while discussing theeffectiveness and limitations of big data. Also, the primary focus of thispaper will be assessing the impact of big data amidst rising challenges. Althoughefficient and effective in its applications, big data still needs improvementin order to survive certain challenges. Going forward, more lights would beshed on these issues to help design ground-breaking solutions. IntroductionThe development of big data technologydates back to the mid 1990’s. Also known as business analytics, John Mashey, aretired scientist among many others was the first to employ such technology(Kitchin & McArdle, 2016, p.
1). Big data can be defined as a method ofcollecting huge and varying sets of data, both structured and unstructured fromdifferent sources for processes and analysis to help identify patterns of meaningfulinformation to enhance decision making. In the past, many companies who dealtwith volumes and varieties of information faced problems organizing,processing, and analyzing them. This was partially due to the absence of asophisticated data analytic system in place. Big data, since its establishment,has changed the way that information is collected, processed and analyzed. From complex to simple, the impact of big datais evident in many areas, including healthcare, education, finance, retail, andmany others.
According to a business article, bigdata if prudently applied, has great value (McGuire, Manyika, & Chui, 2012).Also, it is reported that big data if resourcefully used to improve the qualityof care in the healthcare business, could yield annual value of $300 billion (McGuire, Manyika, & Chui, 2012). The attentiongained recently by big data is quite enormous. In reference to such awareness,IEEE Big Data Service, as part of its objectives for the upcoming Internationalconference in 2018 is determined to promote more research programs and create aplatform that will motivate the exchange of innovative ideas (IEEE ComputerSociety, 2017). Living in a world that is increasinglydriven by data, the demand for big data analytics will remain significant butnotwithstanding, challenges that undermine the effective and efficient use ofbig data is steadily growing and needs to be addressed.
Giving these issuesattention however, will create opportunities to advance big data applications.While reflecting on some of the positive impact of big data, technical issuesinvolving size of data, inadequate storage capacity, security and datadisintegration would be thoroughly discussed along with proposed solutionsthroughout the paper. Importanceof Big Data Theimpact of big data on major institutions and industries to begin with, isnoteworthy. For instance, big data has contributed to the preservation ofnumerous business reputations through the process of identifying patterns oferrors in a data and resolving them. Sometimes traces of errors can affectsound decision making and profit margin of a business. The capability of bigdata in tracking these errors could enhance decision making and also increaserevenue. In addition, big data is very useful in identifying fraud duringanalysis; allow immediate and right measures to be effected in real time.
Further, having high volumes of data couldmake it vulnerable for fraudulent use, especially without employing the rightsystem to speed the process. Data could easily be manipulated and used for thewrong reasons. With the business field constantly growing competitive, big dataanalysis helps in monitoring competition in the market and facilitatesdecisions that could leverage such competition.
Also, big data helps provide “adetailed picture of competitors, such as launching a new product, lowering or increasingprices for a particular duration or focusing on users from a specific location”(Pal, 2017). Furthermore, big data is aneffective technological system that is cost effective. Enterprise andindividual owned organizations that use big data have been able to manage costwhile improving revenue. For instance, using big data, large number andvarieties of data could be analyzed at faster and steady rates whilecontrolling cost. Nevertheless, other technological systems that performsimilar functions are very expensive.
Big data application in businesses helpssave cost on labor. For example, employing big data technology could reducepressure and demand for IT personnel. This allows companies to utilize extra resourceson areas in the company where necessary. From a business standpoint, big datawhen utilized at its maximum capability could save cost.
Cost savings on bigdata technology however, could as well be invested to improve operations of acompany. Additionally, cost reduction in big data is when the system isutilized well to prevent anomalies and also to correct errors. This however,will inform better decision making and improve profits. In the area of healthcare businessfor example, big data has enhanced analytics in identifying people at risk. Theapplication of big data in healthcare helps to design the right interventionmeasures and makes it accessible. For example, in hospitals, big datatechnology makes it possible for information to be made available and accessibleon wireless devices and makes it easy to monitor (Berg, 2015).
The impact ofbig data in delivering correct intervention in the health business helps peopleto understand their own risks; and is able to share important information withtheir primary care providers (Berg, 2015). Big data also provides systemcoordination in healthcare. This way, physicians, nurses, and patients will beable to work together without any difficulties (Berg, 2015). Through big data,healthcare analytics has improved programs and the opportunity to develop newones.
Additionally, the healthcare industry has experienced a remarkablereduction in cost using big data. Big data is essential for dataintegration. Data integration is important in any organization, especially whendealing with huge amounts and varieties of information both internally andexternally. Sometimes accessing variable amount of data that is disorganizedand unstructured could be challenging. Analyzing and transferring becomes anissue when data is not arranged in the order that it is supposed to.
Manycompanies have lost vital information due to the influx of high volumes ofdisorganized data. Nevertheless, the implementation of big data allows for unstructureddata to be simplified and integrated. The integration that big data offersmakes data available and easily accessible. It also ensures flexibility intransferring data either internal or external. Moreover, the structure that bigdata affords data analysis eliminates delays while facilitating easy and smoothtransmission of data (The Benefits of Big Data, 2013).Challenges and SolutionsOn the Contrary, while the impactof big data technology is real and evident, some technical challenges thatinterfere with efficient application of big data needs attention. For instance,the stages involved in big data analysis are very distinct and diverse.However, each of the stage has its own intricacies.
For example, there is agrowing concern over the amount of effort and time spent on analysis ormodeling stage while giving less attention to the subsequent phases of dataanalysis although, they are important too (Agrawal et al). The problem here isthat certain elaborate issues in the “context of multi-tenanted clusters whereseveral users’ programs run concurrently” (Agrawal et al, 2011) areinadequately understood. Regarding this problem, big dataneeds to be managed under a condition that is very noisy and variegated as itmakes it possible to identify errors and also deal with any uncertainties(Agrawal et al, 2011).
Again, considering the different phases during big dataanalysis, it would be important to understand and address critical oversights leastdiscussed yet, significant for success. Additionally, adequately understandingthe processes well could help minimize errors in data analysis and facilitate systemprogress. Another compelling challengesurrounding big data technology is inadequate storage capacity. Storage is amajor conundrum for companies using big data analysis because of the size andtypes of data involved.
Research indicates that most companies still rely onthe traditional way of employing hard disk drive in storing data (slack, 2017).Although good and cost effective, the process tends to be slow and timeconsuming which may affect company’s operations. In contrast, considering analternative means of obtaining much improved data storage seems expensive forsome companies’ especially individual operating firms.
Storage is veryimportant during big data analysis as all data needs to be securely stored tokeep data safe from both internal and external threats. Resolving problems associated withstorage in big data is necessary. As the expansion of data sets (structured orunstructured) grows continuously diverse, the less capable existing storage systemscan manage (Slack, 2012). Considering this problem, big data storage shouldhave the ability to scale and augment the capacity in modules without causingany system dysfunction.
Interestingly, scale-out storage is progressivelybecoming alternate source of storage for big data (Slack, 2012). For example, scale-outclustered architecture allows range of storage capacity with enhanced processingpower and connectivity that can function easily without incorporating thetraditional storage systems (Slack, 2012). Another storage system that canoffer big data the flexibility to increase file numbers to billions withoutfacing similar challenges as the traditional storage system is object-based storagearchitecture (Slack, 2012). Object-based storage system is capable of scaling geographically,while allowing widespread of infrastructures to numerous destinations (Slack,2012). Furthermore, dealing withdifferent kinds of data often times create difficulty integrating separate datasources. This problem is mostly encountered in big data analysis because datacomes from different sources such as “enterprise applications, social mediastreams, email systems, employee-created” and many others (Harvey, 2017). Thechallenge associated with consolidating data tends to affect big data analysis. Without data consolidation, it isimpossible to generate important results to affect good decision.
Manyenterprise companies are turning away from big data and considering a new technologythat has high data integrated capacity (Harvey, 2017). There is no doubt that ahighly disorganized data does not only make quantification difficult, but also makesdata vulnerable to internal and external malicious attacks. The constant patronage of big datatechnology by organizations is increasing because of its diverse impact onbusinesses. In spite of the benefits, big data still needs improvement in dataintegration. This has been a conundrum for companies using the system becauseof the increase in information. With businesses seeking to improve operations,customer retention, product extension, and minimize big data analyticsproblems, new integration methods ought to be adopted rather than thetraditional data integration approach (Data Virtualization, 2017). Cisco datavirtualization platform for instance, offers several approaches that are robustand enhance high functioning data integration in big data (Data Virtualization,2017). For example, massively parallel processing based appliances such as EMCGreenplum, HP vertica, IBM Netezza, SAP Sybase IQ and many others providesefficient data consolidation capability (Data Virtualization, 2017).
Accordingto an IDG report, almost eighty-nine percent of companies are planning oninvesting in a refined big data tools to enhance its integrating capability(Harvey, 2017). Additionally, security is also amajor worry for companies that use big data. Big data storage is increasinglybecoming the target for hackers as hackers continue to hunt big companies inthe area of security to make sure their data remains vulnerable. Othermalicious attacks is also on the rise although, many big data managers areconvinced about the level of security they provide for data. For example, highlydisintegrated information from varying sources is sometimes difficult to keep track,allowing several possibilities for infiltration by hackers (Harvey, 2017). Further,displeased employees could maliciously compromise the security of big data purposelywithout any remorse or being held accountable for it. Privacy invasion isanother security concern in big data as it undermines confidentiality of storedinformation. Security issues concerning bigdata could be resolved using different mechanisms.
For example, companies usingbig data technology could implement security mechanisms such as MandatoryAccess Control (MAC) to restrict employees’ access to a defined set of tasks. Implementingthat would help control employees access levels, especially to classifiedinformation (Parms, 2017). Mandatory access control “adds labels to all file system objects definingthe appropriate access for each object, and users appropriately defined access”(Parms, 2017). This will however, strengthen security and privacy of data andalleviate data vulnerability to internal and external threats. Another importantsecurity measure is to ensure that data is encrypted at every stage. This mustapply to data in transit and data at rest to make sure that information iscompletely protected (Parms, 2017).
Constant monitoring of big data activitiesis also essential to data security. It could be done by continuously checkinglogs and making sure there are no contradictions or anomalies. This type ofsecurity action could help identify any type of threat that may hinderefficient use of big data. DiscussionsAs the demand for big dataincreases, so as it’s compounding problems. There will always be problems withinformation, especially regarding the size of data produced by organizationsglobally. The problem is not necessarily the volume of data, but is mostlydriven by the nature and complexity of organization managing it. Understandinghow big data applies in every aspect of analysis is fundamental to efficientuse of the technology. Using big data effectively eliminates all doubtsconcerning the technical capability of this technology.
Users of big data technology have done littleto understand the full potential of the system, therefore, are unable toimplement it well enough to achieve desired goals and objectives. For example,one purpose for employing big data is to help improve decision making. However,decision making driven by big data could either progress or impede anorganization’s missions. This also means an organization using big data couldpositively or negatively impact revenue depending on how it is applied. A datawhether structured or unstructured could be simplified using big data. Big datathrough different stage processes is able to monitor and identify oversightswhich as a result influence smart decision making. As many organizations seekto increase its dependence on big data, it is also important to search for provenresearched innovations in order to improve on the resourcefulness of big data.
ConclusionData continues to grow almosteverywhere in the world because companies and organizations keeps creatingthem; thus making the demand for big data essential. The dynamics of big data inorganizations are significant as the benefits are numerous. Capable ofanalyzing structured and unstructured data, big data has been valuable in cost reduction,data integration, identifying anomalies, and contributes to smart decision makingwhile improving revenues of an organization.
Again, big data despite itsrelevance in the world of business is faced with challenges and limitationswhich seem to devalue its importance and reliance. Limitations such as data insecurity,insufficient storage space, disintegrated data, and inadequate understanding ofthe process, encumber the progress of big data as the intricacies of big datais driven by the nature of organization managing it. As these problems seem to createdoubts about the dependability of big data, well thought solutions of variable importancewas also shared to help maintain the value of big data. A solution such asobtaining a better understanding of how big data operates was vital to effectiveapplication of the system. In addition, data encryption was emphasized as animportant security measure using big data. As the size of data keepsincreasing, encryption was the best means of ensuring data security and alsominimizing data vulnerabilities. Again, with the growing size of data, dataintegration became critical to organizational information security.
In spite ofthe benefits and challenges of big data, the prospects of this technology inthe future is great because more research platforms are created to motivatediverse innovative ideas to improve and maintain this technology.