BIG DATA - Key points to take into account

So far we have been writing about the added value BIG DATA provides and how it has been implemented in different industries. Today we are going to tell you about the challenges involved when applying BIG DATA into your businesses.  

A lot of businesses are moving their data bases to BIG DATA programs thinking it is an easy project to implement and that by placing all the information they have they will end up with amazing reports and results. In order for BIG DATA to work you first need to process all the data you have, this requires someone to manage and control all the information that is going to be migrated to your BIG DATA base. You cannot put random information and expect to get a great report out of it.   “Big Data is a powerful tool for inferring correlations, not a magic wand for inferring causality.”(Joost Swarte. http://www.newyorker.com/online/blogs/elements/2013/04/steamrolled-by-big-data.html) There are three key points a company should keep in mind; The amount of information that you currently have and the one that will be coming in (Volume), The speed at which this information will come from your providers, could be clients, records, etc. (Velocity) and also the diversity of the information you will have (Variety). 

You need professionals that can help you manage the three V’s of Big Data to obtain big results. It is a process that you will like to do it properly otherwise the end result could be information that has nothing to do with what you expected and a lot of money thrown away.

I will leave you with this short video that talks about the concepts that need to be taken into account.

Sources:

http://www.newyorker.com/online/blogs/elements/2013/04/steamrolled-by-big-data.html

http://www.youtube.com/watch?v=2qutaP5inAc

Big Data and Public Sector

In previous posts, we introduced several interesting big data applications in private sectors and the technologies behind big data. In the information era, facing large volume, high velocity and mixed variety of data, public sector organizations also realized the significance of big data analyzing capability for cost savings and efficiency improvement for their services to reach their goals.

Indeed, harnessing the power of big data technologies would offer enormous benefits to public sector. With big data technologies to track online forum and blog information, government ministry leaders can get more open feedback on their services and policies for improvement, make informed decisions and discover new services that they can provide, which they can hardly achieve through convention ways: this is especially beneficial to countries where citizens prefer online forum for expressing their opinions to traditional feedback channels, due to the political and social restrictions. Thus, big data provides assistance for government ministries to improve operation efficiency and transparency. McKinsey Global Institute’s research shows that big data technologies have the potential to reduce operation costs in Europe’s public sector by around 20%, which is about €300 billion every year [1]. Realizing the benefits, in Mar 2012, US government announced a Big Data initiative with 200 million budgets to improve the big data tools employed [1]. 

Public education institutions also embrace big data analytics to enhance customized e-learning experience for students. Since 2010, National University of Singapore (NUS) invested heavily in man-hours to analyze deeper into the huge amount of student data and hope to understand the user behaviors of its vibrant and diverse student population.  Nonetheless, this is an extremely labor intensive task. Its staff faced challenges in data mining and analyzing uncertainties of the data patterns found using traditional ways. In 2012, NUS deployed Microsoft big data software (SQL Server 2012 Cloud-ready platform) to capture real-time analytics from the campus net. With this solution, the staff can consolidate more accurate data sets and view them in an interactive graphical virtualization to obtain insights across their different level of operations. The data analysis process was claimed to speed up by 50% [2] so that the university can constantly improve their services to meet the need of its diverse groups of students. 

Besides cost reduction and efficiency improvement, big data also brings value to public organizations, especially government agencies, on fraud detection, treat identification and so on. With big data tools, they can track information much more easily to analyze abnormal activities of individual citizens to uncover or predict fraud and thread to society. Tax evasion may also be spot by tracking the social network media. 

Although big data tools deployment in public sector organizations has enormous benefits to fuel the service intelligence, it also has potential risks on which we have concerns. 

Those powerful tools may be misused by some public agency staff to breach the privacy of citizens. You will never know that you are being spied when you are taking to your loved ones. Hence, certain regulations are required to ensure proper usage of the big data tools and reach the original goals of cost savings and efficiency improvement for public sector organizations.

In the next post, we will discuss more on the risks of big data and application in insurance.

[1]  Big data: the next frontier for innovation, McKinsey Global Institute Report, 2011

[2] http://www.futuregov.asia/articles/2013/mar/20/singapore-university-embraces-big-data-analytics

The Technical Architecture of Big Data

In the time it took you to read this sentence, NASA gathered approximately 1.73 gigabytes of data from nearly 100 currently active missions. We do this every hour, every day, every year – and the collection rate is growing exponentially. Handling, storing, and managing this data is a massive challenge. 

The whole idea of big data is still relatively new. In the earlier blogs we have already seen what Big Data actually is and today we will be looking at a holistic view of the Big Data System and some of the technical terminology involved in Big Data.

Below are major Big Data Terms, which will help in understanding the above overview - 

Real Time Streams- These are various sources of information on Internet through which raw data is being made available.

Real Time Processing- Real time processing allows the user to sort through the massive amounts of data and produce information for analysis. While processing, data can be sorted and grouped based on algorithms, but it’s important to understand the limitations and constraints without applying human thought evaluation.

Data Visualization- As data is collected, stored and then analyzed, it needs to be presented in a way that it can be understood and digested. Through Data Visualization programs are able to analyze big data and represent it in a visual display for easier consumption and/or to show results.

Real time Structured Databases-Real time structured databases were created to manage volumes of data that do not have a fixed schema.

NoSQL gained popularity as major companies adopted the system due to an huge volume of data, which could not use the traditional RDBMS solutions. 

Batch processing- This is execution of a series of programs ("jobs") on a computer without manual intervention. Hadoop was developed to enable applications to work with thousands of computational independent computers and petabytes of data.

Interactive Analysis- Interactive Analysis tools dramatically reduces the time required for data analysts/scientists to discover, visualize and explore large volumes of diverse data.

Serialization (Structure & Unstructured Data)- Serialization is the process of converting data structure or object state into a format able to be stored. This stage occurs after the data is collected and when it is being processed. 

Cloud Infrastructure- Cloud infrastructure is the necessary infrastructure required supporting the storage and processing of the big data, which has been gathered. The cloud resources (hardware and software) are usually delivered as a service over a network (typically the Internet). 

References 

http://www.pentahobigdata.com/ecosystem/capabilities/instaview

http://www.greenplum.com/

http://open.nasa.gov/

http://www.wikipedia.org/

http://bigdataarchitecture.com/

Big Data in Business, you need to have it in your company !!!

If you are wondering why should I use “Big Data” in my company, here are some facts:         

1.     Revenue: By increasing the usability of data by just 10%, the average Fortune 100 company could expect an increase in revenues of $2 billion dollars (source: Fathom)

2.     Marketing: 91% marketing leaders believe successful brands use customer data to drive business decisions (source: BRITE/NYAMA)

3.     Competitive Advantage: 27% of organizations have a partial understanding of what Big Data is (source: CIO)

4.     Consumer Behavior: $600 billion potential annual consumer surplus from using personal location data globally (source: McKinsey)

In other words, you increase your revenues, increase your understanding/knowledge about customers, and most important gain a competitive advantage because 1 out of 3 organizations don’t understand big data. Here are some examples how you gain financial value:

And also, depending on which industry you are focused you can have greater gains, greater productivity, as the following charter shows:

After analyzing these charts, you realized it’s important for your business, but how exactly creates values? Here are 3 illustrations:

1.     Time to experiment: as the company collect more and more data, they can analyze it in different ways, approach to situations from different perspectives, and have the information in real time while the experiments are happening.

2.     Customization: After analyzing data you realize that you have a big universe to segment and target your customers. Then you can decide how do you want to approach to these customers, which marketing campaigns, promotions and advertisements work better.

3.     Innovation: when customers use the products/services, the people responsible for the designing and manufacturing of it can obtain more information to improve the product

If you want to see more about creating value, I recommend you to watch this video

º

In conclusion big data is key to your business, adds value through different uses and approaches and gives you competitive advantage.

ºDon’t miss the next posts where we will show another industry example and also go deeply into the technical area of Big Data