Internet of Things,Big Data, Cloud Computing : The Perfect Match

What Is Internet of Things and How Does It Work?

Big Data is quickly becoming the next big asset for many organizations.  It would not be surprising for organizations to begin selling data of all types, including the metrics, knowledge, and insights gained from the data accumulated and analyzed. Riding on this wave of Big Data is Internet of Things (IoT).

Technology, along with low storage costs, is making it easy for organizations to process large amounts of data; as a result, a new trend is emerging known as “capture it all.”  Capture it all means collecting as much data about your customers’ product usage and behaviors as possible because the data collected may be useful in the future.  At a time when organizations are seeing the benefits of Big Data, IoT provides innovative ways of capturing data that can enhance these benefits.

Internet of Things is the concept that things (animals, people or objects) with a unique identifier can automatically transfer data over a network without human interaction.   There are a number of devices that can be connected to the Internet to create a network of ‘things’ that communicate with each other to make intelligent decisions.  This is nothing new; the concept of Ubiquitous computing and sensor networks has been in use for a long time.

Why the Buzz Now?

Four technology trends are fueling the IoT revolution and renewing interest, they include:

  1. Big Data: Big Data’s success is making people realize the value of data, including the ways to identify valuable insights from data once considered junk.   IoT deployments can produce huge amounts of data, as sensors are constantly sensing stimuli and triggering real-time events.  It becomes relatively easy for the data to get accumulated over-time and the big data ecosystem or platform makes it easy to process these huge amounts of data.
  2. Cloud Computing: With the advent of Cloud computing, computing power and data storage has become cheaper and easier to store and process data in Cloud.
  3. Ubiquitous connectivity: With the increase in the usage of smart phones with data plans along with the demand for connectivity to the Internet over smart phones, the infrastructure has been upgraded. Many IoT architectures are piggy backing on connectivity of the smart phone.
  4. Low cost sensors: The cost of Wi-Fi sensors and devices is in gradual decline. Standards like Near Field Communication (NFC), and iBeacon are becoming mainstream and supported by smart phones. As a result, App Developers can use them to creatively build IoT use cases. These improvements in the Bluetooth technology, Bluetooth low energy or BLE, are also becoming a catalyst to the IoT revolution.  In addition to the above, the ability of improved sensors to discreetly capture data is also a stimulus.

How Internet of Things Is Used Today

Internet of Things deployments implemented right have the potential to become Big Data’s killer Application.  Refer to figure 1 for the architecture used for typical IoT deployments.



Sensors that allow sensing of events are delivered to the mother ship on the cloud via servers connected to the Internet.  Data from these sensors is communicated via BLE and temporarily stored on the smartphone.  The use cases and potential for IoT and Big Data is endless as well as incredible.

A number of products based on IoT are getting launched and also receiving overwhelming response and adoption.  Wearable devices like Fitbit, Basis, Smartwatch from Samsung, and Qualcomm are playing an increasing part in IoT awareness.   Google became an early adaptor of IoT with its’ acquisition of the Nest Labs – Smart Thermostat.  Also, Google is gradually getting into home automation with their set-top boxes, NEST, Google Fiber, and Smart watches powered by Android OS.  Currently devices like Fitbit continuously and discreetly capture activity levels and sleep quality and then transmit the data to cloud using the smart phone. The sensors in turn communicate the data to smartphone through BLE.  Just imagine if there is a way to start tracking blood pressure, anxiety levels, stress levels, and heart rate in a similar discreet manner.  We can have personal data that is collected about ones self and then used by physicians in predicting a change in daily routine that can be causing current health issues.  For example, if a patient is unable to sleep properly, having the data collected historically could be invaluable in predicting what could have caused the current problem.  Tools like GOOGLE Nest Smart thermostat, Smart Smoke and CO alarm constantly track or monitor the environment in the house including information about lighting, humidity, daily behavior of the home’s residents, temperature, and air quality.  Smartphone apps like Easily Do also discreetly record day to day activities of the owner by simply keeping track of the GPS on the cell phone.

Potential Uses for Internet of Things

Imagine a world where devices can talk to each other, communicate and exchange information, and make intelligent decisions based on the data collected. For example, if you are coming home from a workout and, based on the data from your Fitbit and Easily Do, information is communicated to Google Nest thermostat that you would be home in a few minutes.  This information can be used to make your house more comfortable and cooler upon your arrival.  Taking this example a little further. What if data is captured from many people and made available to researchers (After anonymizing of Personal Identification Information).  This captured data can provide researchers and scientists with valuable data to study and find correlations between activities/actions that cause people to be susceptible to diseases.  This data can then provide feedback to users who may be susceptible to a disease and allow preventive measures to begin. In addition, trends/patterns may be identified that enable researchers to identify the reasons for diseases like heart attacks and Parkinson’s.  In addition, many of us may have seen sci-fi movies where an Artificial Intelligence system talks back and gives advice by analyzing a situation. Those days may not be far off due to the way technology trends in regards to IoT, Cloud computing, and Big Data are coming together.

Internet of Things’ Future

Gartner predicts that Internet of Things will affect every industry.  As a result, finding top analytics talent qualified to manage massive amounts of data will be difficult in the years ahead.  A yearlong research project conducted by Accenture shows that the United States is projected to create nearly 39,000 new jobs for analytics experts through 2015.  Only 23 percent if these jobs will be filled by qualified candidates. Cisco’s CEO, John Chambers, predicts that during the next decade the impact of Internet of Things will be 5 to 10 times greater than the Internet was on society and believes that IoT opens up a $19 Trillion opportunity during the same period.

IoT is here to stay and will make Big Data even bigger.  Our challenge, as IT professionals, is to discover innovative ways to use this technology that will enhance the general population’s lifestyle as well as benefit companies bottom line.

Simpson, Eh?

Continuing my series on errors in analysis, I’m going do dig into a tricky issue of correlation analysis.

To picture the problem, let’s say I work at a university. It’s called Mock U. Like a lot of universities, Mock U is trying to foster a diverse workforce. In particular, we want to ensure that our faculty positions are filled by more women than men to close an existing gender gap.

Let’s look at the jobs offered to applicants by gender in two colleges:

Men Women Adv
Hired Applied % Hired Applied %
Engineering 2 25 8% 3 30 10% w
Business 5 6 83% 4 4 100% w

Are we meeting our objectives? In Engineering, 25 men applied and 2 were hired (8%). Compare that to the 3 women hired from a pool of 30 applicants (10%). The engineering college favors women in hiring. In business, 5 men were hired from among 6 applicants (83%) while all 4 women who applied were hired (100%). The College of Business favors women in its hiring practices too. This gender gap should be closed in no time, right?

But what happens when we look at both departments together?

Mock U hired a higher percentage of men who applied than women. That’s odd. Can we still say the system is working?

This is known as Simpson’s Paradox. I haven’t found it as common as Berkson’s Paradox, but I think it’s more confusing.

This occurs when you have two populations of data with the same correlation, but there’s a confounding factor. The two populations in this case are different colleges, and the hiring correlation for each of them is positive (my policy appears to be selecting more women from pools of job applicants).

But there’s a confounding factor that’s making the University’s hiring proportion as a whole move in the opposite direction.

Can you see the confounding factor in this example?

It could be a lot of things, but at a glance, it looks like jobs in the College of Engineering are a lot more competitive than jobs in the College of Business (Engineering only hired 9% of applicants while Business hired 69%).

You have slightly more women applying to a much more competitive position, and slightly more men applying to a less competitive one. That’s a recipe for encountering this paradox, and it’s worth watching for as you slice your data.

You have to frame your question carefully to avoid ambiguous answers like this. If our objective were to reshape the hiring of either department alone, we could claim success. If we want to change the whole university, we still have work to do.

This can be a pain in analysis but it can also be useful. It can be helpful to look deeper at a process to see if there’s agreement at multiple levels. Or it can help you notice factors in your data you hadn’t thought of before. But mostly it’s useful for starting arguments about whether it’s even possible for sub-groups to all have a positive correlation while the population at large has a negative one. You can try it with your friends, but don’t say I didn’t warn you.