Google Stackdriver has thousands of build-in metrics to monitor everything from Kubernetes cluster to database or storage. Stackdriver is also not just limited to Google Cloud Platform (GCP), it supports a number of AWS-native services and extensive log monitoring capabilities for a wide array of open source software packages, whether they run in the Cloud or in on premises.

I wanted to use the now generally available Cloud Spanner database to write an app that would track stock prices and social media sentiment to identify potential correlation.

To test even the validity of this approach I put together a Go app that subscribes to Twitter stream for all companies defined in the Stocks table and scores each event against the Google NLP API while comparing the user sentiment against the stock ask price against Yahoo API.

As part of my ramp up on Google APIs I wanted to create a project that would allow me some practical exercise in a context of a real application.

TFeel (short for Twitter Feeling) is a simple sentiment analyses over tweeter data for specific Twitter search terms using Google Cloud services:

I had the opportunity to attend Google Next this year. Week after this event I joined Google. Here are some quick notes in no particular order:

• Registration was a pain, long lines. My first tech conference where I had to go through a metal detector.
• Impressive reminder of the core tech underlining Google Cloud (i.e. Borg) and Google’s experience in operating at scale.
• Serious commitment to ML and Data Science: invest in talent/community (Kaggle), vision/leadership (Fei-Fei Li keynote), powerful APIs (i.e. Label/Face Detection, OCR, Explicit Content etc.)
• Feels like the ease of use for Google is more important than time time market (see the above DS APIs but also Spanner UX).
• Clear understanding of data economy’s potential value (goes with the scale comment but the experience was apparent in many sessions/keynotes)
• Open/partner market makes stronger platform (Sam Ramji’s keynote drilled on this best), early hints of this were probably the main reason why I wanted to join Google Cloud team.
• Size/quality of the network (both, cables & tech), globally distributed data systems (specifically latency) are perhaps the best example of the user value here.

Data is growing at an exponential pace. Based on recent numbers from IDC, the total amount of data in 2015 (4.4ZB) will grow to 44ZB in 2020. Franky, how much is in Zettabyte is almost inconsequential. It is the fact that all of the data generated since the beginning of time (at least the electronic part), will grow 10x in just the next four years that’s shocking!

The presentation that goes along with this post is available here

In my last post I went over the value cycle of machine generated data. In this post, I want to follow up with a few ideas on how to further amplify value of that data by expanding its context beyond the walls of owning organization, in a construct we came to know as Data Exchange, and list a few innovation opportunities in each one of these areas.

This is one of those posts you write on your phone while getting sprayed for 11+ hours with bathroom chemicals in minimally reclining seat in a second to last row of a transatlantic flight. Still, I’m going to try to be as constructive as my thumbs allow.

Over the holidays, as many of us do, I embarked on a little extra-curriculum development effort I called thingz.io. I was driven by the pattern I’d observed in Data Center (DC) monitoring products; although that pattern also exists in many of today’s Internet of Things (IoT) solutions.

As part of my recent solution review, I wanted to compare a few performance metrics specific to multi-node data service deployment on different clouds. This post is about my experience with Google Compute Engine (GCE) as part of that evaluation.

I am excited to share with you today that starting Monday I will be joining the Big Data team at Intel. Yes, Intel. While not traditionally known for its software offerings, Intel has recently entered the Big Data space by introducing their own, 100% open source Hadoop distribution with unique security and monitoring features.