The Pivotal era kicks off this Wednesday April 24th, with Pivotal: A New Platform for a New Era, a live streaming event unveiling this exciting new company. Bringing together selected technology, people and programs from EMC and VMware, Pivotal will unite Greenplum’s products and services with those from Cloud Foundry, Spring, GemFire and other products from the VMware vFabric Suite, Cetas, and Pivotal Labs.

Beginning on Wednesday, we’ll be moving to the Pivotal blog. There you’ll find the Hadoop and data science news, technical deep dives, and articles you’ve come to expect from Datastream, as well as the latest on agile development and the cloud.

Join us on Wednesday, April 24th at gopivotal.com.

Redis is a very cool open-source key-value store that can add instant value to your Hadoop installation. Since keys can contain strings, hashes, lists, sets and sorted sets, Redis can be used as a front end to serve data out of Hadoop, caching your ‘hot’ pieces of data in-memory for fast access when they are needed again. By using a Java client called Jedis, you can ingest and retrieve data with Redis. Combining this simple client with the power of MapReduce will let you write and read data to and from Redis in parallel.

In the code below, we use MapReduce to pull and push key/value pairs to any number of standalone Redis instances. We will be writing to, and reading from, a Redis hash, which maps string fields to string values, much like a Java HashMap. Each hash is uniquely identified by a hash key, similar to the names of tables. Each input and output format has two core configuration parameters: a CSV list of hostnames running a Redis instance, and the hash key. Similar to Hadoop’s default HashPartitioner, (key.hashCode() % number of Redis instances) is used to determine which Redis instance the key is written to. This random distribution will result in even data distribution, so long as your key-space isn’t skewed – but solving that problem is a topic for another post.

Four map tasks outputting to three standalone Redis hash instances. Each record writer has a connection to each Redis instance.

With that said, let’s take a look at all the code. Pay attention to the comments, as they’ll tell you what is going on. First up is an implementation of OutputFormat. This class defines the key/value data types and behavior for writing to Redis instances via Jedis.

 // This output format class is templated to accept a key and value of type Text
public static class RedisHashOutputFormat extends OutputFormat<Text, Text> {

// These static conf variables and methods are used to modify the job configuration.  This is a common pattern for MapReduce related classes to avoid the magic string problem
public static final String REDIS_HOSTS_CONF = "mapred.redishashoutputformat.hosts";
public static final String REDIS_HASH_KEY_CONF = "mapred.redishashinputformat.key";
                    
public static void setRedisHosts(Job job, String hosts) {
job.getConfiguration().set(REDIS_HOSTS_CONF, hosts);                
}
                    
public static void setRedisHashKey(Job job, String hashKey) {
job.getConfiguration().set(REDIS_HASH_KEY_CONF, hashKey);                
}

// This method returns an instance of a RecordWriter for the task.  Note how we are pulling the variables set by the static methods during configuration
public RecordWriter<Text, Text> getRecordWriter(TaskAttemptContext job)
throws IOException, InterruptedException {
String hashKey = job.getConfiguration().get(REDIS_HASH_KEY_CONF);
String csvHosts = job.getConfiguration().get(REDIS_HOSTS_CONF);
return new RedisHashRecordWriter(hashKey, csvHosts);
}
            
// This method is used on the front-end prior to job submission to ensure everything is configured correctly        
public void checkOutputSpecs(JobContext job) throws IOException {
String hosts = job.getConfiguration().get(REDIS_HOSTS_CONF);
if (hosts == null || hosts.isEmpty()) {
throw new IOException(REDIS_HOSTS_CONF + " is not set in configuration.");
}

String hashKey = job.getConfiguration().get(REDIS_HASH_KEY_CONF);
if (hashKey == null || hashKey.isEmpty()) {
throw new IOException(REDIS_HASH_KEY_CONF + " is not set in configuration.");
}
}

// The output committer is used on the back-end to, well, commit output.  Discussion of this class is out of scope, but more info can be found here
public OutputCommitter getOutputCommitter(TaskAttemptContext context)
throws IOException, InterruptedException {
// use a null output committer, since
return (new NullOutputFormat<Text, Text>()).getOutputCommitter(context);
}

public static class RedisHashRecordWriter extends RecordWriter<Text, Text> {
    // implementation of this static nested class is shown shortly
}
} // end RedisHashOutputFormat

The role of OutputFormat is to properly configure the job, ensuring that the RecordWriter implementation has everything it needs to work correctly. Once configured, the RecordWriter is what actually writes key/value pairs wherever you want them to go. A common practice is to make your RecordWriter (or reader) a static nested class, but that isn’t required. Let’s take a look at an implementation of RecordWriter:

// This class is template to write only Text keys and Text values
public static class RedisHashRecordWriter extends RecordWriter<Text, Text> {

// This map is used to map an integer to a Jedis instance
private HashMap<Integer, Jedis> jedisMap = new HashMap<Integer, Jedis>();

// This is the name of the Redis hash
private String hashKey = null;

public RedisHashRecordWriter(String hashKey, String hosts) {
this.hashKey = hashKey;

// Create a connection to Redis for each host
// Map an integer 0-(numRedisInstances - 1) to the instance
int i=0;
for (String host : hosts.split(",")) {
Jedis jedis = new Jedis(host);
jedis.connect();
jedisMap.put(i++, jedis);
}
}

// The write method is what will actually write the key value pairs out to Redis
public void write(Text key, Text value) throws IOException, InterruptedException {
// Get the Jedis instance that this key/value pair will be written to.
Jedis j = jedisMap.get(Math.abs(key.hashCode()) % jedisMap.size());

// Write the key/value pair
j.hset(hashKey, key.toString(), value.toString());
}
                    
public void close(TaskAttemptContext context)
throws IOException, InterruptedException {
// For each jedis instance, disconnect it
for (Jedis jedis : jedisMap.values()) {
jedis.disconnect();
}
}
} // end RedisRecordWriter

This code demonstrates how simple it is to hook into external hosts for output. Such lightweight interfaces allow for endless possibilities, so long as the custom output formats can handle the parallel load of many map or reduce tasks.

Next up, let’s take a look at the InputFormat code to pull data out of our Redis instances. This is a bit more complex, as we’ll use a custom InputSplit implementation as well.

Three map tasks are created, and each record reader pulls from its assigned Redis instance.

We create an InputSplit for each Redis host, and map task is created from each InputSplit. A single map task pulls all the data its assigned Redis instance.

// This input format will read all the data from a given set of Redis hosts
public static class RedisHashInputFormat extends InputFormat<Text, Text> {

// Again, the CSV list of hosts and a hash key variables and methods for configuration
public static final String REDIS_HOSTS_CONF = "mapred.redishashinputformat.hosts";
public static final String REDIS_HASH_KEY_CONF = "mapred.redishashinputformat.key";

public static void setRedisHosts(Job job, String hosts) {
job.getConfiguration().set(REDIS_HOSTS_CONF, hosts);
}

public static void setRedisHashKey(Job job, String hashKey) {
job.getConfiguration().set(REDIS_HASH_KEY_CONF, hashKey);
}

// This method will return a list of InputSplit objects.  The framework uses this to create an equivalent number of map tasks
public List<InputSplit> getSplits(JobContext job) throws IOException {

    // Get our configuration values and ensure they are set
String hosts = job.getConfiguration().get(REDIS_HOSTS_CONF);
if (hosts == null || hosts.isEmpty()) {
throw new IOException(REDIS_HOSTS_CONF + " is not set in configuration.");
}

String hashKey = job.getConfiguration().get(REDIS_HASH_KEY_CONF);
if (hashKey == null || hashKey.isEmpty()) {
throw new IOException(REDIS_HASH_KEY_CONF + " is not set in configuration.");
}

// Create an input split for each Redis instance
// More on this custom split later, just know that one is created per host
List<InputSplit> splits = new ArrayList<InputSplit>();
for (String host : hosts.split(",")) {
splits.add(new RedisHashInputSplit(host, hashKey));
}

return splits;
}

// This method creates an instance of our RedisHashRecordReader
public RecordReader<Text, Text> createRecordReader(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
return new RedisHashRecordReader();
}

public static class RedisHashRecordReader extends RecordReader<Text, Text> {
    // implementation of this static nested class is shown shortly
}

public static class RedisHashInputSplit extends InputSplit implements Writable {
    // implementation of this static nested class is shown shortly
}

} // end RedisHashInputFormat

There are only two methods that adhere to the InputFormat abstract class: getSplits and createRecordReader. The example above demonstrates how simple it is to hook into external sources for output. The remaining static methods and variables are used to configure the job for the needs of the InputFormat and RecordReader implementations.

// This custom RecordReader will pull in all key/value pairs from a Redis instance for a given hash
public static class RedisHashRecordReader extends RecordReader<Text, Text> {

// A number of member variables to iterate and store key/value pairs from Redis
private Iterator<Entry<String, String>> keyValueMapIter = null;
private Text key = new Text(), value = new Text();
private float processedKVs = 0, totalKVs = 0;
private Entry<String, String> currentEntry = null;

// Initialize is called by the framework and given an InputSplit to process
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {

// Get the host location from the InputSplit
String host = split.getLocations()[0];
String hashKey = ((RedisHashInputSplit) split).getHashKey();

// Create a new connection to Redis
Jedis jedis = new Jedis(host);
jedis.connect();
jedis.getClient().setTimeoutInfinite();
    
// Get all the key/value pairs from the Redis instance and store them in memory
totalKVs = jedis.hlen(hashKey);
keyValueMapIter = jedis.hgetAll(hashKey).entrySet().iterator();
LOG.info("Got " + totalKVs + " from " + hashKey); jedis.disconnect();
}

// This method is called by Mapper’s run method to ensure all key/value pairs are read
public boolean nextKeyValue() throws IOException, InterruptedException {
if (keyValueMapIter.hasNext()) {
// Get the current entry and set the Text objects to the entry
currentEntry = keyValueMapIter.next();
key.set(currentEntry.getKey());
value.set(currentEntry.getValue());
return true;                
} else {
return false;    
}
}

// The next two methods are to return the current key/value pairs.  Best practice is to re-use objects rather than create new ones, i.e. don’t use “new”
public Text getCurrentKey() throws IOException, InterruptedException {
return key;
}

public Text getCurrentValue() throws IOException, InterruptedException {
return value;
}

// This method is used to report the progress metric back to the framework.  It is not required to have a true implementation, but it is recommended.
public float getProgress() throws IOException, InterruptedException {
return processedKVs / totalKVs;                
}

public void close() throws IOException {
/* nothing to do */
}

} // end RedisHashRecordReader

Now that we’ve implemented a RecordReader, we need to determine what data is read by it. This is defined as an InputSplit implementation, which initializes the reader. The number of input splits determines the number of map tasks created by the framework.

This is a pretty simple task with Redis. We will create one map task for each Redis instance, which hosts a shard of our total data set. Each mapper will then connect to a single Redis instance and pull all of the data for the hash. This will all happen in parallel, similar to how MapReduce reads a file in parallel by reading its blocks. This split means we won’t overload a single Redis instance with too many connections. This is where InputSplit comes in.¹

public static class RedisHashInputSplit extends InputSplit implements Writable {

// Two member variables, the hostname and the hash key (table name)
private String location = null;
private String hashKey = null;

public RedisHashInputSplit() {
// Default constructor required for reflection
}

public RedisHashInputSplit(String redisHost, String hash) {
this.location = redisHost;
this.hashKey = hash;
}

public String getHashKey() {
return this.hashKey;
}

// The following two methods are used to serialize the input information for an individual task
public void readFields(DataInput in) throws IOException {
this.location = in.readUTF();
this.hashKey = in.readUTF();
}

public void write(DataOutput out) throws IOException {
out.writeUTF(location);
out.writeUTF(hashKey);
}

// This gets the size of the split so the framework can sort them by size.  This isn’t that important here, but we could query a Redis instance and get the bytes if we desired
public long getLength() throws IOException, InterruptedException {
return 0;
}

// This method returns hints to the framework of where to launch a task for data locality
public String[] getLocations() throws IOException, InterruptedException {
return new String[] { location };
}

} // end RedisHashInputSplit

This demonstrates how to customize input and output using the MapReduce framework for Redis. Though it’s often overlooked, customizing I/O is a useful way to make MapReduce more flexible. If I’ve piqued your interest and you want to know more about customizing MapReduce I/O, check out chapter seven of MapReduce Design Patterns (O’Reilly 2012), “Input and Output Patterns.”

When implementing custom formats for yourself for other external sources, be mindful of how well these sources can scale, and what would happen if a task fails and is tried again. In some cases, such as this one, that doesn’t really matter. Data that was already written to Redis will just be overwritten with a new copy, and data pulled from Redis will simply be pulled again on the next attempt. Other cases, for example if we were writing to a Redis list rather than a hash, would require a little bit more effort. In this scenario, task retries would add duplicate entries to the list. It would take additional engineering to roll back committed entries, but worth the effort to ensure more fault-tolerant external outputs.

Now get out there and write some custom formats of your own!

1. Note that custom InputSplits must inherit from Writable in order for the framework to serialize them.↩

Just as consumer-facing web giants such as Google and Facebook have done, enterprise companies increasingly need to store and analyze massive amounts of data cost-effectively, ingest huge numbers of events in real time, reason over the data, and react rapidly. To meet this need, Pivotal will host a live streaming event on April 24th with a special announcement and an unveiling of its plans to build a platform that makes the consumer grade enterprise a reality.

Billed as "A New Platform for a New Era", the Pivotal platform will unite data, application, and cloud fabrics. Prioritizing big and fast data, agile development, and cloud independence, the Pivotal platform will enable enterprises to understand more, react quicker, build smarter, and achieve at an even greater scale. To this end, the new company brings together a prodigious set of technologies and talent from a number of EMC and VMware entities, which include Greenplum, Cloud Foundry, Spring, GemFire and other products from the VMware vFabric Suite, Cetas, and Pivotal Labs.

Paul Maritz, the Pivotal Leadership Team, and special guests will unveil this platform, and make a special announcement during a live streaming event on Wednesday, April 24th at 10:00 am Pacific/1:00 pm Eastern. Sign up for the event at gopivotal.com and follow @gopivotal on Twitter for updates.

No matter how much data you amass, or how ingenious your models may be, your efforts are only as effective as how you communicate the insights revealed. Data visualization is not a new art, but one which has grown significantly more important in recent years, as organizations must respond to an increasing amount of data faster than ever.

Two recent articles demonstrate the wide range of uses for data visualization — from business intelligence to social advocacy — and how form and function vary widely depending on the data visualized and the intended purpose. Co.Design features Periscopic, an information design studio that uses visualization as a persuasive storytelling form ripe for innovation and experimentation. In contrast, a recent Harvard Business Review article details Procter & Gamble’s flexible yet robust framework of visualization tools, designed to rapidly deliver actionable business insight to decision makers.

Despite the ascent of the form, the notion of telling stories with data visualization was a tougher sell when Periscopic formed in 2004. At the time, co-founder Kim Rees explains to Co.Design, “We were seeing more interest in large databases, but there was a lag in the sense-making part of it.” Still, Rees and her partner Dino Citraro had a sense of the coming data deluge, and knew they were on to something. “If you store a lot of data it does you no good unless you can understand it,” she says.

Periscopic became an in-demand information design firm and advocacy group by building tools that demonstrated their vision, telling stories out of complex sets of data through interactive interfaces which fostered agency and engagement among users. Rees tells Co.Design that Periscopic’s “goal is to communicate what the data is saying and allow people to have their own ownership of that data.” Conversely, the studio produces starkly simple interface when the topic demands such an approach. One exemplary recent examples of this is Periscopic’s sobering animation of the years lost to gun deaths in the U.S., which drives home the human toll of each individual death, rather than taking an approach that abstracts the statistics.

In terms of subject matter, methods, and intent, Procter & Gamble’s robust analytics platform differs widely from Periscopic’s work. Its “Decision Cockpit” is accessible by over 50,000 decision makers within the company, clearly presenting a profusion of critical information on business operations. The dashboard is instantly accessible and highly visible in management meeting rooms in over 50 Procter & Gamble offices.

The company’s CIO tells Harvard Business Review that the purpose of its analytics platform is to turn data into knowledge and results, by “getting beyond the what to the why and the how.” A global company with a huge portfolio of products, Procter & Gamble’s management need tools that let them drill deep into the minutiae of a particular market, without losing sight of the bigger picture. IT Director Guy Peri explains, “With visual analytics, we are able to quickly focus business decision makers on the businesses issues that are material.”

Despite the vast differences, both Periscopic and Procter & Gamble demonstrate that there is no single set of techniques for effective data visualization. Form must not only follow function, but also context and intent. Read more at Co.Design and Harvard Business Review, which is running a month-long series on visualizing data.

“A robot took my job!” The familiar refrain gets at a long-running cultural anxiety that machines will render human workers redundant. “My father had worked for the same firm for twelve years,” begins a memorable gag from Woody Allen’s early standup career. “They replaced him with a tiny gadget, this big, that does everything my father does, only it does it much better. The depressing thing is, my mother ran out and bought one.” But according to a recent Economist article about the role of big data and predictive analytics in Human Resources departments, an increasing number of workers may instead have reason to declare, “A robot gave me my job!”

The notion of machine-aided HR departments might seem troubling to some job seekers, evoking a future where algorithms cull through countless resumes for candidates that fulfill a pre-selected set of metrics. Yet the approach is revealing surprising correlations that not only identify promising job candidates, but may benefit prospective employees in unexpected ways.

Some of the correlations are intriguing yet untested — for example, candidates who apply for jobs using a non pre-installed web browser like Chrome or Firefox tend to perform better and stay at jobs longer. But deep analysis of a wealth of applicant and HR data can also reveal insights that meaningfully contradict deeply ingrained HR biases:

For instance, firms routinely cull job candidates with a criminal record. Yet the data suggest that for certain jobs there is no correlation with work performance…Likewise, many HR departments automatically eliminate candidates who have hopped from job to job. But a recent analysis of 100,000 call-centre workers showed that those who had job-hopped in the past were no more likely to quit quickly than those who had not.

Other insights seem to make sly, intuitive sense — on the whole, honest people tend to be more productive and dependable employees, but less effective salespeople. And as the article details, biases and human error can still taint the accuracy of results. But for many job seekers, data mining and algorithms may increase, rather than limit, new opportunities. Read more at The Economist.

Pivotal, the new company uniting EMC Greenplum’s decade of big data R & D, agile development trailblazers Pivotal Labs, and VMWare’s cloud services and app framework acumen, is ready to launch. In a letter to employees sent Monday, April 1st, CEO Paul Maritz announced Pivotal as “a new and exciting company with great promise,” and outlined the Pivotal vision. In the letter, titled “Pivotal: One Team. One Platform. One Company,” Maritz wrote:

As you recall, on December 4, 2012 we shared with you our intentions to bring together products and people from EMC and VMware to form a “new entity” in an effort to pursue a unique opportunity in the marketplace. On March 13th, we publicly outlined our intentions to the analyst community. We knew that getting this far would take effort and we still have much work to do, however, at this time I would like to recognize everyone’s hard work, patience, and spirit that has brought us to this day. Thank you.

When I think of Pivotal, one word consistently comes to mind: privilege.

We are all privileged to have the opportunity to work on a big and important idea: to enable customers to build a new class of applications, leveraging big and fast data, and do all of this with the power of cloud-independence. Building a new platform for a new era is an incredible challenge and opportunity. If we can succeed we will have something to be rightly proud of.

On April 29th, Pivotal will launch, promising a unified platform to enable “the consumer-grade enterprise,” boasting robust data analytics and applications fabrics, cloud independence, and the tools and skills to develop customer experiences in a rapid, iterative fashion. For the time being, you can learn more about Pivotal at gopivotal.com, and get more details in the Pivotal FAQ.

In an interview with CRN, Maritz emphasizes the cloud independence aspect of the Pivotal platform:

“‘We don’t want this world to be like the bad old days of the mainframe: when you wrote a COBOL CICS app, you were condemned to pay IBM (NYSE:IBM) a tax for all eternity,’ said Maritz, who is slated to publicly unveil the Pivotal Initiative in an April 29 press conference. ‘We don’t want to make it so when you write an app in Amazon you are condemned to pay Amazon a tax for all eternity.’”

(snip)

“Thus, Pivotal is aiming to deliver a platform as a service on top of a wide array of infrastructure-as-a-service offerings, from Amazon’s EC2 Web Service to Microsoft’s Azure and other platforms too. ‘If infrastructure as a service is the new hardware, we are the new OS on top of it,’ said Maritz.”

“‘At the end of the day, we intend to be a platform provider,” said Maritz. “I hesitate to use the word [OS] because it isn’t a really good analogy, [but] we want to provide the operating system for the cloud era.’”

Some additional responses in the press:

EMC’s Amazon Challenger Comes Out | NYTimes.com

“Pivotal has drawn talent from both companies, in particular a division of EMC specialized in data analysis and prediction, and another group that works on writing software applications within cloud computing.

“In a letter to employees, Mr. Maritz talked about Pivotal’s goal ‘to enable customers to build a new class of applications, leveraging big and fast data, and do all of this with the power of cloud independence.’ Those applications would be running on privately run clouds rich in EMC and VMware products.

(snip)

“Mr. Maritz’s last great act at VMware was spending $1.26 billion on a networking software company that needs a lot of private clouds to succeed. Pivotal, if it can supply lots of cloud applications and services faster than Amazon can make them, could be a way of ensuring that future.”

EMC, VMware Love Child Pivotal Poses Challenge to AWS | eWeek

“Does Amazon and the Web services sector need another competitor? Of course they would say no, but it’s a big Internet world out there. With the financial and development muscle of EMC—a $50 billion per year company—and VMware (nearly $5 billion), to go with the know-how and experience of Maritz, Pivotal stands as good a chance as any to make a real mark.”

“One supposes that in the marketplace of the Tier 1 Web service providers, with all things being relatively equal, it might simply come down trust and security. EMC, with its RSA Security arm and its generation-old data protection background, brings a lot to the table with Pivotal in that case.”

The Pivotal Initiative – Aiming for the PaaS Crown | SYS-CON MEDIA

The Pivotal Initiative will aim to deliver the market a data analysis platform capable of capturing large volumes of data, quickly addressing and querying it and then producing near real time answers that can be stored in a large scale-out storage system. It would be naïve to think this is an initiative aimed just at existing VMware customers. This is an attempt to not only enter but also become relevant in the software led infrastructure arena that competes with the likes of Amazon.

“In essence the Pivotal Initiative is a brave yet necessary move from both EMC and VMware to embrace the challenge of change as the legacy of traditional infrastructure faces the daunting prospect of new software paradigms. Whether the Pivotal Initiative can be successful and achieve it’s $1bn rate in its projected five years depends on a number of factors. One thing is certain is that the first challenge to remaining relevant in the IT industry is to acknowledge and adapt to change. The masters behind the Pivotal Initiative have already achieved that.”

EMC: Big Strides Ahead | Seeking Alpha

“The Pivotal venture is expected to be spun-off in the future and possibly made a public company, majority-owned by EMC (69%). It is expected to accrue $1 billion in revenue by 2017, according to this presentation. I believe this new development will help accelerate growth prospects for all these three entities.”

And some of the reactions on Twitter:

Pivotal: A New Platform for a New Era. Ready. Set. #gopivotal gopivotal.com Not much to say, but I like the promo

— Doug Henschen (@DHenschen) April 1, 2013

Day 1 of Pivotal: A New Cloud Platform for a New Era. Ready. Set. #gopivotal gopivotal.com

— Brent Byrnes (@Brent_Byrnes) April 1, 2013

Learn more at gopivotal.com.

Logan Lee at Data Science London. Image via Data Science London (@ds_ldn)

Over the past few years, organizations like Data Science London have sprung up in major metropolitan cities all over the world. This is yet another sign of increasing momentum behind the data science community. It’s a community capable of transformative impact, one enabled by the dramatic improvements in technologies to effectively analyze and model massive data volume. Greenplum feels privileged to be invited from time to time to present information about our products and learnings, and to participate in collaborative discussion with a group representative of the markets we serve.

Recently, I had the opportunity to present a workshop on Greenplum Chorus at Data Science London. The group is one of the largest data science communities in Europe, with over 1,600 data professionals from a diverse range of organizations and companies. They meet regularly to discuss data science concepts and technologies used to analyze large-scale data, extract predictive insight, and exploit business opportunities from data products.

On this night, I introduced the three primary concepts of Chorus:

• The first concept is self-service access to data from disparate sources for data scientists to speed up their work.
• The second concept is how to avoid over-investing in each new data science project by providing a standard analytical environment to prove out the idea.
• The third concept is the value of disseminating knowledge about data sources and datasets across an organization to speed up each team’s ability to work with the data they need.

The workshop was composed of a simulated data science project, where participants randomly formed teams and played the roles of Data Miner, Data Scientist, or Subject Matter Experts (SMEs). The Data Miner’s unique skills were technical aptitude to access and prepare datasets. The Data Scientists were the only team members able to create a statistical model using various programming languages. The Subject Matter Experts (SMEs) possessed critical domain knowledge of the project’s business constraints and a detailed understanding of the source data that materially affected what data should be used and how the model should be built.

The roles of data miner, data scientist, and subject matter experts are real-life roles, but data scientists (with unique statistical and computer science skills) are often forced to play all three roles, spending up to 80% of their time on tasks others could have performed. Effectively bringing others into a collaborative and secure environment is a principle element to how we see data science transforming.

Image via Data Science London (@ds_ldn)

Using Chorus, the Data Miner on the project was given instructions to identify and prepare training data for a model from a range of datasets stored within a Greenplum database. Instructions were intentionally vague and the Data Miner did not have 100% of the necessary knowledge to prepare the data on their own. Through Chorus’ data management and notation capabilities, the Data Miner was able to effectively surface potential training sets, solicit input from the Data Scientist and SMEs on the project, and quickly revise the training data based on input received from the other two roles. All the data preparation work was done without the need for shuttling files back and forth or setting up individual environments to inspect and evaluate data stored in the database. With each team member navigating through the virtual workspace to the appropriate objects, Chorus made accessing the source of truth easy.

Once training data was prepared, the Data Scientist on the team was called into action to apply their unique domain knowledge to create sophisticated statistical models to predict the desired outcome. Chorus includes an integrated console for developing and versioning code that runs against the Greenplum data platform. Because datasets relevant to the project are readily available through the Chorus web interface, specifying what data to interact with is simple and intuitive. Here too, Chorus facilitated easy verification by SMEs, who provided key points related to the project’s goals or the nature of the underlying data that materially affected the data scientist’s choices. Chorus allowed all members of the team to run the unfinished model, saving iterations as new versions to be discussed and quickly improved.

These activities highlighted the value Chorus offers, streamlining the process of ”baton-handing” between team members, an important capability as organizations scale their data science activities. I was happy to see that even though the participants did not know which other people in the room were their team members, they were able to effectively collaborate in Chorus’ web interface through the phases of their work.

Although each team worked in independent workspaces within Chorus, the contextual knowledge they accumulated and disseminated among themselves was exposed across teams, due to the use of common source datasets. A few astute teams quickly realized they could actually make use of the knowledge derived by other teams to improve their own results. This drove home the second important Chorus concept: how to make the knowledge accessible to others? By wrapping the questions, answers, comments, and feedback around the data assets, or modeling code itself, users can discover and use that knowledge as they search or browse for objects relevant to their work.

Feedback at the end of our workshop was positive and I was glad to see participants thinking about how they could apply the concepts Chorus enables in their own organization. As Chorus’ Director of Product Management,it was extremely useful to me to gain direct feedback on how first time users use our product, and where to effectively invest in our goal to further transform the practice of data science for the better.

If you run a data science group and are interested in learning more about Chorus or Greenplum, please contact us. We’d like to learn more about you as well.

eBay’s Beth Axelrod

The challenges of hiring skilled data scientists is well-documented, but what of Big Data’s impact on talent recruitment within other fields? According to Beth Axelrod, eBay’s Senior Vice President of Human Resources and the co-author of the new book The War for Talent, Human Resources is the latest department to be significantly disrupted within the data-driven enterprise.

In an interview at Forbes, Axelrod explains, “There’s a lot of value to be created and added through data analytics…Whether it’s doing a better job spotting talent outside to attract to the company, or doing predictive analysis of who is likely to leave and what are the factors, so you can intervene before that point is reached to try to change the trajectory.” She points to recruitment startup Gild as exemplifying the changes to come. Not unlike a Kaggle in reverse, the service analyzes and evaluates open source code available on the web, and then connects prospective employers with the developers writing the top-ranked code.

Axelrod notes that the greatest barrier for companies embracing data-driven employee recruitment is often the folks in the Human Resources department, who may lack the skills or desire to perform sophisticated analytics. This mismatch should encourage companies to reach outside of their HR departments for recruitment efforts, she says, and engage in-house analytics talent when considering new hires.

Read more at Forbes.

Over the past decade, consumer-facing web giants such as Google and Facebook have invested heavily in user data — storing it, analyzing it, and using the insights revealed to rapidly iterate on products. These companies recognized early that user data is a treasure trove, and devoted significant in-house resources to reap its value. In this way, increasingly data-driven and data-dependent enterprises are playing catch-up. In a post at GigaOm, Pivotal Initiative leader Paul Maritz articulates his vision for “the consumer-grade enterprise,” taking its cues not from the offerings of traditional enterprise vendors, but instead from the likes of Google, Facebook, and Amazon.

The factors driving this shift, Maritz explains, include the ascent of the cloud and the profusion of data, which will only increase in years ahead as consumers adopt “The Internet of Things” technologies. As the amount of data increases, enterprises will not only need to store and analyze it, but transform it into actionable insight at an ever-faster speed. He writes:

Enterprise companies will need ways to store and analyze massive amounts of data cost-effectively, ingest huge numbers of events in real time, reason over the data and events, and react in real time. Teams will need to be able to develop rapidly the new solutions that exploit these underlying capabilities. The need for these capabilities can be seen across a wider set of industries — from industrial control to telecommunications to retail, and even to modern agriculture.

To address these challenges, Maritz calls for a new cloud-enabled platform that is free from vendor lock-in, capable of handling massive amounts of data, which delivers insights in real time. Read more about Maritz’s vision at GigaOm.

Running the most data-driven Presidential campaign to date presented unique challenges. The high-stakes contest that is a Presidential campaign demands constant iteration. Reed stated that multivariate user testing and focusing on user experience was key, demanding that his team reap insights from a huge amount of data. This required people who thrived in a high-pressure, constantly changing work environment, facing a hard deadline and with no room for error. It demanded a technology infrastructure that wouldn’t go down on election day. And it required that the campaign run analytics on a wealth of data from many sources — social media, SMS, volunteer canvasing, poll results — and deliver actionable insights in real time.

Before the keynote, Reed spoke with Datastream about how the campaign found the right problem solvers for the job, and his reflections on using data to push the conversation closer to individual voters.

Building a Team of Problem Solvers

“Working for the President, every conversation was sort of like a mic drop. People would ask, ‘What are you doing,’ and I’d say, ‘well, I’m working for the President,’ and they’d drop the phone and show up. For the most part, people wanted the job, so it wasn’t very tricky to hire — people were excited about the job and the problems. What we did to make sure we got the right people is that we really celebrated the problem solving and what we were trying to do. We had 18 months to build a giant platform, it was going to be this billion dollar company, and then we were going to shut it down, and it couldn’t fail or go down.”

“Oftentimes, when you read job descriptions, they talk about your responsibilities. That doesn’t really tell you: ‘What are the awesome problems? What is the culture of problem solving? Why are we here? What are we trying to accomplish?’ We had this big architecture, and we’d see it was going to go down in six months, so we had to make sure it didn’t do that. So how were we going to do that? We used that to sort of lure them in.”

“There’s a book by Daniel Pink called Drive that talks about the motivations of problem solvers and really smart people. What we found was that by creating a culture where they could be autonomous, and very specifically not trying to control how they solved problems, instead making sure that we had goals that they could achieve — for example, ‘this needs to be highly available’ — one team might build that differently than another team.”

“We had to have frameworks for them to do that. One example of what not to do is say, ‘you have to use this tool to solve this problem,’ because then you find people who know the tools, but not how the solve the problems. We focused aggressively on ‘not the tools’. So if you wanted to use Rails, that was great, but if you happened to be really good at Python, there was a whole team that was solving problems in Python, and you could easily do that. It was more about enabling them to do what they needed to do to solve the problem, and not focusing on asking, ’are they solving it the right way?’”

“That was hard, because anyone in this world can name 100 reasons why that’s risky. You’re segmenting your technology, making it harder to maintain, all these things. We had a very scaled-back operation, but like any business, we couldn’t fail.”

twitpic.com/c6tew1 @harper reed, Obama 2012 Campaign CTO, crushing it as usual. #hadoopSF @greenplum

— Kim Bassett (@kimbrob) February 25, 2013

Big Answers Rather Than Big Data

“It seems to me that the conversation about Big Data came out of it being hard to store. Greenplum is in this business, EMC is in this business. But the thing is, storage doesn’t matter, because these and other companies have solved this problem. Because it’s largely solved, it pushes us to this idea that I think people invest less in, which is Big Answers: How do you couple this data with answers? I think people say Big Data when they really mean answers. This should be a conversation of how you get the best answers.”

“On the campaign, it was very important to us that we focused on the answers, so our analytics team was all about giving us answers. Every day they would give us a brief that would say, ‘we need to put more people in Florida’ or ‘we need to do more media here,’ or ‘when you’re on the TV, here is who your audience is.’ It was about giving us actual information that we could react to and act upon, answers to an actual question. The question wasn’t ‘how big is your database?’ The questions we focus on as technologists, we forget that the reason we’re here is to get better answers and insights.”

From Many-to-Many to Many-to-One: How Data Can Push Conversations Closer to Individuals

“I realized that my entire career has been about asking, ‘how do we push the conversation closer and closer to an individual?’ The idea of microlistening came from Tim O’Reilly. I was at a Foo Camp, and Tim O’Reilly said, ‘I’m tired of targeting, I want more listening.’ I started looking closely at all the listening we were doing, whether it was on Twitter or knocking on doors.”

“If you knock on a door and somebody says, ‘I’m really interested in health care,’ then when you ask, ‘who’s interested in health care in this area,’ you have that person and you have something to react to, and you can move that conversation closer. You can then target them to have a loop, a conversation."

"People are focusing more on, ‘how can I show them an ad?’ They’ll say that a person needs to see it 12 times or 10 times or whatever to impact them, and that’s great — we still need ads and microtargeting is cool — but I think what’s important is asking, ’how do we have a conversation, and how do we make it so it’s on an individual level?’ So for example, it’s between EMC and Harper Reed, EMC and a programmer, EMC and a data guy — that’s the advertising cycle, that is the loop.”

Trusting Your Users

“If you have microtargeting in your organization, how do you reflect that into listening? First of all, you have to trust your users. You have to want to have that conversation. In a campaign, that is the most important thing, for us to listen to people, because we’re representing people — we’re trying to participate in a representative government.”

“That’s where I think it gets interesting, and technology is bringing us towards that, where we can target so specifically that you start to have microconversations and these tiny, many-to-one interactions. Twitter obviously does this on a grand social scale, and we can take advantage of that, but there’s a lot of opportunities there that people are missing.”

Watch Harper Reed’s keynote on The Power of Data, from Hadoop: The Foundation for Change.