Columnar Data Storage

Hasso Plattner, SAP
Image by dfarber via Flickr

At the SAP academic research conference yesterday Hasso Plattner spent a lot of time talking about database design and why it’s still important. More significantly, he drilled into why re-architecting applications to take advantage of a fundamentally differently database than what we are used to with relational databases is critical if we are to simplify code bases and develop new generations of applications that take off where the current state of the art ends.

This area has a pretty steep learning curve so to get started here’s a good explanation of the distinction between columnar vs. row database architecture.

Reblog this post [with Zemanta]

SAP’s John Schwarz on Analytics

This is Part 2 in my Fortune Brainstorm Tech series. Part 1 covered a conversation with Vishal Sikka.

Vinnie Mirchandani, Oliver Marks, Dennis Howlett and I sat down with John Schwarz, who recently came to SAP via the Business Objects acquisition and has quickly emerged as the man with the plan. If you have been following SAP’s financial results this year you already know that Business Objects is fueling SAP’s growth as their core ERP business has been hit by the global economic recession and stalled.

That analytics is fueling new business growth is not surprising, SAP’s business has historically been driven by transaction and system of record applications (think of all the large three letter acronym categories) but as those systems mature the demand for upgrades subsides and as a consequence of having been at this for so many years there is simple marketplace saturation and “going downmarket” hasn’t exactly worked out very well because SaaS applications have emerged as preferred solutions for small and mid tier customers.

Fast forward to today and analytics is looking to be a really attractive market for SAP because of the strength of the Business Objects offerings and good integration with SAP’s application products. Competitors include IBM and Oracle but neither of those vendors are particularly well positioned because IBM is selling database oriented solutions and Oracle hasn’t achieved broad integration of the analytics assets they have acquired, so they are selling pieces and parts rather than a solution.

Schwarz is responsible for this group within SAP and I think it’s pretty safe to say that they have integrated the company and their 6k employees, or at least have done the bulk of the integration heavy lifting. The question that now exists is “where are you going to take this thing now that you have the keys and a full tank of gas?”.

The answer is two pronged, the first being that Schwarz makes a tacit acknowledgement that most of what all these companies have been selling in years past is more “reporting” and less “analytics”, and even in that frame customers have not demonstrated great competence at utilization of these capabilities. Another reality is that the best systems have been applied against cubed data (OLAP) systems as opposed to real time operational systems, and by definition has been targeted at specialized use cases (like how your credit card company analyzes potential fraud).

Schwarz’s answer to this is moving the state of the art on two separate axis, the first being “analytics for everyone” by delivering solutions that have a very refined user experience for reporting, ad hoc query, and historical and predictive analytics. The second axis is potentially more disruptive to the market but one that will most certainly present more headwind in the form of mature competition that SAP will have to overcome while on a steep learning curve, and that is unstructured data (free form text).

The first axis, analytics for everyone, has high probability for success because SAP & Business Objects are really well positioned to expand their footprint in customer sites by going direct to the business user, who according to Schwarz is the decision maker for analytics purchases as opposed to IT. Schwarz and Sikka both referred to the “consumerization” trend in enterprise apps and it was refreshing to hear them say it because this has been a central theme in my writing for several years now and it is simply undeniable, business users expect business applications to be like the applications and services they use in the consumer world.

The move into unstructured data is more problematic for SAP on three fronts. First and foremost, SAP doesn’t have any packaged data feed bundles they can sell and relying on partners has proven problematic for other companies in this space. I asked Schwarz if this suggests that SAP will be acquiring wholesale content providers that they can layer on top of applications and he acknowledged that it is currently a “raging debate within the company”.

While I think SAP should acquire a content aggregator and perhaps even someone like Gnip, I doubt they will do it, instead going to companies like Factiva and Thomson Reuters to provide bundled content for SAP customers and in the long run they will come to the conclusion that trying to be a neutral player while bringing together content providers is simply cumbersome and foists upon the customer too many contingencies. However, I will acknowledge that given the state of their progress on this initiative, acquiring a content provider now does not give them a strategic advantage therefore time is on their side.

A second concern I have is that unstructured text analysis is very difficult to do well and even specialist companies that have been at this for years find it difficult, so to suggest that SAP will simply do this because they set their mind to it is optimistic to say the least. Entity extraction, semantic tagging and triple building, sentiment analysis, quality, and content authority are extremely challenging, especially as content feeds shift to activity streams which provide far less context for the content to be analyzed against. To then marry this to structured data analysis is something that has been accomplished in highly specialized fields like government intelligence and applications for financial traders. Don’t get me wrong, all of the things I have just written about are exciting and well worth doing, I simply think you have to be prepared to fail more than you succeed, even if you are as accomplished as Business Objects is.

Lastly, there is one very big obstacle in the way of anyone attempting to capture unstructured data for analytics, office applications. The single greatest store of unstructured data in any enterprise is email, followed closely by documents created by personal productivity applications. Microsoft is certainly well positioned here, their entire Sharepoint roadmap reads like a plan to integrate MS Office data to collaboration and transaction systems but even Microsoft is finding that bringing intelligence to email is a tough rock to push. Similarly, companies like Gist, Xobni, Clear Context, Kwaga, and Postbox have emerged and will certainly develop leadership long before SAP enters the market.

So what can we conclude from all of the above, well I think it’s pretty clear that Business Objects under Schwarz’s leadership is well positioned to execute on the vision he has established, from the standpoint of having a solution, market credibility, and access to a substantial customer base, but it’s also clear that they will have to make some acquisitions to fill gaps and build some durable partnerships with companies that matter.

On the partnership front, we spent quite a bit of time talking about the widely known issues that SAP has with the idea of partnering, and by extension the Palo Alto versus Walldorf mentality that has marked their expansion into Silicon Valley. Schwarz insists that these issues have been resolved but I remain unconvinced for many reasons, but mostly because NIH is such an integral part of SAP’s culture that I don’t think they will ever shed it. Secondly, as accomplished as John Wookey is, I don’t believe he can be the force at SAP that he was at Oracle because to do so would involve a massive realignment of turf that would strip many in Walldorf of their fiefdoms and we all know that at it’s core power in any organization is a function of turf.

Personally I think that SAP should stop having an inferiority complex about partnering and instead just focus on building the best tools for partners to take advantage of. Every company that sells into the enterprise wants to align with SAP at some level (hell even Oracle does to support their database business) so instead of trying to sell everyone on “hey look at us, we’re cool like Google and we want to be your friend”, just go out and build kick ass tools that partners can use to build great solutions and then get the hell out of the way.

On the competitive front Schwarz had some interesting things to say about known competitors IBM and Oracle, but his most insightful comments were saved for Microsoft and Google. On Microsoft, Schwarz confirmed that they are looking at Azure with great caution but generalized Microsoft’s competitive weakness as “trying unsuccessfully to be all things to all people”. It’s hard to disagree with that statement, nor to ignore the fact that Microsoft has not been as successful with applications as they would have predicted so for SAP the truth is the devil you know is better than the one you don’t.

Regarding the devil they don’t know, it’s clear that Google has emerged as a competitor for SAP to be concerned about. Schwarz’s general view seems to be that Google doesn’t have the maturity and product management expertise to be successful in the enterprise, pointing to companies like FAST and Autonomy doing enterprise search far better than Google, despite the fact that search is the crown jewel of the Google empire. From where I sit, the single biggest threat that Google presents is their ability to climb the learning curve very quickly and that the lack of rigid product management has demonstrated to be a remarkably effective quality for quickly iterating products that then gain broad consumer acceptance.

Whether or not that agile process is repeatable in the enterprise is debatable but it’s clear that for all the talk about innovations at SAP, the fact remains that we see very little of it in the product pipeline. This isn’t due, IMO, to the stuff not working but rather the tendency for SAP development groups to set the bar impossibly high for a product to be delivered. While I don’t think Google will present a serious threat to SAP for many years, I do believe that SAP can learn a lot from Google that will make them a better company as a result.

To conclude, I really enjoyed meeting Schwarz and it’s clear that he has a good line of sight on where his business unit needs to be focusing not just for revenue growth but also to protect the flanks as the market continues to evolve. While there is much to like, there is also much to be concerned about but on balance the concerns are more tactical than strategic so I’m willing to give him the benefit of doubt at this point.

Vishal Sikka on SAP’s Challenges, Futures, and Realities

The odds of sitting down with a senior executive of a major technology company and being surprised and enlightened are one in a million because typically they sit down with a prepared agenda and, with the benefit of years and years of media training, religiously stay on message turning every question back to the thing they want to talk about.

On Wednesday Vinnie Mirchandani, Oliver Marks and I sat down with my old friend Vishal Sikka, CTO of SAP at the Fortune Brainstorm Tech event in Pasadena for a discussion that was free wheeling and definitely without the benefit of a script, meaning it was actually interesting. The fact that Vishal is one of the smartest and most level headed people you will ever have the privilege of meeting no doubt played a big part in why our lunch meeting was so engaging.

Vishal has a tough job because he is responsible for charting the technology roadmap for a company that prides itself on being innovative yet is faced with the reality that their customers want stability and continuity. In Vishal’s own words, SAP’s customers want the company to be like a night watchman who ensures that everything is safe and secure when no one else is watching.

At the heart of this night watchman metaphor lies the fundamental criticism levied against the company, and others like them, which is that despite their vigorous pronouncements to the contrary, the fact is that they won’t change, dramatically lower the cost of their solutions, or embrace third party offerings that are better than their own products. I am sure that there is some truth to this but like other mature industries with a small cluster of dominant vendors, I’m not sure it really matters to customers.

The problems that SAP, Oracle and Microsoft are solving are complex and massively consequential; Vishal made the comment that SAP is at the center of more global trade than any other company on earth and I have little reason to doubt that but with that realization comes significant responsibility that goes well beyond the legal dance called compliance.

The financial industry is the ultimate example of a modern industry that exists because of technology, the ability to exploit information latency for profit as a result of massive processing capacity and well honed software algorithms… but with that immense power the ability for the oldest of human motives, greed, to corrupt is exponentially greater than at any time in history and we now live with what feels like ritual boom and bust cycles as a result. Ironically, we now have a new class of financial software applications that act as circuit breakers to essentially slow down automated trading that can wreak havoc in a market on any given day.

When I confronted Vishal with the above observation and applied it to supply chain rather than financial markets he acknowledged the problem and made an interesting that global trade is a great human equalizer and when nations intertwine economically they are inherently less prone to making war with each other. His point is valid, that whatever damage any bad actors can do with technology, the benefits far outweigh the downsides, a fact that is hard to disagree with.

In the years ahead I think I am on solid ground in suggesting that companies like SAP will come under increasing scrutiny to provide not only full compliance offerings and features, but also overarching transparency that exposes more of the underbelly that all organizations have.

The point about transparency is something that Vishal brought up in our conversation, his exact quote was “radical transparency that reinforces clarity of purpose”. The question that prompted this was based on the observation that unlike Oracle, SAP has always been obsessively consensus oriented, which is really difficult to sustain when involving innovations that have the result of reducing the power of individuals who are expected to grant consensus.

In such an environment how do you breed a generation of leaders who balance risk, customer expectations, commitment to excellence, and financial returns against their own personal agendas? I don’t think you do but Vishal made the point that it’s essential to not forget where you come from, which is just another way of saying you have to always keep in the forefront what your core business is and what it is that has made you successful in that over the years. This ties back to the night watchman metaphor because Vishal is right to assert that SAP’s customers don’t want them chasing every new trend that emerges, in the process foisting upon them disruptive and expensive change for benefits that are still largely promises.

What this does mean is that SAP can often be late to the party or resisting of developments that others have pioneered, but that tradeoff doesn’t seem to bother Vishal very much. A potentially more damaging consequence is that because, as has been said before, “when you are SAP every new market is a niche” and that allows oxygen for other companies to develop offerings that shape the market to SAP’s detriment. I don’t think SAP is alone in this regard, the same could be said of Microsoft, but the question must remain at the forefront of Vishal’s thinking as he is responsible for all of the self-described innovation groups at SAP.

It really is a treat to talk with Vishal, there are few people in his stratosphere who bring his smarts, rational perspective on the market, and most of all, a confidence in what he is good at that never crosses over to arrogance.

PS- I have more notes that cover Vishal’s thinking on cloud computing, the analytics market, enterprise 2.0 technologies, and much more. Rather than cover everything in one post, I will follow up with several posts on each topic.