NoSQL: The Love Child of Google, Amazon and … Lotus Notes
- 12.05.12
- 7:30 AM
Katz and the Couch
Damien Katz joined Lotus as a summer intern in 1995, around the time it was acquired by IBM, and after working with Lotus Notes consulting outfit for a while, he returned to the company, joining the Iris team, which Lotus had formally acquired.
At Iris, Katz worked his way into the guts of Lotus Notes. Among other things, he rewrote the engine that powers Formula, the scripting language used for developing Notes applications. Katz says he was massively under qualified for the job, but he also sees himself as someone who was born to code. “Each @function I completed was like a hit of a drug and I was a junkie looking for the next fix,” he later wrote on his blog.
He left Lotus in 2005, joining a startup called Koobie, but shortly thereafter, he started an effort to bring the Lotus Notes ethos into the modern age, and this eventually morphed into CouchDB. In an early blog post the project, he wrote: “Couch is Lotus Notes built from the ground up for the web.”
The original version of CouchDB used a Formula-like programming language. But he soon moved the project in a new direction, turning platform into a dedicated database. “MySQL was at the height of its popularity,” Katz says. “And telling people you were working on something that was like Lotus Notes made them go ‘ugh!’”
There were bumps along the way. In early 2007, with a new baby on the way, Katz went to work for the MySQL team at Sun Microsystems and quit working on CouchDB. But the open source project had attracted other developers, notably Jan Lehnardt and Noah Slater, who kept plugging away.
Slater introduced JSON, then a new format for structuring data in text files, and while on paternity leave from Sun, Katz ended up replacing the entire CouchDB storage engine, substituting JSON for XML. At that point, Katz realized that using JavaScript — the standard language for web applications — might be a better idea than using the Formula-style engine. “Once we introduced JavaScript,” he says, “the project took off.”
Couch Goes Commercial
In 2007, the revitalized CouchDB attracted IBM’s attention, and soon, Katz was back on the company’s payroll, developing CouchDB full-time. Crucially, IBM agreed to donate the project to the non-profit Apache Foundation, which meant that IBM also had to grant the use of the company’s relevant patents to developers and users of CouchDB. This meant that IBM wouldn’t be able to sue CouchDB for infringing on Lotus Notes related patents.
Meanwhile, the NoSQL movement was in full flight. The Google and Amazon papers helped popularize this model — already advocated by various open source developers — and provided some insight into how to make it work in the real world.
In 2007, a company called 10gen started work on a NoSQL document database called MongoDB, using BigTable as a model. “It was completely independent, there are not a lot of parallels between MongoDB and Couch and Lotus Notes,” says 10gen founder Dwight Merriman. That same year Neo4j, a graph database, was released. A year later, Facebook open sourced Cassandra, a NoSQL database that incorporated concepts from both Dynamo and BigTable. And by 2009, as CouchDB, Cassandra, MongoDB, and others gathered steam, the tech blog ReadWriteWeb asked whether the relational database was doomed.
Meanwhile, Johan Oskarsson, then a Last.fm employee, hosted the first NoSQL meetup, accidentally giving the loosely defined movement a name.
Amidst all the hype, Katz, Lehnardt and J. Chris Anderson founded Couch.io to commercialize CouchDB. By this time, a team of MIT physicists had already started a CouchDB company called Cloudant, and they were hard at work on their own version of the database, called BigCouch, and though Couch.io, later renamed CouchOne, struggled to find it’s place in the world, it would soon find its footing by merging with another NoSQL outfit called Membase.
Membase needed a new CTO. CouchOne needed a CEO. Couch needed a better way to scale to large numbers of machines, which Membase could provide. Membase needed a better data structure, which CouchDB offered. And perhaps most importantly, Membase had what Katz saw as a sustainable business model. Both the new company and the new database were called Couchbase.
But the merger led to a messy divorce with Apache. “We made a real effort to keep the changes in sync,” Katz says. “But eventually we reached a point in which we needed to move quicker than the Apache project could move.” Ultimately, Katz decided to move on from the project he founded and focus his efforts on Couchbase. In January 2012, a year after the merger, he posted a strongly worded farewell letter on his blog, writing: “What’s the future of CouchDB? It’s Couchbase.”
Slater, who had become part of the Apache project’s management team, responded with a single tweet: “The future of CouchDB is CouchDB.”
Katz acknowledges that he could have been more diplomatic, but ultimately, the story shows just how vibrant the NoSQL has become. Developers are still plugging away on CouchDB, even without Katz’ involvement. Cloudant remains committed to CouchDB, having vowed to contribute the BigCouch code back into the project. And Couchbase is on the verge of launching version 2.0 of its database, after landing big name customers such as NTT DoCoMo and AOL. The idea of a document database is now cemented in the minds of developers, thanks not only to CouchDB and its many offshoots, but also the popularity of MongoDB.
Meanwhile, IBM is discontinuing the Lotus brand name. Notes will live on, at least for now. Perhaps its best years are behind it, but it set the stage for so much more.
The post has been updated to correct and clarify the functioning of PLATO Notes
