“senseidb VS. Solr VS. elasticsearch (Incomplete)”

fujohnwang

2012-02-27

比较的时候，主要关注以下几个方面：

Clustering
- Scalability on Storage and Service
- High Availability Considerations
Features
Flexibility

1. Solr

很显然， Solr跟Lucene是一家，所以，对Lucene做了很多扩展，与lucene的集成也比较好，而且，业界貌似求稳的都会选择Solr来构建他们的搜索体系。

但SolrCloud才是最终的理想解决方案，而SolrCloud还没有production-ready。

下面是Solr相关的架构图：

Features

Solr的首页上对自己的特性罗列阐述的很详细了，这里不再赘述。

Pros & Cons

Pros
- 成熟且验证过的方案
- 文档资料丰富
- 社区活跃
- plugin extension points
Cons
- 貌似体系比较庞杂， replication的架构扩展有稍许问题？！

References

New SolrCloud Design
Scaling Lucene and Solr
Turbocharging Solr Index Replication with BitTorrent
- funny and sparkling idea by introducing BitTorrent replication mechanism *****
Distributed Searching
Carrot2-OSS framework for building search clustering engine
- Solr search results clustering is based on the Carrot2 real-time document clustering engine.
Clustering Component
- 结果集的分类
New SolrCloud Design
SolrCloud
UniqueKey
Solr Near Realtime Search
- will be added in Solr4, currently available in trunk
Scaling Solr Indexing with SolrCloud, Hadoop and Behemoth

2. Senseidb

Features

主要解决高速索引更新的问题;
- 底层是zoie的“2-swapping-in-memory-index + 1-on-disk-index”索引结构支持
需要定义schema;
通过Gateway可以接入多种数据源;
通过BQL或者REST API，甚至各种语言bindings进行数据查询；
支持通过hadoop MR job批量更新数据索引；

Pros & Cons

Pros
- 高速索引更新
- 多数据源接入
- 灵活的访问接口
- 与hadoop生态的集成
- 优秀的分布式扩展能力
Cons
- static schema
- application side versioning maitaining

为何没有直接用Solr？

摘录在John Wang的访谈片段：

Sensei leverages Lucene.

We weren’t able to leverage Solr because of the following requirements:

    * High update requirement, 10’s of thousands updates per second in to the system
    * Real distributed solution, current Solr’s distributed story has a SPOF at the master, and Solr Cloud is not yet completed.
    * Complex faceting support. Not just your standard terms based faceting. We needed to facet on social graph, dynamic time ranges and many other interesting faceting scenarios. Faceting behavior also needs to be highly customizable, which is not available via Solr.

##References 1. Introducing SenseiDB 1.0: an open-source, distributed, realtime, semi-structured database 2. Sensei: distributed, realtime, semi-structured database

3. elasticsearch

很新，当前0.19RC3版本，文档缺乏

不过， ES确实有很多值得喝彩的地方。

Features

Schema-Free | Schemaless
feed index engine with JSON formatted documents
Query by Lucene based query string or JSON based query DSL over HTTP or Native Java;
shards and replicas, LB and routings
cloud integration
multiple search types
multiple data sources integration with River
many more…

Pros & Cons

Pros
- 许多灵活，优秀的特性（见features列表）
- 作者拥有多年在搜索领域的涉猎经验
- senseidb的pros它也基本都有
Cons
- 文档不足
- 后端没有大的商业机构支持

References

quick intro to elastic search
Flume, Hive and realtime indexing via ElasticSearch
The Future of Compass & ElasticSearch
Elastic Search: Distributed, Lucene-based Search Engine
ElasticSearch at berlinbuzzwords 2010
Elastic Search Vs. Apache Solr
- 这篇貌似倾向于ES比较多一些
Your Data, Your Search
Search Engine Time Machine
- transient状态与持久化状态的结合， write behind策略
NoSQL, Yes Search
- 多种数据源类型的平滑接入
Geo Location and Search
- 基于geo进行排序的特性很新颖
Zero Conf Persistency
- Local Gateway (Local Storage | Local FileSystem)
The River
- ES里River的概念跟Senseidb里Gateway的概念相近，是数据源通道的意思，可以根据不同的数据源给出不同的River实现，比如基于MysqlBinlog的River，基于Hbase的River，或者RabbitMQ River，CouchDB River etc.
Percolator
- 这个Percolator是ES里的概念，不要跟Google的Percolator混淆
Versioning
- Optimistic Concurrency Control
New Search Types
- Introduce count and scan search types, the latter can be used to scroll large result set
Data Visualization with ElasticSearch and Protovis
Distributed Diagram (Video)
Road to a Distributed Search Engine (Video)*****

4. Conclusion

All are based on Lucene.
All are distributed.
- senseidb shards with multi-write?!
- solr shards with master-slaves and slave pull strategy;
- elasticsearch shards with primary-secondary push strategy;
All do partitioning in document granularity, All require some unique key for each document(optional for some situations);
Sensei is good at real-time index update; Solr is good at stable and wide adoption; Elasticsearch is good at flexible and good ideas;