“senseidb VS. Solr VS. elasticsearch (Incomplete )” fujohnwang 2012-02-27
比较的时候,主要关注以下几个方面:
Clustering
Scalability on Storage and Service
High Availability Considerations
Features
Flexibility
很显然, Solr跟Lucene是一家,所以,对Lucene做了很多扩展,与lucene的集成也比较好,而且,业界貌似求稳的都会选择Solr来构建他们的搜索体系。
但SolrCloud才是最终的理想解决方案,而SolrCloud还没有production-ready。
下面是Solr相关的架构图:
Features
Solr的首页上对自己的特性罗列阐述的很详细了,这里不再赘述。
Pros & Cons
Pros
成熟且验证过的方案
文档资料丰富
社区活跃
plugin extension points
Cons
貌似体系比较庞杂, replication的架构扩展有稍许问题?!
References
New SolrCloud Design
Scaling Lucene and Solr
Turbocharging Solr Index Replication with BitTorrent
funny and sparkling idea by introducing BitTorrent replication mechanism *****
Distributed Searching
Carrot2-OSS framework for building search clustering engine
Solr search results clustering is based on the Carrot2 real-time document clustering engine.
Clustering Component
New SolrCloud Design
SolrCloud
UniqueKey
Solr Near Realtime Search
will be added in Solr4, currently available in trunk
Scaling Solr Indexing with SolrCloud, Hadoop and Behemoth
architecture of sensei
Features
主要解决高速索引更新的问题;
底层是zoie的“2-swapping-in-memory-index + 1-on-disk-index ”索引结构支持
需要定义schema;
通过Gateway可以接入多种数据源;
通过BQL或者REST API,甚至各种语言bindings进行数据查询;
支持通过hadoop MR job批量更新数据索引;
Pros & Cons
Pros
高速索引更新
多数据源接入
灵活的访问接口
与hadoop生态的集成
优秀的分布式扩展能力
Cons
static schema
application side versioning maitaining
为何没有直接用Solr?
摘录在John Wang的访谈片段:
Sensei leverages Lucene.
We weren’t able to leverage Solr because of the following requirements:
* High update requirement, 10’s of thousands updates per second in to the system
* Real distributed solution, current Solr’s distributed story has a SPOF at the master, and Solr Cloud is not yet completed.
* Complex faceting support. Not just your standard terms based faceting. We needed to facet on social graph, dynamic time ranges and many other interesting faceting scenarios. Faceting behavior also needs to be highly customizable, which is not available via Solr.
##References 1. Introducing SenseiDB 1.0: an open-source, distributed, realtime, semi-structured database 2. Sensei: distributed, realtime, semi-structured database
很新,当前0.19RC3版本, 文档缺乏
不过, ES确实有很多值得喝彩的地方。
Features
Schema-Free | Schemaless
feed index engine with JSON formatted documents
Query by Lucene based query string or JSON based query DSL over HTTP or Native Java;
shards and replicas, LB and routings
cloud integration
multiple search types
multiple data sources integration with River
many more…
Pros & Cons
Pros
许多灵活, 优秀的特性(见features列表)
作者拥有多年在搜索领域的涉猎经验
senseidb的pros它也基本都有
Cons
References
quick intro to elastic search
Flume, Hive and realtime indexing via ElasticSearch
The Future of Compass & ElasticSearch
Elastic Search: Distributed, Lucene-based Search Engine
ElasticSearch at berlinbuzzwords 2010
Elastic Search Vs. Apache Solr
Your Data, Your Search
Search Engine Time Machine
transient状态与持久化状态的结合, write behind策略
NoSQL, Yes Search
Geo Location and Search
Zero Conf Persistency
Local Gateway (Local Storage | Local FileSystem)
The River
ES里River的概念跟Senseidb里Gateway的概念相近, 是数据源通道的意思,可以根据不同的数据源给出不同的River实现,比如基于MysqlBinlog的River, 基于Hbase的River,或者RabbitMQ River ,CouchDB River etc.
Percolator
这个Percolator是ES里的概念,不要跟Google的Percolator混淆
Versioning
Optimistic Concurrency Control
New Search Types
Introduce count and scan search types, the latter can be used to scroll large result set
Data Visualization with ElasticSearch and Protovis
Distributed Diagram (Video)
Road to a Distributed Search Engine (Video)*****
4. Conclusion
All are based on Lucene.
All are distributed.
senseidb shards with multi-write?!
solr shards with master-slaves and slave pull strategy;
elasticsearch shards with primary-secondary push strategy;
All do partitioning in document granularity, All require some unique key for each document(optional for some situations);
Sensei is good at real-time index update; Solr is good at stable and wide adoption; Elasticsearch is good at flexible and good ideas;
其它参考文献
Lily架构简介
在自己的lily node里实现了multiwrite + wal+ message queue的数据分发, 没有充分利用现有系统中各个组件/系统的能力(虽然是基于hbase的table实现的), 部分上来讲把事情搞复杂了。