100 open source Big Data architecture papers for data professionals

r囧r小猫 2022-08-01 11:25 186阅读 0赞

转载于：https://www.linkedin.com/pulse/100-open-source-big-data-architecture-papers-anil-madan

Big Data technology has been extremely disruptive with open source playing a dominant role in shaping its evolution. While on one hand it has been disruptive, on the other it has led to a complex ecosystem where new frameworks, libraries and tools are being released pretty much every day, creating confusion as technologists struggle and grapple with the deluge.

If you are a Big Data enthusiast or a technologist ramping up (or scratching your head), it is important to spend some serious time deeply understanding the architecture of key systems to appreciate its evolution. Understanding the architectural components and subtleties would also help you choose and apply the appropriate technology for your use case. In my journey over the last few years, some literature has helped me become a better educated data professional. My goal here is to not only share the literature but consequently also use the opportunity to put some sanity into the labyrinth of open source systems.    
  
One caution, most of the reference literature included is hugely skewed towards deep architecture overview (in most cases original research papers) than simply provide you with basic overview. I firmly believe that deep dive will fundamentally help you understand the nuances, though would not provide you with any shortcuts, if you want to get a quick basic overview.

Jumping right in…

![AAEAAQAAAAAAAAMLAAAAJDRmYmE3MDY2LTcyODQtNDFhNS1iMDhlLTczMWJlMGQzNjgwNA.jpg][]

Key architecture layers

*  File Systems\- Distributed file systems which provide storage, fault tolerance, scalability, reliability, and availability.
 *  Data Stores– Evolution of application databases into [Polyglot][] storage with application specific databases instead of one size fits all. Common ones are Key-Value, Document, Column and Graph.
 *  Resource Managers– provide resource management capabilities and support schedulers for high utilization and throughput.
 *  Coordination– systems that manage state, distributed coordination, consensus and lock management.
 *  Computational Frameworks– a lot of work is happening at this layer with highly specialized compute frameworks for Streaming, Interactive, Real Time, Batch and Iterative Graph (BSP) processing. Powering these are complete computation runtimes like [BDAS][] (Spark) & Flink.
 *  DataAnalytics –Analytical (consumption) tools and libraries, which support exploratory, descriptive, predictive, statistical analysis and machine learning.
 *  Data Integration– these include not only the orchestration tools for managing pipelines but also metadata management.
 *  Operational Frameworks – these provide scalable frameworks for monitoring & benchmarking.

### Architecture Evolution ###

The modern data architecture is evolving with a goal of reduced latency between data producers and consumers. This consequently is leading to real time and low latency processing, bridging the traditional batch and interactive layers into hybrid architectures like Lambda and Kappa.

*  [Lambda][] - Established architecture for a typical data pipeline. [Mor][]e details.
 *  [Kappa][]– An alternative architecture which moves the processing upstream to the Stream layer.
 *  [SummingBird][]– a reference model on bridging the online and traditional processing models.

Before you deep dive into the actual layers, here are some general documents which can provide you a great background on NoSQL, Data Warehouse Scale Computing and Distributed Systems.

*  [Data center as a computer][]– provides a great background on warehouse scale computing.
 *  [NOSQL Data Stores][]– background on a diverse set of key-value, document and column oriented stores.
 *  [NoSQL Thesis][]– great background on distributed systems, first generation NoSQL systems.
 *  [Large Scale Data Management][]\- covers the data model, the system architecture and the consistency model, ranging from traditional database vendors to new emerging internet-based enterprises. 
 *  [Eventual Consistency][]– background on the different consistency models for distributed systems.
 *  [CAP Theorem][]– a nice background on CAP and its evolution.

There also has been in the past a fierce debate between traditional Parallel DBMS with Map Reduce paradigm of processing. Pro [parallel DBMS][] ([another][]) paper(s) was rebutted by the pro [MapReduce][] one. Ironically the  Hadoop community from then has come full circle with the introduction of MPI style shared nothing based processing on Hadoop - [SQL on Hadoo][]p.

### File Systems  ###

As the focus shifts to low latency processing, there is a shift from traditional disk based storage file systems to an  emergence of in memory file systems - which drastically reduces the I/O & disk serialization cost. Tachyon and Spark [RDD][] are examples of that evolution.

*  [Google File System][]\- The seminal work on Distributed File Systems which shaped the Hadoop File System.
 *  [Hadoop File System][]– Historical context/architecture on evolution of HDFS.
 *  [Ceph File System][]– An [alternative][] to HDFS. 
 *  [Tachyon][]– An in memory storage system to handle the modern day low latency data processing.

File Systems have also seen an evolution on the file formats and compression techniques. The following references gives you a great background on the merits of row and column formats and the shift towards newer nested column oriented formats which are highly efficient for Big Data processing. Erasure codes are using some innovative techniques to reduce the triplication (3 replicas) schemes without compromising data recoverability and availability.

*  [Column Oriented vs Row-Stores][]– good overview of data layout, compression and materialization.
 *  [RCFile][]– Hybrid PAX structure which takes the best of both the column and row oriented stores.
 *  [Parquet][]– column oriented format first covered in Google’s Dremel’s paper.
 *  [ORCFile][]– an improved column oriented format used by Hive.
 *  [Compression][]– compression techniques and their comparison on the Hadoop ecosystem.
 *  [Erasure Codes][]– background on erasure codes and techniques; improvement on the default triplication on [Hadoop][] to reduce storage cost.

### Data Stores ###

Broadly, the distributed data stores are classified on ACID & BASE stores depending on the continuum of strong to weak consistency respectively. BASE further is classified into KeyValue, Document, Column and Graph - depending on the underlying schema & supported data structure. While there are multitude of systems and offerings in this space, I have covered few of the more prominent ones. I apologize if I have missed a significant one...

BASE  
  
Key Value Stores  
[Dynamo][] – key-value distributed storage system  
[Cassandra][] – Inspired by Dynamo; a multi-dimensional key-value/column oriented data store.  
[Voldemort][] – another one inspired by Dynamo, developed at LinkedIn.  
  
Column Oriented Stores  
[BigTable][] – seminal paper from Google on distributed column oriented data stores.  
[HBase][] – while there is no definitive paper , this provides a good overview of the technology.  
[Hypertable][] – provides a good overview of the architecture.

Document Oriented Stores  
[CouchDB][] – a popular document oriented data store.  
[MongoDB][] – a good introduction to MongoDB architecture.

Graph  
[Neo4j][] – most popular Graph database.  
[Titan][] – open source Graph database under the Apache license.

ACID  
I see a lot of evolution happening in the open source community which will try and catch up with what Google has done – 3 out of the prominent papers below are from Google , they have solved the globally distributed consistent data store problem.

[Megastore][] – a highly available distributed consistent database. Uses Bigtable as its storage subsystem.  
[Spanner][] – Globally distributed synchronously replicated linearizable database which supports SQL access.  
[MESA][] – provides consistency, high availability, reliability, fault tolerance and scalability for large data and query volumes.  
[CockroachDB][] – An open source version of Spanner (led by former engineers) in active development.

### Resource Managers ###

While the first generation of Hadoop ecosystem started with monolithic schedulers like YARN, the evolution now is towards hierarchical schedulers (Mesos), that can manage distinct workloads, across different kind of compute workloads, to achieve higher utilization and efficiency.  
  
[YARN][] – The next generation Hadoop compute framework.  
[Mesos][] – scheduling between multiple diverse cluster computing frameworks.  
  
These are loosely coupled with schedulers whose primary function is schedule jobs based on scheduling policies/configuration.  
  
Schedulers  
[Capacity Scheduler ][Capacity Scheduler]\- introduction to different features of capacity scheduler.   
[FairShare Scheduler][] - introduction to different features of fair scheduler.  
[Delayed Scheduling][] - introduction to Delayed Scheduling for FairShare scheduler.  
[Fair & Capacity schedulers][Fair _ Capacity schedulers] – a survey of Hadoop schedulers.

### Coordination    ###

These are systems that are used for coordination and state management across distributed data systems.  
[Paxos][] – a simple version of the [classical][] paper; used for distributed systems consensus and coordination.   
[Chubby][] – Google’s distributed locking service that implements Paxos.  
[Zookeeper][] – open source version inspired from Chubby though is general coordination service than simply a locking service

### Computational Frameworks    ###

The execution runtimes provide an environment for running distinct kinds of compute. The most common runtimes are

[Spark][] – its popularity and adoption is challenging the traditional Hadoop ecosystem.[  
][RDD][Flink][] – very similar to Spark ecosystem; strength over Spark is in iterative processing.  
  
The frameworks broadly can be classified based on the model and latency of processing  
  
Batch  
[MapReduce][MapReduce 1] – The seminal paper from Google on MapReduce.  
[MapReduce Survey][] – A dated, yet a good paper; survey of Map Reduce frameworks.

Iterative (BSP)  
[Pregel][] – Google’s paper on large scale graph processing  
[Giraph][] - large-scale distributed Graph processing system modelled around Pregel  
[GraphX][] - graph computation framework that unifies graph-parallel and data parallel computation.  
[Hama][] - general BSP computing engine on top of Hadoop  
[Open source graph processing ][Open source graph processing] survey of open source systems modelled around Pregel BSP.

Streaming  
[Stream Processing][Stream_Processing] – A great overview of the distinct real time processing systems   
[Storm][] – Real time big data processing system  
[Samza][]  - stream processing framework from LinkedIn  
[Spark Streaming][] – introduced the micro batch architecture bridging the traditional batch and interactive processing.

Interactive  
[Dremel][] – Google’s paper on how it processes interactive big data workloads, which laid the groundwork for multiple open source SQL systems on Hadoop.  
[Impala][] – MPI style processing on make Hadoop performant for interactive workloads.  
[Drill][] – A open source implementation of Dremel.  
[Shark][] – provides a good introduction to the data analysis capabilities on the Spark ecosystem.  
[Shark][Shark 1] – another great paper which goes deeper into SQL access.  
[Dryad][] – Configuring & executing parallel data pipelines using DAG.  
[Tez][] – open source implementation of Dryad using YARN.  
[BlinkDB][] - enabling interactive queries over data samples and presenting results annotated with meaningful error bars

RealTime  
[Druid][] – a real time OLAP data store. Operationalized time series analytics databases  
[Pinot][] – LinkedIn OLAP data store very similar to Druid.

### Data Analysis    ###

The analysis tools range from declarative languages like SQL to procedural languages like Pig. Libraries on the other hand are supporting out of the box implementations of the most common data mining and machine learning libraries.

Tools  
[Pig][] – Provides a good overview of Pig Latin.  
[Pig][Pig 1] – provide an introduction of how to build data pipelines using Pig.  
[Hive][] – provides an introduction of Hive.  
[Hive][Hive 1] – another good paper to understand the motivations behind Hive at Facebook.  
[Phoenix][] – SQL on Hbase.  
[Join Algorithms for Map Reduce][] – provides a great introduction to different join algorithms on Hadoop.   
[Join Algorithms for Map Reduce][Eventual Consistency] – another great paper on the different join techniques.

Libraires  
[MLlib][] – Machine language framework on Spark.  
[SparkR][] – Distributed R on Spark framework.  
[Mahout][] – Machine learning framework on traditional Map Reduce.

Data Integration  
Data integration frameworks provide good mechanisms to ingest and outgest data between Big Data systems. It ranges from orchestration pipelines to metadata framework with support for lifecycle management and governance.

Ingest/Messaging  
[Flume][] – a framework for collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.  
[Sqoop][]– a tool to move data between Hadoop and Relational data stores.  
[Kafka][] – distributed messaging system for data processing

ETL/Workflow  
[Crunch][] – library for writing, testing, and running MapReduce pipelines.  
[Falcon][] – data management framework that helps automate movement and processing of Big Data.  
[Cascading][] – data manipulation through scripting.  
[Oozie][] – a workflow scheduler system to manage Hadoop jobs.

Metadata  
[HCatalog][] - a table and storage management layer for Hadoop.

Serialization  
[ProtocolBuffers][] – language neutral serialization format popularized by Google. [Avro][] – modeled around Protocol Buffers for the Hadoop ecosystem.

### Operational Frameworks    ###

Finally the operational frameworks provide capabilities for metrics, benchmarking and performance optimization to manage workloads.

Monitoring Frameworks  
[OpenTSDB][] – a time series metrics systems built on top of HBase.  
[Ambari][] - system for collecting, aggregating and serving Hadoop and system metrics

Benchmarking  
[YCSB][] – performance evaluation of NoSQL systems.  
[GridMix][] – provides benchmark for Hadoop workloads by running a mix of synthetic jobs  
[Background][] on big data benchmarking with the key challenges associated.

### Summary    ###

I hope that the papers are useful as you embark or strengthen your journey. I am sure there are few hundred more papers that I might have inadvertently missed and a whole bunch of systems that  I might be unfamiliar with - apologies in advance as don't mean to offend anyone though happy to be educated....

[AAEAAQAAAAAAAAMLAAAAJDRmYmE3MDY2LTcyODQtNDFhNS1iMDhlLTczMWJlMGQzNjgwNA.jpg]: https://media.licdn.com/mpr/mpr/shrinknp_800_800/AAEAAQAAAAAAAAMLAAAAJDRmYmE3MDY2LTcyODQtNDFhNS1iMDhlLTczMWJlMGQzNjgwNA.jpg
[Polyglot]: http://bigbe.su/lectures/2014/15.3.pdf
[BDAS]: http://datasys.cs.iit.edu/events/CCGrid2014/CCGrid-May25-Stoica.pdf
[Lambda]: http://jameskinley.tumblr.com/post/37398560534/the-lambda-architecture-principles-for
[Mor]: http://lambda-architecture.net/
[Kappa]: http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html
[SummingBird]: http://www.vldb.org/pvldb/vol7/p1441-boykin.pdf
[Data center as a computer]: http://www.morganclaypool.com/doi/pdf/10.2200/S00516ED2V01Y201306CAC024
[NOSQL Data Stores]: http://www.cattell.net/datastores/Datastores.pdf
[NoSQL Thesis]: http://www.christof-strauch.de/nosqldbs.pdf
[Large Scale Data Management]: http://webdocs.cs.ualberta.ca/~lengdong/papers/JCST14.pdf
[Eventual Consistency]: http://grail.csuohio.edu/~sschung/cis612/hadoopjoin_sigmod2010.pdf
[CAP Theorem]: http://www.cs.berkeley.edu/~rxin/db-papers/CAP.pdf
[parallel DBMS]: http://database.cs.brown.edu/sigmod09/benchmarks-sigmod09.pdf
[another]: http://database.cs.brown.edu/papers/stonebraker-cacm2010.pdf
[MapReduce]: http://www.cs.princeton.edu/courses/archive/spr11/cos448/web/docs/week10_reading2.pdf
[SQL on Hadoo]: http://www.vldb.org/pvldb/vol7/p1295-floratou.pdf
[RDD]: https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
[Google File System]: http://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf
[Hadoop File System]: http://zoo.cs.yale.edu/classes/cs422/2014fa/readings/papers/shvachko10hdfs.pdf
[Ceph File System]: http://ceph.com/papers/weil-ceph-osdi06.pdf
[alternative]: https://www.usenix.org/legacy/publications/login/2010-08/openpdfs/maltzahn.pdf
[Tachyon]: http://www.cs.berkeley.edu/~haoyuan/papers/2014_socc_tachyon.pdf
[Column Oriented vs Row-Stores]: http://db.csail.mit.edu/projects/cstore/abadi-sigmod08.pdf
[RCFile]: http://web.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-4.pdf
[Parquet]: https://github.com/Parquet/parquet-mr/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper
[ORCFile]: http://web.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-14-2.pdf
[Compression]: http://www.ijritcc.org/IJRITCC%20Vol_2%20Issue_3/A%20Survey%20on%20Compression%20Algorithms%20%20in%20Hadoop.pdf
[Erasure Codes]: http://web.eecs.utk.edu/~plank/plank/papers/Login-2013.pdf
[Hadoop]: http://anrg.usc.edu/~maheswaran/Xorbas.pdf
[Dynamo]: http://www.cs.ucsb.edu/~agrawal/fall2009/dynamo.pdf
[Cassandra]: https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf
[Voldemort]: http://static.usenix.org/events/fast/tech/full_papers/Sumbaly.pdf
[BigTable]: http://static.googleusercontent.com/media/research.google.com/en/us/archive/bigtable-osdi06.pdf
[HBase]: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.303.752&rep=rep1&type=pdf
[Hypertable]: http://www.hypertable.com/collateral/whitepaper-hypertable-architecture.pdf
[CouchDB]: https://media.readthedocs.org/pdf/couchdb/latest/couchdb.pdf
[MongoDB]: http://s3.amazonaws.com/info-mongodb-com/MongoDB_Architecture_Guide.pdf
[Neo4j]: http://info.neotechnology.com/rs/neotechnology/images/GraphDatabases.pdf
[Titan]: http://s3.thinkaurelius.com/docs/titan/0.9.0-M2/
[Megastore]: http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf
[Spanner]: http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-osdi2012.pdf
[MESA]: http://www.vldb.org/pvldb/vol7/p1259-gupta.pdf
[CockroachDB]: https://github.com/cockroachdb/cockroach/blob/master/docs/design.md
[YARN]: https://54e57bc8-a-62cb3a1a-s-sites.googlegroups.com/site/2013socc/home/program/a5-vavilapalli.pdf?attachauth=ANoY7co94J9PVjpjD5GD4z-S8e1O7YrLsqHssH7aeFReTJaoOBLbvLhq9HeDNb-PQz2jQvPUeQbDjJa2bctooZz5_zHCKWXAKZrYqAy_mVCLIQqU0Cc-sNQBHOJNsUTyVPfEdpHQ5yoIGVdIzoCnQwsFjbSX2ztS9b0OBNI2SjDCdvLE7Hsi5ktJINChoFa7w0ELgFvir4sEAJaL-G1qgmUglhOjVjHgwXYsqHH7FOPXrTVC-csZelo%3D&attredirects=0
[Mesos]: http://people.csail.mit.edu/matei/papers/2011/nsdi_mesos.pdf
[Capacity Scheduler]: https://hadoop.apache.org/docs/stable1/capacity_scheduler.pdf
[FairShare Scheduler]: http://www.valleytalk.org/wp-content/uploads/2013/03/fair_scheduler_design_doc.pdf
[Delayed Scheduling]: http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-55.pdf
[Fair _ Capacity schedulers]: http://arxiv.org/ftp/arxiv/papers/1207/1207.0780.pdf
[Paxos]: http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf
[classical]: http://research.microsoft.com/en-us/um/people/lamport/pubs/lamport-paxos.pdf
[Chubby]: http://static.googleusercontent.com/media/research.google.com/en/us/archive/chubby-osdi06.pdf
[Zookeeper]: https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf
[Spark]: https://www.usenix.org/system/files/login/articles/zaharia.pdf
[Flink]: http://events.linuxfoundation.org/sites/events/files/slides/flink_apachecon_small.pdf
[MapReduce 1]: http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf
[MapReduce Survey]: http://www.cs.arizona.edu/~bkmoon/papers/sigmodrec11.pdf
[Pregel]: http://kowshik.github.io/JPregel/pregel_paper.pdf
[Giraph]: http://researcher.ibm.com/researcher/files/us-heq/Large%20Scale%20Graph%20Processing%20with%20Apache%20Giraph.pdf
[GraphX]: https://amplab.cs.berkeley.edu/wp-content/uploads/2014/02/graphx.pdf
[Hama]: http://csl.skku.edu/papers/CS-TR-2010-330.pdf
[Open source graph processing]: http://www.vldb.org/pvldb/vol7/p1047-han.pdf
[Stream_Processing]: https://www.ucviden.dk/portal/files/26907191/Survey_of_real_time_processing_systems_for_big_data_Draft_.pdf
[Storm]: https://cs.brown.edu/courses/cs227/papers/ss-storm.pdf
[Samza]: http://www.jfokus.se/jfokus15/preso/ApacheSamza.pdf
[Spark Streaming]: http://people.csail.mit.edu/matei/papers/2013/sosp_spark_streaming.pdf
[Dremel]: http://www.vldb.org/pvldb/vldb2010/papers/R29.pdf
[Impala]: http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper28.pdf
[Drill]: http://wiki.apache.org/incubator/DrillProposal?action=AttachFile&do=get&target=Drill+slides.pdf
[Shark]: http://www.cs.berkeley.edu/~matei/papers/2012/sigmod_shark_demo.pdf
[Shark 1]: http://people.csail.mit.edu/matei/papers/2013/sigmod_shark.pdf
[Dryad]: http://www.news.cs.nyu.edu/~jinyang/sp07/papers/dryad.pdf
[Tez]: http://www.datanubes.com/mediac/ApacheTezPrimer.pdf
[BlinkDB]: https://www.cs.berkeley.edu/~sameerag/blinkdb_eurosys13.pdf
[Druid]: http://static.druid.io/docs/druid.pdf
[Pinot]: https://github.com/linkedin/pinot/wiki/Architecture
[Pig]: http://infolab.stanford.edu/~olston/publications/sigmod08.pdf
[Pig 1]: http://paperhub.s3.amazonaws.com/a7b584c04b61fabb8d10333e91989120.pdf
[Hive]: http://infolab.stanford.edu/~ragho/hive-icde2010.pdf
[Hive 1]: http://www.vldb.org/pvldb/2/vldb09-938.pdf
[Phoenix]: http://phoenix.apache.org/presentations/OC-HUG-2014-10-4x3.pdf
[Join Algorithms for Map Reduce]: http://www.ispras.ru/proceedings/docs/2012/23/isp_23_2012_285.pdf
[MLlib]: http://stanford.edu/~rezab/sparkworkshop/slides/xiangrui.pdf
[SparkR]: http://files.meetup.com/3138542/SparkR-meetup.pdf
[Mahout]: http://openresearch.baidu.com/u/cms/www/201210/30144944cqmu.pdf
[Flume]: https://blogs.apache.org/flume/entry/flume_ng_architecture
[Sqoop]: https://cwiki.apache.org/confluence/download/attachments/27361435/Cecho_Ting_SqoopBigDataTechCon.pdf?version=1&modificationDate=1366107169000&api=v2
[Kafka]: http://notes.stephenholiday.com/Kafka.pdf
[Crunch]: http://events.linuxfoundation.org/sites/events/files/slides/Simplifying%20Big%20Data%20with%20Apache%20Crunch.pdf
[Falcon]: http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Falcon/2.0.6.0-76/FalconHortonworksTechnicalPreview.pdf
[Cascading]: http://smokinn.com/files/cascading_notes/cascading.pdf
[Oozie]: http://oozie.apache.org/docs/4.2.0/index.html
[HCatalog]: https://cwiki.apache.org/confluence/display/Hive/HCatalog+UsingHCat
[ProtocolBuffers]: http://homepages.lasige.di.fc.ul.pt/~vielmo/notes/2014_02_12_smalltalk_protocol_buffers.pdf
[Avro]: http://mil-oss.org/resources/mil-oss-wg3_an-introduction-to-apache-avro_douglas-creager.pdf
[OpenTSDB]: http://opentsdb.net/overview.html
[Ambari]: https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf
[YCSB]: http://research.ijais.org/volume7/number8/ijais14-451229.pdf
[GridMix]: https://hadoop.apache.org/docs/stable1/gridmix.pdf
[Background]: http://arxiv.org/ftp/arxiv/papers/1402/1402.5194.pdf