Join Optimization in Apache Hive

矫情吗；* 2022-08-23 11:59 212阅读 0赞

## 这个是很老的资料了，还是决定转进来，原文在此 https://www.facebook.com/note.php?note\_id=470667928919 ##

PPT在此 https://cwiki.apache.org/confluence/download/attachments/27362054/Hive+Summit+2011-join.pdf?version=1&modificationDate=1309986642000

## Join Optimization in Apache Hive ##

由  [Liyin Tang][] 于 2010年12月15日下午 12:04 发布

![164858_480172672199_1760925_n.jpg][]With more than 500 million users sharing a billion pieces of content daily, Facebook stores a vast amount of data, and needs a solid infrastructure to store and retrieve that data. This is why we use [Apache Hive][] and [Apache Hadoop][] so widely at Facebook. Hive is a data warehouse infrastructure built on top of Hadoop that can compile SQL queries as[MapReduce][] jobs and run the jobs in the cluster.

As performant as Hive and Hadoop are, there is always room for improvement. I was so excited that my internship project was to optimize performance of join, a very common SQL operation, in Hive.

**How Joins Work Today**

First, let's discuss how join works in Hive. A common join operation will be compiled to a MapReduce task, as shown in figure 1.  A common join task involves a map stage and a reduce stage. A mapper reads from join tables and emits the join key and join value pair into an intermediate file. Hadoop sorts and merges these pairs in what's called the shuffle stage. The reducer takes the sorted results as input and does the actual join work. The shuffle stage is really expensive since it needs to sort and merge. Saving the shuffle and reduce stages improves the task performance.

![Figure 1: Common Join in Hive][Figure 1_ Common Join in Hive]Figure 1: Common Join in Hive

The motivation of map join is to save the shuffle and reduce stages and do the join work only in the map stage. By doing so, when one of the join tables is small enough to fit into the memory, all the mappers can hold the data in memory and do the join work there. So all the join operations can be finished in the map stage. However there are some scaling problems with this type of map join. When thousands of mappers read the small join table from the [Hadoop Distributed File System (HDFS)][Hadoop Distributed File System _HDFS] into memory at the same time, the join table easily becomes the performance bottleneck, causing the mappers to time out during the read operations.

**Using the Distributed Cache**

[Hive-1641][] solves this scaling problem. The basic idea of optimization is to create a new MapReduce local task just before the original join MapReduce task. This new task reads the small table data from HDFS to an in-memory hash table. After reading, it serializes the in-memory hash table into a hashtable file. In the next stage, when the MapReduce task is launching, it uploads this hashtable file to the Hadoop distributed cache, which populates these files to each mapper's local disk. So all the mappers can load this persistent hashtable file back into memory and do the join work as before. The execution flow of the optimized map join is shown in figure 2. After optimization, the small table needs to be read just once. Also if multiple mappers are running on the same machine, the distributed cache only needs to push one copy of the hashtable file to this machine.

![Figure 2: The Optimized Map Join][Figure 2_ The Optimized Map Join]Figure 2: The Optimized Map Join

Since map join is faster than the common join, it's better to run the map join whenever possible. Previously, Hive users needed to give a hint in the query to specify the small table. For example, `select /*+mapjoin(a)*/ * from src1 x join src2 y on x.key=y.key;`.

This isn't a good user experience because sometimes the user may give the wrong hint or may not give any hint at all. It's much better to convert the common join into a map join without user hints.

**Converting Joins to Map Joins Based on Size**

[Hive-1642][] solves this problem by converting the common join into a map join automatically. For the map join, the query processor should know which input table is the big table. The other input tables are recognized as the small tables during the execution stage, and these tables need to be held in the memory. However, in general, the query processor has no idea of input file size during compilation time because some of the tables may be intermediate tables generated from sub-queries. So the query processor can only figure out the input file size during the execution time.

![Figure 3: The Join Execution Flow][Figure 3_ The Join Execution Flow]Figure 3: The Join Execution Flow

As shown in figure 3, the left side flow shows the previous common join execution flow, which is very straightforward. On the other side, the right side flow is the new common join execution flow. During compilation time, the query processor generates a conditional task containing a list of tasks; one of these tasks gets resolved to run during execution time. First, the original common join task should be put into the task list. Then the query processor generates a series of map join tasks by assuming each of the input tables may be the big table. For example, `select * from src1 x join src2y on x.key=y.key`. Because both tables src2 and src1 can be the big table, the processor generates two map join tasks, with one assuming src1 is the big table and the other assuming src2 is the big table.

During the execution stage, the conditional task knows the exact file size of each input table, even if the table is an intermediate one. If all the tables are too large to be converted into map join, then just run the common join task as previously. If one of the tables is large and others are small enough to run map join, then the conditional task will pick the corresponding map join local task to run. By this mechanism, it can convert the common join into a map join automatically and dynamically.

Currently, if the total size of small tables is larger than 25MB, then the conditional task will choose the original common join to run. 25MB is a very conservative number and you can change this number with set hive.smalltable.filesize.

**Measuring Performance Improvement**

Now let's see how much of a performance improvement we can get after the map join optimization.

![Table 1: The comparison between the previous map join with the new optimized map join][Table 1_ The comparison between the previous map join with the new optimized map join]Table 1: The comparison between the previous map join with the new optimized map join

As shown in table 1, the optimized map join is 12 to 26 times faster than the previous one. Most of map join performance improvement comes from removing the [JDBM][] component.

Also, let's see how much performance improvement we can get if a common join can be converted into map join. All the join operations in the benchmarks can be converted into map join.

![Table 2: The comparison between the previous join with the new optimized join][Table 2_ The comparison between the previous join with the new optimized join]Table 2: The comparison between the previous join with the new optimized join

From the results shown in table 2, if the new common join can be converted into map join, it will get 57% - 163% performance improvements.

In order to measure the benefits from this project - we would like to track all instances where join operations are converted to map joins (and similarly cases where the map-join optimization fails because the mappers run out of memory). [Hive-1792][] allows us to capture such tracking information in a generic way. For our Hive deployment, I developed a Hive execution hook to read the tracking information provided by this issue and record it in an internal database. After this work, we can know exactly how many common joins have been converted into map joins everyday and how much CPU time is saved in the cluster.

**Looking Ahead**

There are a few optimizations remaining to be done. One tricky aspect is setting up the number of replicas for the compressed hashtable files in HDFS. Currently, the number of replicas is initially set to 3 and (asynchronously) increased to 10. But 10 replicas may be insufficient when thousands of nodes need to read this file (and load it into the distributed cache). In addition, the lag time to increase the replication from 3 to 10 is non-deterministic and causes large variations in mapper performance. So we need to figure out the replication factor  dynamically based on the number of mappers as well as achieve the higher replication factor in a deterministic and synchronous manner.

We also bounded the table size of small table to be a maximum of 25MB. We were conservative in coming up with this limit because in the worst case, two factors may hold true. First, the file may be compressed. Second, all the rows in the file may be unique. In this worst case scenario, a file of this size can contain more than 100 million rows. Because of this conservative limit, we may lose many opportunities to convert joins into map joins. A smarter strategy may be to sample the input file to better estimate the number of rows and decide to try the map join strategy based on this estimate.

Finally, the local task can hold 2 million unique key/value pairs in memory, but it consumes 1.47GB of memory. By optimizing the code of hashtable implementations to be more memory efficient, the local task can hold more data in memory, allowing more join queries to be converted into map joins. There are many ideas to explore here, like using hash tables optimized for holding primitive values, which use fewer objects.

*Liyin is an intern on our data infrastructure engineering team.*

(Originally [posted on the Hive wiki][].)

[赞][Link 1] ·  [评论][Link 1] ·  [分享][Link 2]

*  Alexandra Paun 、 Anoop Raj 、 Stephen Clv 和其他 567 人赞了
 *  [29次分享][29]
 *  39 / 68
    
    [查看之前的评论][Link 3]
 *  ![1560384_10151930941332916_1617142730_n.jpg][]
    
    **Yader Hernandez**   @Phill there is a function available to remove a request. When you go to the person's profile page, there is an "x" next to the request. Click it to remove the request. I know it's not the most obvious thing  
    
    [2010年12月16日上午 5:23][2010_12_16_ 5_23]
 *  ![405274_455984084435543_1853072031_n.jpg][]
    
    **Saurav Verma**   Yeah..Hadoop was inspired by Google's MapReduce but its Hive and Pig projects are now one of the industry standards being used in most of the top organizations around the World.  
    I wonder waht is in store for next...maybe a Galaxy project now  
    
    [2010年12月16日上午 5:51][2010_12_16_ 5_51]
 *  ![1604378_705291046159239_1174458786_n.jpg][]
    
    **Tarkeshwar Chauragade**   great
    
    [2010年12月16日上午 5:55][2010_12_16_ 5_55]
 *  ![10298775_699244083468973_2534807801777734243_n.jpg][]
    
    **Subhash Kumar**   Its really amazing.......!
    
    [2010年12月16日上午 7:58][2010_12_16_ 7_58]
 *  ![581026_368906283151976_1564157159_n.jpg][]
    
    **Dewan Shapon**   thanks for sharing
    
    [2010年12月16日上午 8:16][2010_12_16_ 8_16]
 *  ![37009_136700026342588_3132826_n.jpg][]
    
    **Charles Daniels**   
    
    [2010年12月16日下午 4:41][2010_12_16_ 4_41]  ·  [1][]
 *  ![10448822_10202410799410511_6628766414743125371_n.jpg][]
    
    **Arshad Mahmood**   nice!!
    
    [2010年12月17日上午 4:46][2010_12_17_ 4_46]
 *  ![10517558_692817970788020_661091984096377165_n.jpg][]
    
    **Syueb Ghani**   I like
    
    [2010年12月17日上午 5:55][2010_12_17_ 5_55]
 *  ![10269357_10204229086672107_8809710982669442069_n.jpg][]
    
    **Abisoye Falabi**   Must Confess This Is Great,You Guys Should keep Up The Good Work..Better Still Would Love to Learn All that is Been Used..
    
    [2010年12月24日上午 10:34][2010_12_24_ 10_34]
 *  ![1920266_10151985384848284_1190774018_n.jpg][]
    
    **Sathiya N Sundarajan**   Great stuff, can't appreciate enough for sharing..
    
    [2010年12月26日下午 5:09][2010_12_26_ 5_09]
 *  ![168490_175544295823110_5611881_n.jpg][]
    
    **Smarty Smart**   b 4 facebook gud 2 use in mobile now very difficult 2 use each time rply then go back big prob. Make easier how b 4 thanks fb
    
    [2011年1月8日上午 12:02][2011_1_8_ 12_02]
 *  ![10430505_10152316439730415_2149715188909542610_n.jpg][]
    
    **Abu Baker Siddique**   like it
    
    [2011年1月11日下午 9:29][2011_1_11_ 9_29]
 *  ![10264732_10150002137498343_3701109639787196793_n.jpg][]
    
    **Sarah Yu**   Hadoop and Hive are awesome! Nice job!
    
    [2011年1月23日下午 8:37][2011_1_23_ 8_37]
 *  ![63374_495028280537792_1709504568_n.jpg][]
    
    **Raghav Prabhu**   Nice to see that joins are using in map reduce arena...  
    
    [2011年2月5日上午 10:34][2011_2_5_ 10_34]
 *  ![10390289_10203191667384331_8629569840883688716_n.jpg][]
    
    **Martin Konicek**   In Figure 2, how do the mappers know the offsets of corresponding keys in "Big table data"? Is the big table sorted by key or indexed?
    
    [2011年4月25日下午 1:05][2011_4_25_ 1_05]
 *  ![1174697_3482169590878_900155_n.jpg][]
    
    **Trang van Anh**   that's great! thanks so much for sharing
    
    [2011年10月14日下午 8:19][2011_10_14_ 8_19]
 *  ![316921_105749946198796_416854969_n.jpg][]
    
    **Insung Jung**   gooooooooooooooooood~!
    
    [2011年10月17日上午 3:17][2011_10_17_ 3_17]
 *  ![10492228_10204320920891486_7909458462685807644_n.jpg][]
    
    **James Jara**   veeryy good
    
    [2012年2月7日下午 2:38][2012_2_7_ 2_38]
 *  ![535551_363314270371070_696216533_n.jpg][]
    
    **Samiksha Chavan**   We have an 5 Node cluster , trying to join on 500 GB table with small table of 4MB and query is taking 1 Hours 15Mins after using Hasp Map Join, Is there any other way to improve the performance?
    
    [2012年2月9日下午 10:49][2012_2_9_ 10_49]
 *  ![10273597_10203967003406268_1071355877160337145_n.jpg][]
    
    **Abhishek Sagar**   does somebody has any useful literature/code to perform SQL-joins in hadoopDB clusters ?
    
    [2012年2月11日下午 11:10][2012_2_11_ 11_10]
 *  ![379496_2318497176007_46813453_n.jpg][]
    
    **Srinivasan Rajivelu**   Excellent idea.. Very Clean Design.. Thanks for Sharing..
    
    [2012年3月19日上午 9:33][2012_3_19_ 9_33]
 *  ![1148858_10151525340646780_806446190_n.jpg][]
    
    **Roberto Congiu**   where should the hint go if the query is written as FROM( select a,b from t1) tj1 JOIN tj1 ON (tj1.id = tj2.id) INSERT OVERWRITE TABLE destable PARTITION(dt='2012-01-01') SELECT \* ?
    
    [2012年3月26日下午 4:03][2012_3_26_ 4_03]
 *  ![10455965_10152136747961695_4056934285683281102_n.jpg][]
    
    **Arjun Prashant**   Can any one let me know the year when the first edition of hive was released ??
    
    [2012年4月12日下午 11:19][2012_4_12_ 11_19]
 *  ![954801_10150002137498316_604636659114323291_n.jpg][]
    
    **DarkAngelvn VietNam**   Wow,greater
    
    [2012年4月29日上午 4:23][2012_4_29_ 4_23]
 *  ![1551699_731556130189781_2093700256_n.jpg][]
    
    **Ratan Kumar Nath**   Hi, Is there any alternative available for ORDER BY. Because it only use one reducer?
    
    [2012年5月28日上午 5:00][2012_5_28_ 5_00]
 *  ![321331_415164905258426_489101201_n.jpg][]
    
    **Gaurav Bhasin**   how to reduce the latency time in hive query (basic select )
    
    [2012年5月29日上午 5:45][2012_5_29_ 5_45]
 *  ![1477618_10202752563671857_818220001_n.jpg_oh_695ca56e2cd832273fca875346050bf2_oe_543AB8CC_gda_1414293546_1fe80ae4ce48ef800363d4cc00f82d48][]
    
    **Veleti Kishore**   Great Article!!! Thanks for the detailed documentation.
    
    [2012年8月25日下午 5:56][2012_8_25_ 5_56]
 *  ![60783_10151163203822075_540207340_n.jpg_oh_df0c411e5195a7b6a785af09ea4c6e3a_oe_54556800_gda_1414719189_0a65af051299dc9b3f115004362ba077][]
    
    **Sanjith Jacob Thomas**   excellent!!!
    
    [2012年11月17日上午 7:18][2012_11_17_ 7_18]
 *  ![1982168_10202374385893784_1958536803_n.jpg][]
    
    **Krishnendu Mitra**   Good to Know , how about a scenarios , given that both the tables are Big tables say 15 billion records each with file size of say 450 GB , how would it effect increasing the hive.smalltable.filesize ?
    
    [2012年12月3日上午 3:41][2012_12_3_ 3_41]
 *  ![10463869_10152560371557450_8089651904684954446_n.jpg][]
    
    **Balaji Gandhi**   Brilliant! Thanks for sharing.
    
    [2013年1月18日下午 8:27][2013_1_18_ 8_27]
 *  ![1743541_10201945475571894_1033798319_n.jpg][]
    
    **Ali Shan Zaidi**   great!!!! thanks for sharing "Liyin Tang"  .
    
    [2013年2月9日上午 12:22][2013_2_9_ 12_22]
 *  ![1795737_10151995508386376_1142804125_n.jpg][]
    
    **Namit Jain**   The correct parameter is hive.mapjoin.smalltable.size, not hive.smalltable.size as mentioned in the note.
    
    [2013年3月27日下午 10:49][2013_3_27_ 10_49]
 *  ![522484_3196775532807_73466768_n.jpg][]
    
    **Hassan Raja Azad**   very nice
    
    [2013年4月15日上午 2:56][2013_4_15_ 2_56]
 *  ![10154442_557410811042820_2408872431171489894_n.jpg][]
    
    **Linh Nguyễn Mạnh**   great! thank a lot!
    
    [2013年5月3日上午 3:08][2013_5_3_ 3_08]
 *  ![1240518_520603324680966_1250299486_n.jpg][]
    
    **YoonMin Nam**   Very helpful articles for my interest!
    
    [2013年5月5日下午 6:39][2013_5_5_ 6_39]
 *  ![954801_10150002137498316_604636659114323291_n.jpg][]
    
    **Abhishek Gupta**   There is one question about the effectiveness of the Distributed cache. As first case all the Mappers were reading the data from the HDFS because of which HDFS became the bottleneck for system. Similarly the distributed cache can become the bottleneck at some point. Can any one explain more on this issue. In what scenario distributed cache might be a bad option ?
    
    [2013年6月10日上午 5:50][2013_6_10_ 5_50]
 *  ![954801_10150002137498316_604636659114323291_n.jpg][]
    
    **Abhishek Gupta**   Please suggest any link where I can find details about Bucket Map join.
    
    [2013年6月10日上午 5:56][2013_6_10_ 5_56]
 *  ![10563176_600786113373485_7394130963474234951_n.jpg][]
    
    **Mukesh Chakarwarti**   pls suggest with any video...
    
    [2013年7月10日上午 1:30][2013_7_10_ 1_30]
 *  ![561472_360978893985661_1666469339_n.jpg_oh_4d3057aef51dd2ead7d1f280f6f177b9_oe_544A09C6_gda_1414418932_ad9a8c6ab5840e04b258ff225e41cffe][]
    
    **Marco Villarreal**   Hive is a really awesome platform, it would be nice more use cases like this, it makes you analyze what you do and how to improve it. A bucket Mapjoin video would be great!!!!
    
    [2013年12月10日下午 6:29][2013_12_10_ 6_29]

[Liyin Tang]: https://www.facebook.com/liyin.tang
[164858_480172672199_1760925_n.jpg]: https://scontent-a-ord.xx.fbcdn.net/hphotos-xap1/t1.0-9/s180x540/164858_480172672199_1760925_n.jpg
[Apache Hive]: http://l.facebook.com/l.php?u=http%3A%2F%2Fhadoop.apache.org%2Fhive%2F&h=8AQHnKEbE&s=1
[Apache Hadoop]: http://l.facebook.com/l.php?u=http%3A%2F%2Fhadoop.apache.org%2F&h=7AQGm4u3Y&s=1
[MapReduce]: http://l.facebook.com/l.php?u=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FMapReduce&h=cAQGxypcG&s=1
[Figure 1_ Common Join in Hive]: https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-xap1/t1.0-9/164612_480172777199_5365634_n.jpg
[Hadoop Distributed File System _HDFS]: http://l.facebook.com/l.php?u=http%3A%2F%2Fhadoop.apache.org%2Fhdfs%2F&h=LAQEnfUjo&s=1
[Hive-1641]: https://www.facebook.com/l.php?u=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-1641&h=2AQGgNFb8&s=1
[Figure 2_ The Optimized Map Join]: https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-xfa1/t1.0-9/164179_480173072199_3345662_n.jpg
[Hive-1642]: https://www.facebook.com/l.php?u=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-1642&h=PAQH3ssyt&s=1
[Figure 3_ The Join Execution Flow]: https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-xpf1/t1.0-9/156668_480173297199_8376805_n.jpg
[Table 1_ The comparison between the previous map join with the new optimized map join]: https://fbcdn-sphotos-d-a.akamaihd.net/hphotos-ak-prn2/t1.0-9/155070_480173432199_2480453_n.jpg
[JDBM]: http://l.facebook.com/l.php?u=http%3A%2F%2Fjdbm.sourceforge.net%2F&h=lAQGL2eWN&s=1
[Table 2_ The comparison between the previous join with the new optimized join]: https://scontent-b-ord.xx.fbcdn.net/hphotos-xfa1/t1.0-9/163066_480173572199_3267285_n.jpg
[Hive-1792]: https://www.facebook.com/l.php?u=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHIVE-1792&h=oAQFJhlR_&s=1
[posted on the Hive wiki]: http://l.facebook.com/l.php?u=http%3A%2F%2Fwiki.apache.org%2Fhadoop%2FHive%2FJoinOptimization&h=_AQHjGVjD&s=1
[Link 1]: https://www.facebook.com/r.php?fbpage_id=9445547199&r=111
[Link 2]: https://www.facebook.com/ajax/sharer/?s=4&appid=2347471856&p[0]=9445547199&p[1]=470667928919&share_source_type=unknown&__av=
[29]: https://www.facebook.com/shares/view?id=470667928919
[Link 3]: https://www.facebook.com/note.php?note_id=470667928919#
[1560384_10151930941332916_1617142730_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/p32x32/1560384_10151930941332916_1617142730_n.jpg
[2010_12_16_ 5_23]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14256755&offset=0&total_comments=68
[405274_455984084435543_1853072031_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-frc3/t1.0-1/c28.28.345.345/s32x32/405274_455984084435543_1853072031_n.jpg
[2010_12_16_ 5_51]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14256855&offset=0&total_comments=68
[1604378_705291046159239_1174458786_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c11.0.32.32/p32x32/1604378_705291046159239_1174458786_n.jpg
[2010_12_16_ 5_55]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14256870&offset=0&total_comments=68
[10298775_699244083468973_2534807801777734243_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpf1/t1.0-1/p32x32/10298775_699244083468973_2534807801777734243_n.jpg
[2010_12_16_ 7_58]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14257483&offset=0&total_comments=68
[581026_368906283151976_1564157159_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/p32x32/581026_368906283151976_1564157159_n.jpg
[2010_12_16_ 8_16]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14257599&offset=0&total_comments=68
[37009_136700026342588_3132826_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xaf1/t1.0-1/c97.22.269.269/s32x32/37009_136700026342588_3132826_n.jpg
[2010_12_16_ 4_41]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14259643&offset=0&total_comments=68
[1]: https://www.facebook.com/browse/likes?id=471271733919
[10448822_10202410799410511_6628766414743125371_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c6.0.32.32/p32x32/10448822_10202410799410511_6628766414743125371_n.jpg
[2010_12_17_ 4_46]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14262451&offset=0&total_comments=68
[10517558_692817970788020_661091984096377165_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/t1.0-1/c5.0.32.32/p32x32/10517558_692817970788020_661091984096377165_n.jpg
[2010_12_17_ 5_55]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14262766&offset=0&total_comments=68
[10269357_10204229086672107_8809710982669442069_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c9.0.32.32/p32x32/10269357_10204229086672107_8809710982669442069_n.jpg
[2010_12_24_ 10_34]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14305146&offset=0&total_comments=68
[1920266_10151985384848284_1190774018_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/c0.3.32.32/p32x32/1920266_10151985384848284_1190774018_n.jpg
[2010_12_26_ 5_09]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14317036&offset=0&total_comments=68
[168490_175544295823110_5611881_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/t1.0-1/c22.10.127.127/s32x32/168490_175544295823110_5611881_n.jpg
[2011_1_8_ 12_02]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14389001&offset=0&total_comments=68
[10430505_10152316439730415_2149715188909542610_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/p32x32/10430505_10152316439730415_2149715188909542610_n.jpg
[2011_1_11_ 9_29]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14413770&offset=0&total_comments=68
[10264732_10150002137498343_3701109639787196793_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpf1/t1.0-1/c9.0.32.32/p32x32/10264732_10150002137498343_3701109639787196793_n.jpg
[2011_1_23_ 8_37]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14484551&offset=0&total_comments=68
[63374_495028280537792_1709504568_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/t1.0-1/p32x32/63374_495028280537792_1709504568_n.jpg
[2011_2_5_ 10_34]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=14563629&offset=0&total_comments=68
[10390289_10203191667384331_8629569840883688716_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/p32x32/10390289_10203191667384331_8629569840883688716_n.jpg
[2011_4_25_ 1_05]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=15827208&offset=0&total_comments=68
[1174697_3482169590878_900155_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/t1.0-1/c115.58.730.730/s32x32/1174697_3482169590878_900155_n.jpg
[2011_10_14_ 8_19]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=18819830&offset=0&total_comments=68
[316921_105749946198796_416854969_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xaf1/t1.0-1/c45.0.371.371/s32x32/316921_105749946198796_416854969_n.jpg
[2011_10_17_ 3_17]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=18851187&offset=0&total_comments=68
[10492228_10204320920891486_7909458462685807644_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/c0.0.32.32/p32x32/10492228_10204320920891486_7909458462685807644_n.jpg
[2012_2_7_ 2_38]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=20562624&offset=0&total_comments=68
[535551_363314270371070_696216533_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-frc3/t1.0-1/c49.49.617.617/s32x32/535551_363314270371070_696216533_n.jpg
[2012_2_9_ 10_49]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=20595441&offset=0&total_comments=68
[10273597_10203967003406268_1071355877160337145_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c0.0.32.32/p32x32/10273597_10203967003406268_1071355877160337145_n.jpg
[2012_2_11_ 11_10]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=20619971&offset=0&total_comments=68
[379496_2318497176007_46813453_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/c50.50.621.621/s32x32/379496_2318497176007_46813453_n.jpg
[2012_3_19_ 9_33]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21082646&offset=0&total_comments=68
[1148858_10151525340646780_806446190_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-frc3/t1.0-1/p32x32/1148858_10151525340646780_806446190_n.jpg
[2012_3_26_ 4_03]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21172789&offset=0&total_comments=68
[10455965_10152136747961695_4056934285683281102_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/p32x32/10455965_10152136747961695_4056934285683281102_n.jpg
[2012_4_12_ 11_19]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21377446&offset=0&total_comments=68
[954801_10150002137498316_604636659114323291_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfp1/t1.0-1/c9.0.32.32/p32x32/954801_10150002137498316_604636659114323291_n.jpg
[2012_4_29_ 4_23]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21571291&offset=0&total_comments=68
[1551699_731556130189781_2093700256_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c5.0.32.32/p32x32/1551699_731556130189781_2093700256_n.jpg
[2012_5_28_ 5_00]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21903196&offset=0&total_comments=68
[321331_415164905258426_489101201_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/c170.50.621.621/s32x32/321331_415164905258426_489101201_n.jpg
[2012_5_29_ 5_45]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=21914354&offset=0&total_comments=68
[1477618_10202752563671857_818220001_n.jpg_oh_695ca56e2cd832273fca875346050bf2_oe_543AB8CC_gda_1414293546_1fe80ae4ce48ef800363d4cc00f82d48]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpf1/v/t1.0-1/c8.0.32.32/p32x32/1477618_10202752563671857_818220001_n.jpg?oh=695ca56e2cd832273fca875346050bf2&oe=543AB8CC&__gda__=1414293546_1fe80ae4ce48ef800363d4cc00f82d48
[2012_8_25_ 5_56]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=22810317&offset=0&total_comments=68
[60783_10151163203822075_540207340_n.jpg_oh_df0c411e5195a7b6a785af09ea4c6e3a_oe_54556800_gda_1414719189_0a65af051299dc9b3f115004362ba077]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/v/t1.0-1/c204.44.551.551/s32x32/60783_10151163203822075_540207340_n.jpg?oh=df0c411e5195a7b6a785af09ea4c6e3a&oe=54556800&__gda__=1414719189_0a65af051299dc9b3f115004362ba077
[2012_11_17_ 7_18]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=23695223&offset=0&total_comments=68
[1982168_10202374385893784_1958536803_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/t1.0-1/c12.0.32.32/p32x32/1982168_10202374385893784_1958536803_n.jpg
[2012_12_3_ 3_41]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=23856419&offset=0&total_comments=68
[10463869_10152560371557450_8089651904684954446_n.jpg]: https://scontent-b-ord.xx.fbcdn.net/hprofile-xpf1/l/t1.0-1/p32x32/10463869_10152560371557450_8089651904684954446_n.jpg
[2013_1_18_ 8_27]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=24332435&offset=0&total_comments=68
[1743541_10201945475571894_1033798319_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c1.0.32.32/p32x32/1743541_10201945475571894_1033798319_n.jpg
[2013_2_9_ 12_22]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=24541718&offset=0&total_comments=68
[1795737_10151995508386376_1142804125_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xaf1/t1.0-1/c5.0.32.32/p32x32/1795737_10151995508386376_1142804125_n.jpg
[2013_3_27_ 10_49]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=24964409&offset=0&total_comments=68
[522484_3196775532807_73466768_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-frc1/t1.0-1/c5.0.32.32/p32x32/522484_3196775532807_73466768_n.jpg
[2013_4_15_ 2_56]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=25117307&offset=0&total_comments=68
[10154442_557410811042820_2408872431171489894_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpf1/t1.0-1/c8.0.32.32/p32x32/10154442_557410811042820_2408872431171489894_n.jpg
[2013_5_3_ 3_08]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=25315904&offset=0&total_comments=68
[1240518_520603324680966_1250299486_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xpa1/t1.0-1/c30.30.371.371/s32x32/1240518_520603324680966_1250299486_n.jpg
[2013_5_5_ 6_39]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=25344020&offset=0&total_comments=68
[2013_6_10_ 5_50]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=25738326&offset=0&total_comments=68
[2013_6_10_ 5_56]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=25738373&offset=0&total_comments=68
[10563176_600786113373485_7394130963474234951_n.jpg]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfa1/t1.0-1/c2.0.32.32/p32x32/10563176_600786113373485_7394130963474234951_n.jpg
[2013_7_10_ 1_30]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=26066863&offset=0&total_comments=68
[561472_360978893985661_1666469339_n.jpg_oh_4d3057aef51dd2ead7d1f280f6f177b9_oe_544A09C6_gda_1414418932_ad9a8c6ab5840e04b258ff225e41cffe]: https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xap1/v/t1.0-1/c113.33.414.414/s32x32/561472_360978893985661_1666469339_n.jpg?oh=4d3057aef51dd2ead7d1f280f6f177b9&oe=544A09C6&__gda__=1414418932_ad9a8c6ab5840e04b258ff225e41cffe
[2013_12_10_ 6_29]: https://www.facebook.com/notes/facebook-engineering/join-optimization-in-apache-hive/470667928919?comment_id=27720657&offset=0&total_comments=68