Download e-book for kindle: Apache Flume: Distributed Log Collection for Hadoop - Second by Steve Hoffman

About This Book

• Construct a chain of Flume brokers utilizing the Apache Flume provider to successfully gather, combination, and circulation quite a lot of occasion data
• Configure failover paths and cargo balancing to take away unmarried issues of failure
• Use this step by step consultant to movement logs from program servers to Hadoop's HDFS

Who This publication Is For

If you're a Hadoop programmer who desires to find out about Flume so that it will stream datasets into Hadoop in a well timed and replicable demeanour, then this booklet is perfect for you. No earlier wisdom approximately Apache Flume is critical, yet a simple wisdom of Hadoop and the Hadoop dossier approach (HDFS) is assumed.

What you are going to Learn

• Understand the Flume structure, and likewise the right way to obtain and set up open resource Flume from Apache
• Follow alongside an in depth instance of transporting weblogs in close to actual Time (NRT) to Kibana/Elasticsearch and archival in HDFS
• Learn advice and tips for transporting logs and information on your creation environment
• Understand and configure the Hadoop dossier process (HDFS) Sink
• Use a morphline-backed Sink to feed information into Solr
• Create redundant facts flows utilizing sink groups
• Configure and use a variety of assets to ingest data
• Inspect information files and circulate them among a number of locations in response to payload content
• Transform info en-route to Hadoop and video display your facts flows

In Detail

Apache Flume is a disbursed, trustworthy, and to be had provider used to successfully gather, combination, and circulate quite a lot of log facts. it's used to circulate logs from program servers to HDFS for advert hoc analysis.

This publication starts off with an architectural evaluate of Flume and its logical parts. It explores channels, sinks, and sink processors, via assets and channels. via the top of this publication, you may be totally built to build a chain of Flume brokers to dynamically shipping your circulate facts and logs out of your platforms into Hadoop.

A step by step e-book that courses you thru the structure and elements of Flume overlaying diversified methods, that are then pulled jointly as a real-world, end-to-end use case, progressively going from the easiest to the main complex features.