NPN Training - Page 3 of 8 - Training is the essence of success & we are committed to it.

November 16, 2018

Hadoop Quiz Set 01

Uncategorized

November 11, 2018

Ideal size of HDFS block to get maximum performance

Table of Contents1 HDFS2 Block3 Block Size4 What happens when the block size is small5 What happens when the block size is large6 Important Points to consider while choosing Block Size HDFS HDFS Stands for Hadoop Distributed File System is the worlds most reliable Distributed Storage System. HDFS is a FileSystem designed for storing very large files. Block In Hadoop...

Hadoop

October 28, 2018

repartition() vs coalesce() in Spark

Spark splits data into partitions and computation is done in parallel for each partition. It is very important to understand how data is partitioned and when you need to manually modify the partitioning to run spark application efficiently. In Spark RDD API there are 2 methods available to increase or decrease the number of partitions. repartition() method and coalesce() method...

Apache Spark/RDD

October 26, 2018

cache() vs persist() methods in Spark

cache() and persist() are 2 methods available in Spark to improve performance of spark computation. These methods help to save intermediate results so they can be reused in subsequent stages. These interim results as RDDs are thus kept in Memory (default) or Solid storage like disk and/or replicated. RDDs can be cached using cache operation and they can also be persisted using...

Apache Spark/RDD

October 12, 2018

Kafka Commands

How to start a zookeeper ? > zkServer.sh start How to start a Kafka Broker ? > kafka-server-start.sh /home/npntraining/opt/kafka_2.11-0.10.2.1/config/server.properties How to create a topic ? > kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic Hello-Kafka

Kafka

October 11, 2018

Aggregation on Text Fields in the MapReduce Example

Table of Contents1 Introduction2 Here is how to set things up to run the above MapReduce job:2.1 Create an Executable Jar containing your MapReduce classes2.2 Create a working Hadoop instance2.3 You must first have a working Hadoop installation to run this on. I personally like to create a Docker container using the sequenceiq/docker-spark image.2.4 Create an HDFS directory for your...

Hadoop/MapReduce

October 11, 2018

Compressing Intermediary Results of Map Output in Hadoop

It is generally recommended to always compress Intermediary map output. This is because IO operations and network transfers are biggest bottlenecks in Hadoop, and compression can help with both of these issues. Map output is written to local disk, and then transferred (shuffled) across the network to reducer nodes. At this point in a MapReduce job, we are no longer...

Hadoop/MapReduce

October 11, 2018

Changing the Output Delimiter in t

Table of Contents1 Introduction2 Output Delimiter Configuration Property3 Example Introduction Hadoop’s default output delimiter (character isolating the output key and value) is tab (“\t”). This post explains the best approach to alter the default Hadoop output delimiter. Output Delimiter Configuration Property The output delimiter of a Hadoop job can easily be altered by Changing the mapred.textoutputformat.separator configuration property. This property can...

Hadoop/MapReduce

October 10, 2018

WRITING DATA FROM APACHE KAFKA TO TEXT FILE

When working with Apache Kafka you might want to write data from a Kafka topic to a local text file. This is actually very easy to do with Kafka Connect. Kafka Connect is a framework that provides scalable and reliable streaming of data to and from Apache Kafka. With Kafka Connect, writing a topic’s content to a local text file...

Kafka

October 7, 2018

Working with Nested JSON in Spark

JSON could be a quite common way to store information. however JSON will get untidy and parsing it will get tough. Here are some samples of parsing nested data structures in JSON Spark DataFrames (examples here finished Spark one.6.0). Sample JSON File: { “user”: “gT35Hhhre9m”, “dates”: [“2016-01-29”, “2016-01-28”], “status”: “OK”, “reason”: “some reason”, “content”: [{ “foo”: 123, “bar”: “val1″...

Apache Spark/Spark SQL