Docker

I recently discovered a solution (a product) called Docker. This piece of software isolates a given job and its dependencies. Similar to virtual machine environment but this is light weight, and lot more storage friendly. This piece of software could store an application or a service in a bin. They call it a container. A…

Some notes on hadoop cluster

One way Passwordless SSH from Master to worker nodes: 1. Generate Key: ssh-keygen -t rsa 2. Create folder in worker node: ssh [user]@255.255.255.255 mkdir -p .ssh 3. Copy key to worker node: ssh-copy-id id_rsa.pub [user]@255.255.255.255 4. Enable permissions to worker: ssh [user]@255.255.255.255 "chmod 700 .ssh; chmod 640 .ssh/authorized_keys" 700 -- user can read, write and execute. group…

developing text processing data products: part I

Folks, this is going to be a series of information pieces (more than one blog post about same topic) about text processing. In this series, if intend to discuss some of my experiences and also take this moment to organize the discussion on my blog. In the past, if touched upon some of the text…

Setting up Apache Spark: Part II

Now that we have Hadoop YARN, Scala and other pre-requistes setup, we are now ready to install apache spark. If you are visiting this blog post for the first time, please do have a look at the earlier post Setting up Apache Spark: quick look at the Hadoop, Scala and SBT You definitely need maven,…

Setting up Apache Spark: Quick look at Hadoop, Scala and SBT

We are going to look at installing spark on a Hadoop. Lets try to setup hadoop yarn here once again with screenshots from scratch, as i received some comments that my installation needs more screenshots so i am doing one with screenshots. In this post, we will look at creating a new user account on…

Apache Spark: data processing engine for cluster computing

May I present Apache Spark, another apache licensed top-level project that could perform large scale data processing way faster than Hadoop (I am referring to MR1.0 here). It is possible due to Resilient Distributed Datasets concept that is behind this fast data processing. RDD is basically a collection of objects, spraed across a cluster stored…

Hadoop 2: YARN — Overview and Setup

We have already seen MapReduce. Now, lets dig deep into the new data processing model of Hadoop. Hadoop comes with a advanced resource management tool called YARN, and packaged as Hadoop-2.x. What is YARN? YARN stands for Yet Another Resource Navigator. YARN is also known as Hadoop Data Operating System -- YARN enables data processing…

Book Review: Optimizing Hadoop for MapReduce

I had a chance to review another book titled “Optimizing Hadoop for MapReduce” and must say this book is an good resource for devops professionals who build MapReduce programs in Hadoop. The book is well organized -- starts off with introducing basic concepts, identifying system bottlenecks and resource weaknesses, suggesting ways to fix and optimize them, followed…

Big Data Logistics: data transfer using Apache Sqoop from RDBMS

Apache Sqoop is a connectivity tool to perform data transfer between Hadoop and traditional databases (RDBMS) which contains structured data. Using sqoop, one can import data to Hadoop Distributed File System from RDBMS like Oracle, Teradata, MySQL, etc... and also export the data from Hadoop to any RDBMS in form of CSV file or direct…

Real-time analytics using distributed computing system Storm: Part II

How to setup a Storm cluster ? We will look at how to set up a single node cluster of storm project. The following are the prerequisites for setting up: Java 6 or above Python 2.6 Zookeeper ZeroMQ JZMQ any other dependencies (unzip, git, etc...) Zookeeper: Apache zookeeper project gives you a set of tools…