Creating HDFS policy in Ranger User interface   Apache Ranger is a policy based security tool for Hadoop eco system tools. Ranger provides security policies for tools like HDFS YARN Hive Knox HBase and Storm. In this article we will learn how to create HDFS policy in Apache Ranger UI.1) Create a folder in HDFS. We will create an HDFS directory /user/hdfs/ranger to test Ranger HDFS policies. We will be creating directory /user/hdfs/ranger using hdfs user. hdfs dfs -mkdir /user/hdfs/ranger…

Creating HDFS policy in Ranger User interface Apache Ranger is a policy based security tool for Hadoop eco system tools. Ranger provides security policies for tools like HDFS YARN Hive Knox HBase and Storm. In this article we will learn how to create HDFS policy in Apache Ranger UI.1) Create a folder in HDFS. We will create an HDFS directory /user/hdfs/ranger to test Ranger HDFS policies. We will be creating directory /user/hdfs/ranger using hdfs user. hdfs dfs -mkdir /user/hdfs/ranger…

Starting and stopping Ambari agents   In this article we will learn how to work with Ambari agents.We will learn how to start stop  restart and some more operations of ambari agents from command line. If you are not using root user you have to prefix sudo to all commands listed in below steps.1) Check the status of Ambari agents. This command tells us wether Ambari agent is in running state or not. If Ambari agent is in running state  Command also gives us pid of the Ambari agent process…

Starting and stopping Ambari agents In this article we will learn how to work with Ambari agents.We will learn how to start stop restart and some more operations of ambari agents from command line. If you are not using root user you have to prefix sudo to all commands listed in below steps.1) Check the status of Ambari agents. This command tells us wether Ambari agent is in running state or not. If Ambari agent is in running state Command also gives us pid of the Ambari agent process…

Starting and stopping Ambari-server   In this article we will learn how to work with Ambari server.We will learn how to start stop  restart and some more operations of ambari server from command line. If you are not using root user you have to prefix sudo to all commands listed in below steps.1) Check the status of Ambari server. You can use any one of the commands to check status of ambari server. ambari-server status OR sudoambari-server status ORservice ambari-server status This command…

Starting and stopping Ambari-server In this article we will learn how to work with Ambari server.We will learn how to start stop restart and some more operations of ambari server from command line. If you are not using root user you have to prefix sudo to all commands listed in below steps.1) Check the status of Ambari server. You can use any one of the commands to check status of ambari server. ambari-server status OR sudoambari-server status ORservice ambari-server status This command…

Checking versions of Hadoop eco system tools   We need to know versions of hadoop technologies while we are trouble shooting any Hadoop issues. This article talks how to check versions of Hadoop ecosystem technologies.1) Hadoop version Hadoop version can be found using below command. hadoop version  2) Hive version Hive version can be found using below command. hive version   3) Pig version Pig version can be found using below command. pig version  4) Sqoop version Sqoop version can be found…

Checking versions of Hadoop eco system tools We need to know versions of hadoop technologies while we are trouble shooting any Hadoop issues. This article talks how to check versions of Hadoop ecosystem technologies.1) Hadoop version Hadoop version can be found using below command. hadoop version 2) Hive version Hive version can be found using below command. hive version 3) Pig version Pig version can be found using below command. pig version 4) Sqoop version Sqoop version can be found…

Ambari agent installation   In this article  We will learn how to install Ambari agent for Ambari on different operating systems.1) Installing ambari agent We use yum command if operating system is CentOS or Redhat. We use zypper command if operating system is SLES or apt-get command if operating system is Ubuntu. Commands below needs to be run as root user.CentOS or RedHat :yum install ambari-agentSLES (Suse Linux )zypper install ambari-agentUbuntuapt-get install ambari-agent The picture…

Ambari agent installation In this article We will learn how to install Ambari agent for Ambari on different operating systems.1) Installing ambari agent We use yum command if operating system is CentOS or Redhat. We use zypper command if operating system is SLES or apt-get command if operating system is Ubuntu. Commands below needs to be run as root user.CentOS or RedHat :yum install ambari-agentSLES (Suse Linux )zypper install ambari-agentUbuntuapt-get install ambari-agent The picture…

Exploring snapshots in HDFS   HDFS snapshot is saved copy of an existing directory. Snapshots will be useful for restoring the corrupt data. In this article we will learn how to manage HDFS snapshots. Practice below commands to get practical understanding of HDFS snapshots.1) Create a local file with sample numbers.  2) Create a folder on hdfs and upload local file to HDFS directory The following commands create a folder called numbers in HDFS directory /user/hdfs and upload local file…

Exploring snapshots in HDFS HDFS snapshot is saved copy of an existing directory. Snapshots will be useful for restoring the corrupt data. In this article we will learn how to manage HDFS snapshots. Practice below commands to get practical understanding of HDFS snapshots.1) Create a local file with sample numbers. 2) Create a folder on hdfs and upload local file to HDFS directory The following commands create a folder called numbers in HDFS directory /user/hdfs and upload local file…

Enabling and disabling ACLs in HDFS   ACLs commands setfacl and getfacl provide advanced permission management in HDFS. We will learn how to enable/disable ACLs in HDFS using Apache Ambari. ACLs are disabled by default . We need to modify/add ACLs property to enable ACLs in HDFS. 1) Search hdfs config for dfs.namenode.acls.enabled property in Ambari  you get no results if property not defined yet. goto HDFS -------- Configs --------- enter dfs.namenode.acls.enabled in filter box  2) We need…

Enabling and disabling ACLs in HDFS ACLs commands setfacl and getfacl provide advanced permission management in HDFS. We will learn how to enable/disable ACLs in HDFS using Apache Ambari. ACLs are disabled by default . We need to modify/add ACLs property to enable ACLs in HDFS. 1) Search hdfs config for dfs.namenode.acls.enabled property in Ambari you get no results if property not defined yet. goto HDFS -------- Configs --------- enter dfs.namenode.acls.enabled in filter box 2) We need…

Working with databases in Apache Hive    In this article We will learn how to work on databases in Apache Hive. We will learn how to create drop change and use database in Apache Hive.1) Check existing databases. We check existing databases in Hive using show databases command. Apache Hive comes with a database called default. Command : show databases;  2) Creating a new database; We can create a new database in Apache Hive using create command. Command syntax: create database [if not exist]…

Working with databases in Apache Hive In this article We will learn how to work on databases in Apache Hive. We will learn how to create drop change and use database in Apache Hive.1) Check existing databases. We check existing databases in Hive using show databases command. Apache Hive comes with a database called default. Command : show databases; 2) Creating a new database; We can create a new database in Apache Hive using create command. Command syntax: create database [if not exist]…

Starting HDFSMAPREDUCE2 and YARN processes manually   We start HDFSMAPREDUCE2 and YARN services using either Cloudera manager or Apache Ambari if we use CDH or HDP.Many a times Cloudera manager and Apache Ambari do not display complete error message if services fail to start.We can go to log directories and search for startup errors. Searching startup errors in log files is also not easy as log files are huge.One easy way to find startup errors is starting processes manually from command…

Starting HDFSMAPREDUCE2 and YARN processes manually We start HDFSMAPREDUCE2 and YARN services using either Cloudera manager or Apache Ambari if we use CDH or HDP.Many a times Cloudera manager and Apache Ambari do not display complete error message if services fail to start.We can go to log directories and search for startup errors. Searching startup errors in log files is also not easy as log files are huge.One easy way to find startup errors is starting processes manually from command…

Using where clause in Apache Hive query   In this article We will learn where clause in Apache Hive. Where clause is used to filter the column data that satisfies given condition.We will learn how to use where clause in different ways in the following steps. 1) Create table called employee in Apache Hiveusing this article. 2) Check the data in Hive table employee. Select  from employee;  3) Where clause in Hive supports many operators.  The query below uses equal operator and filters the…

Using where clause in Apache Hive query In this article We will learn where clause in Apache Hive. Where clause is used to filter the column data that satisfies given condition.We will learn how to use where clause in different ways in the following steps. 1) Create table called employee in Apache Hiveusing this article. 2) Check the data in Hive table employee. Select from employee; 3) Where clause in Hive supports many operators. The query below uses equal operator and filters the…


More ideas
Pinterest
Search