如何使用Ansible清除Hadoop YARN缓存?

Question

我想删除Hadoop集群中的所有缓存文件夹。我已经写下了一些步骤，但我不确定我应该如何使用 Ansible 脚本。总的来说，我了解 Ansible，但我有点困惑我应该使用哪些模块来满足我的需求。

步骤：

1. 停止 YARN 服务。

   $ sudo service hadoop-mapreduce-historyserver stop
   $ sudo service hadoop-yarn-resourcemanager stop
   $ sudo service hadoop-yarn-nodemanager stop
   

2. 登录到所有节点并删除用户缓存目录的内容。例如：

   for i in cat list_of_nodes_in_cluster; do ssh $i rm -rf
   /prod/HDOP/data/13/daas.na/yarn/nm/usercache/*; done
   

3. 验证所有节点中的所有用户缓存目录是否为空。

4. 启动 YARN 服务。

   $ sudo service hadoop-mapreduce-historyserver start
   $ sudo service hadoop-yarn-resourcemanager start
   $ sudo service hadoop-yarn-nodemanager start
   

5. 将消息">YARN 缓存已刷新">发送到我们的组 ID。

Answer 1

您可以使用此 Ansible 剧本来执行您的任务。

1. 停止 YARN 服务。

   - name: service hadoop-mapreduce-historyserver stop
   service: 
   name: hadoop-mapreduce-historyserver
   state: stopped
   - name: service hadoop-yarn-resourcemanager stop
   service: 
   name: hadoop-yarn-resourcemanager
   state: stopped
   - name: service hadoop-yarn-nodemanager stop
   service: 
   name: hadoop-yarn-nodemanager
   state: stopped
   

2. 登录到所有节点并删除用户缓存目录的内容。

   - name: delete the content of the usercache directories
   file: path="{{ item }}" state=absent
   with_fileglob:
   - /prod/HDOP/data/13/daas.na/yarn/nm/usercache/*
   

3. 验证所有节点中的所有用户缓存目录是否为空。

   - name:  Verify all usercache directories are empty.
   shell: /usr/bin/ls ~/soniya/ | wc -l
   register: total_size
   - debug:
   msg: "Directory total size is {{ total_size.stdout_lines }}"
   when: total_size.rc == 0
   

4. 启动 YARN 服务。

   - name: service hadoop-mapreduce-historyserver start
   service: 
   name: hadoop-mapreduce-historyserver
   state: started
   - name: service hadoop-yarn-resourcemanager start
   service: 
   name: hadoop-yarn-resourcemanager
   state: started
   - name: service hadoop-yarn-nodemanager start
   service: 
   name: hadoop-yarn-nodemanager
   state: started
   

   5. 将按摩作为"YARN 缓存已刷新"发送到我们的组 ID

      您可以使用邮件模块来执行此操作。