Skip to content

Commit

Permalink
Hadoop integration adaptation has been completed. (#1016)
Browse files Browse the repository at this point in the history
  • Loading branch information
@TJxiaobao @tomsun28
TJxiaobao authored and tomsun28 committed Mar 10, 2024
1 parent 6d30f88 commit bff3cea
Show file tree
Hide file tree
Showing 3 changed files with 481 additions and 0 deletions.
87 changes: 87 additions & 0 deletions home/docs/help/hadoop.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
id: hadoop
title: Monitoring: Hadoop
sidebar_label: Hadoop keywords: [Open Source Monitoring System, Open Source Java Monitoring, Hadoop JVM Monitoring]
---

> Collect and monitor general performance metrics for the JVM virtual machine in Hadoop.
**Protocol used: JMX**

### Pre-monitoring steps

>You need to enable JMX service in the Hadoop application before monitoring. HertzBeat uses the JMX protocol to collect performance metrics from Hadoop's JVM.
### Steps to enable JMX protocol in the Hadoop application
Add JVM parameters when the application starts. ⚠️Note that you can customize the exposed port and external IP.

- 1.Enter the hadoop-env.sh configuration file and enter the following command in the terminal:
```shell
vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
```
- 2.Add the following parameters, where <port> is the number of the custom-exposed port:
```shell
export HADOOP_OPTS= "$HADOOP_OPTS
-Djava.rmi.server.hostname=对外ip地址
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false "
```
- 3.Save and exit, and then execute "start-all.sh" in the "$HADOOP_HOME/sbin" directory to restart the service.


### Configuration parameter

| Parameter name | Parameter help description |
|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Monitoring Host | Monitored IPV4, IPV6 or domain name. Note⚠️Without protocol header (eg: https://, http://) |
| Monitoring name | Identify the name of this monitoring. The name needs to be unique |
| Port | Port provided by JMX |
| Username | JMX connection user name, optional |
| Password | JMX connection password, optional |
| Collection interval | Interval time of monitor periodic data collection, unit: second, and the minimum interval that can be set is 30 seconds |
| Whether to detect | Whether to detect and check the availability of monitoring before adding monitoring. Adding and modifying operations will continue only after the detection is successful |
| Description remarks | For more information about identifying and describing this monitoring, users can note information here |

### Collection Metrics

#### Metrics Set：memory_pool

| Metric name | Metric unit | Metric help description |
|-------------|-------------|-------------------------|
| name | | metrics name |
| committed | kb | total size |
| init | kb | init size |
| max | kb | max size |
| used | kb | used size |

#### Metrics Set：code_cache (Only Support JDK8)

| Metric name | Metric unit | Metric help description |
|-------------|-------------|-------------------------|
| committed | kb | total size |
| init | kb | init size |
| max | kb | max size |
| used | kb | used size |

#### Metrics Set：class_loading

| Metric name | Metric unit | Metric help description |
|-----------------------|-------------|--------------------------|
| LoadedClassCount | | Loaded Class Count |
| TotalLoadedClassCount | | Total Loaded Class Count |
| UnloadedClassCount | | Unloaded Class Count |


#### Metrics Set：thread

| Metric name | Metric unit | Metric help description |
|-------------------------|-------------|----------------------------|
| TotalStartedThreadCount | | Total Started Thread Count |
| ThreadCount | | Thread Count |
| PeakThreadCount | | Peak Thread Count |
| DaemonThreadCount | | Daemon Thread Count |
| CurrentThreadUserTime | ms | Current Thread User Time |
| CurrentThreadCpuTime | ms | Current Thread Cpu Time |


93 changes: 93 additions & 0 deletions home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/hadoop.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
id: hadoop
title: 监控：Hadoop
sidebar_label: Hadoop
keywords: [开源监控系统, 开源JAVA监控, Hadoop的JVM虚拟机监控]
---

> 对Hadoop的JVM虚拟机的通用性能指标进行采集监控
**使用协议：JMX**

### 监控前操作

> 您需要在 Hadoop 应用开启 `JMX` 服务，HertzBeat 使用 JMX 协议对Hadoop中的 JVM 进行指标采集。
#### Hadoop应用开启JMX协议步骤

应用启动时添加JVM参数 ⚠️注意可自定义暴露端口,对外IP

- 1.进入hadoop-env.sh配置文件，在命令行终端输入以下命令：

```shell
vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
```

- 2.添加如下参数，`<port>` 为自定义暴露端口号。

```shell
export HADOOP_OPTS= "$HADOOP_OPTS
-Djava.rmi.server.hostname=对外ip地址
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false "
```

- 3.保存并退出，并在 $HADOOP_HOME/sbin 目录下执行 `start-all.sh` 重启服务。

### 配置参数

| 参数名称 | 参数帮助描述 |
| ----------- | ----------- |
| 监控Host | 被监控的对端IPV4，IPV6或域名。注意⚠️不带协议头(eg: https://, http://)。 |
| 监控名称 | 标识此监控的名称，名称需要保证唯一性。 |
| 查询超时时间 | 设置JVM连接的超时时间，单位ms毫秒，默认3000毫秒。 |
| 用户名 | JMX连接用户名 |
| 密码 | JMX连接密码 |
| 采集间隔 | 监控周期性采集数据间隔时间，单位秒，可设置的最小间隔为30秒 |
| 是否探测 | 新增监控前是否先探测检查监控可用性，探测成功才会继续新增修改操作 |
| 描述备注 | 更多标识和描述此监控的备注信息，用户可以在这里备注信息 |

### 采集指标


#### 指标集合：memory_pool

| 指标名称 | 指标单位 | 指标帮助描述 |
|-----------| ----------- | ----------- |
| name | 无 | 指标名称 |
| committed | kb | 总量 |
| init | kb | 初始化大小 |
| max | kb | 最大 |
| used | kb | 已使用 |

#### 指标集合：code_cache (限JDK8及以下)

| 指标名称 | 指标单位 | 指标帮助描述 |
|-----------| ----------- | ----------- |
| committed | kb | 总量 |
| init | kb | 初始化大小 |
| max | kb | 最大 |
| used | kb | 已使用 |

#### 指标集合：class_loading

| 指标名称 | 指标单位 | 指标帮助描述 |
|-----------------------| ----------- | ----------- |
| LoadedClassCount | 个 | 已加载类数量 |
| TotalLoadedClassCount | 个 | 历史已加载类总量 |
| UnloadedClassCount | 个 | 未加载类数量 |


#### 指标集合：thread

| 指标名称 | 指标单位 | 指标帮助描述 |
|-------------------------| ----------- | ----------- |
| TotalStartedThreadCount | 个 | 已经开始的线程数量 |
| ThreadCount | 个 | 线程数 |
| PeakThreadCount | 个 | 未加载类数量 |
| DaemonThreadCount | 个 | 守护进程数 |
| CurrentThreadUserTime | ms | 使用时间 |
| CurrentThreadCpuTime | ms | 使用CPU时间 |


Loading

0 comments on commit bff3cea

Please sign in to comment.