Chinese | English

Maintainer: openGauss AI-SIG

As a part of the openGauss database, DBMind provides automatic driving capabilities for the openGauss database, and is a leading open source database autonomous operation and maintenance platform. With DBMind, you can easily discover database problems, and at the same time implement second-level root cause analysis of database problems.

Features of DBMind:

• DBMind adopts an advanced plug-in architecture and supports massive plug-in extensions;
• Support multiple operating modes, with command line interactive operation and service operation;
• Designed for cloud native, supports Prometheus, and provides a variety of rich exporter plug-ins;
• Provides rich docking modes, which can be easily connected with existing management systems, and supports RESTful API, Python SDK, command line, Prometheus protocol and other modes;
• Supports end-to-end full-process database autonomous operation and maintenance capabilities, including slow SQL root cause analysis, workload index recommendation, multi-index correlation mining, fault self-repair, anomaly detection and root cause analysis, etc.;

Start using DBMind

Download and install DBMind

DBMind is implemented based on the Python language. When using DBMind, it is necessary to have a Python virtual machine in the operating environment and install the required third-party dependencies.

Method 1: Directly download code deployment

DBMind is mainly written in Python language, so after downloading and obtaining the source code of DBMind, it can be run directly using the Python virtual machine installed on the operating system, but the third-party dependencies in this process need to be manually installed by the user.

Users can pass git clone command to download code from Gitee or Github, for example:

git clone --depth 1 https://gitee.com/opengauss/openGauss-DBMind.git

It can also be downloaded through the zip package download path provided by Gitee or Github, and then decompress the zip package.

After downloading DBMind, a file named openGauss-DBMind directory, add the path of the directory to the environment variablePATH, you can call the executable file in the directory. For example, you can execute the following command to complete:

chmod +x openGauss-DBMind/gs_dbmind

echo PATH=`pwd`/openGauss-DBMind:'$PATH' >> ~/.bashrc
echo 'export PATH' >> ~/.bashrc

source ~/.bashrc

Method 2: Deploy using the installation package

DBMind will regularly publish the DBMind installation package on the release page of the openGauss-DBMind project, and you can install and deploy it by downloading the DBMind installation package. The installation package will automatically decompress DBMind to the specified directory and configure the environment variables.

The download address of the installation package is:

https://gitee.com/opengauss/openGauss-DBMind/releases

About the Python operating environment

A version of at least Python 3.7 is required. Although the implementation of DBMind is as compatible as possible with environments below Python 3.7, these low-version Python environments are laxly tested and may cause unexpected exceptions. At the same time, when DBMind starts, it will also try to verify the Python version. If the Python version does not meet the requirements, it will not continue to perform subsequent actions by default.

The Python version of DBMind is constrained by the variables in the constant file in the root directory

If your environment needs to install multiple versions of the Python runtime, and they may cause conflicts, then we recommend that you install the Python runtime environment required by DBMind to the root directory of DBMind. python directory, DBMind will give priority to using it in its root directory python environment in the directory.which is gs_dbmind The command will first be in thepython/bin search under directory python3 command to execute subsequent Python functions.

About third-party dependence

The third-party dependencies used by DBMind can be accessed through the root directory of DBMind requirements-xxx.txt file designation. For x86 architecture (amd64) and ARM architecture (aarch64), different file names are used for identification. This is because the ARM platform is not friendly to some third-party dependencies, and a specific version must be specified before it can be installed.

Third-party dependencies can be installed using the pip tool. Similar to the situation mentioned above, if your current operating system has to install multiple Python runtime environments, then DBMind also supports the priority selection of third-party dependencies.That is, the third-party dependency library can be stored in the root directory of DBMind 3rd in the directory.in passing gs_dbmind When the command uses the DBMind function, it will give priority to the 3rd The third-party dependent libraries in the directory are loaded.

Taking the x86 environment as an example, you can use the followingpipCommand to install DBMind’s third-party dependency library:

python3 -m pip install -r requirements-x86.txt

If you want to specify the address of the downloaded third-party dependent library, you can pass --target or -t options, such as

python3 -m pip install -r requirements-x86.txt -t 3rd

Use DBMind

Deploy Prometheus

You can obtain the download method through the Prometheus official website, download and deploy Prometheus, so as to collect the monitoring results of the openGauss instance.

Deploy Node Exporter

Download and start the Prometheus node exporter.

Node exporter can be used to monitor Linux systems, so only one instance needs to be deployed in each Linux environment (or container).

Start the DBMind component

If you want to run DBMind as a background service, the following DBMind components must be installed, otherwise you cannot get the monitoring information of the database.In order to obtain a higher security mechanism, the exporter provided by DBMind uses the Https protocol by default. If you think that you do not need to use the Https protocol in your scenario, you can pass --disable-https option disabled.

openGauss Exporter

openGauss exporter reads the data of the system table (or system view) from the openGauss database and stores it through Prometheus.Since the openGauss exporter needs to read the system table information of the monitoring database, it should at least have monadmin authority.For example, the following SQL statement can be used for the named dbmind_monitor User grants permissions:

ALTER USER dbmind_monitor monadmin;

use gs_dbmind component opengauss_exporter ... command to start the openGauss exporter component.For example, you can monitor a database with the following command, by --url The parameter specifies the monitored database instance address:

gs_dbmind component opengauss_exporter --url postgresql://username:password@host:port/database --web.listen-address 0.0.0.0 --web.listen-port 9187 --log.level warn --disable-https ...

--url Indicates the DSN address of the database, and its format can be referred to here.

You can check whether the openGauss exporter has started with the following command:

curl -vv http://localhost:9187/metrics

Reprocessing Exporter

The reprocessing exporter is an exporter for secondary processing of data. Since the data saved in Prometheus by node exporter and openGauss exporter is real-time monitoring information, it is impossible to reflect the instantaneous incremental information of some indicators only through this information. Such as TPS, iops information, etc. Therefore, the reprocessing exporter can be used to calculate incremental information or aggregate results, etc.

Since reprocessing obtains indicator data from Prometheus, performs secondary processing and then returns it to Prometheus. Therefore, it corresponds to Prometheus one by one, that is, if there is only one Prometheus service, only one reprocessing exporter is required. For example, the reprocessing exporter can be started with the following command:

gs_dbmind component reprocessing_exporter 127.0.0.1 9090 --web.listen-address 0.0.0.0 --web.listen-port 9189

If your Prometheus usesbasic authorizationFor login verification, you need to specify additionally --prometheus-auth-user as well as --prometheus-auth-password The value of the option.

Configure and start

DBMind background service is memory resident. Therefore, you need to configure a configuration file directory first, and save multiple DBMind configuration files in this directory.able to pass gs_dbmind service command to generate the configuration file directory and start the service. The instructions for using this command are:

$ gs_dbmind service --help
usage:  service [-h] -c DIRECTORY [--only-run {...}] [--interactive | --initialize] {setup,start,stop}

positional arguments:
  {setup,start,stop}    perform an action for service

optional arguments:
  -h, --help            show this help message and exit
  -c DIRECTORY, --conf DIRECTORY
                        set the directory of configuration files
  --only-run {slow_query_diagnosis,forecast}
                        explicitly set a certain task running in the backend
  --interactive         configure and initialize with interactive mode
  --initialize          initialize and check configurations after configuring.

Next, the generation of the configuration file directory and the start and stop operations of the service are introduced respectively.

Configure DBMind

DBMind provides two ways to generate configuration files.One is interactive, via --interactive The option is specified; the other requires the user to manually modify it, which is also the default method.

interactive configuration

Here are some usage examples, here we use CONF_DIRECTORY Identify our configuration file directory:

gs_dbmind service setup -c CONF_DIRECTORY --interactive

Through the above command, the user can enter the openGauss instance information and parameters to be monitored in the interactive interface according to the prompt information.

Manual configuration

The following command demonstrates how to configure DBMind manually:

gs_dbmind service setup -c CONF_DIRECTORY

After executing the above command, a file named CONF_DIRECTORY directory, which contains many configuration files.However, users need to configure CONF_DIRECTORY/dbmind.conf file. After the user configures the file, you need to execute the following command, and DBMind will initialize the DBMind system according to the information just configured by the user:

gs_dbmind service setup -c CONF_DIRECTORY --initialize

Start and stop DBMind service

After the user configures the DBMind database, the DBMind background service can be started directly through the following command:

gs_dbmind service start -c CONF_DIRECTORY

Close the DBMind service with the following command:

gs_dbmind service stop -c CONF_DIRECTORY

Components of DBMind

As mentioned above, DBMind is based on a plug-in design, and this component is the plugin provided by DBMind. Through the plug-in design, DBMind can expand functions arbitrarily.If you want to use the functionality of a component, you need to executecomponentsubcommand.For example a namedxtunerComponents can perform data parameter tuning, then you can execute the following command to usextunerfunction.

gs_dbmind component xtuner --help

Run DBMind with Docker

DBMind supports Docker, and will regularly publish openGauss-DBMind docker images on Docker Hub. The address of the image is:

https://hub.docker.com/r/dbmind/opengauss_dbmind

The image can be pulled with the following command:

docker pull dbmind/opengauss_dbmind

Create a Docker image

In some cases, you may wish to manually create a docker image of DBMind, such as when you want to create an image based on the latest code. Then, it can be created through the Dockerfile file in the root directory of the DBMind code.For example, execute the following command in the root directory of DBMind to create a file named opengauss_dbmind mirror image of:

docker build -t opengauss_dbmind .

Use of Docker images

The default executable file of DBMind’s docker image is docker_run.pythe startup script can start most of the dependent services required by DBMind in the container, including Prometheus, openGauss exporter, reprocessing exporter. However, the node exporter cannot be run in the mirror container to monitor the information on the remote server.

Users can pass the openGauss service information to be monitored to the docker image of DBMind through the following environment variables:

OPENGAUSS_DSNS: 需要监控的openGauss数据库实例的DSN信息，多个DSN信息用逗号(,)隔开
NODE_EXPORTERS: openGauss数据库实例所在机器的node exporter地址，多个地址用逗号(,)隔开
METADATABASE: 可选，将DBMind的离线计算结果存储起来的位置，用DSN形式标识数据库的连接信息；若为空，则默认使用SQLite进行存储
SCRAPE_INTERVAL: 可选，指标信息的采集间隔，单位是秒；默认为15秒
MASTER_USER: 可选，具有管理员权限的数据库用户名，可以用来执行某些数据库变更动作或者查询当前数据库的即时状态信息；若为空，则采用 OPENGAUSS_DSNS 中提供的用户
MASTER_USER_PWD: 可选，上述 MASTER_USER 对应的用户密码

Note: For the configuration format of DSN, please refer to the instructions in FAQ.

usedocker runof-vThe parameter can map the path, and the logs in the docker container are uniformly written to /log In the directory, the persistent data is stored in the /data in the directory.use -p The parameter can map the port number in the container, the port in the container of Prometheus is 9090, and the web service of DBMind uses port 8080. The following is an example of starting the docker service:

docker run -it \
    -e OPENGAUSS_DSNS="dbname=postgres user=dbmind_monitor password=DBMind@123 port=6789 host=192.168.1.100, dbname=postgres user=dbmind_monitor password=DBMind@123 port=6789 host=192.168.1.101, dbname=postgres user=dbmind_monitor password=DBMind@123 port=6789 host=192.168.1.102" \
    -e NODE_EXPORTERS="http://192.168.1.100:9100,http://192.168.1.101:9100,http://192.168.1.102:9100" \
    -e METADATABASE='postgresql://dbmind_metadb:DBMind%40123@192.168.1.100:6789/dbmind_metadb' \
    -e MASTER_USER='dbmind_sys' \
    -e MASTER_USER_PWD='DBMind@123' \
    -e SCRAPE_INTERVAL=30 \
    -p 38080:8080 -p 39090:9090 \
    -v `pwd`/data:/data -v `pwd`/log:/log \
    dbmind/opengauss_dbmind 

The above example is the deployment form of one master and two standby nodes, and their IP addresses are respectively192.16.1.100,192.16.1.101as well as192.16.1.102the port number of the database is 6789. We used three users above. For the convenience of demonstration, their passwords are set toDBMind@123.indbmind_monitorResponsible for grabbing indicator monitoring from the openGauss database, need to have monitor adminauthority;dbmind_sys at least need to have monitor adminpermission so that the immediate state of the database can be obtained, if availablesysadminpermissions, you can complete some database change actions, such as slow SQL killing;dbmind_metadb It is only responsible for data storage, and it is enough to have the permission to use the specified database; at the same time, port and directory mapping is also performed here.

If you want to use the command line to run DBMind, you can directly call it in the docker image gs_dbmind The command is enough, and the Python runtime and third-party dependencies have been packaged in the docker image, so there is no need to install it again. For example, if you want to use the functions provided by the parameter tuning component of DBMind, you can execute the following command:

docker run -it dbmind/opengauss_dbmind \
   gs_dbmind component xtuner recommend \
   --database tpcds \
   --db-host 192.168.1.100 \
   --host-user omm \
   --db-user tpcds \
   --db-port 16000

Note: in usedocker run command to run gs_dbmind need to specify -it parameters in order to create a tty.

common problem

Format description of DSN

DSN is the abbreviation of Database Source Name, here supports two formats, one is KV format, such asdbname=postgres user=username password=password_value port=6789 host=127.0.0.1; The other is the URL form, such aspostgresql://username:password_value@127.0.0.1:6789/postgres; For DSNs in URL format, since@and other special characters are used to separate the content of each part of the URL string, so URL encoding (URL encode) is required.e.g. a userdbmindThe password forDBMind@123the DSN in URL form can bepostgresql://dbmind:DBMind%40123@127.0.0.1:6789about to@The character encoding is%40. Similarly, the characters that need to be encoded also include other characters that may cause ambiguity, such as/, \, ?, &.

Relevant information

DBMind is a part of openGauss, which empowers openGauss to carry the autonomous operations and maintenance capabilities. DBMind is leading and open-source. Through DBMind, users can easily discover database problems and the root causes of the problems in seconds.

Getting Started

Prerequisites

In order to run DBMind, the following components should be configured and running.

Python Runtime

At least Python 3.7.

Third-party Dependencies

Use pip install to install the python dependencies. Type the pip install command with dependencies according to the environment you are running:

pip install -r requirements-aarch64.txt | requirements-x86.txt

Prometheus up and running

Download and run the Prometheus time-series database.

Node Exporter

Download and run the Prometheus node exporter. Node-exporter is to monitor the Linux system. Hence, one Linux environment only needs to deploy one node-exporter.

DBMind Components

The following DBMind components are required:

Note: If you want to get higher security, you should use the HTTPS scheme.

openGauss Exporter

The openGauss-exporter reads data from the database and places it on the Prometheus time-series database. OpenGauss-exporter is to monitor only one database instance. So if your deployment environment has not only one instance, you should start multiple openGauss-exporters to correspond to monitor multiple database instances. It needs database access with a user having the role of at least monadmin (monitoring administrator) granted to run it. For example, you can grant monadmin privilege to role dbmind as below:

ALTER USER dbmind monadmin;

Use the following command with the parameters below:

gs_dbmind component opengauss_exporter ...

You can get detailed explanations of this component through passing --help:

gs_dbmind component opengauss_exporter --help

For example, the following command starts it:

gs_dbmind component opengauss_exporter --url postgresql://username:password@host:port/database --web.listen-address 0.0.0.0 --web.listen-port 9187 --log.level warn --disable-https ...

To test that the exporter is up, type the following command on its host (or use change the localhost to the server address):

curl -vv http://localhost:9187/metrics

Reprocessing Exporter

Reprocessing-exporter is a re-processing module for metrics stored in the Prometheus server. It helps Prometheus to reprocess the metric data then dump the new data into Prometheus. Therefore, only one needs to be started in a deployment environment. To run it use the command below:

gs_dbmind component reprocessing_exporter ...

Users can see usage by using --help too.

See this example for running the exporter in a single machine development environment:

gs_dbmind component reprocessing_exporter 127.0.0.1 9090 --web.listen-address 0.0.0.0 --web.listen-port 9189

Use the following command to check that the service is up:

curl http://127.0.0.1:9189/metrics

Configure, Start and Stop the DBMind Service

DBMind service is a memory-resident backend service. Therefore, users should configure it first then start or stop the service by using the configuration.

Service usages:

$ gs_dbmind service --help
usage:  service [-h] -c DIRECTORY [--only-run {...}] [--interactive | --initialize] {setup,start,stop}

positional arguments:
  {setup,start,stop}    perform an action for service

optional arguments:
  -h, --help            show this help message and exit
  -c DIRECTORY, --conf DIRECTORY
                        set the directory of configuration files
  --only-run {slow_query_diagnosis,forecast}
                        explicitly set a certain task running in the backend
  --interactive         configure and initialize with interactive mode
  --initialize          initialize and check configurations after configuring.

Configure

DBMind offers two methods to configure. The one is an interactive mode by using --interactive argument, the other is a modification by hands.

See this example for configuring in the interactive mode:

gs_dbmind service setup -c CONF_DIRECTORY --interactive

Then users can type parameters into the shell terminal.

See the following example for configuring by hands:

gs_dbmind service setup -c CONF_DIRECTORY

After executing the above command, the directory CONF_DIRECTORY will generate too many configuration files. Therefore, users should modify these parameters in the CONF_DIRECTORY/dbmind.conf. While users finish configuring, this command needs to be run to initialize DBMind according to the CONF_DIRECTORY/dbmind.conf.

gs_dbmind service setup -c CONF_DIRECTORY --initialize

Start or Stop the DBMind Service

After configuring, specify your CONF_DIRECTORY, users can start or stop the service directly.

gs_dbmind service start/stop -c CONF_DIRECTORY

Component

If users want to use a specific component offline. They can use the sub-command component:

gs_dbmind component xxx ...

xxx is the name of a component. Users can also get the component list by using the --help argument.

For example, use the following component to tune the knobs of a database:

gs_dbmind component xtuner --help

Mulan PSL v2

1. https://en.wikipedia.org/wiki/Percent-encoding
2. https://dba.stackexchange.com/questions/243219/in-postgresql-url-i-cant-use-a-password-containing-special-characters