NODBS

How to ensure TiDB cluster can use TiCDC to synchronize data?

2024-08-03T00:00:00+08:00

This article focuses on the more difficult to determine parts of the TiCDC’s limitations.

Currently, TiCDC unsupported scenarios and limitations as below:

A TiKV cluster that uses RawKV alone.
The CREATE SEQUENCE DDL operation and the SEQUENCE function in TiDB. When the upstream TiDB uses SEQUENCE, TiCDC ignores SEQUENCE DDL operations/functions performed upstream. However, DML operations using SEQUENCE functions can be correctly replicated.
Currently, performing BR data recovery and TiDB Lightning physical import imports on tables and databases that are being replicated by TiCDC is not supported. For more information, see Why does replication using TiCDC stall or even stop after data restore using TiDB Lightning and BR from upstream.
TiCDC only replicates tables that have at least one valid index

For details, refer to the official documentation unsupported scenarios and best practices.

The most difficult part of these limitations to be identified is the valid index:

TiCDC only replicates tables that have at least one valid index. A valid index is defined as follows:

A primary key (PRIMARY KEY) is a valid index.

A unique index (UNIQUE INDEX) is valid if every column of the index is explicitly defined as non-nullable (NOT NULL) and the index does not have a virtual generated column (VIRTUAL GENERATED COLUMNS).

If there are tables in the TiDB cluster without at least one valid index, then these tables cannot use TiCDC to synchronize data. How to find out which tables in the cluster do not have any valid index?

1.Find tables that have primary key

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys');

2.Find tables have unique index and every column of the index is explicitly defined as non-nullable (NOT NULL) and the index does not have a virtual generated column

select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

3.Combine the result of the above two sets are the tables have valid index

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')
union all
select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

Conclusion

Tables Without Valid Index = All Tables - Tables With Valid Index, if the result set of the follow SQL is not empty, means the TiDB cluster contains tables can not be replicated by TiCDC.

select table_schema, table_name from information_schema.tables where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') and table_type='BASE TABLE' and (table_schema, table_name) not in (select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') union all select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION=''));

Others

About another limitation not supporting SEQUENCE related operations. It is very simple to find out whether SEQUENCE is used in the cluster. The SQL is as follows. If the query result is not empty, it means that SEQUENCE is used:

SELECT table_schema, table_name, table_type from information_schema.tables where table_type='SEQUENCE';

中文版

本文主要围绕 TiCDC 的限制中比较难确定的部分内容，TiCDC 官方文档中已经清楚的描述了不支持的场景以及必须满足的条件。

目前 TiCDC 暂不支持的场景和限制如下：

暂不支持单独使用 RawKV 的 TiKV 集群。
暂不支持在 TiDB 中创建 SEQUENCE 的 DDL 操作和 SEQUENCE 函数。在上游 TiDB 使用 SEQUENCE 时，TiCDC 将会忽略掉上游执行的 SEQUENCE DDL 操作/函数，但是使用 SEQUENCE 函数的 DML 操作可以正确地同步。
暂不支持在同步的过程中对 TiCDC 正在同步的表和库进行 BR 数据恢复和 TiDB Lightning 物理导入。详情请参考为什么在上游使用了 TiDB Lightning 和 BR 恢复了数据之后，TiCDC 同步会出现卡顿甚至卡住。
TiCDC 同步的表需要至少存在一个有效索引的表。

详见官方文档中暂不支持的场景和最佳实践。

这些限制中最不好判断的是有效索引这一条：

TiCDC同步的表需要至少存在一个有效索引的表，有效索引的定义如下：

主键 (PRIMARY KEY) 为有效索引。

唯一索引 (UNIQUE INDEX) 中每一列在表结构中明确定义非空 (NOT NULL) 且不存在虚拟生成列 (VIRTUAL GENERATED COLUMNS)。

如果 TiDB 集群中有表没有有效索引，那么这些表是不能利用 TiCDC 来同步数据的，如何集群中哪些表没有有效索引呢？推导如下：

1.找出有主键的表

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys');

2.找出有唯一索引并且索引中每一列在表结构中明确定义非空 (NOT NULL) 且不存在虚拟生成列的表

select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

3.以上两种表的并集就是含有有效索引的表。

select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') 
union all 
select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION='');

总结

没有有效索引的表 = 所有的表 - 含有有效索引的表，如果以下 SQL 的查询结果为非空，则表示 TiDB 集群中包含有不能使用 TiCDC 来同步数据的表：

select table_schema, table_name from information_schema.tables where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') and table_type='BASE TABLE' and (table_schema, table_name) not in (select distinct table_schema,table_name from information_schema.tidb_indexes where KEY_NAME='PRIMARY' and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys') union all select distinct a.table_schema, a.table_name from (select table_schema,table_name,column_name from information_schema.tidb_indexes where KEY_NAME<>'PRIMARY' and NON_UNIQUE=0 and table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) a left join (select TABLE_SCHEMA,TABLE_NAME,COLUMN_NAME,IS_NULLABLE,GENERATION_EXPRESSION from information_schema.columns where table_schema not in ('METRICS_SCHEMA','mysql','PERFORMANCE_SCHEMA','INFORMATION_SCHEMA','sys')) b on a.table_schema=b.table_schema and a.table_name=b.table_name and a.column_name=b.column_name where (b.IS_NULLABLE='NO' and b.GENERATION_EXPRESSION=''));

其他

还有一个关于不支持 SEQUENCE 相关操作的限制，找出集群中是否使用了 SEQUENCE 是很简单的，SQL 如下，如果查询结果为非空，则说明使用了 SEQUENCE：

SELECT table_schema, table_name, table_type from information_schema.tables where table_type='SEQUENCE';

How to ensure TiDB cluster can use TiCDC to synchronize data? was originally published by Sherry, Yin Xi at NODBS on August 03, 2024.

TiDB Operator Enable TLS For MySQL Client With cert-manager

2024-06-06T00:00:00+08:00

TiDB Operatior supports enable and disable TLS for MySQL client for an existing TiDB cluster. This article will show you how to enable it. Certificates can be issued by cfssl and cert-manager, here I will choose the cloud native way cert-manager.

We need a self-signed CA
Use the CA to sign certificate for TiDB cluster

Prepare CA

Install cert-manager

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.15.2 --set crds.enabled=true

It will take some time, be patient.

Prepare Issuers

TiDB is not a internet facing public services, self-signed issuer and CA is enough. There’s no naming restrictions for the issuers for TiDB Operator.
1.Create tidb-server-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tidb-selfsigned-ca-issuer
  namespace: tidb-cluster 
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tidb-ca
  namespace: tidb-cluster
spec:
  secretName: tidb-ca-secret                  # specify a secret to store the CA certificate and private key
  commonName: "TiDB CA"
  isCA: true                                  # Make this certificate is valid for certificate signing
  duration: 87600h # 10yrs
  renewBefore: 720h # 30d
  issuerRef:
    name: tidb-selfsigned-ca-issuer           # Must specify an issuer for the certificate, we want a CA certificate, the issuer should be self-signed issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tidb-issuer                           # This is the issuer issues TLS certificates for the TiDB cluster
  namespace: tidb-cluster
spec:
  ca:
    secretName: tidb-ca-secret

2.Create Issuers

kubectl apply -f tidb-server-issuer.yaml

kubectl get issuer -n tidb-cluster

Output:

NAMESPACE      NAME                        READY   AGE
tidb-cluster   tidb-issuer                 True    18s
tidb-cluster   tidb-selfsigned-ca-issuer   True    18s

Enable TiDB Cluster TLS for Client

Perpare Server Certificates

We need to issue two sets of certificates:

A set of server-side certificates for TiDB server
A set of client-side certificates for MySQL client.

Certificate and keys will be stored in secrets, ${cluster_name}-tidb-server-secret and ${cluster_name}-tidb-client-secrete.

The Secret objects you created must follow the above naming convention. Otherwise, the deployment of the TiDB cluster will fail.

1.Create tidb-server-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: advanced-test-tidb-server-secret
  namespace: tidb-cluster
spec:
  secretName: advanced-test-tidb-server-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - Sherry Yin Xi
  commonName: "TiDB Server"
  usages:
    - server auth
  dnsNames:
    - "advanced-test-tidb"
    - "advanced-test-tidb.tidb-cluster"
    - "advanced-test-tidb.tidb-cluster.svc"
    - "*.advanced-test-tidb"
    - "*.advanced-test-tidb.tidb-cluster"
    - "*.advanced-test-tidb.tidb-cluster.svc"
    - "*.advanced-test-tidb-peer"
    - "*.advanced-test-tidb-peer.tidb-cluster"
    - "*.advanced-test-tidb-peer.tidb-clsuter.svc"
  ipAddresses:
    - 127.0.0.1
    - ::1
  issuerRef:
    name: tidb-issuer
    kind: Issuer
    group: cert-manager.io

2.Issue Certificate for TiDB Server

kubectl apply -f tidb-server-cert.yaml

Perpare Client Certificates

1.Create tidb-client-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: advanced-test-tidb-client-secret
  namespace: tidb-cluster
spec:
  secretName: advanced-test-tidb-client-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - Sherry Yin Xi
  commonName: "TiDB Client"
  usages:
    - client auth
  issuerRef:
    name: tidb-issuer
    kind: Issuer
    group: cert-manager.io

2.Issue Cerficate for MySQL Client

kubectl apply -f tidb-client-cert.yaml

3.Other Client Components Please refer to official documentaiton.

Enable TLS for Client

For an existing TiDB cluster, just update your tidb cluster manifest set .spec.tidb.tlsClient.enabled to true as below.

spec:
...
  tidb:
    ...
    tlsClient:                  
      enabled: true             
  ...

After apply the above changes, the tidb and pd pods of your TiDB cluster will rolling restart.

Check the logs of PD and TiDB pods

kubectl logs advanced-test-pd-0 -n tidb-cluster |grep -i tls
[2024/06/06 02:37:20.061 +00:00] [INFO] [server.go:249] ["PD config"] [config="{\"client-urls\":\"http://0.0.0.0:2379\",\"peer-urls\":\"http://0.0.0.0:2380\",\"advertise-client-urls\":\"http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2379\",\"advertise-peer-urls\":\"http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2380\",\"name\":\"advanced-test-pd-0\",\"data-dir\":\"/var/lib/pd\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"advanced-test-pd-0=http://advanced-test-pd-0.advanced-test-pd-peer.tidb-cluster.svc:2380\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"\",\"lease\":3,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null,\"error-output-path\":\"\"},\"max-concurrent-tso-proxy-streamings\":5000,\"tso-proxy-recv-from-client-timeout\":\"1h0m0s\",\"tso-save-interval\":\"3s\",\"tso-update-physical-interval\":\"50ms\",\"enable-local-tso\":false,\"metric\":{\"job\":\"advanced-test-pd-0\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":20,\"max-merge-region-keys\":0,\"split-merge-interval\":\"1h0m0s\",\"switch-witness-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"max-store-preparing-time\":\"48h0m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"witness-schedule-limit\":4,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"enable-tikv-split-region\":\"true\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-witness\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"transfer-witness-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7,\"max-movable-hot-peer-size\":512,\"enable-diagnostic\":\"true\",\"enable-witness\":\"false\",\"slow-store-evicting-affected-store-ratio-threshold\":0.3,\"store-limit-version\":\"v1\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3,\"min-resolved-ts-persistence-interval\":\"1s\",\"server-memory-limit\":0,\"server-memory-limit-gc-trigger\":0.7,\"enable-gogc-tuner\":\"false\",\"gc-tuner-threshold\":0.6,\"block-safe-point-v1\":\"false\"},\"cluster-version\":\"0.0.0\",\"labels\":{},\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"max-request-bytes\":157286400,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null,\"SSLCABytes\":null,\"SSLCertBytes\":null,\"SSLKEYBytes\":null,\"redact-info-log\":false,\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"168h0m0s\",\"master-key\":{\"type\":\"plaintext\",\"key-id\":\"\",\"region\":\"\",\"endpoint\":\"\",\"path\":\"\"}}},\"label-property\":null,\"WarningMsgs\":null,\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\",\"dashboard\":{\"tidb-cacert-path\":\"/var/lib/tidb-client-tls/ca.crt\",\"tidb-cert-path\":\"/var/lib/tidb-client-tls/tls.crt\",\"tidb-key-path\":\"/var/lib/tidb-client-tls/tls.key\",\"public-path-prefix\":\"\",\"internal-proxy\":true,\"enable-telemetry\":false,\"enable-experimental\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"pause-region-split\":\"false\"}},\"keyspace\":{\"pre-alloc\":null,\"wait-region-split\":true,\"wait-region-split-timeout\":\"30s\",\"check-region-split-interval\":\"50ms\"},\"controller\":{\"degraded-mode-wait-duration\":\"0s\",\"ltb-max-wait-duration\":\"30s\",\"request-unit\":{\"read-base-cost\":0.125,\"read-per-batch-base-cost\":0.5,\"read-cost-per-byte\":0.0000152587890625,\"write-base-cost\":1,\"write-per-batch-base-cost\":1,\"write-cost-per-byte\":0.0009765625,\"read-cpu-ms-cost\":0.3333333333333333}}}"]

kubectl logs advanced-test-tidb-0 -c tidb -n tidb-cluster |grep tidb-server-tls
[2024/06/06 03:38:53.325 +00:00] [INFO] [printer.go:52] ["loaded config"] [config="{\"host\":\"0.0.0.0\",\"advertise-address\":\"advanced-test-tidb-0.advanced-test-tidb-peer.tidb-cluster.svc\",\"port\":4000,\"cors\":\"\",\"store\":\"tikv\",\"path\":\"advanced-test-pd:2379\",\"socket\":\"/tmp/tidb-4000.sock\",\"lease\":\"45s\",\"split-table\":true,\"token-limit\":1000,\"temp-dir\":\"/tmp/tidb\",\"tmp-storage-path\":\"/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage\",\"tmp-storage-quota\":-1,\"server-version\":\"\",\"version-comment\":\"\",\"tidb-edition\":\"\",\"tidb-release-version\":\"\",\"keyspace-name\":\"\",\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":null,\"enable-timestamp\":null,\"disable-error-stack\":null,\"enable-error-stack\":null,\"file\":{\"filename\":\"\",\"max-size\":300,\"max-days\":0,\"max-backups\":3},\"slow-query-file\":\"/var/log/tidb/slowlog\",\"expensive-threshold\":10000,\"query-log-max-len\":4096,\"enable-slow-log\":true,\"slow-threshold\":0,\"record-plan-in-slow-log\":1,\"timeout\":0},\"instance\":{\"tidb_general_log\":false,\"tidb_pprof_sql_cpu\":false,\"ddl_slow_threshold\":300,\"tidb_expensive_query_time_threshold\":60,\"tidb_expensive_txn_time_threshold\":600,\"tidb_stmt_summary_enable_persistent\":false,\"tidb_stmt_summary_filename\":\"tidb-statements.log\",\"tidb_stmt_summary_file_max_days\":3,\"tidb_stmt_summary_file_max_size\":64,\"tidb_stmt_summary_file_max_backups\":0,\"tidb_enable_slow_log\":true,\"tidb_slow_log_threshold\":0,\"tidb_record_plan_in_slow_log\":1,\"tidb_check_mb4_value_in_utf8\":true,\"tidb_force_priority\":\"NO_PRIORITY\",\"tidb_memory_usage_alarm_ratio\":0.8,\"tidb_enable_collect_execution_info\":true,\"plugin_dir\":\"/data/deploy/plugin\",\"plugin_load\":\"\",\"max_connections\":0,\"tidb_enable_ddl\":true,\"tidb_rc_read_check_ts\":false,\"tidb_service_scope\":\"\"},\"security\":{\"skip-grant-table\":false,\"ssl-ca\":\"/var/lib/tidb-server-tls/ca.crt\",\"ssl-cert\":\"/var/lib/tidb-server-tls/tls.crt\",\"ssl-key\":\"/var/lib/tidb-server-tls/tls.key\",\"cluster-ssl-ca\":\"\",\"cluster-ssl-cert\":\"\",\"cluster-ssl-key\":\"\",\"cluster-verify-cn\":null,\"session-token-signing-cert\":\"\",\"session-token-signing-key\":\"\",\"spilled-file-encryption-method\":\"plaintext\",\"enable-sem\":false,\"auto-tls\":false,\"tls-version\":\"\",\"rsa-key-size\":4096,\"secure-bootstrap\":false,\"auth-token-jwks\":\"\",\"auth-token-refresh-interval\":\"1h0m0s\",\"disconnect-on-expired-password\":true},\"status\":{\"status-host\":\"0.0.0.0\",\"metrics-addr\":\"\",\"status-port\":10080,\"metrics-interval\":15,\"report-status\":true,\"record-db-qps\":false,\"record-db-label\":false,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-concurrent-streams\":1024,\"grpc-initial-window-size\":2097152,\"grpc-max-send-msg-size\":2147483647},\"performance\":{\"max-procs\":0,\"max-memory\":0,\"server-memory-quota\":0,\"stats-lease\":\"3s\",\"stmt-count-limit\":5000,\"pseudo-estimate-ratio\":0.8,\"bind-info-lease\":\"3s\",\"txn-entry-size-limit\":6291456,\"txn-total-size-limit\":104857600,\"tcp-keep-alive\":true,\"tcp-no-delay\":true,\"cross-join\":true,\"distinct-agg-push-down\":false,\"projection-push-down\":false,\"max-txn-ttl\":3600000,\"index-usage-sync-lease\":\"0s\",\"plan-replayer-gc-lease\":\"10m\",\"gogc\":100,\"enforce-mpp\":false,\"stats-load-concurrency\":5,\"stats-load-queue-size\":1000,\"analyze-partition-concurrency-quota\":16,\"plan-replayer-dump-worker-concurrency\":1,\"enable-stats-cache-mem-quota\":true,\"committer-concurrency\":128,\"run-auto-analyze\":true,\"force-priority\":\"NO_PRIORITY\",\"memory-usage-alarm-ratio\":0.8,\"enable-load-fmsketch\":false,\"lite-init-stats\":true,\"force-init-stats\":true},\"prepared-plan-cache\":{\"enabled\":true,\"capacity\":100,\"memory-guard-ratio\":0.1},\"opentracing\":{\"enable\":false,\"rpc-metrics\":false,\"sampler\":{\"type\":\"const\",\"param\":1,\"sampling-server-url\":\"\",\"max-operations\":0,\"sampling-refresh-interval\":0},\"reporter\":{\"queue-size\":0,\"buffer-flush-interval\":0,\"log-spans\":false,\"local-agent-host-port\":\"\"}},\"proxy-protocol\":{\"networks\":\"\",\"header-timeout\":5,\"fallbackable\":false},\"pd-client\":{\"pd-server-timeout\":3},\"tikv-client\":{\"grpc-connection-count\":4,\"grpc-keepalive-time\":10,\"grpc-keepalive-timeout\":3,\"grpc-compression-type\":\"none\",\"commit-timeout\":\"41s\",\"async-commit\":{\"keys-limit\":256,\"total-key-size-limit\":4096,\"safe-window\":2000000000,\"allowed-clock-drift\":500000000},\"max-batch-size\":128,\"overload-threshold\":200,\"max-batch-wait-time\":0,\"batch-wait-size\":8,\"enable-chunk-rpc\":true,\"region-cache-ttl\":600,\"store-limit\":0,\"store-liveness-timeout\":\"1s\",\"copr-cache\":{\"capacity-mb\":1000},\"copr-req-timeout\":60000000000,\"ttl-refreshed-txn-size\":33554432,\"resolve-lock-lite-threshold\":16},\"binlog\":{\"enable\":false,\"ignore-error\":false,\"write-timeout\":\"15s\",\"binlog-socket\":\"\",\"strategy\":\"range\"},\"compatible-kill-query\":false,\"pessimistic-txn\":{\"max-retry-count\":256,\"deadlock-history-capacity\":10,\"deadlock-history-collect-retryable\":false,\"pessimistic-auto-commit\":false,\"constraint-check-in-place-pessimistic\":true},\"max-index-length\":3072,\"index-limit\":64,\"table-column-count-limit\":1017,\"graceful-wait-before-shutdown\":0,\"alter-primary-key\":false,\"treat-old-version-utf8-as-utf8mb4\":true,\"enable-table-lock\":false,\"delay-clean-table-lock\":0,\"split-region-max-num\":1000,\"top-sql\":{\"receiver-address\":\"\"},\"repair-mode\":false,\"repair-table-list\":[],\"isolation-read\":{\"engines\":[\"tikv\",\"tiflash\",\"tidb\"]},\"new_collations_enabled_on_first_bootstrap\":true,\"experimental\":{\"allow-expression-index\":false},\"skip-register-to-dashboard\":false,\"enable-telemetry\":false,\"labels\":{},\"enable-global-index\":false,\"deprecate-integer-display-length\":false,\"enable-enum-length-limit\":true,\"stores-refresh-interval\":60,\"enable-tcp4-only\":false,\"enable-forwarding\":false,\"max-ballast-object-size\":0,\"ballast-object-size\":0,\"transaction-summary\":{\"transaction-summary-capacity\":500,\"transaction-id-digest-min-duration\":2147483647},\"enable-global-kill\":true,\"enable-32bits-connection-id\":true,\"initialize-sql-file\":\"\",\"enable-batch-dml\":false,\"mem-quota-query\":1073741824,\"oom-action\":\"cancel\",\"oom-use-tmp-storage\":true,\"check-mb4-value-in-utf8\":true,\"enable-collect-execution-info\":true,\"plugin\":{\"dir\":\"/data/deploy/plugin\",\"load\":\"\"},\"max-server-connections\":0,\"run-ddl\":true,\"disaggregated-tiflash\":false,\"autoscaler-type\":\"aws\",\"autoscaler-addr\":\"tiflash-autoscale-lb.tiflash-autoscale.svc.cluster.local:8081\",\"is-tiflashcompute-fixed-pool\":false,\"autoscaler-cluster-id\":\"\",\"use-autoscaler\":false,\"tidb-max-reuse-chunk\":64,\"tidb-max-reuse-column\":256,\"tidb-enable-exit-check\":false,\"in-mem-slow-query-topn-num\":30,\"in-mem-slow-query-recent-num\":500}"]

MySQL Client Connect to TiDB Cluster with TLS

Save CA/Client Certificate/Private Key

kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.tls\.crt}' | base64 --decode > client-tls.crt
kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.tls\.key}' | base64 --decode > client-tls.key
kubectl get secret -n tidb-cluster advanced-test-tidb-client-secret  -ojsonpath='{.data.ca\.crt}'  | base64 --decode > client-ca.crt

Connect with MySQL Client

mysql --comments -uroot -p -P 4000 -h ${tidb_host} --ssl-cert=client-tls.crt --ssl-key=client-tls.key --ssl-ca=client-ca.crt

mysql> SHOW STATUS LIKE "Ssl%";
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name         | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Ssl_cipher            | TLS_AES_128_GCM_SHA256                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Ssl_cipher_list       | RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:AES128-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256: |
| Ssl_server_not_after  | Jun  6 02:22:01 2025 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Ssl_server_not_before | Jun  6 02:22:01 2024 UTC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Ssl_verify_mode       | 5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Ssl_version           | TLSv1.3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6 rows in set (0.10 sec)

TiDB Operator Enable TLS For MySQL Client With cert-manager was originally published by Sherry, Yin Xi at NODBS on June 06, 2024.

S3 Compatable MinIO Operator Deployment

2024-04-12T00:00:00+08:00

Object store service is a common service nowdays, especially S3 compatible. The introduction of MinIO can be found here.

This article will demostrate how to setup MinIO Operator for testing purpose, this is NOT a comprehensive guide for production scenario.

Deploy MinIO Operator
Deploy MinIO Tenant
Create bucket in MinIO Console

Depoly MinIO Operator

We use Helm to deploy the operator.

helm repo add minio-operator https://operator.min.io
helm install --namespace minio-operator  --create-namespace operator minio-operator/operator

Deploy MinIO Tenant

1.Download tenant CR file tenant-values.yaml

curl -sLo tenant-values.yaml https://raw.githubusercontent.com/minio/operator/master/helm/tenant/values.yaml

2.Configure Tenant Update tenant-values.yaml

tenant:
  name: myminio
  image:
    repository: quay.io/minio/minio
    tag: RELEASE.2024-07-16T23-46-41Z
    pullPolicy: IfNotPresent
  imagePullSecret: { }
  scheduler: { }
  configuration:
    name: myminio-env-configuration
    configSecret:
      name: myminio-env-configuration
      accessKey: minio
      secretKey: minio123
  pools:
    - servers: 1
      name: pool-0
      volumesPerServer: 1
      size: 10Gi
      storageAnnotations: { }
      annotations: { }
      labels: { }
      tolerations: [ ]
      nodeSelector: { }
      affinity: { }
      resources: { }
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        fsGroupChangePolicy: "OnRootMismatch"
        runAsNonRoot: true
      containerSecurityContext:
        runAsUser: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints: [ ]
  mountPath: /export
  subPath: /data
  metrics:
    enabled: false
    port: 9000
    protocol: http
  certificate:
    externalCaCertSecret: [ ]
    externalCertSecret: [ ]
    requestAutoCert: true
    certConfig: { }
  features:
    bucketDNS: false
    domains: { }
    enableSFTP: false
  buckets: [ ]
  users: [ ]
  podManagementPolicy: Parallel
  liveness: { }
  readiness: { }
  startup: { }
  lifecycle: { }
  exposeServices: { }
  serviceAccountName: ""
  prometheusOperator: false
  logging: { }
  serviceMetadata: { }
  env: [ ]
  priorityClassName: ""
  additionalVolumes: [ ]
  additionalVolumeMounts: [ ]

ingress:
  api:
    enabled: false
    ingressClassName: ""
    labels: { }
    annotations: { }
    tls: [ ]
    host: minio.local
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: ""
    labels: { }
    annotations: { }
    tls: [ ]
    host: minio-console.local
    path: /
    pathType: Prefix

3.Deploy tenant

helm install  --namespace myminio --create-namespace --values tenant-values.yaml myminio minio-operator/tenant

4.Expose console service

kubectl port-forward svc/myminio-console 9443 -n myminio --address=0.0.0.0

5.Access MinIO console and create bucket

Username: minio (accessKey)
Password: minio123 (secretKey)

S3 Compatable MinIO Operator Deployment was originally published by Sherry, Yin Xi at NODBS on April 12, 2024.

TiDB Operator Logging EFK Integration

2024-04-07T00:00:00+08:00

❗ This is not a production setup, it’s only for testing and demo purpose

Introduction

Since TiDB components deployed by TiDB Operator output the logs in the stdout and stderr of the container by default, as long as you have EFK deployed in your Kubernetes cluster, all the TiDB components logs will be collected by Fluentd running on each Kubernetes node, you can find TiDB logs same as other Kubernetes serviceslogs.

This article demonstrates how to collect TiDB logs using Fluent-bit in Kubernetes based on EFK, especially for slow logs which need to be properly formated before streaming to EFK, for the EFK deployment please refer to my previous blog Kubernetes Logging EFK Deployment

Enviornments

Fluent-bit v1.8 (needed by multiline parser)
TiDB operator v1.5.2
TiDB v7.5

Step by Step Setup

1. Deploy TiDB Operator

kubectl create -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.5.2/manifests/crd.yaml
kubectl create namespace tidb-admin
helm repo add pingcap https://charts.pingcap.org/
helm install --namespace tidb-admin tidb-operator pingcap/tidb-operator --version v1.5.2

2. Deploy TiDB Cluster with Fluent-bit

Fluent bit running as sidecar in TiDB pod, you can use Fluentd as well. I choose fluent-bit as it’s light weight.

2.1 Prepare Fluent bit config

For demo purpose, I will manually mount Fluent-bit config. You may customize your image for a clean deployment, as slow query log is multiline format, we need to handle it properly, mutiline format parser is included in the config.

1.Create fluent-bit ConfigMap

kubectl apply -f tidb-fluentbit-cm-slowlog.yaml

tidb-fluentbit-cm-slowlog.yaml can be found here.

2.2 Deploy TiDB Cluster with fluent-bit as sidecar container

Option 1: fluent-bit in additional container

This deployment will keep the default slowlog container, fluent-bit as an additional container, tidb pod in total 3 containers.

kubectl apply -f tidb-advanced-cluster-additional-container.yaml

tidb-advanced-cluster-additional-container.yaml can be found here.

Option2: Fluent-bit Merged with default slowlog container

We customize the default slowlog container to run Fluent bit by merging the default slowlog container and additional container. tidb pod only 2 containers

kubectl apply -f tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml

tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml can be found here.

3. View TiDB Slow Query Logs in Kibana

1.You will find tidb-yyyy.mm.dd index

2.Create data view for tidb index

3.View TiDB Slow Logs

4.Formated TiDB Slow Log

TiDB Operator Logging EFK Integration was originally published by Sherry, Yin Xi at NODBS on April 07, 2024.

Kubernetes Logging EFK Deployment

2024-04-02T00:00:00+08:00

❗ This is not a production setup, it’s only for testing and demo purpose

Envrionments

minikube v1.32.0
Kubernetes v1.24
ECK v2.12
Elasticsearch + Kibana v8.13.0
Fluentd v1.16

Introduction

This is a simple infra level logging system setup, Fluentd as a daemonset running on each Kubernetes node as an agent to collect all containers’ logs to elasticseach. Again, this is not for production setup, just meet basic log collection requirement, production setup colud be more conprehensive.

A glance of the architecture

Step By Step Setup

1. Prepare Kubernetes

minikube start --cpus=3 --memory=6G --disk-size=25G --kubernetes-version=v1.24

2. Deploy ECK

Refer to official documentation Deploy ECK in your Kubernetes cluster

2.1 Deploy Elastic Operator

1.Create CRD

kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml

2.Create Operator

kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml

2.2 Deploy Elasticsearch

1.Create namespace eck for ECK

2.Deploy Elaisticsearch

kubectl apply -f elasticsearch.yaml

elasticsearch.yaml can be found here.

Make sure HEALTH is green, and PHASE is Ready.

2.3 Deploy Kibana

kubectl apply -f kibana.yaml

kibana.yaml can be found here.

2.4 Access Kibana

1.Get password

kubectl get secret quickstart-es-elastic-user -n eck -o go-template='{{.data.elastic | base64decode}}'

2.Expose kibana service access from outside Kubernetes

kubectl port-forward service/quickstart-kb-http -n eck 5601 --address=0.0.0.0

3.Login Access Kibana URL https://{your host}:5601/

3. Deploy Fluentd

kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml

fluentd-daemonset-elasticsearch-rbac.yaml can be found here.

Replace the environment varibles accordingly, especically FLUENT_ELASTICSEARCH_PASSWORD

4. Create Data View in Kibana

5. View K8S Logs in Kibana

Enjoy EFK!!!

Kubernetes Logging EFK Deployment was originally published by Sherry, Yin Xi at NODBS on April 02, 2024.

Rust Begin Here

2023-08-15T00:00:00+08:00

Local Rust Installation

Install

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

More details refer to the guide.

Components installed

cargo: the Rust’s build system and package manager.
rustfmt: a tool for formatting Rust code according to style guidelines.

Hello World!

New

Create a new project with Cargo

cargo new hello_world

Output: “Created binary (application) hello_world package”

A quick glance

cat hello_world/src/main.rs

Output:

fn main() {
    println!("Hello, world!");
}

Build

cd hello_world
cargo build

Output: “binary hello_world was created under target/debug”

Run

target/debug/hello_world

output: “Hello, world!”

cargo run

output:

    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `target/debug/hello_world`
Hello, world!

Project Directory

tree
.
├── Cargo.lock
├── Cargo.toml
├── src
│   └── main.rs
└── target
    ├── CACHEDIR.TAG
    └── debug
        ├── build
        ├── deps
        │   ├── hello_world-a65d4c6029deb9db
        │   ├── hello_world-a65d4c6029deb9db.2m06qpscokrdldr2.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.337tnzf71uan4bv8.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3lakkzjgkq4s2ufz.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3nyq6f5mboo59omw.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.3txvt6tt1wrkqp6g.rcgu.o
        │   ├── hello_world-a65d4c6029deb9db.d
        │   └── hello_world-a65d4c6029deb9db.okmjk6ajfymokvw.rcgu.o
        ├── examples
        ├── hello_world
        ├── hello_world.d
        └── incremental
            └── hello_world-27bxg3u8cn054
                ├── s-gq9ubsyjer-cu1tnq-8pp2iqcnvjen9b1yyqrks0gg0
                │   ├── 2m06qpscokrdldr2.o
                │   ├── 337tnzf71uan4bv8.o
                │   ├── 3lakkzjgkq4s2ufz.o
                │   ├── 3nyq6f5mboo59omw.o
                │   ├── 3txvt6tt1wrkqp6g.o
                │   ├── dep-graph.bin
                │   ├── okmjk6ajfymokvw.o
                │   ├── query-cache.bin
                │   └── work-products.bin
                └── s-gq9ubsyjer-cu1tnq.lock

10 directories, 24 files

IDE Recommendation

VSCode

Some plugins to recommend:

Rust Syntax: provides a TextMate grammar for Rust.
rust-analyzer: an implementation of Language Server Protocol for the Rust programming language. It provides features like completion and goto definition for many code editors, including VS Code, Emacs and Vim.
crates: helps Rust developers managing dependencies with Cargo.toml.
Even Better Toml: a TOML language support extension.
Rust Test Lens: adds a code lens to quickly run or debug a single test for your Rust code.

RustRover

From JetBrains, currently RustRover is free to use during the public preview, download.

Rust Begin Here was originally published by Sherry, Yin Xi at NODBS on August 15, 2023.