kafka接入clickhouse

╰半夏微凉° 2023-01-03 04:23 161阅读 0赞

1.ClickHouse 提供了Kafka Engine 作为访问Kafka集群的一个接口（数据流）

[https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/?spm=a2c4g.11186623.2.15.54096144mc3uXv][https_clickhouse.tech_docs_en_engines_table-engines_integrations_kafka_spm_a2c4g.11186623.2.15.54096144mc3uXv]

缺点：不可单独按表设置入库大小，比如你想只有1条kafka数据落地clickhouse，可以实现，但比较麻烦

2.[clickhouse\_sinker][clickhouse_sinker]

[https://github.com/housepower/clickhouse\_sinker][clickhouse_sinker]

3.[gohangout][]

[https://github.com/childe/gohangout][gohangout](官方)

[https://gitee.com/betas/gohangout-plus][https_gitee.com_betas_gohangout-plus](个人优化gitee)

[https://github.com/jsntian/gohangout-plus][https_github.com_jsntian_gohangout-plus](个人优化github)

做了以下优化：

1.  ipip.go中简写了 country\_name -> country,province\_name -> province,city\_name -> city,由于业务不需需，删除了isp、latitude等
2.  根目下上传了 ipipfree.ipdb 文件
3.  clickhouse\_output.go中实现在自动加载表中所有字段，无需在配置文件中配置fields，对json解析时数值类型不匹配的错误，自动根据数据库类型转化(相关代码类型请见clickhouse\_output.go432行)，无需再配置文件中使用json:not\_usenumber加Convert处理

4.官网推荐各种接入clickhouse的第三方库

[https://clickhouse.tech/docs/zh/interfaces/third-party/integrations/][https_clickhouse.tech_docs_zh_interfaces_third-party_integrations]

[https_clickhouse.tech_docs_en_engines_table-engines_integrations_kafka_spm_a2c4g.11186623.2.15.54096144mc3uXv]: https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/?spm=a2c4g.11186623.2.15.54096144mc3uXv
[clickhouse_sinker]: https://github.com/housepower/clickhouse_sinker
[gohangout]: https://github.com/childe/gohangout
[https_gitee.com_betas_gohangout-plus]: https://gitee.com/betas/gohangout-plus
[https_github.com_jsntian_gohangout-plus]: https://github.com/jsntian/gohangout-plus
[https_clickhouse.tech_docs_zh_interfaces_third-party_integrations]: https://clickhouse.tech/docs/zh/interfaces/third-party/integrations/