5 Star 33 Fork 20

DiDi-opensource/KnowAgent

Create your Gitee Account
Explore and code with more than 12 million developers,Free private repositories !:)
Sign up
Clone or Download
contribute
Sync branch
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README
Apache-2.0

KnowAgent 一站式日志采集平台

​ 阅读本文档,您可以了解到 KnowAgent 的用户群体、产品定位等信息,并通过体验地址,快速体验以应用为采集粒度,从应用维度批量下发采集任务全流程。若KnowAgent已在贵司的生产环境进行使用,并想要获得官方更好地支持和指导,可以通过添加微信号William_Hu1989PenceXie,加入官方交流平台。

1 产品简介

KnowAgent 源于滴滴多年来在各核心场景沉淀的超大规模采集引擎集群运维实践经验。经历过多方考验,对于大规模 Agent 日志采集的治理管控,采集任务与 Agent 的可观测性、健康度巡检、故障诊断已经沉淀出一套完善的解决方案。KnowAgent 以应用为采集粒度,满足了从业务层面创建采集任务的需求,并易于将文件采集作为流式计算的一种可靠数据源进行接入、管理。

1.1 前置申明

1.1.1 Agent 与采集任务管控规模

KnowAgent 基于最小依赖、便于体验,采用 MySQL 存储 Agent 的 Metrics 与 Error Logs 数据。受制于 MySQL 性能瓶颈,在单个采集任务对应一个采集路径、Metrics 与 Error Logs 数据保存周期为一周的情况下,支持 50 个 Agent 与 50 个采集任务的管控。如需要管控更多的 Agent 与采集任务,参见《如何替换Agent的Metrics与Error Logs数据存储引擎以纳管更大的Agent集群》

1.1.2 容器采集

KnowAgent目前对容器日志采集未实现平台化支持,未来将提供全面的容器日志采集平台化支持,参见《KnowAgent一站式日志采集平台介绍》展望部分。

1.2 快速体验地址

1.3 用户体验地图

  • 我的工作台
    • 运营大盘:展示运营相关的核心指标数据。
    • 运维大盘:展示运维相关的核心指标数据。
  • 元数据中心
    • 应用管理:维护应用相关信息、与应用-主机关联关系。
    • 接收端管理:维护 Agent 数据流、指标流、错误日志流的数据需要写入的下游接收端相关信息。
    • 元数据管理:维护用户上传的元数据 Excel 文件(元数据 Excel 文件内容含主机与应用信息),并提供元数据文件内容预览、元数据导入功能。
  • Agent 中心
    • Agent 版本管理:维护各个版本的 Agent 安装包信息。
    • Agent 管理:维护主机相关信息与 Agent 配置、查看 Agent 健康度与故障原因(健康度为黄或红时显示故障原因)。Agent 启动后,将携带宿主机信息自动向管理平台进行注册,注册成功后,即可对其进行管理。
  • 采集任务管理:维护采集任务相关信息,查看采集任务健康度与故障原因(健康度为黄或红时显示故障原因)。
  • 监控中心:
    • Agent 指标看板:查看 Agent 运行时全景指标(含:系统、进程、与 Agent 自身业务指标)。
    • 采集任务指标看板:用于查看采集任务在各个主机上的运行时全景指标。
  • 运维中心
    • 指标探查:查看 Agent 与采集任务的运行时全景指标,支持多 Agent、多采集任务进行对比分析、一体化观测。

1.4 核心优势

一站式日志采集方案

​ 高可靠、高性能、具备全方位可观测性的采集引擎,与面向应用的易管控、易观测、易治理的管理平台,大幅降低日志数据采集接入成本,大幅提升日志数据采集接入效率。

易融入流式计算

​ 面向应用的采集任务、高可靠的采集引擎可确保在任意情况下的数据完整性、完善的监控指标,支撑可靠的采集任务的数据完整性校验,这使得采集任务可作为流式计算的一种可靠的数据源,形成整个流式计算链路的数据完整性闭环。

生产级特性

​ 经过长时间在各场景下大规模采集引擎集群运维沉淀的专家经验,形成了全方位的可观测性与完善的故障、风险感知与诊断能力,使较少的无需采集引擎先验知识的普通运维人员亦能轻松运维庞大的采集引擎集群。

高性能

​ 优异的持续采集性能,基于jdk11+,处理单行、单文件、无解析场景下发送日志数据至kafka,持续吞吐量可达100MB/s以上,极限吞吐量可达150MB/s以上,CPU限制单核情况下,持续吞吐量可达40MB/s以上。

同类对比

维度 Flume FileBeat Logi-Agent
可靠性 一般 一般 高可靠
采集性能 Mb级 十Mb级 百Mb级
数据完整性 无法确保数据完整性 无法确保数据完整性 可确保数据完整性
采集任务级的租户隔离 无法做到采集任务级的租户隔离 无法做到采集任务级的租户隔离 可做到采集任务级的租户隔离
可扩展性 良好 一般 一般
资源消耗 CPU消耗与FileBeat趋近(大采集量场景下CPU消耗较FileBeat更小),内存消耗较FileBeat略大(但在一个数量级上)
监控指标的完善程度 较少指标 较少指标 完善的指标体系
可观测性 具备全方位的可观测性
是否容易配置
大规模运维复杂度 极高 极高

1.5 KnowAgent 架构

KnowAgent 是一站式的日志采集平台,包括两个组件:

  • Agent:是基于 Java 的高性能、支持多租户隔离的采集引擎,它能在任意情况下(除:待采集数据在被采集前被移除,例如:日志文件被归档或日志文件滚动速度太快导致)保证采集数据的完整性,持续采集速度可达150MB+/秒。
  • Agent Manager:是针对 Agent 的管理平台。用于管理相关元数据、Agent与采集任务,自动巡检 Agent 与采集任务的健康度、故障诊断,指标展示。
KnowAgent架构

1.5.1 Agent 架构

Agent架构

1.5.2 Agent Manager 架构

AgentManager架构

2 相关文档

2.1 产品文档

《KnowAgent一站式日志采集平台介绍》

《快速搭建一站式KnowAgent体验环境》

《KnowAgent安装部署手册》

《KnowAgent用户使用手册》

《KnowAgent开发手册》

《如何替换Agent的Metrics与Error Logs数据存储引擎以纳管更大的Agent集群》

《KnowAgent代码贡献指南》

《KnowAgent常见问题》

3 KnowAgent开源用户交流群

​ 请添加微信号:mike_zhangliang 或 William_Hu1989,回复”KnowAgent 加群“。

4 协议

KnowAgent基于Apache-2.0协议进行分发和使用,更多信息参见协议文件

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright (C) 2017 Beijing Didi Infinity Technology and Development Co.,Ltd. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

一站式日志采集平台 expand collapse
Apache-2.0
Cancel

Releases

No release

Contributors

All

Activities

Load More
can not load any more
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
1
https://gitee.com/didiopensource/KnowAgent.git
git@gitee.com:didiopensource/KnowAgent.git
didiopensource
KnowAgent
KnowAgent
master

Search