Apache SkyWalking – Tracing

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Wed, 08 Apr 2026 00:00:00 +0000

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Apache SkyWalking introduced TraceQL support in version 10.4.0, implementing Grafana Tempo’s HTTP query APIs so that Grafana can query and visualize traces stored in SkyWalking without any additional plugins. This means you can now use the familiar Grafana Tempo data source to search, filter, and drill into both SkyWalking native traces and Zipkin-compatible traces — all served by your existing SkyWalking OAP server.

Architecture Overview

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

The TraceQL Service sits inside the OAP server and exposes the Tempo-compatible HTTP API on port 3200 (default). It converts traces from their native format into Tempo’s format, where the trace detail part (Trace message) reuses OTLP Trace definitions.

Limitations and Supported TraceQL Features

TraceQL is a rich query language, but SkyWalking currently implements a practical subset. The following features are supported:

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

The following features are not yet supported:

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

Important: SkyWalking native trace support in TraceQL is based on the Query Traces V2 API. Currently, only BanyanDB storage implements this API. Other storage backends (e.g., Elasticsearch, MySQL, PostgreSQL) do not support SkyWalking native trace queries via TraceQL. Zipkin-compatible traces are not subject to this restriction.

Trace Format Conversion

Since the trace detail part of Tempo’s format reuses OTLP Trace definitions, the conversion descriptions below refer to OTLP field names (e.g., span kind, status code).

SkyWalking Native Trace

Trace ID Encoding

SkyWalking native trace IDs are arbitrary strings (e.g., 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539), while Grafana Tempo requires pure hex-encoded trace IDs. The TraceQL Service encodes each UTF-8 byte of the original trace ID as two lowercase hex characters:

Original:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
Encoded:   32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

This encoded hex trace ID is what appears in all API responses and in Grafana. When you click a trace ID in Grafana, the TraceQL Service automatically decodes it back to the original SkyWalking trace ID for the internal query.

Span Kind Mapping

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

Status Mapping

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents are converted to OTLP span events, with tags mapped as string attributes and summary mapped as numeric attributes (serialized as strings).

Zipkin Trace

Span Kind Mapping

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

Status Mapping

If the otel.status_code tag is present, it is used directly.
Otherwise, if the error tag equals true, the status is STATUS_CODE_ERROR.
If neither tag is present, the status defaults to STATUS_CODE_UNSET.

Endpoint and Annotation Mapping

Zipkin endpoint fields are mapped to OTLP attributes (e.g., localEndpoint.ipv4 → net.host.ip), and Zipkin annotations are converted to OTLP span events.

For the full conversion details, see the TraceQL Service documentation.

How to Enable TraceQL

Step 1: Enable the TraceQL Module

By default, the TraceQL module is disabled (selector: ${SW_TRACEQL:-}). To enable it, set the selector to default:

# In application.yml
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

Or via environment variables:

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

If you want to query Zipkin traces, you also need to enable the Zipkin receiver so that SkyWalking can ingest Zipkin trace data:

# In application.yml
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

Or via environment variable:

export SW_RECEIVER_ZIPKIN=default

Full Configuration Reference

For the complete list of all configuration options and their default values, see the Configuration section of the TraceQL Service documentation.

Configuring Grafana Tempo Data Source

Prerequisite: Grafana 12 or later is required.

Each trace backend (SkyWalking native / Zipkin) needs its own Tempo data source in Grafana, because each is served under a different context path.

Context Paths

The two backends are served under separate context paths on the same port:

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

Setting Up the SkyWalking Data Source

In Grafana, go to Configuration → Data Sources → Add data source.
Choose Tempo.
Set the URL to http://<oap-host>:3200/skywalking.
Disable the Streaming option (SkyWalking does not support streaming mode).

Save and test the data source.

Setting Up the Zipkin Data Source

Same as above, but set the URL to http://<oap-host>:3200/zipkin.

Configuring Trace List Result Tags

When you search for traces in Grafana, the trace list panel shows a summary of each trace. The tracesListResultTags configuration controls which span tags are included in the search result and displayed as columns in the trace list.

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

Note that service.name and span.kind are always included regardless of this setting.

These tags appear as attribute columns in the Grafana Tempo trace search results, making it easier to identify and group traces at a glance:

SkyWalking native trace list:

Zipkin trace list:

You can customize these tags based on your application’s instrumentation. For example, if your services heavily use messaging, you might add mq.destination or messaging.system to the list.

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Go to the Explore page in Grafana.
Select the Tempo data source you configured for SkyWalking (e.g., SkyWalkingTraceQL).
Run a test query, then click Add to dashboard and save it as SkyWalking Trace.

Step 2: Configure Variables

Add dashboard variables so users can filter traces dynamically (e.g., by service name):

Step 3: Add a Trace Panel

Choose a Table chart (or edit the panel you saved).
Set Query type to Search.
Set the Service Name query condition to the variable $Service.
Add other query conditions as needed (e.g., duration, span name, tags).
Test and save.

Step 4: View Trace Details

Click any trace ID in the trace panel to jump to the Explore page showing the full trace waterfall view with all spans, tags, and events:

Zipkin Trace Dashboard

The setup for Zipkin traces is identical to SkyWalking native traces — just use the Zipkin Tempo data source you configured (e.g., ZipkinTraceQL).

Zipkin trace detail view:

Summary

With TraceQL support in SkyWalking 10.4.0, you can now leverage Grafana’s powerful Tempo data source to query and visualize both SkyWalking native traces and Zipkin-compatible traces. The key points to remember:

Enable the TraceQL module by setting SW_TRACEQL=default and enabling the desired backends.
Configure separate Tempo data sources in Grafana for each backend (/skywalking and /zipkin).
Disable the Streaming option in the Grafana Tempo data source settings.
Customize result tags via SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS and SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS to control what’s shown in search results.
SkyWalking native trace queries require BanyanDB storage (Zipkin traces work with all storage backends).

For the complete API reference and conversion details, see the TraceQL Service documentation. For Grafana integration details, see Use Grafana As The UI.

Zh: 使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Wed, 08 Apr 2026 00:00:00 +0000

使用 TraceQL 查询 SkyWalking 和 Zipkin 链路追踪数据并在 Grafana 中可视化

Apache SkyWalking 在 10.4.0 版本中引入了 TraceQL 支持，实现了 Grafana Tempo 的 HTTP 查询 API，使 Grafana 无需任何额外插件即可查询和可视化 SkyWalking 中存储的链路追踪数据。这意味着你现在可以使用熟悉的 Grafana Tempo 数据源来搜索、过滤和深入分析 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪 —— 所有数据都由现有的 SkyWalking OAP 服务器提供。

架构概览

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

TraceQL Service 位于 OAP 服务器内部，在端口 3200（默认）上暴露 Tempo 兼容的 HTTP API。它将链路追踪数据从原生格式转换为 Tempo 的格式，其中链路追踪详情部分（Trace 消息）复用了 OTLP Trace 定义。

支持的 TraceQL 特性与限制

TraceQL 是一种功能丰富的查询语言，但 SkyWalking 目前实现了一个实用的子集。以下特性已支持：

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

以下特性暂不支持：

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

重要提示：TraceQL 中的 SkyWalking 原生链路追踪支持基于 Query Traces V2 API。目前只有 BanyanDB 存储实现了该 API。其他存储后端（如 Elasticsearch、MySQL、PostgreSQL）不支持通过 TraceQL 查询 SkyWalking 原生链路追踪数据。 Zipkin 兼容链路追踪不受此限制。

Trace 格式转换

由于 Tempo 格式的链路追踪详情部分复用了 OTLP Trace 定义，以下转换描述使用 OTLP 字段名称（如 span kind、status code）。

SkyWalking 原生链路追踪

Trace ID 编码

SkyWalking 原生 trace ID 是任意字符串（例如 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539），而 Grafana Tempo 要求纯十六进制编码的 trace ID。TraceQL Service 将原始 trace ID 的每个 UTF-8 字节编码为两个小写十六进制字符：

原始值:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
编码后:  32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

编码后的十六进制 trace ID 会出现在所有 API 响应和 Grafana 中。当你在 Grafana 中点击 trace ID 时，TraceQL Service 会自动将其解码回原始的 SkyWalking trace ID 进行内部查询。

Span Kind 映射

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

状态映射

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents 被转换为 OTLP span events，其中 tags 映射为字符串属性，summary 映射为数值属性（序列化为字符串）。

Zipkin 链路追踪

Span Kind 映射

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

状态映射

如果存在 otel.status_code 标签，则直接使用。
否则，如果 error 标签等于 true，则状态为 STATUS_CODE_ERROR。
如果以上标签都不存在，则状态默认为 STATUS_CODE_UNSET。

Endpoint 与 Annotation 映射

Zipkin endpoint 字段被映射为 OTLP 属性（例如 localEndpoint.ipv4 → net.host.ip）， Zipkin annotations 被转换为 OTLP span events。

完整的转换详情请参阅 TraceQL Service 文档。

如何启用 TraceQL

步骤 1：启用 TraceQL 模块

默认情况下，TraceQL 模块是禁用的（selector: ${SW_TRACEQL:-}）。要启用它，将 selector 设置为 default：

# 在 application.yml 中
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

或通过环境变量设置：

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

步骤 2：启用 Zipkin 接收器（仅用于 Zipkin 链路追踪）

如果你需要查询 Zipkin 链路追踪数据，还需要启用 Zipkin 接收器，以便 SkyWalking 能够接收 Zipkin 链路追踪数据：

# 在 application.yml 中
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

或通过环境变量设置：

export SW_RECEIVER_ZIPKIN=default

完整配置参考

所有配置选项及其默认值的完整列表，请参阅 TraceQL Service 文档的配置章节。

配置 Grafana Tempo 数据源

前提条件：需要 Grafana 12 或更高版本。

每个链路追踪后端（SkyWalking 原生 / Zipkin）需要在 Grafana 中配置各自独立的 Tempo 数据源，因为它们分别在不同的上下文路径下提供服务。

上下文路径

两个后端在同一端口上使用不同的上下文路径提供服务：

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

配置 SkyWalking 数据源

在 Grafana 中，前往 Configuration → Data Sources → Add data source。
选择 Tempo。
将 URL 设置为 http://<oap-host>:3200/skywalking。
禁用 Streaming 选项（SkyWalking 不支持流式模式）。

保存并测试数据源。

配置 Zipkin 数据源

与上述步骤相同，但将 URL 设置为 http://<oap-host>:3200/zipkin。

配置链路追踪列表结果标签

在 Grafana 中搜索链路追踪时，链路追踪列表面板会显示每条追踪的摘要信息。 tracesListResultTags 配置控制哪些 span 标签会包含在搜索结果中并作为列显示在追踪列表中。

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

注意，无论此设置如何，service.name 和 span.kind 始终包含在结果中。

这些标签在 Grafana Tempo 链路追踪搜索结果中显示为属性列，方便快速识别和分组追踪数据：

SkyWalking 原生追踪列表：

Zipkin 追踪列表：

你可以根据应用程序的埋点情况自定义这些标签。例如，如果你的服务大量使用消息队列，可以在列表中添加 mq.destination 或 messaging.system。

在 Grafana 中构建链路追踪仪表板

SkyWalking 原生追踪仪表板

步骤 1：探索并保存

前往 Grafana 的 Explore 页面。
选择你为 SkyWalking 配置的 Tempo 数据源（例如 SkyWalkingTraceQL）。
运行一个测试查询，然后点击 Add to dashboard 并保存为 SkyWalking Trace。

步骤 2：配置变量

添加仪表板变量，以便用户可以动态过滤追踪数据（例如按服务名称过滤）：

步骤 3：添加追踪面板

选择 Table 图表（或编辑你保存的面板）。
将 Query type 设置为 Search。
将 Service Name 查询条件设置为变量 $Service。
根据需要添加其他查询条件（如 duration、span name、tags）。
测试并保存。

步骤 4：查看追踪详情

点击追踪面板中的任意 trace ID，即可跳转到 Explore 页面，查看完整的追踪瀑布图，包含所有 span、标签和事件：

Zipkin 追踪仪表板

Zipkin 追踪的设置与 SkyWalking 原生追踪完全相同 —— 只需使用你配置的 Zipkin Tempo 数据源（例如 ZipkinTraceQL）。

Zipkin 追踪详情视图：

总结

通过 SkyWalking 10.4.0 中的 TraceQL 支持，你现在可以利用 Grafana 强大的 Tempo 数据源来查询和可视化 SkyWalking 原生链路追踪和 Zipkin 兼容链路追踪数据。需要记住的要点：

启用 TraceQL 模块：设置 SW_TRACEQL=default 并启用所需的后端。
在 Grafana 中配置独立的 Tempo 数据源：为每个后端分别配置（/skywalking 和 /zipkin）。
禁用 Streaming 选项：在 Grafana Tempo 数据源设置中关闭流式模式。
自定义结果标签：通过 SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS 和 SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS 控制搜索结果中显示的内容。
SkyWalking 原生追踪查询需要 BanyanDB 存储（Zipkin 追踪支持所有存储后端）。

完整的 API 参考和转换详情，请参阅 TraceQL Service 文档。 Grafana 集成详情，请参阅使用 Grafana 作为 UI。

Blog: SkyWalking Ruby Quick Start and Principle Introduction

Thu, 06 Mar 2025 00:00:00 +0000

Background

Ruby is a dynamic, object-oriented programming language with concise and elegant syntax, supporting multiple programming paradigms, including object-oriented, functional, and metaprogramming. Leveraging its powerful metaprogramming capabilities, Ruby allows modifying the behavior of classes and objects at runtime. SkyWalking provides a Ruby gem to facilitate integration with Ruby projects, and this gem supports many out-of-the-box frameworks and gems.

This article is based on skywalking-ruby-v0.1. We will guide you on how to quickly integrate the skywalking-ruby project into Ruby projects and briefly introduce the implementation principle of SkyWalking Ruby’s auto-instrumentation plugins using redis-rb as an example.

The demonstration includes the following steps:

Deploy SkyWalking: This involves setting up the SkyWalking backend and UI programs to enable you to see the final results.
Integrate SkyWalking into Different Ruby Projects: This section explains how to integrate SkyWalking into different Ruby projects.
Application Deployment: You will export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend.
Visualization on SkyWalking UI: Finally, you will send requests and observe the results in the SkyWalking UI.

Deploy SkyWalking

Please download the SkyWalking APM program from the official SkyWalking website, and then you can start all the required services using the quick start script.

Next, you can access the address http://localhost:8080/. At this point, since no applications have been deployed, you will not see any data.

Integrate SkyWalking into Different Ruby Projects It is recommended to use Bundler to install and manage SkyWalking dependencies. Simply declare it in the Gemfile and run bundle install to complete the installation.

# Gemfile
source "https://rubygems.org"

gem "skywalking"

Integration in Rails Projects

For Rails projects, it is recommended to use the following command to automatically generate the configuration file:

bundle exec rails generate skywalking:start

This command will automatically generate a skywalking.rb file in the config/initializers directory, where you can configure the startup parameters.

Integration in Sinatra Projects

For Sinatra projects, you need to manually call Skywalking.start when the application starts. For example:

require 'sinatra'
require 'skywalking'

Skywalking.start

get '/sw' do
  "Hello SkyWalking!"
end

In the Gemfile, place skywalking after sinatra and use Bundler.require during initialization, or call require 'skywalking' after the sinatra gem is loaded. Note that the skywalking gem needs to be placed after other gems (such as redis, elasticsearch).

Application Deployment

Before starting the application deployment, you can change the service name of the current application in SkyWalking through environment variables. You can also modify its configuration, such as the server-side address. For more details, please refer to the documentation.

Here, we will change the current service name to sw-ruby.

Next, you can start the application. Here is an example using sinatra:

export SW_AGENT_SERVICE_NAME=sw-ruby

ruby sinatra.rb

Visualization on SkyWalking UI

Now, send requests to the application and observe the results in the SkyWalking UI.

After a few seconds, revisit the SkyWalking UI at http://localhost:8080. You will be able to see the deployed demo service on the homepage.

Additionally, on the tracing page, you can see the request you just sent.

Plugin Implementation Mechanism

To understand the implementation mechanism of Ruby Agent’s auto-instrumentation plugins, it is essential to understand the concept of the ancestor chain in Ruby. The ancestor chain is an ordered list, and in Ruby, each class or module has an ancestor chain that includes all its parent classes and mixin modules (modules mixed in via include, prepend, or extend). When Ruby looks up a method, it searches in the order of the ancestor chain until it finds the target method or throws a NoMethodError.

class User
end

We have defined a User class, and its ancestor chain is as shown in the following figure:

Next, mix in a module using the prepend method:

module Dapper
  def brave
    "Hello from brave"
  end
end

class User
  prepend Dapper
end

p User.new.brave # => "Hello from brave"

prepend will insert at position 1 in the above figure. Ruby first looks for the brave method in the Dapper module, finds it, and calls it. If the brave method is not found in Dapper, Ruby continues to search in the User class. If it is not found in the User class, Ruby continues to search in Object, and so on.

Based on this mechanism, let’s briefly introduce how we instrument the redis-rb method. The following code is the target method to be instrumented:

# lib/redis/client.rb
class Redis
  class Client < ::RedisClient
    def call_v(command, &block)
      super(command, &block)
    rescue ::RedisClient::Error => error
      Client.translate_error!(error)
    end
  end
end

Below is the core code for instrumentation:

module Skywalking
  module Plugins
    class Redis5 < PluginsManager::SWPlugin
      module Redis5Intercept
        def call_v(args, &block)
          operation = args[0] rescue "UNKNOWN"
          return super if operation == :auth

          Tracing::ContextManager.new_exit_span(
            operation: "Redis/#{operation.upcase}"
          ) do |span|
            # Omitted handling of span 
            super(args, &block) # Call the original method
          end
        end
      end

      def install
        ::Redis::Client.prepend Redis5Intercept
      end
    end
  end
end

Here, we define a Redis5Intercept module and prepend it to ::Redis::Client. According to Ruby’s method lookup mechanism, when the call_v method of Redis::Client is called, Ruby will first execute the call_v method in Redis5Intercept. The order of the ancestor chain is as follows:

Redis5Intercept -> Redis::Client -> ... (other parent classes and modules)

At the same time, in the call_v method of Redis5Intercept, super(args, &block) will find the next method with the same name along the ancestor chain, which in this case is the original call_v method in Redis::Client, while passing the original arguments and block.

Conclusion

This article explained the integration methods of SkyWalking Ruby in Ruby projects and briefly introduced the implementation mechanism of SkyWalking Ruby’s auto-instrumentation plugins.

Currently, the Ruby auto-instrumentation is in the early stages of development. In the future, we will continue to expand the functionality of SkyWalking Ruby and add support for more plugins. So, stay tuned!

Zh: SkyWalking Ruby 快速开始与原理介绍

Thu, 06 Mar 2025 00:00:00 +0000

背景

Ruby 是一种动态、面向对象的编程语言，它的语法简洁优雅，支持多种编程范式，包括面向对象、函数式和元编程。其中依靠强大的元编程能力，Ruby 允许在运行时修改类和对象的行为。 SkyWalking 提供了 Ruby gem，方便 Ruby 项目集成, 该 gem 支持许多开箱即用的框架和 gem。

本文基于 skywalking-ruby-v0.1，我们将指导你如何快速将 skywalking-ruby 项目集成到 Ruby 项目中，并以 redis-rb 为例，简要地介绍 SkyWalking Ruby 对插件自动探针的实现原理。

演示部分包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
为不同 Ruby 项目集成 skywalking：这里介绍了不同的 Ruby 项目如何集成 skywalking。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序，然后可以根据快速启动脚本启动所有所需服务。

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

为不同 Ruby 项目集成 SkyWalking

推荐使用 Bundler 来安装和管理 skywalking 的依赖。只需在 Gemfile 中声明，然后运行 bundle install 即可完成安装。

# Gemfile
source "https://rubygems.org"

gem "skywalking"

在 Rails 项目中集成

对于 Rails 项目，推荐使用以下命令自动生成配置文件：

bundle exec rails generate skywalking:start

该命令会在 config/initializers 目录下自动生成 skywalking.rb 文件，你可以在其中配置启动参数。

在 Sinatra 项目中集成

对于 Sinatra 项目，你需要手动在应用启动时调用 Skywalking.start。例如：

require 'sinatra'
require 'skywalking'

Skywalking.start

get '/sw' do
  "Hello SkyWalking!"
end

在 Gemfile 中，将 skywalking 放在 sinatra 之后，并在初始化时使用 Bundler.require，或者在 sinatra gem 加载后调用 require ‘skywalking’。注意，skywalking gem 需要位于其他 gem（如 redis、elasticsearch）之后。

应用部署

在开始部署应用程序之前，你可以通过环境变量更改 SkyWalking 中当前应用程序的服务名称。你还可以更改其配置，例如服务器端的地址。有关详细信息，请参阅文档。

在这里，我们将当前服务的名称更改为 sw-ruby。

接下来，你可以启动应用程序，这里以 sinatra 作为示例：

export SW_AGENT_SERVICE_NAME=sw-ruby

ruby sinatra.rb

在 SkyWalking UI 上可视化

现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

插件实现机制

要了解 Ruby Agent 对插件自动探针的实现机制，首先要了解 Ruby 中祖先链的概念。祖先链是一个有序的列表，在 Ruby 中，每个类或模块都有一个祖先链，它包含了一个类或模块的所有父类以及 mixin 模块（通过 include、prepend 或 extend 混入的模块）。 Ruby 在查找方法时，会按照祖先链的顺序依次查找，直到找到目标方法或抛出 NoMethodError。

class User
end

我们定义了一个 User 类，那么它的祖先链是如下图：

接下来用 prepend 方法混入一个模块：

module Dapper
  def brave
    "Hello from brave"
  end
end

class User
  prepend Dapper
end

p User.new.brave # => "Hello from brave"

prepend 会在上图 1 处进行插入，Ruby 首先在 Dapper 模块中查找 brave 方法，找到并调用，如果 Dapper 中没有 brave 方法， Ruby 会继续查找 User 类。如果 User 类中也没有，Ruby 会继续查找 Object，依此类推。

根据这样的机制，简单介绍下我们如何对 redis-rb 进行方法插桩，下面代码是要进行插桩的目标方法：

# lib/redis/client.rb
class Redis
  class Client < ::RedisClient
    def call_v(command, &block)
     super(command, &block)
    rescue ::RedisClient::Error => error
     Client.translate_error!(error)
    end
  end
end

下面是进行插桩的核心代码：

module Skywalking
  module Plugins
    class Redis5 < PluginsManager::SWPlugin
      module Redis5Intercept
        def call_v(args, &block)
          operation = args[0] rescue "UNKNOWN"
          return super if operation == :auth

          Tracing::ContextManager.new_exit_span(
            operation: "Redis/#{operation.upcase}"
          ) do |span|
            # 省略对 span 的处理 
            super(args, &block) # 调用原方法
          end
        end
      end

      def install
        ::Redis::Client.prepend Redis5Intercept
      end
    end
  end
end

这里我们定义了一个 Redis5Intercept 模块，并将其作为 ::Redis::Client 的前置模块，根据 Ruby 方法查找机制，当 Redis::Client 的 call_v 方法被调用时，Ruby 会首先会执行 Redis5Intercept 中的 call_v 方法，这里祖先链的顺序如下：

Redis5Intercept -> Redis::Client -> ...（其他父类和模块）

同时在 Redis5Intercept 中的 call_v 方法中，super(args, &block) 会沿着祖先链找到下一个同名方法，在这里也就是 Redis::Client 中的原始 call_v 方法，同时传递原始的参数和代码块。

总结

本文讲述了 Skywalking Ruby 在 Ruby 项目中的集成方法，并简要介绍了 SkyWalking Ruby 对插件自动探针的实现机制。

目前 Ruby 探针处于早期的开发阶段，未来我们将继续扩展 SkyWalking Ruby 的功能，添加更多插件支持。所以，请继续关注！

Blog: Introduce R3 to recognition RESTFul URI

Thu, 17 Oct 2024 00:00:00 +0000

Background

In modern applications, services are typically provided through RESTFul HTTP URIs. Using RESTFul HTTP URIs (as unique resource identifiers) offers high readability, making it easier for both clients and servers to understand. However, in the observability field, this approach poses several challenges:

A large number of endpoints (HTTP URI): Browsing through all externally provided endpoints becomes more difficult, making it hard to identify problematic endpoints.
Metrics are difficult to collect: It becomes particularly challenging to categorize similar endpoints and generate observability metrics.

In existing solutions, this issue can be resolved following these application-level resolutions:

Agent Detection: In certain frameworks, rules are often declared to handle RESTFul requests. For example, in Java’s Spring Web, annotations like @GET can be used, which can then be linked to current requests using a Java Agent.
OpenAPI: Predefined files can be associated with the application, allowing the observability system to be aware of the URIs that may be used.

Both resolutions are tightly coupled with application settings, which can be limiting for unknown applications or applications where the agent cannot be monitored. Therefore, we need to consider whether there is a more general solution to identify URIs and merge metrics generated from similar URIs for better representation.

R3

R3(RESTFul Pattern Recognition) is a high-performance RESTFul URI recognition tool inspired by Drain3. It can be deployed as a standalone application on the observability server and communicate with the SkyWalking OAP.

R3 can accept a URI list via the gRPC protocol and aggregate similar URIs into a specific format. The aggregated (formatted) URI list can also be queried using the gRPC protocol.

Data Interaction Flow

OAP receives and caches unformatted URI list: OAP receives observability data through different protocols and identifies all unformatted URIs. These URIs are stored in a temporary list categorized by the service they belong to.
OAP sends URIs to be formatted to R3: OAP periodically batches the URIs that need formatting and sends them to the R3 service.
R3 receives and parses the URI list: R3 asynchronously analyzes the similarity of the received URIs and stores (persists) the results on the local disk to allow features like recovery after a restart.
OAP queries formatted URI list from R3: OAP periodically queries R3 for the detected formatted URIs and saves the results in memory.
OAP formats URIs: When OAP receives new observability data, it matches the URIs against the formatted URIs retrieved from R3. If a match is found, the formatted URI is used for subsequent metric calculations.

Scenarios

In R3, the following scenarios are primarily addressed. For URIs identified as duplicates, R3 would replace the variable parts with {var} to standardize them.

ID Matching

A common practice in RESTFul APIs is to include various IDs in the URI paths, which leads to a large number of unique URI endpoints. For example, paths like the following will be aggregated by R3 into a standardized format: /api/users/{var}.

/api/users/cbf11b02ea464447b507e8852c32190a
/api/users/5e363a4a18b7464b8cbff1a7ee4c91ca
/api/users/44cf77fc351f4c6c9c4f1448f2f12800
/api/users/38d3be5f9bd44f7f98906ea049694511
/api/users/5ad14302e7924f4aa1d60e58d65b3dd2

Word Detection

In RESTFul URIs, operations on an entity are usually specified using HTTP methods, but often additional types are needed. This is addressed by including specific nouns in the path. To handle this, R3 implements word parsing: when R3 detects specific words in the path, it will not format that part. For example, URIs like the following would not be considered similar and therefore will not be merged:

/api/sale
/api/product_sale
/api/ProductSale

Low Sample

To prevent incorrect judgments due to insufficient sample sizes, R3 allows the configuration of a combine min URI count parameter in the configuration file. This parameter sets the minimum number of similar paths required before proceeding with the analysis.

Such as the threshold is 3, the following URI would keep the original URI, not parameterized.

/api/fetch1
/api/fetch2

But the following URI would be parametrized to /api/{var}, since the sample count is bigger than the threshold.

/api/fetch1
/api/fetch2
/api/fetch3

Version API

In real-world scenarios, we often encounter URIs with multiple versions. R3 addresses this by ensuring that if a specified path contains a v\\d+ parameter (indicating version information), that part would not be parameterized. For example, the following URIs will be separately parsed into /test/v1/{var} and /test/v999/{var}.

/test/v1/cbf11b02ea464447b507e8852c32190a
/test/v1/5e363a4a18b7464b8cbff1a7ee4c91ca
/test/v1/38d3be5f9bd44f7f98906ea049694511
/test/v999/1
/test/v999/2
/test/v999/3

Demo

Next, let’s quickly demonstrate how to use R3 to format observed endpoints, so you can understand more specifically what it accomplishes.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, R3 service and UI.

export FEATURE_FLAGS=java-agent-injector,single-node,elasticsearch,r3
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Trigger RESTFul Requests

In R3, a scheduled task is started by default to generate RESTFul traffic at regular intervals. However, you can also manually trigger this process using the following command:

kubectl exec -n sample-services $(kubectl get pod -n sample-services --selector=app=gateway -o jsonpath='{.items[0].metadata.name}') -- /bin/bash -c 'for i in $(seq 1 200); do curl http://rating/songs/$i/reviews/$((i+1)); sleep 1; done'

In the above command, R3 would automatically locate the gateway node and send requests in RESTFul format to the rating service within that node. This allows R3 to generate and test traffic patterns that simulate real RESTFul requests to the target service.

Check Formatted URIs

Once the RESTFul requests are triggered, you can view the aggregated endpoints in the UI.

Note: Since the formatted endpoints are generated asynchronously, some of the earlier requests may not yet be formatted. You may need to wait for some time before the UI shows only the formatted addresses.

Conclusion

In this article, we discussed in detail how SkyWalking utilizes the R3 service to format RESTFul URIs and aggregate related metrics upon receiving them. Currently, it applies to most RESTFul scenarios, and if more cases need to be supported, we can extend it further as needed.

Blog: Detailed explanation of SkyWalking Go Toolkit Trace

Wed, 18 Oct 2023 00:00:00 +0000

Background

SkyWalking Go is an open-source, non-intrusive Golang agent used for monitoring, tracing, and data collection within distributed systems. It enables users to observe the flow and latency of requests within the system, collect performance data from various system components for performance monitoring, and troubleshoot issues by tracing the complete path of requests.

In version v0.3.0, Skywalking Go introduced the toolkit trace tool. Trace APIs allow users to include critical operations, functions, or services in the tracing scope in situations where plugins do not support them. This inclusion enables tracking and monitoring of these operations and can be used for fault analysis, diagnosis, and performance monitoring.

Before diving into this, you can learn how to use the Skywalking Go agent by referring to the SkyWalking Go Agent Quick Start Guide.

The following sections will explain how to use these interfaces in specific scenarios.

Introducing the Trace Toolkit

Execute the following command in the project’s root directory:

go get github.com/apache/skywalking-go/toolkit

To use the toolkit trace interface, you need to import the package into your project:

"github.com/apache/skywalking-go/toolkit/trace"

Manual Tracing

A Span is the fundamental unit of an operation in Tracing. It represents an operation within a specific timeframe, such as a request, a function call, or a specific action. It records essential information about a particular operation, including start and end times, the operation’s name, tags (key-value pairs), and relationships between operations. Multiple Spans can form a hierarchical structure.

In situations where Skywalking-go doesn’t support a particular framework, users can manually create Spans to obtain tracing information.

(Here, I have removed the supported frameworks for the sake of the example. These are only examples. You should reference this when using the APIs in private and/or unsupported frameworks)

For example, when you need to trace an HTTP response, you can create a span using trace.CreateEntrySpan() within the method handling the request, and end the span using trace.StopSpan() after processing. When sending an HTTP request, use trace.CreateExitSpan() to create a span, and end the span after the request returns.

Here are two HTTP services named consumer and provider. When a user accesses the consumer service, it receives the user’s request internally and then accesses the provider to obtain resources.

// consumer.go
package main

import (
	"io"
	"net/http"

	_ "github.com/apache/skywalking-go"
	"github.com/apache/skywalking-go/toolkit/trace"
)

func getProvider() (*http.Response, error) {
	// Create an HTTP request
	req, err := http.NewRequest("GET", "http://localhost:9998/provider", http.NoBody)
	// Create an ExitSpan before sending the HTTP request.
	trace.CreateExitSpan("GET:/provider", "localhost:9999",
		func(headerKey, headerValue string) error {
			// Injector adds specific header information to the request.
			req.Header.Add(headerKey, headerValue)
			return nil
		})
	// Finish the ExitSpan and ensure it executes when the function returns using defer.
	defer trace.StopSpan()

	// Send the request.
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	return resp, nil
}

func consumerHandler(w http.ResponseWriter, r *http.Request) {
	// Create an EntrySpan to trace the execution of the consumerHandler method.
	trace.CreateEntrySpan(r.Method+"/consumer", func(headerKey string) (string, error) {
		// Extractor retrieves the header information added to the request.
		return r.Header.Get(headerKey), nil
	})
	// Finish the EntrySpan.
	defer trace.StopSpan()

	// Prepare to send an HTTP request.
	resp, err := getProvider()

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return
	}
	_, _ = w.Write(body)
}

func main() {
	http.HandleFunc("/consumer", consumerHandler)

	_ = http.ListenAndServe(":9999", nil)
}

// provider.go
package main

import (
	"net/http"

	_ "github.com/apache/skywalking-go"
	"github.com/apache/skywalking-go/toolkit/trace"
)

func providerHandler(w http.ResponseWriter, r *http.Request) {
	//Create an EntrySpan to trace the execution of the providerHandler method.
	trace.CreateEntrySpan("GET:/provider", func(headerKey string) (string, error) {
		return r.Header.Get(headerKey), nil
	})
	// Finish the EntrySpan.
	defer trace.StopSpan()

	_, _ = w.Write([]byte("success from provider"))
}

func main() {
	http.HandleFunc("/provider", providerHandler)

	_ = http.ListenAndServe(":9998", nil)
}

Then, in the terminal, execute:

go build -toolexec="/path/to/go-agent" -a -o consumer ./consumer.go
./consumer

go build -toolexec="/path/to/go-agent" -a -o provider ./provider.go
./provider

curl 127.0.0.1:9999/consumer

At this point, the UI will display the span information you created.

If you need to trace methods that are executed only locally, you can use trace.CreateLocalSpan(). If you don’t need to monitor information or states from the other end, you can change ExitSpan and EntrySpan to LocalSpan.

The usage examples provided are for illustration purposes, and users can decide the tracing granularity and where in the program they need tracing.

Please note that if a program ends too quickly, it may cause tracing data to be unable to be asynchronously sent to the SkyWalking backend.

Populate The Span

When there’s a necessity to record additional information, including creating/updating tags, appending logs, and setting a new operation name of the current traced Span, these APIs should be considered. These actions are used to enhance trace information, providing a more detailed and precise contextual description, which aids in better understanding the events or operations being traced.

Toolkit trace APIs provide a convenient way to access and manipulate trace data, including:

Setting Tags: SetTag()
Adding Logs: AddLog()
Setting Span Names: SetOperationName()
Getting various IDs: GetTraceID(), GetSegmentID(), GetSpanID()

For example, if you need to record the HTTP status code in a span, you can use the following interfaces while the span is not yet finished:

trace.CreateExitSpan("GET:/provider", "localhost:9999", func(headerKey, headerValue string) error {
	r.Header.Add(headerKey, headerValue)
	return nil
})
resp, err := http.Get("http://localhost:9999/provider")
trace.SetTag("status_code", fmt.Sprintf("%d", resp.StatusCode))
spanID := trace.GetSpanID()
trace.StopSpan()

It’s important to note that when making these method calls, the current thread should have an active span.

Async APIs

Async APIs work for manipulating spans across Goroutines. These scenarios might include:

Applications involving concurrency or multiple goroutines where operating on Spans across different execution contexts is necessary.
Updating or logging information for a Span during asynchronous operations.
Requiring a delayed completion of a Span.

To use it, follow these steps:

Obtain the return value of CreateSpan, which is SpanRef.
Call spanRef.PrepareAsync() to prepare for operations in another goroutine.
When the current goroutine’s work is done, call trace.StopSpan() to end the span (affecting only in the current goroutine).
Pass the spanRef to another goroutine.
After the work is done in any goroutine, call spanRef.AsyncFinish().

Here’s an example:

spanRef, err := trace.CreateLocalSpan("LocalSpan")
if err != nil {
	return
}
spanRef.PrepareAsync()
go func(){
	// some work
    spanRef.AsyncFinish()
}()
// some work
trace.StopSpan()

Correlation Context

Correlation Context is used to pass parameters within a Span, and the parent Span will pass the Correlation Context to all its child Spans. It allows the transmission of information between spans across different applications. The default number of elements in the Correlation Context is 3, and the content’s length cannot exceed 128 bytes.

Correlation Context is commonly applied in the following scenarios:

Passing Information Between Spans: It facilitates the transfer of critical information between different Spans, enabling upstream and downstream Spans to understand the correlation and context between each other.
Passing Business Parameters: In business scenarios, it involves transmitting specific parameters or information between different Spans, such as authentication tokens, business transaction IDs, and more.

Users can set the Correlation Context using trace.SetCorrelation(key, value) and then retrieve the corresponding value in downstream spans using value := trace.GetCorrelation(key).

For example, in the code below, we store the value in the tag of the span, making it easier to observe the result:

package main

import (
	_ "github.com/apache/skywalking-go"
	"github.com/apache/skywalking-go/toolkit/trace"
	"net/http"
)

func providerHandler(w http.ResponseWriter, r *http.Request) {
	ctxValue := trace.GetCorrelation("key")
	trace.SetTag("result", ctxValue)
}

func consumerHandler(w http.ResponseWriter, r *http.Request) {
	trace.SetCorrelation("key", "value")
	_, err := http.Get("http://localhost:9999/provider")
	if err != nil {
		return
	}
}

func main() {
	http.HandleFunc("/provider", providerHandler)

	http.HandleFunc("/consumer", consumerHandler)

	_ = http.ListenAndServe(":9999", nil)
}

Then, in the terminal, execute:

export SW_AGENT_NAME=server
go build -toolexec="/path/to/go-agent" -a -o server ./server.go
./server

curl 127.0.0.1:9999/consumer

Finally, in the providerHandler() span, you will find the information from the Correlation Context:

Conclusion

This article provides an overview of Skywalking Go’s Trace APIs and their practical application. These APIs empower users with the ability to customize tracing functionality according to their specific needs.

For detailed information about the interfaces, please refer to the documentation: Tracing APIs.

Welcome everyone to try out the new version.

Blog: Activating Automatical Performance Analysis -- Continuous Profiling

Sun, 25 Jun 2023 00:00:00 +0000

Background

In previous articles, We have discussed how to use SkyWalking and eBPF for performance problem detection within processes and networks. They are good methods to locate issues, but still there are some challenges:

The timing of the task initiation: It’s always challenging to address the processes that require performance monitoring when problems occur. Typically, manual engagement is required to identify processes and the types of performance analysis necessary, which cause extra time during the crash recovery. The root cause locating and the time of crash recovery conflict with each other from time to time. In the real case, rebooting would be the first choice of recovery, meanwhile, it destroys the site of crashing.
Resource consumption of tasks: The difficulties to determine the profiling scope. Wider profiling causes more resources than it should. We need a method to manage resource consumption and understand which processes necessitate performance analysis.
Engineer capabilities: On-call is usually covered by the whole team, which have junior and senior engineers, even senior engineers have their understanding limitation of the complex distributed system, it is nearly impossible to understand the whole system by a single one person.

The Continuous Profiling is a new created mechanism to resolve the above issues.

Automate Profiling

As profiling is resource costing and high experience required, how about introducing a method to narrow the scope and automate the profiling driven by polices creates by senior SRE engineer? So, in 9.5.0, SkyWalking first introduced preset policy rules for specific services to be monitored by the eBPF Agent in a low-energy manner, and run profiling when necessary automatically.

Policy

Policy rules specify how to monitor target processes and determine the type of profiling task to initiate when certain threshold conditions are met.

These policy rules primarily consist of the following configuration information:

Monitoring type: This specifies what kind of monitoring should be implemented on the target process.
Threshold determination: This defines how to determine whether the target process requires the initiation of a profiling task.
Trigger task: This specifies what kind of performance analysis task should be initiated.

Monitoring type

The type of monitoring is determined by observing the data values of a specified process to generate corresponding metrics. These metric values can then facilitate subsequent threshold judgment operations. In eBPF observation, we believe the following metrics can most directly reflect the current performance of the program:

Monitor Type	Unit	Description
System Load	Load	System load average over a specified period.
Process CPU	Percentage	The CPU usage of the process as a percentage.
Process Thread Count	Count	The number of threads in the process.
HTTP Error Rate	Percentage	The percentage of HTTP requests that result in error responses (e.g., 4xx or 5xx status codes).
HTTP Avg Response Time	Millisecond	The average response time for HTTP requests.

Monitoring network type metrics is not as simple as obtaining basic process information. It requires the initiation of eBPF programs and attaching them to the target process for observation. This is similar to the principles of network profiling task we introduced in the previous article, except that we no longer collect the full content of the data packets. Instead, we only collect the content of messages that match specified HTTP prefixes.

By using this method, we can significantly reduce the number of times the kernel sends data to the user space, and the user-space program can parse the data content with less system resource usage. This ultimately helps in conserving system resources.

Metrics collector

The eBPF agent would report metrics of processes periodically as follows to indicate the process performance in time.

Name	Unit	Description
process_cpu	(0-100)%	The CPU usage percent
process_thread_count	count	The thread count of process
system_load	count	The average system load for the last minute, each process have same value
http_error_rate	(0-100)%	The network request error rate percentage
http_avg_response_time	ms	The network average response duration

Threshold determination

For the threshold determination, the judgement is made by the eBPF Agent based on the target monitoring process in its own memory, rather than relying on calculations performed by the SkyWalking backend. The advantage of this approach is that it doesn’t have to wait for the results of complex backend computations, and it reduces potential issues brought about by complicated interactions.

By using this method, the eBPF Agent can swiftly initiate tasks immediately after conditions are met, without any delay.

It includes the following configuration items:

Threshold: Check if the monitoring value meets the specified expectations.
Period: The time period(seconds) for monitoring data, which can also be understood as the most recent duration.
Count: The number of times(seconds) the threshold is triggered within the detection period, which can also be understood as the total number of times the specified threshold rule is triggered in the most recent duration(seconds). Once the count check is met, the specified Profiling task will be started.

Trigger task

When the eBPF Agent detects that the threshold determination in the specified policy meets the rules, it can initiate the corresponding task according to pre-configured rules. For each different target performance task, their task initiation parameters are different:

On/Off CPU Profiling: It automatically performs performance analysis on processes that meet the conditions, defaulting to 10 minutes of monitoring.
Network Profiling: It performs network performance analysis on all processes in the same Service Instance on the current machine, to prevent the cause of the issue from being unrealizable due to too few process being collected, defaulting to 10 minutes of monitoring.

Once the task is initiated, no new profiling tasks would be started for the current process for a certain period. The main reason for this is to prevent frequent task creation due to low threshold settings, which could affect program execution. The default time period is 20 minutes.

Data Flow

The figure 1 illustrates the data flow of the continuous profiling feature:

Figure 1: Data Flow of Continuous Profiling

eBPF Agent with Process

Firstly, we need to ensure that the eBPF Agent and the process to be monitored are deployed on the same host machine, so that we can collect relevant data from the process. When the eBPF Agent detects a threshold validation rule that conforms to the policy, it immediately triggers the profiling task for the target process, thereby reducing any intermediate steps and accelerating the ability to pinpoint performance issues.

Sliding window

The sliding window plays a crucial role in the eBPF Agent’s threshold determination process, as illustrated in the figure 2:

Figure 2: Sliding Window in eBPF Agent

Each element in the array represents the data value for a specified second in time. When the sliding window needs to verify whether it is responsible for a rule, it fetches the content of each element from a certain number of recent elements (period parameter). If an element exceeds the threshold, it is marked in red and counted. If the number of red elements exceeds a certain number, it is deemed to trigger a task.

Using a sliding window offers the following two advantages:

Fast retrieval of recent content: With a sliding window, complex calculations are unnecessary. You can know the data by simply reading a certain number of recent array elements.
Solving data spikes issues: Validation through count prevents situations where a data point suddenly spikes and then quickly returns to normal. Verification with multiple values can reveal whether exceeding the threshold is frequent or occasional.

eBPF Agent with SkyWalking Backend

The eBPF Agent communicates periodically with the SkyWalking backend, involving three most crucial operations:

Policy synchronization: Through periodic policy synchronization, the eBPF Agent can keep processes on the local machine updated with the latest policy rules as much as possible.
Metrics sending: For processes that are already being monitored, the eBPF Agent periodically sends the collected data to the backend program. This facilitates real-time query of current data values by users, who can also compare this data with historical values or thresholds when problems arise.
Profiling task reporting: When the eBPF detects that a certain process has triggered a policy rule, it automatically initiates a performance task, collects relevant information from the current process, and reports it to the SkyWalking backend. This allows users to know when, why, and what type of profiling task was triggered from the interface.

Demo

Next, let’s quickly demonstrate the continuous profiling feature, so you can understand more specifically what it accomplishes.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.5.0
export SW_UI_IMAGE=apache/skywalking-ui:9.5.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.5.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Create Continuous Profiling Policy

Currently, continues profiling feature is set by default in the Service Mesh panel at the Service level.

Figure 3: Continuous Policy Tab

By clicking on the edit button aside from the Policy List, the polices of current service could be created or updated.

Figure 4: Edit Continuous Profiling Policy

Multiple polices are supported. Every policy has the following configurations.

Target Type: Specifies the type of profiling task to be triggered when the threshold determination is met.
Items: For profiling task of the same target, one or more validation items can be specified. As long as one validation item meets the threshold determination, the corresponding performance analysis task will be launched.
1. Monitor Type: Specifies the type of monitoring to be carried out for the target process.
2. Threshold: Depending on the type of monitoring, you need to fill in the corresponding threshold to complete the verification work.
3. Period: Specifies the number of recent seconds of data you want to monitor.
4. Count: Determines the total number of seconds triggered within the recent period.
5. URI Regex/List: This is applicable to HTTP monitoring types, allowing URL filtering.

Done

After clicking the save button, you can see the currently created monitoring rules, as shown in the figure 5:

Figure 5: Continuous Profiling Monitoring Processes

The data can be divided into the following parts:

Policy list: On the left, you can see the rule list you have created.
Monitoring Summary List: Once a rule is selected, you can see which pods and processes would be monitored by this rule. It also summarizes how many profiling tasks have been triggered in the last 48 hours by the current pod or process, as well as the last trigger time. This list is also sorted in descending order by the number of triggers to facilitate your quick review.

When you click on a specific process, a new dashboard would show to list metrics and triggered profiling results.

Figure 6: Continuous Profiling Triggered Tasks

The current figure contains the following data contents:

Task Timeline: It lists all profiling tasks in the past 48 hours. And when the mouse hovers over a task, it would also display detailed information:
1. Task start and end time: It indicates when the current performance analysis task was triggered.
2. Trigger reason: It would display the reason why the current process was profiled and list out the value of the metric exceeding the threshold when the profiling was triggered. so you can quickly understand the reason.
Task Detail: Similar to the CPU Profiling and Network Profiling introduced in previous articles, this would display the flame graph or process topology map of the current task, depending on the profiling type.

Meanwhile, on the Metrics tab, metrics relative to profiling policies are collected to retrieve the historical trend, in order to provide a comprehensive explanation of the trigger point about the profiling.

Figure 7: Continuous Profiling Metrics

Conclusion

In this article, I have detailed how the continuous profiling feature in SkyWalking and eBPF works. In general, it involves deploying the eBPF Agent service on the same machine where the process to be monitored resides, and monitoring the target process with low resource consumption. When it meets the threshold conditions, it would initiate more complex CPU Profiling and Network Profiling tasks.

In the future, we will offer even more features. Stay tuned!

Twitter, ASFSkyWalking
Slack. Send Request to join SkyWalking slack mail to the mail list(dev@skywalking.apache.org), we will invite you in.
Subscribe to our medium list.

Zh: 自动化性能分析——持续剖析

Sun, 25 Jun 2023 00:00:00 +0000

背景

在之前的文章中，我们讨论了如何使用 SkyWalking 和 eBPF 来检测性能问题，包括进程和网络。这些方法可以很好地定位问题，但仍然存在一些挑战：

任务启动的时间: 当需要进行性能监控时，解决需要性能监控的进程始终是一个挑战。通常需要手动参与，以标识进程和所需的性能分析类型，这会在崩溃恢复期间耗费额外的时间。根本原因定位和崩溃恢复时间有时会发生冲突。在实际情况中，重新启动可能是恢复的第一选择，同时也会破坏崩溃的现场。
任务的资源消耗: 确定分析范围的困难。过宽的分析范围会导致需要更多的资源。我们需要一种方法来管理资源消耗并了解哪些进程需要性能分析。
工程师能力: 通常由整个团队负责呼叫，其中有初级和高级工程师，即使是高级工程师也对复杂的分布式系统有其理解限制，单个人几乎无法理解整个系统。

持续剖析（Continuous Profiling） 是解决上述问题的新机制。

自动剖析

由于性能分析的资源消耗和高经验要求，因此引入一种方法以缩小范围并由高级 SRE 工程师创建策略自动剖析。因此，在 9.5.0 中，SkyWalking 首先引入了预设策略规则，以低功耗方式监视特定服务的 eBPF 代理，并在必要时自动运行剖析。

策略

策略规则指定了如何监视目标进程并确定在满足某些阈值条件时应启动何种类型的分析任务。

这些策略规则主要包括以下配置信息：

监测类型: 这指定了应在目标进程上实施什么样的监测。
阈值确定: 这定义了如何确定目标进程是否需要启动分析任务。
触发任务: 这指定了应启动什么类型的性能分析任务。

监测类型

监测类型是通过观察指定进程的数据值来生成相应的指标来确定的。这些指标值可以促进后续的阈值判断操作。在 eBPF 观测中，我们认为以下指标最能直接反映程序的当前性能：

监测类型	单位	描述
系统负载	负载	在指定时间段内的系统负载平均值。
进程 CPU	百分比	进程的 CPU 使用率百分比。
进程线程计数	计数	进程中的线程数。
HTTP 错误率	百分比	导致错误响应（例如，4xx 或 5xx 状态代码）的 HTTP 请求的百分比。
HTTP 平均响应时间	毫秒	HTTP 请求的平均响应时间。

指标收集器

eBPF 代理会定期报告以下进程度量，以指示进程性能：

名称	单位	描述
process_cpu	(0-100)%	CPU 使用率百分比
process_thread_count	计数	进程中的线程数
system_load	计数	最近一分钟的平均系统负载，每个进程的值相同
http_error_rate	(0-100)%	网络请求错误率百分比
http_avg_response_time	毫秒	网络平均响应持续时间

阈值确定

对于阈值的确定，eBPF 代理是基于其自身内存中的目标监测进程进行判断，而不是依赖于 SkyWalking 后端执行的计算。这种方法的优点在于，它不必等待复杂后端计算的结果，减少了复杂交互所带来的潜在问题。

通过使用此方法，eBPF 代理可以在条件满足后立即启动任务，而无需任何延迟。

它包括以下配置项：

阈值: 检查监测值是否符合指定的期望值。
周期: 监控数据的时间周期（秒），也可以理解为最近的持续时间。
计数: 检测期间触发阈值的次数（秒），也可以理解为最近持续时间内指定阈值规则触发的总次数（秒）。一旦满足计数检查，指定的分析任务将被开始。

触发任务

当 eBPF Agent 检测到指定策略中的阈值决策符合规则时，根据预配置的规则可以启动相应的任务。对于每个不同的目标性能任务，它们的任务启动参数都不同：

On/Off CPU Profiling: 它会自动对符合条件的进程进行性能分析，缺省情况下监控时间为 10 分钟。
Network Profiling: 它会对当前机器上同一 Service Instance 中的所有进程进行网络性能分析，以防问题的原因因被收集进程太少而无法实现，缺省情况下监控时间为 10 分钟。

一旦任务启动，当前进程将在一定时间内不会启动新的剖析任务。主要原因是为了防止因低阈值设置而频繁创建任务，从而影响程序执行。缺省时间为 20 分钟。

数据流

图 1 展示了持续剖析功能的数据流：

图 1: 持续剖析的数据流

eBPF Agent进行进程跟踪

首先，我们需要确保 eBPF Agent 和要监测的进程部署在同一台主机上，以便我们可以从进程中收集相关数据。当 eBPF Agent 检测到符合策略的阈值验证规则时，它会立即为目标进程触发剖析任务，从而减少任何中间步骤并加速定位性能问题的能力。

滑动窗口

滑动窗口在 eBPF Agent 的阈值决策过程中发挥着至关重要的作用，如图 2 所示：

图 2: eBPF Agent 中的滑动窗口

数组中的每个元素表示指定时间内的数据值。当滑动窗口需要验证是否负责某个规则时，它从最近的一定数量的元素 (period 参数) 中获取每个元素的内容。如果一个元素超过了阈值，则标记为红色并计数。如果红色元素的数量超过一定数量，则被认为触发了任务。

使用滑动窗口具有以下两个优点：

快速检索最近的内容：使用滑动窗口，无需进行复杂的计算。你可以通过简单地读取一定数量的最近数组元素来了解数据。
解决数据峰值问题：通过计数进行验证，可以避免数据点突然增加然后快速返回正常的情况。使用多个值进行验证可以揭示超过阈值是频繁还是偶然发生的。

eBPF Agent与OAP后端通讯

eBPF Agent 定期与 SkyWalking 后端通信，涉及三个最关键的操作：

策略同步：通过定期的策略同步，eBPF Agent 可以尽可能地让本地机器上的进程与最新的策略规则保持同步。
指标发送：对于已经被监视的进程，eBPF Agent 定期将收集到的数据发送到后端程序。这就使用户能够实时查询当前数据值，用户也可以在出现问题时将此数据与历史值或阈值进行比较。
剖析任务报告：当 eBPF 检测到某个进程触发了策略规则时，它会自动启动性能任务，从当前进程收集相关信息，并将其报告给 SkyWalking 后端。这使用户可以从界面了解何时、为什么和触发了什么类型的剖析任务。

演示

接下来，让我们快速演示持续剖析功能，以便你更具体地了解它的功能。

部署 SkyWalking Showcase

SkyWalking Showcase 包含完整的示例服务，并可以使用 SkyWalking 进行监视。有关详细信息，请查看官方文档。

在此演示中，我们只部署服务、最新发布的 SkyWalking OAP 和 UI。

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.5.0
export SW_UI_IMAGE=apache/skywalking-ui:9.5.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.5.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

部署完成后，请运行以下脚本以打开 SkyWalking UI：http://localhost:8080/。

kubectl port-forward svc/ui 8080:8080 --namespace default

创建持续剖析策略

目前，持续剖析功能在 Service Mesh 面板的 Service 级别中默认设置。

图 3: 持续策略选项卡

通过点击 Policy List 旁边的编辑按钮，可以创建或更新当前服务的策略。

图 4: 编辑持续剖析策略

支持多个策略。每个策略都有以下配置。

Target Type：指定符合阈值决策时要触发的剖析任务的类型。
Items：对于相同目标的剖析任务，可以指定一个或多个验证项目。只要一个验证项目符合阈值决策，就会启动相应的性能分析任务。
1. Monitor Type：指定要为目标进程执行的监视类型。
2. Threshold：根据监视类型的不同，需要填写相应的阈值才能完成验证工作。
3. Period：指定你要监测的最近几秒钟的数据数量。
4. Count：确定最近时间段内触发的总秒数。
5. URI 正则表达式/列表：这适用于 HTTP 监控类型，允许 URL 过滤。

完成

单击保存按钮后，你可以看到当前已创建的监控规则，如图 5 所示：

图 5: 持续剖析监控进程

数据可以分为以下几个部分：

策略列表：在左侧，你可以看到已创建的规则列表。
监测摘要列表：选择规则后，你可以看到哪些 pod 和进程将受到该规则的监视。它还总结了当前 pod 或进程在过去 48 小时内触发的性能分析任务数量，以及最后一个触发时间。该列表还按触发次数降序排列，以便你快速查看。

当你单击特定进程时，将显示一个新的仪表板以列出指标和触发的剖析结果。

图 6: 持续剖析触发的任务

当前图包含以下数据内容：

任务时间轴：它列出了过去 48 小时的所有剖析任务。当鼠标悬停在任务上时，它还会显示详细信息：
1. 任务的开始和结束时间：它指示当前性能分析任务何时被触发。
2. 触发原因：它会显示为什么会对当前进程进行剖析，并列出当剖析被触发时超过阈值的度量值，以便你快速了解原因。
任务详情：与前几篇文章介绍的 CPU 剖析和网络剖析类似，它会显示当前任务的火焰图或进程拓扑图，具体取决于剖析类型。

同时，在 Metrics 选项卡中，收集与剖析策略相关的指标以检索历史趋势，以便在剖析的触发点提供全面的解释。

图 7: 持续剖析指标

结论

在本文中，我详细介绍了 SkyWalking 和 eBPF 中持续剖析功能的工作原理。通常情况下，它涉及将 eBPF Agent 服务部署在要监视的进程所在的同一台计算机上，并以低资源消耗监测目标进程。当它符合阈值条件时，它会启动更复杂的 CPU 剖析和网络剖析任务。

在未来，我们将提供更多功能。敬请期待！

Twitter：ASFSkyWalking
Slack：向邮件列表 (dev@skywalking.apache.org) 发送“Request to join SkyWalking Slack”，我们会邀请你加入。
订阅我们的 Medium 列表。

Zh: 使用SkyWalking go agent快速实现Dubbo Go监控

Mon, 05 Jun 2023 00:00:00 +0000

本文演示如何将 Dubbo-Go 应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。

以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（ Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。

在本文中，我们将指导你快速将 skywalking-go 项目集成到 dubbo-go 项目中。

演示包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
使用 SkyWalking Go 编译程序：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序。然后执行以下两个命令来启动服务:

# 启动 OAP 后端
> bin/oapService.sh
# 启动 UI
> bin/webappService.sh

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

使用 SkyWalking GO 编译 Dubbo Go 程序

这里将演示如何将 Dubbo-go 程序与SkyWalking Go Agent集成。请依次执行如下命令来创建一个新的项目:

# 安装dubbo-go基础环境
> export GOPROXY="https://goproxy.cn"
> go install github.com/dubbogo/dubbogo-cli@latest
> dubbogo-cli install all
# 创建demo项目
> mkdir demo && cd demo
> dubbogo-cli newDemo .
# 升级dubbo-go依赖到最新版本
> go get -u dubbo.apache.org/dubbo-go/v3

在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：

go get github.com/apache/skywalking-go

接下来，请分别在服务端和客户端的main包中引入。包含之后，代码将会更新为：

// go-server/cmd/server.go
package main

import (
	"context"
)

import (
	"dubbo.apache.org/dubbo-go/v3/common/logger"
	"dubbo.apache.org/dubbo-go/v3/config"
	_ "dubbo.apache.org/dubbo-go/v3/imports"

	"helloworld/api"
	
	// 引入skywalking-go
	_ "github.com/apache/skywalking-go"
)

type GreeterProvider struct {
	api.UnimplementedGreeterServer
}

func (s *GreeterProvider) SayHello(ctx context.Context, in *api.HelloRequest) (*api.User, error) {
	logger.Infof("Dubbo3 GreeterProvider get user name = %s\n", in.Name)
	return &api.User{Name: "Hello " + in.Name, Id: "12345", Age: 21}, nil
}

// export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-server/conf/dubbogo.yaml
func main() {
	config.SetProviderService(&GreeterProvider{})
	if err := config.Load(); err != nil {
		panic(err)
	}
	select {}
}

在客户端代码中除了需要引入skywalking-go之外，还需要在main方法中的最后一行增加主携程等待语句，以防止因为客户端快速关闭而无法将Tracing数据异步发送到SkyWalking后端：

package main

import (
	"context"
)

import (
	"dubbo.apache.org/dubbo-go/v3/common/logger"
	"dubbo.apache.org/dubbo-go/v3/config"
	_ "dubbo.apache.org/dubbo-go/v3/imports"

	"helloworld/api"

	// 引入skywalking-go
	_ "github.com/apache/skywalking-go"
)

var grpcGreeterImpl = new(api.GreeterClientImpl)

// export DUBBO_GO_CONFIG_PATH= PATH_TO_SAMPLES/helloworld/go-client/conf/dubbogo.yaml
func main() {
	config.SetConsumerService(grpcGreeterImpl)
	if err := config.Load(); err != nil {
		panic(err)
	}

	logger.Info("start to test dubbo")
	req := &api.HelloRequest{
		Name: "laurence",
	}
	reply, err := grpcGreeterImpl.SayHello(context.Background(), req)
	if err != nil {
		logger.Error(err)
	}
	logger.Infof("client response result: %v\n", reply)
	// 增加主携程等待语句
	select {}
}

接下来，请从官方 SkyWalking 网站下载 Go Agent 程序。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec="/path/to/go-agent -a 参数。例如，请使用以下命令：

# 进入项目主目录
> cd demo
# 分别编译服务端和客户端
# -toolexec 参数定义为go-agent的路径
# -a 参数用于强制重新编译所有依赖项
> cd go-server && go build -toolexec="/path/to/go-agent" -a -o go-server cmd/server.go && cd ..
> cd go-client && go build -toolexec="/path/to/go-agent" -a -o go-client cmd/client.go && cd ..

应用部署

在这里，我们分别启动两个终端窗口来分别启动服务端和客户端。

在服务端，将服务的名称更改为dubbo-server：

# 导出dubbo-go服务端配置文件路径
export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-server/conf/dubbogo.yaml
# 导出skywalking-go的服务名称
export SW_AGENT_NAME=dubbo-server

./go-server/go-server

在客户端，将服务的名称更改为dubbo-client：

# 导出dubbo-go客户端配置文件路径
export DUBBO_GO_CONFIG_PATH=/path/to/demo/go-client/conf/dubbogo.yaml
# 导出skywalking-go的服务名称
export SW_AGENT_NAME=dubbo-client

./go-client/go-client

在 SkyWalking UI 上可视化

现在，由于客户端会自动像服务器端发送请求，现在就可以在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 dubbo-server 和 dubbo-client 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

并可以在拓扑图页面中看到服务之间的关系。

总结

在本文中，我们指导你快速开发dubbo-go服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的任意 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档。

将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！

Blog: Quick start with SkyWalking Go Agent

Thu, 01 Jun 2023 00:00:00 +0000

Background

Previously, if you wanted to monitor a Golang application in SkyWalking, you would integrate your project with the go2sky project and manually write various frameworks with go2sky plugins. Now, we have a brand-new project (Skywalking Go) that allows you to integrate your Golang projects into SkyWalking with almost zero coding, while offering greater flexibility and scalability.

In this article, we will guide you quickly integrating the skywalking-go project into your Golang project.

Quick start

This demonstration will consist of the following steps:

Deploy SkyWalking: This involves setting up the SkyWalking backend and UI programs, enabling you to see the final effect.
Compile Golang with SkyWalking Go: Here, you’ll compile the SkyWalking Go Agent into the Golang program you wish to monitor.
Application Deployment: You’ll export environment variables and deploy the application to facilitate communication between your service and the SkyWalking backend.
Visualization on SkyWalking UI: Finally, you’ll send requests and observe the effects within the SkyWalking UI.

Deploy SkyWalking

Please download the SkyWalking APM program from the official SkyWalking website. Then execute the following two commands to start the service:

# startup the OAP backend
> bin/oapService.sh
# startup the UI
> bin/webappService.sh

Next, you can access the address at http://localhost:8080/. At this point, as no applications have been deployed yet, you will not see any data.

Compile Golang with SkyWalking GO

Here is a simple business application here that starts an HTTP service.

package main

import "net/http"

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

Execute the following command in the project’s root directory. This command will download the dependencies required for skywalking-go:

go get github.com/apache/skywalking-go

Also, include it in the main package of the project. After the inclusion, the code will update to:

package main

import (
	"net/http"

    // This is an important step. DON'T MISS IT.
	_ "github.com/apache/skywalking-go"
)

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

Next, please download the Go Agent program from the official SkyWalking website. When you compile with the go build command, find the agent program that matches your current operating system in the bin directory, and add the -toolexec="/path/to/go-agent -a parameter. For example, use the following command:

# Build application with SkyWalking go agent
# -toolexec parameter define the path of go-agent
# -a parameter is used to force rebuild all packages
> go build -toolexec="/path/to/go-agent" -a -o test .

Application Deployment

Before you start to deploy the application, you can change the service name of the current application in SkyWalking through environment variables. You can also change its configuration such as the address with the server-side. For specific details, please refer to the documentation.

Here, we’re just changing the name of the current service to demo.

# Change the service name
> export SW_AGENT_NAME=demo

Next, you can start the application:

# Start the application
> ./test

Visualization on SkyWalking UI

Now, you can send a request to the application and observe the results in the SkyWalking UI.

# Send a request
> curl http://localhost:8000/hello

After a few seconds, you can revisit the SkyWalking UI at http://localhost:8080. You will be able to see the demo service you deployed on the homepage.

Moreover, on the Trace page, you can see the request you just sent.

Conclusion

In this article, we’ve guided you to quickly develop a demo service and integrate it with SkyWalking Go Agent. This process is also applicable to your own Golang services. Ultimately, you can view the display effect in the SkyWalking service. If you’re interested in learning which frameworks the SkyWalking Go agent currently supports, please refer to this documentation.

In the future, we will continue to expand the functionality of SkyWalking Go, adding more plugin support. So, stay tuned!

Zh: SkyWalking Go Agent 快速开始指南

Thu, 01 Jun 2023 00:00:00 +0000

本文演示如何将应用程序与 SkyWalking Go 集成，并在 SkyWalking UI 中查看结果。

以前，如果你想要在 SkyWalking 中监控 Golang 应用程序，需要将项目与 go2sky 项目集成，并手动编写各种带有 go2sky 插件的框架。现在，我们有一个全新的项目（Skywalking Go ），允许你将 Golang 项目集成到 SkyWalking 中，几乎不需要编码，同时提供更大的灵活性和可扩展性。

在本文中，我们将指导你快速将 skywalking-go 项目集成到 Golang 项目中。

演示包括以下步骤：

部署 SkyWalking：这涉及设置 SkyWalking 后端和 UI 程序，使你能够看到最终效果。
使用 SkyWalking Go 编译 Golang：在这里，你将把 SkyWalking Go Agent 编译到要监控的 Golang 程序中。
应用部署：你将导出环境变量并部署应用程序，以促进你的服务与 SkyWalking 后端之间的通信。
在 SkyWalking UI 上可视化：最后，你将发送请求并在 SkyWalking UI 中观察效果。

部署 SkyWalking

请从官方 SkyWalking 网站下载 SkyWalking APM 程序。然后执行以下两个命令来启动服务:

# 启动 OAP 后端
> bin/oapService.sh
# 启动 UI
> bin/webappService.sh

接下来，你可以访问地址 http://localhost:8080/ 。此时，由于尚未部署任何应用程序，因此你将看不到任何数据。

使用 SkyWalking GO 编译 Golang

这里有一个简单的业务应用程序，启动了一个 HTTP 服务。

package main

import "net/http"

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

在项目的根目录中执行以下命令。此命令将下载 skywalking-go 所需的依赖项：

go get github.com/apache/skywalking-go

接下来，请将其包含在项目的 main 包中。包含之后，代码将会更新为：

package main

import (
	"net/http"
	_ "github.com/apache/skywalking-go"
)

func main() {
	http.HandleFunc("/hello", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello World"))
	})
	err := http.ListenAndServe(":8000", nil)
	if err != nil {
		panic(err)
	}
}

接下来，请从官方 SkyWalking 网站下载 Go Agent 程序。当你使用 go build 命令进行编译时，请在 bin 目录中找到与当前操作系统匹配的代理程序，并添加 -toolexec="/path/to/go-agent" -a 参数。例如，请使用以下命令：

go build -toolexec="/path/to/go-agent" -a -o test .

应用部署

在这里，我们只是将当前服务的名称更改为 demo。

接下来，你可以启动应用程序：

export SW_AGENT_NAME=demo

./test

在 SkyWalking UI 上可视化

现在，向应用程序发送请求并在 SkyWalking UI 中观察结果。

几秒钟后，重新访问 http://localhost:8080 的 SkyWalking UI。能够在主页上看到部署的 demo 服务。

此外，在追踪页面上，可以看到刚刚发送的请求。

总结

在本文中，我们指导你快速开发 demo 服务，并将其与 SkyWalking Go Agent 集成。这个过程也适用于你自己的 Golang 服务。最终，可以在 SkyWalking 服务中查看显示效果。如果你有兴趣了解 SkyWalking Go 代理当前支持的框架，请参阅此文档。

将来，我们将继续扩展 SkyWalking Go 的功能，添加更多插件支持。所以，请继续关注！

Blog: eBPF enhanced HTTP observability - L7 metrics and tracing

Thu, 12 Jan 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source Application Performance Management system that helps users collect and aggregate logs, traces, metrics, and events for display on a UI. In the previous article, we introduced how to use Apache SkyWalking Rover to analyze the network performance issue in the service mesh environment. However, in business scenarios, users often rely on mature layer 7 protocols, such as HTTP, for interactions between systems. In this article, we will discuss how to use eBPF techniques to analyze performance bottlenecks of layer 7 protocols and how to enhance the tracing system using network sampling.

This article will show how to use Apache SkyWalking with eBPF to enhance metrics and traces in HTTP observability.

HTTP Protocol Analysis

HTTP is one of the most common Layer 7 protocols and is usually used to provide services to external parties and for inter-system communication. In the following sections, we will show how to identify and analyze HTTP/1.x protocols.

Protocol Identification

In HTTP/1.x, the client and server communicate through a single file descriptor (FD) on each side. Figure 1 shows the process of communication involving the following steps:

Connect/accept: The client establishes a connection with the HTTP server, or the server accepts a connection from the client.
Read/write (multiple times): The client or server reads and writes HTTPS requests and responses. A single request-response pair occurs within the same connection on each side.
Close: The client and server close the connection.

To obtain HTTP content, it’s necessary to read it from the second step of this process. As defined in the RFC, the content is contained within the data of the Layer 4 protocol and can be obtained by parsing the data. The request and response pair can be correlated because they both occur within the same connection on each side.

Figure 1: HTTP communication timeline.

HTTP Pipeline

HTTP pipelining is a feature of HTTP/1.1 that enables multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. This feature is important because it ensures that the order of the responses on the server side matches the order of the requests.

Figure 2 illustrates how this works. Consider the following scenario: an HTTP client sends multiple requests to a server, and the server responds by sending the HTTP responses in the same order as the requests. This means that the first request sent by the client will receive the first response from the server, the second request will receive the second response, and so on.

When designing HTTP parsing, we should follow this principle by adding request data to a list and removing the first item when parsing a response. This ensures that the responses are processed in the correct order.

Figure 2: HTTP/1.1 pipeline.

Metrics

Based on the identification of the HTTP content and process topology diagram mentioned in the previous article, we can combine these two to generate process-to-process metrics data.

Figure 3 shows the metrics that currently support the analysis between the two processes. Based on the HTTP request and response data, we can analyze the following data:

Metrics Name	Type	Unit	Description
Request CPM(Call Per Minute)	Counter	count	The HTTP request count
Response Status CPM(Call Per Minute)	Counter	count	The count of per HTTP response status code
Request Package Size	Counter/Histogram	Byte	The request package size
Response Package Size	Counter/Histogram	Byte	The response package size
Client Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the client side
Server Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the server side

Figure 3: Process-to-process metrics.

HTTP and Trace

During the HTTP process, if we unpack the HTTP requests and responses from raw data, we can use this data to correlate with the existing tracing system.

Trace Context Identification

In order to track the flow of requests between multiple services, the trace system usually creates a trace context when a request enters a service and passes it along to other services during the request-response process. For example, when an HTTP request is sent to another server, the trace context is included in the request header.

Figure 4 displays the raw content of an HTTP request intercepted by Wireshark. The trace context information generated by the Zipkin Tracing system can be identified by the “X-B3” prefix in the header. By using eBPF to intercept the trace context in the HTTP header, we can connect the current request with the trace system.

Figure 4: View of HTTP headers in Wireshark.

Trace Event

We have added the concept of an event to traces. An event can be attached to a span and consists of start and end times, tags, and summaries, allowing us to attach any desired information to the Trace.

When performing eBPF network profiling, two events can be generated based on the request-response data. Figure 5 illustrates what happens when a service performs an HTTP request with profiling. The trace system generates trace context information and sends it in the request. When the service executes in the kernel, we can generate an event for the corresponding trace span by interacting with the request-response data and execution time in the kernel space.

Previously, we could only observe the execution status in the user space. However, by combining traces and eBPF technologies, we can now also get more information about the current trace in the kernel space, which would impact less performance for the target service if we do similar things in the tracing SDK and agent.

Figure 5: Logical view of profiling an HTTP request and response.

Sampling

To ensure efficient data storage and minimize unnecessary data sampling, we use a sampling mechanism for traces in our system. This mechanism triggers sampling only when certain conditions are met. We also provide a list of the top N traces, which allows users to quickly access the relevant request information for a specific trace.

To help users easily identify and analyze relevant events, we offer three different sampling rules:

Slow Traces: Sampling is triggered when the response time for a request exceeds a specified threshold.
Response Status [400, 500): Sampling is triggered when the response status code is greater than or equal to 400 and less than 500.
Response Status [500, 600): Sampling is triggered when the response status code is greater than or equal to 500 and less than 600.

In addition, we recognize that not all request or response raw data may be necessary for analysis. For example, users may be more interested in requesting data when trying to identify performance issues, while they may be more interested in response data when troubleshooting errors. As such, we also provide configuration options for request or response events to allow users to specify which type of data they would like to sample.

Profiling in a Service Mesh

The SkyWalking and SkyWalking Rover projects have already implemented the HTTP protocol analyze and trace associations. How do they perform when running in a service mesh environment?

Deployment

Figure 6 demonstrates the deployment of SkyWalking and SkyWalking Rover in a service mesh environment. SkyWalking Rover is deployed as a DaemonSet on each machine where a service is located and communicates with the SkyWalking backend cluster. It automatically recognizes the services on the machine and reports metadata information to the SkyWalking backend cluster. When a new network profiling task arises, SkyWalking Rover senses the task and analyzes the designated processes, collecting and aggregating network data before ultimately reporting it back to the SkyWalking backend service.

Figure 6: SkyWalking rover deployment topology in a service mesh.

Tracing Systems

Starting from version 9.3.0, the SkyWalking backend fully supports all functions in the Zipkin server. Therefore, the SkyWalking backend can collect traces from both the SkyWalking and Zipkin protocols. Similarly, SkyWalking Rover can identify and analyze trace context in both the SkyWalking and Zipkin trace systems. In the following two sections, network analysis results will be displayed in the SkyWalking and Zipkin UI respectively.

SkyWalking

When SkyWalking performs network profiling, similar to the TCP metrics in the previous article, the SkyWalking UI will first display the topology between processes. When you open the dashboard of the line representing the traffic metrics between processes, you can see the metrics of HTTP traffic from the “HTTP/1.x” tab and the sampled HTTP requests with tracing in the “HTTP Requests” tab.

As shown in Figure 7, there are three lists in the tab, each corresponding to a condition in the event sampling rules. Each list displays the traces that meet the pre-specified conditions. When you click on an item in the trace list, you can view the complete trace.

Figure 7: Sampled HTTP requests within tracing context.

When you click on an item in the trace list, you can quickly view the specified trace. In Figure 8, we can see that in the current service-related span, there is a tag with a number indicating how many HTTP events are related to that trace span.

Since we are in a service mesh environment, each service involves interacting with Envoy. Therefore, the current span includes Envoy’s request and response information. Additionally, since the current service has both incoming and outgoing requests, there are events in the corresponding span.

Figure 8: Events in the trace detail.

When the span is clicked, the details of the span will be displayed. If there are events in the current span, the relevant event information will be displayed on a time axis. As shown in Figure 9, there are a total of 6 related events in the current Span. Each event represents a data sample of an HTTP request/response. One of the events spans multiple time ranges, indicating a longer system call time. It may be due to a blocked system call, depending on the implementation details of the HTTP request in different languages. This can also help us query the possible causes of errors.

Figure 9: Events in one trace span.

Finally, we can click on a specific event to see its complete information. As shown in Figure 10, it displays the sampling information of a request, including the SkyWalking trace context protocol contained in the request header from the HTTP raw data. The raw request data allows you to quickly re-request the request to solve any issues.

Figure 10: The detail of the event.

Zipkin

Zipkin is one of the most widely used distributed tracing systems in the world. SkyWalking can function as an alternative server to provide advanced features for Zipkin users. Here, we use this way to bring the feature into the Zipkin ecosystem out-of-box. The new events would also be treated as a kind of Zipkin’s tags and annotations.

To add events to a Zipkin span, we need to do the following:

Split the start and end times of each event into two annotations with a canonical name.
Add the sampled HTTP raw data from the event to the Zipkin span tags, using the same event name for corresponding purposes.

Figures 11 and 12 show annotations and tags in the same span. In these figures, we can see that the span includes at least two events with the same event name and sequence suffix (e.g., “Start/Finished HTTP Request/Response Sampling-x” in the figure). Both events have separate timestamps to represent their relative times within the span. In the tags, the data content of the corresponding event is represented by the event name and sequence number, respectively.

Figure 11: Event timestamp in the Zipkin span annotation.

Figure 12: Event raw data in the Zipkin span tag.

Demo

In this section, we demonstrate how to perform network profiling in a service mesh and complete metrics collection and HTTP raw data sampling. To follow along, you will need a running Kubernetes environment.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0
export SW_UI_IMAGE=apache/skywalking-ui:9.3.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Start Network Profiling Task

Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.

In figure 13, we have selected an instance with a list of tasks in the network profiling tab.

Figure 13: Network Profiling tab in the Data Plane.

When we click the Start button, as shown in Figure 14, we need to specify the sampling rules for the profiling task. The sampling rules consist of one or more rules, each of which is distinguished by a different URI regular expression. When the HTTP request URI matches the regular expression, the rule is used. If the URI regular expression is empty, the default rule is used. Using multiple rules can help us make different sampling configurations for different requests.

Each rule has three parameters to determine if sampling is needed:

Minimal Request Duration (ms): requests with a response time exceeding the specified time will be sampled.
Sampling response status code between 400 and 499: all status codes in the range [400-499) will be sampled.
Sampling response status code between 500 and 599: all status codes in the range [500-599) will be sampled.

Once the sampling configuration is complete, we can create the task.

Figure 14: Create network profiling task page.

Done!

After a few seconds, you will see the process topology appear on the right side of the page.

When you click on the line between processes, you can view the data between the two processes, which is divided into three tabs:

TCP: displays TCP-related metrics.
HTTP/1.x: displays metrics in the HTTP 1 protocol.
HTTP Requests: displays the analyzed request and saves it to a list according to the sampling rule.

Figure 16: TCP metrics in a network profiling task.

Figure 17: HTTP/1.x metrics in a network profiling task.

Figure 18: HTTP sampled requests in a network profiling task.

Conclusion

In this article, we detailed the overview of how to analyze the Layer 7 HTTP/1.x protocol in network analysis, and how to associate it with existing trace systems. This allows us to extend the scope of data we can observe from just user space to also include kernel-space data.

In the future, we will delve further into the analysis of kernel data, such as collecting information on TCP packet size, transmission frequency, network card, and help on enhancing distributed tracing from another perspective.

Additional Resources

Blog: Boost Root Cause Analysis Quickly With SkyWalking’s New Trace-Metrics Association Feature

Mon, 19 Dec 2022 00:00:00 +0000

Observability for modern distributed applications work is critical for understanding how they behave under a variety of conditions and for troubleshooting and resolving issues when they arise. Traces, metrics, and logs are regarded as fundamental parts of the observability stack. Traces are the footprints of distributed system executions, meanwhile, metrics measure system performance with numbers in the timeline. Essentially, they measure the performance from two dimensions. Being able to quickly visualize the connection between traces and corresponding metrics makes it possible to quickly diagnose which process flows are correlated to potentially pathological behavior. This powerful new capability is now available in SkyWalking 9.3.0.

The SkyWalking project started only with tracing, with a focus on 100% sampling-based metrics and topology analysis since 2018. When users face anomaly trends of time-series metrics, like a peak on the line chart, or histogram shows a larger gap between p95 and p95, the immediate question is, why is this happening? One of SkyWalking’s latest features, the trace-metric association, makes it much easier to answer that question and to address the root cause.

How Are Metrics Generated?

SkyWalking provides three ways to calculate metrics:

Metrics built from trace spans, depending on the span’s layer, kind, and tags.
Metrics extracted from logs—a kind of keyword and tags-based metrics extraction.
Metrics reported from mature and mainstream metrics/meter systems, such as OpenTelemetry, Prometheus, and Zabbix.

Tracing tracks the processes of requests between an application’s services. Most systems that generate traffic and performance-related metrics also generate tracing data, either from server-side trace-based aggregations or through client SDKs.

Use SkyWalking to Reduce the Traditional Cost of Trace Indexing

Tracing data and visualization are critical troubleshooting tools for both developers and operators alike because of how helpful they are in locating issue boundaries. But, because it has traditionally been difficult to find associations between metrics and traces, teams have added increasingly more tags into the spans, and search through various combinations. This trend of increased instrumentation and searching has required increased infrastructure investment to support this kind of search. SkyWalking’s metrics and tracing association capabilities can help reduce the cost of indexing and searching that data.

Find the Associated Trace

When looking for association between metrics and traces, the kind of metrics we’re dealing with determines their relationships to traces. Let’s review the standard request rate, error, and duration (RED) metrics to see how it works.

Success Rate Metrics

The success rate is determined by the return code, RPC response code, or exceptions of the process. When the success rate decreases, looking for errors in the traces of this service or pod are the first place to look to find clues.

Figure 1: The success rate graph from SkyWalking’s 9.3.0 dashboard with the option to view related traces at a particular time.

Drilling down from the peak of the success rate, SkyWalking lists all traces and their error status that were collected in this particular minute (Figure 2):

Figure 2: SkyWalking shows related traces with an error status.

Requests to /test can be located from the trace, and the span’s tag indicates a 404 response code of the HTTP request.

Figure 3: A detail view of a request to http://frontend/test showing that the URI doesn’t exist.

By looking at the trace data, it becomes immediately clear that the drop in success rate is caused by requests to a nonexistent URI.

Average Response Time

The average response time metric provides a general overview of service performance. When average response time is unstable, this usually means that the system is facing serious performance impacts.

Figure 4: SkyWalking’s query UI for searching for related traces showing traces for requests that exceed a particular duration threshold.

When you drill down from this metric, this query condition (Figure 4) will reveal the slowest traces of the service in this specific minute. Notice, at least 168ms is added as a condition automatically, to avoid scanning a large number of rows in the Database.

Apdex

Apdex—the Application Performance Index—is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times (Figure 5). The response time is measured from an asset request to completed delivery back to the requestor.

Figure 5: The Apdex formula

A user defines a response time tolerating threshold T. All responses handled in T or less time satisfy the user.

For example, if T is 1.2 seconds and a response completes in 0.5 seconds, then the user is satisfied. All responses greater than 1.2 seconds dissatisfy the user. Responses greater than 4.8 seconds frustrate the user.

When the Apdex score decreases, we need to find related traces from two perspectives: slow traces and error status traces. SkyWalking’s new related tracing features offers a quick way to view both (Figure 6) directly from the Apdex graph.

Figure 6: Show slow trace and error status traces from the Apdex graph

Service Response Time

Percentile MetricThe percentile graph (Figure 7) provides p50, p75, p90, p95, and p99 latency ranks to measure the long-tail issues of service performance.

Figure 7: The service response time percentile graph helps to highlight long-tail issues of service performance.

This percentile graph shows a typical long-tail issue. P99 latency is four times slower than the P95. When we use the association, we see the traces with latency between P95 - P99 and P99 - Infinity.

The traces of requests causing this kind of long-tail phenomena are automatically listing from there.

Figure 8: Query parameters to search for traces based on latency.

Are More Associations Available?

SkyWalking provides more than just associations between between traces and metrics to help you find possible causal relationships and to avoid looking for the proverbial needle in a haystack.

Currently, SkyWalking 9.3.0 offers two more associations: metric-to-metric associations and event-to-metric associations.

Metric-to-metric Associations

There are dozens of metrics on the dashboard—which is great for getting a complete picture of application behavior. During a typical performance issue, the peaks of multiple metrics are affected simultaneously. But, trying to correlate peaks across all of these graphs can be difficult…

Now in SkyWalking 9.3.0, when you click the peak of one graph, the pop-out box lets you see associated metrics.

Figure 9: SkyWalking’s option to view associated metrics.

When you choose that option, all associated metrics graphs will show axis pointers (the dotted vertical lines) in all associated graphs like in Figure 10. This makes it easier to correlate the peaks in different graphs with each other. Often, these correlated peaks with have the same root cause.

Figure 10: Axis pointers (vertical dotted lines) show associations between peaks across multiple metrics graphs.

Event-to-Metric Associations

SkyWalking provides the event concept to associate possible service performance impacted by the infrastructure, such as new deployment even from k8s. Or, the anomaly had been detected by alerting or integrated AIOps engine.

The event to metrics association is also automatically, it could cover the time range of the event on the metric graphs(blue areas). If the area of event and peaks are matched, most likely this event covered this anomaly.

Figure 11: SkyWalking’s event to metric association view.

SkyWalking Makes it Easier and Faster to Find Root Causes

SkyWalking now makes it easy to find associations between metrics, events, and traces, ultimately making it possible to identify root causes and fix problems fast. The associations we’ve discussed in this article are available out-of-box in the SkyWalking 9.3.0 release.

Figure 12: Just click on the dots to see related traces and metrics associations.

Click the dots on any metric graph, and you will see a View Related Traces item pop-out if this metric has logical mapping traces.

Conclusion

In this blog, we took a look at the newly-added association feature between metrics and traces. With this new visualization, it’s now much easier to find key traces to identify root cause of issues.Associations in SkyWalking can go even deeper. Associations from metrics to traces is not the end of diagnosing system bottleneck. In the next post, we will introduce an eBPF powered trace enhancement where you’ll be able to see HTTP request and response details associated with tracing spans from network profiling. Stay tuned.

Blog: Apache ShenYu(incubating) plugin implementation principles and observability practices

Sun, 08 May 2022 00:00:00 +0000

Content

1. Introduction of SkyWalking and ShenYu

1.1 SkyWalking

SkyWalking is an Application Performance Monitoring (APM) and Observability Analysis Platform (OAP) for microservices, distributed systems, and cloud natives, Has powerful features that provide a multi-dimensional means of application performance analysis, including distributed topology diagrams, application performance metrics, distributed link tracing, log correlation analysis and alerts. Also has a very rich ecology. Widely used in various companies and open source projects.

1.2 Apache ShenYu (incubating)

Apache ShenYu (incubating) High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes. Rich protocol support: Http, Spring Cloud, gRPC, Dubbo, SOFARPC, Motan, Tars, etc.

2. Apache ShenYu plugin implementation principle

ShenYu’s asynchrony is a little different from previous exposure to asynchrony, it is a full-link asynchrony, the execution of each plug-in is asynchronous, and thread switching is not a single fixed situation (and the individual plug-in implementation is related). The gateway initiates service calls of various protocol types, and the existing SkyWalking plugins create ExitSpan (synchronous or asynchronous) when they initiate service calls. The gateway receives the request and creates an asynchronous EntrySpan. The asynchronous EntrySpan needs to be concatenated with the synchronous or asynchronous ExitSpan, otherwise the link will be broken.

There are 2 types of tandem solutions：

Snapshot Delivery:
Pass the snapshot after creating the EntrySpan to the thread that created the ExitSpan in some way.
Currently this approach is used in the asynchronous WebClient plugin, which can receive asynchronous snapshots. shenYu proxy Http service or SpringCloud service is to achieve span concatenation through snapshot passing.
LocalSpan transit:
Other RPC class plugins do not receive snapshots for concatenation like Asynchronous WebClient. Although you can modify other RPC plugins to receive snapshots for concatenation, it is not recommended or necessary to do so. This can be achieved by creating a LocalSpan in the thread where the ExitSpan is created, and then connecting the asynchronous EntrySpan and LocalSpan by snapshot passing. This can be done without changing the original plugin code.

The span connection is shown below:

You may ask if it is possible to create LocalSpan inside a generic plugin, instead of creating one separately for ShenYu RPC plugin? The answer is no, because you need to ensure that LocalSpan and ExitSpan are in the same thread, and ShenYu is fully linked asynchronously. The code to create LocalSpan is reused in the implementation.

3. Adding generalized call tracking to the gRPC plugin and keeping it compatible

The existing SkyWalking gRPC plugin only supports calls initiated by way of stubs. For the gateway there is no proto file, the gateway takes generalized calls (not through stubs), so tracing RPC requests, you will find that the link will break at the gateway node. In this case, it is necessary to make the gRPC plugin support generalized calls, while at the same time needing to remain compatible and not affect the original tracing method. This is achieved by determining whether the request parameter is a DynamicMessage, and if it is not, then the original tracing logic through the stub is used. If not, then the original tracing logic via stubs is used, and if not, then the generalized call tracing logic is used. The other compatibility is the difference between the old and new versions of gRPC, as well as the compatibility of various cases of obtaining server-side IP, for those interested in the source code.

4. ShenYu Gateway Observability Practice

The above explains the principle of SkyWalking ShenYu plug-in implementation, the following deployment application to see the effect. SkyWalking powerful, in addition to the link tracking requires the development of plug-ins, other powerful features out of the box. Here only describe the link tracking and application performance analysis part, if you want to experience the power of SkyWalking features, please refer to the SkyWalking official documentation.
Version description:

skywalking-java: 8.11.0-SNAPSHOT source code build. Note: The shenyu plugin will be released in version 8.11.0, and will probably release it initially in May or June. the Java agent is in the regular release phase.
skywalking: 9.0.0 V9 version

Usage instructions:
SkyWalking is designed to be very easy to use. Please refer to the official documentation for configuring and activating the shenyu plugin.

4.1 Sending requests to the gateway

Initiate various service requests to the gateway via the postman client or other means.

4.2 Request Topology Diagram

4.3 Request Trace (in the case of gRPC)

Normal Trace：

Abnormal Trace：

Click on the link node to see the corresponding node information and exception information

Service Provider Span

Gateway request span

4.4 Service Metrics Monitoring

4.5 Gateway background metrics monitoring

Database Monitoring:

Thread pool and connection pool monitoring:

4.6 JVM Monitoring

4.7 Endpoint Analysis

4.8 Exception log and exception link analysis

See official documentation for log configuration

Log monitoring

Distributed link trace details corresponding to exception logs

5. Summary

SkyWalking has very comprehensive support for metrics, link tracing, and logging in observability, and is powerful, easy to use, and designed for large distributed systems, microservices, cloud-native, container architectures, and has a rich ecosystem. Using SkyWalking to provide powerful observability support for Apache ShenYu (incubating) gives ShenYu a boost. Finally, if you are interested in high-performance responsive gateways, you can follow Apache ShenYu (incubating). Also, thanks to SkyWalking such an excellent open source software to the industry contributions.

Blog: SourceMarker: Continuous Feedback for Developers

Tue, 16 Mar 2021 00:00:00 +0000

SourceMarker is an open-source continuous feedback IDE plugin built on top of Apache SkyWalking, a popular open-source APM system with monitoring, tracing, and diagnosing capabilities for distributed software systems. SkyWalking, a truly holistic system, provides the means for automatically producing, storing, and querying software operation metrics. It requires little to no code changes to implement and is lightweight enough to be used in production. By itself, SkyWalking is a formidable force in the realm of continuous monitoring technology.

SourceMarker, leveraging the continuous monitoring functionality provided by SkyWalking, creates continuous feedback technology by automatically linking software operation metrics to source code and displaying feedback directly inside of the IDE. While currently only supporting JetBrains-based IDEs and JVM-based programming languages, SourceMarker may be extended to support any number of programming languages and IDEs. Using SourceMarker, software developers can understand and validate software operation inside of their IDE. Instead of charts that indicate the health of the application, software developers can view the health of individual source code components and interpret software operation metrics from a much more familiar perspective. Such capabilities improve productivity as time spent continuously context switching from development to monitoring would be eliminated.

Logging

The benefits of continuous feedback technology are immediately apparent with the ability to view and search logs directly from source code. Instead of tailing log files or viewing logs through the browser, SourceMarker allows software developers to navigate production logs just as easily as they navigate source code. By using the source code as the primary perspective for navigating logs, SourceMarker allows software developers to view logs specific to any package, class, method, or line directly from the context of the source code which resulted in those logs.

Tracing

Furthermore, continuous feedback technology offers software developers a deeper understanding of software by explicitly tying the implicit software operation to source code. Instead of visualizing software traces as Gantt charts, SourceMarker allows software developers to step through trace stacks while automatically resolving trace tags and logs. With SourceMarker, software developers can navigate production software traces in much the same way one debugs local applications.

Alerting

Most importantly, continuous feedback technology keeps software developers aware of production software operation. Armed with an APM-powered IDE, every software developer can keep track of the behavior of any method, class, package, and even the entire application itself. Moreover, this allows for source code to be the medium through which production bugs are made evident, thereby creating the feasibility of source code with the ability to self-diagnose and convey its own health.

Download SourceMarker

SourceMarker aims to bridge the theoretical and empirical practices of software development through continuous feedback. The goal is to make developing software with empirical data feel natural and intuitive, creating more complete software developers that understand the entire software development cycle.

https://github.com/sourceplusplus/sourcemarker

This project is still early in its development, so if you think of any ways to improve SourceMarker, please let us know.

Blog: Apache SkyWalking: How to propagate context between threads when using ThreadPoolExecutor

Tue, 09 Feb 2021 00:00:00 +0000

When using SkyWalking java agent, people usually propagate context easily. They even do not need to change the business code. However, it becomes harder when you want to propagate context between threads when using ThreadPoolExecutor. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace. This way you must change your code. The developer manager usually don’t like this because there may be lots of projects, or lots of runnable code. If they don’t use SkyWalking some day, the code added will be superfluous and inelegant.

Is there a way to propagate context without changing the business code? Yes.

Skywalking java agent enhances a class by add a field and implement an interface. The ThreadPoolExecutor is a special class that is used widely. We even don’t know when and where it is loaded. Most JVMs do not allow changes in the class file format for classes that have been loaded previously. So SkyWalking should not enhance the ThreadPoolExecutor successfully by retransforming when the ThreadPoolExecutor has been loaded. However, we can apply advice to the ThreadPoolExecutor#execute method and wrap the Runnable param using our own agent, then enhance the wrapper class by SkyWalking java agent. An advice do not change the layout of a class.

Now we should decide how to do this. You can use the RunnableWrapper in the maven artifact org.apache.skywalking:apm-toolkit-trace to wrap the param, but you need to face another problem. This RunnableWrapper has a plugin whose active condition is checking if there is @TraceCrossThread. Agent core uses net.bytebuddy.pool.TypePool.Default.WithLazyResolution.LazyTypeDescription to find the annotations of a class. The LazyTypeDescription finds annotations by using a URLClassLoader with no urls if the classloader is null(bootstrap classloader). So it can not find the @TraceCrossThread class unless you change the LocationStrategy of SkyWalking java agent builder.

In this project, I write my own wrapper class, and simply add a plugin with a name match condition. Next, Let me show you how these two agents work together.

Move the plugin to the skywalking “plugins” directory.
Add this agent after the SkyWalking agent since the wrapper class should not be loaded before SkyWalking agent instrumentation have finished. For example,

java -javaagent:/path/to/skywalking-agent.jar -javaagent:/path/to/skywalking-tool-agent-v1.0.0.jar …
When our application runs
- SkyWalking java agent adds a transformer by parsing the plugin for enhancing the wrapper class in the tool agent.
- The tool agent loads the wrapper class into bootstrap classloader. This triggers the previous transformer.
- The tool agent applies an advice to the ThreadPoolExecutor class, wrapping the java.lang.Runnable param of “execute” method with the wrapper class.
- Now SkyWalking propagates the context with the wrapper class.

Enjoy tracing with ThreadPoolExecutor in SkyWalking!

Blog: Apache SkyWalking: Use Profiling to Fix the Blind Spot of Distributed Tracing

Mon, 13 Apr 2020 00:00:00 +0000

This post originally appears on The New Stack

This post introduces a way to automatically profile code in production with Apache SkyWalking. We believe the profile method helps reduce maintenance and overhead while increasing the precision in root cause analysis.

Limitations of the Distributed Tracing

In the early days, metrics and logging systems were the key solutions in monitoring platforms. With the adoption of microservice and distributed system-based architecture, distributed tracing has become more important. Distributed tracing provides relevant service context, such as system topology map and RPC parent-child relationships.

Some claim that distributed tracing is the best way to discover the cause of performance issues in a distributed system. It’s good at finding issues at the RPC abstraction, or in the scope of components instrumented with spans. However, it isn’t that perfect.

Have you been surprised to find a span duration longer than expected, but no insight into why? What should you do next? Some may think that the next step is to add more instrumentation, more spans into the trace, thinking that you would eventually find the root cause, with more data points. We’ll argue this is not a good option within a production environment. Here’s why:

There is a risk of application overhead and system overload. Ad-hoc spans measure the performance of specific scopes or methods, but picking the right place can be difficult. To identify the precise cause, you can “instrument” (add spans to) many suspicious places. The additional instrumentation costs more CPU and memory in the production environment. Next, ad-hoc instrumentation that didn’t help is often forgotten, not deleted. This creates a valueless overhead load. In the worst case, excess instrumentation can cause performance problems in the production app or overload the tracing system.
The process of ad-hoc (manual) instrumentation usually implies at least a restart. Trace instrumentation libraries, like Zipkin Brave, are integrated into many framework libraries. To instrument a method’s performance typically implies changing code, even if only an annotation. This implies a re-deploy. Even if you have the way to do auto instrumentation, like Apache SkyWalking, you still need to change the configuration and reboot the app. Otherwise, you take the risk of GC caused by hot dynamic instrumentation.
Injecting instrumentation into an uninstrumented third party library is hard and complex. It takes more time and many won’t know how to do this.
Usually, we don’t have code line numbers in the distributed tracing. Particularly when lambdas are in use, it can be difficult to identify the line of code associated with a span. Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Regardless of the above choices, to dive deeper requires collaboration with your Ops or SRE team, and a shared deep level of knowledge in distributed tracing.

Profiling in Production

Introduction

To reuse distributed tracing to achieve method scope precision requires an understanding of the above limitations and a different approach. We called it PROFILE.

Most high-level languages build and run on a thread concept. The profile approach takes continuous thread dumps. We merge the thread dumps to estimate the execution time of every method shown in the thread dumps. The key for distributed tracing is the tracing context, identifiers active (or current) for the profiled method. Using this trace context, we can weave data harvested from profiling into existing traces. This allows the system to automate otherwise ad-hoc instrumentation. Let’s dig deeper into how profiling works:

We consider a method invocation with the same stack depth and signature (method, line number etc), the same operation. We derive span timestamps from the thread dumps the same operation is in. Let’s put this visually:

Above, represents 10 successive thread dumps. If this method is in dumps 4-8, we assume it started before dump 4 and finished after dump 8. We can’t tell exactly when the method started and stopped. but the timestamps of thread dumps are close enough.

To reduce overhead caused by thread dumps, we only profile methods enclosed by a specific entry point, such as a URI or MVC Controller method. We identify these entry points through the trace context and the APM system.

The profile does thread dump analysis and gives us:

The root cause, precise to the line number in the code.
Reduced maintenance as ad-hoc instrumentation is obviated.
Reduced overload risk caused by ad-hoc instrumentation.
Dynamic activation: only when necessary and with a very clear profile target.

Implementing Precise Profiling with Apache SkyWalking 7

Distributed profiling is built-into Apache SkyWalking application performance monitoring (APM). Let’s demonstrate how the profiling approach locates the root cause of the performance issue.

final CountDownLatchcountDownLatch= new CountDownLatch(2);
 
threadPool.submit(new Task1(countDownLatch));
threadPool.submit(new Task2(countDownLatch));
 
try {
   countDownLatch.await(500, TimeUnit.MILLISECONDS);
} catch (InterruptedExceptione) {
}

Task1 and Task2 have a race condition and unstable execution time: they will impact the performance of each other and anything calling them. While this code looks suspicious, it is representative of real life. People in the OPS/SRE team are not usually aware of all code changes and who did them. They only know something in the new code is causing a problem.

To make matters interesting, the above code is not always slow: it only happens when the condition is locked. In SkyWalking APM, we have metrics of endpoint p99/p95 latency, so, we are easy to find out the p99 of this endpoint is far from the avg response time. However, this is not the same as understanding the cause of the latency. To locate the root cause, add a profile condition to this endpoint: duration greater than 500ms. This means faster executions will not add profiling load.

This is a typical profiled trace segment (part of the whole distributed trace) shown on the SkyWalking UI. We now notice the “service/processWithThreadPool” span is slow as we expected, but why? This method is the one we added the faulty code to. As the UI shows that method, we know the profiler is working. Now, let’s see what the profile analysis result say.

This is the profile analysis stack view. We see the stack element names, duration (include/exclude the children) and slowest methods have been highlighted. It shows clearly, “sun.misc.Unsafe.park” costs the most time. If we look for the caller, it is the code we added: CountDownLatch.await.

The Limitations of the Profile Method

No diagnostic tool can fit all cases, not even the profile method.

The first consideration is mistaking a repeatedly called method for a slow method. Thread dumps are periodic. If there is a loop of calling one method, the profile analysis result would say the target method is slow because it is captured every time in the dump process. There could be another reason. A method called many times can also end up captured in each thread dump. Even so, the profile did what it is designed for. It still helps the OPS/SRE team to locate the code having the issue.

The second consideration is overhead, the impact of repeated thread dumps is real and can’t be ignored. In SkyWalking, we set the profile dump period to at least 10ms. This means we can’t locate method performance issues if they complete in less than 10ms. SkyWalking has a threshold to control the maximum parallel degree as well.

Understanding the above keeps distributed tracing and APM systems useful for your OPS/SRE team.

How to Try This

Everything we discussed, including the Apache SkyWalking Java Agent, profile analysis code, and UI, could be found in our GitHub repository. We hope you enjoyed this new profile method, and love Apache SkyWalking. If so, give us a star on GitHub to encourage us.

SkyWalking 7 has just been released. You can contact the project team through the following channels:

Follow SkyWalking twitter.
Subscribe mailing list: dev@skywalking.apache.org. Send to dev-subscribe@kywalking.apache.org to subscribe to the mail list.

Co-author Sheng Wu is a Tetrate founding engineer and the founder and VP of Apache SkyWalking. He is solving the problem of observability for large-scale service meshes in hybrid and multi-cloud environments.

Adrian Cole works in the Spring Cloud team at VMware, mostly on Zipkin

Han Liu is a tech expert at Lagou. He is an Apache SkyWalking committer

Blog: Understand distributed trace easier in the incoming 6-GA

Tue, 01 Jan 2019 00:00:00 +0000

Background

Distributed tracing is a necessary part of modern microservices architecture, but how to understand or use distributed tracing data is unclear to some end users. This blog overviews typical distributed tracing use cases with new visualization features in SkyWalking v6. We hope new users will understand more through these examples.

Metric and topology

Trace data underpins in two well known analysis features: metric and topology

Metric of each service, service instance, endpoint are derived from entry spans in trace. Metrics represent response time performance. So, you could have average response time, 99% response time, success rate, etc. These are broken down by service, service instance, endpoint.

Topology represents links between services and is distributed tracing’s most attractive feature. Topologies allows all users to understand distributed service relationships and dependencies even when they are varied or complex. This is important as it brings a single view to all interested parties, regardless of if they are a developer, designer or operator.

Here’s an example topology of 4 projects, including Kafka and two outside dependencies.

Topology in SkyWalking optional UI, RocketBot

Trace

In a distributed tracing system, we spend a lot of resources(CPU, Memory, Disk and Network) to generate, transport and persistent trace data. Let’s try to answer why we do this? What are the typical diagnosis and system performance questions we can answer with trace data?

SkyWalking v6 includes two trace views:

TreeMode: The first time provided. Help you easier to identify issues.
ListMode: Traditional view in time line, also usually seen in other tracing system, such as Zipkin.

Error occurred

In the trace view, the easiest part is locating the error, possibly caused by a code exception or network fault. Both ListMode and TreeMode can identify errors, while the span detail screen provides details.

ListMode error span

TreeMode error span

Slow span

A high priority feature is identifying the slowest spans in a trace. This uses execution duration captured by application agents. In the old ListMode trace view, parent span almost always includes the child span’s duration, due to nesting. In other words, a slow span usually causes its parent to also become slow. In SkyWalking 6, we provide Top 5 of slow span filter to help you locate the spans directly.

Top 5 slow span

The above screenshot highlights the top 5 slow spans, excluding child span duration. Also, this shows all spans’ execution time, which helps identify the slowest ones.

Too many child spans

In some cases, individual durations are quick, but the trace is still slow, like this one:

Trace with no slow span

To understand if the root problem is related to too many operations, use Top 5 of children span number. This filter shows the amount of children each span has, highlighting the top 5.

13 database accesses of a span

In this screenshot, there is a span with 13 children, which are all Database accesses. Also, when you see overview of trace, database cost 1380ms of this 2000ms trace.

1380ms database accesses

In this example, the root cause is too many database accesses. This is also typical in other scenarios like too many RPCs or cache accesses.

Trace depth

Trace depth is also related latency. Like the too many child spans scenario, each span latency looks good, but the whole trace is slow.

Trace depth

Here, the slowest spans are less than 500ms, which are not too slow for a 2000ms trace. When you see the first line, there are four different colors representing four services involved in this distributed trace. Every one of them costs 100~400ms. For all four, there nearly 2000ms. From here, we know this slow trace is caused by 3 RPCs in a serial sequence.

At the end

Distributed tracing and APM tools help users identify root causes, allowing development and operation teams to optimize accordingly. We hope you enjoyed this, and love Apache SkyWalking and our new trace visualization. If so, give us a star on GitHub to encourage us.

SkyWalking 6 is scheduled to release at the end of January 2019. You can contact the project team through the following channels:

Follow SkyWalking twitter
Subscribe mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list.
Join Gitter room.