Apache SkyWalking – Metrics

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Thu, 30 Apr 2026 00:00:00 +0000

Mini programs are a major part of the mobile experience in China, but the open-source observability ecosystem has long focused on web browsers and native apps. SkyWalking already covers browser (client-js), iOS, and the server side; mini programs and Android were the remaining gaps. With SkyAPM/mini-program-monitor joining the SkyWalking ecosystem, the mini-program half of that gap is closed — one SDK supports both WeChat and Alipay, and the matching OAP-side component IDs, MAL rules, and UI templates are merged on main and will ship with 10.5.0.

This post is for teams that already run a SkyWalking backend and want to bring their mini programs into the same observability stack. The interesting parts aren’t that the project exists — they are how the data flows from a mini program to a SkyWalking dashboard, how the two platforms coexist, and what design trade-offs you should know about before rolling this out.

Data path

The SDK uses two protocols:

OTLP HTTP (error logs, performance metrics, request metrics) → OAP /v1/logs, /v1/metrics
SkyWalking native (distributed tracing segments, optional) → OAP /v3/segments

Why not a single protocol? OTLP already covers logs and metrics, so there’s no point reinventing native endpoints for those. But for tracing, OAP’s native SegmentObject maps more cleanly onto SkyWalking’s trace model, and sw8 header propagation to the backend works without any conversion. So traces go native, everything else goes OTLP, and neither side has to translate.

OTLP defaults to protobuf; JSON is available for debugging. The SDK has zero runtime dependencies.

Two platforms, two independent Layers and dashboards

Many teams maintain a WeChat mini program and an Alipay mini program against a shared backend. Rather than collapsing them into a single tagged service, the design promotes each platform to its own Layer — WECHAT_MINI_PROGRAM and ALIPAY_MINI_PROGRAM — with its own dashboard set. The SDK tags every signal with a resource attribute miniprogram.platform = wechat | alipay and assigns each platform its own component ID (WeChat = 10002, Alipay = 10003).

On the OAP side, the MAL rule’s filter routes data into the right Layer at ingest:

metricPrefix: meter_wechat_mp
filter: "{ tags -> tags.miniprogram_platform == 'wechat' }"

The Alipay rule mirrors this with 'alipay'. The two rules are mutually exclusive — no double counting — and produce distinct metric prefixes (meter_wechat_mp_* vs meter_alipay_mp_*) that feed each Layer’s dashboards. Even when both platforms use the same service.name (e.g. mini-program-demo), the UI exposes two completely separate entry points.

Asymmetric metric semantics

This is the design choice I want to highlight. WeChat’s base library exposes PerformanceObserver, which gives you renderer-authoritative timings: app launch, first render, route navigation, script execution, sub-package load — all real measurements. Alipay’s base library doesn’t offer an equivalent, so the SDK falls back to lifecycle hooks: the App.onLaunch → App.onShow delta is used as an approximation of launch time, and renderer-level timings simply aren’t available.

So the two MAL rule sets are deliberately not the same:

WeChat: app_launch_duration, first_render_duration, route_duration, script_duration, package_load_duration, request_duration_percentile, request_cpm
Alipay: app_launch_duration, first_render_duration, request_duration_percentile, request_cpm

The Alipay app_launch_duration is a lifecycle approximation and is not directly comparable to WeChat’s renderer timing — the dashboard tooltip says so explicitly. Putting the two numbers side by side is comparing two different measurement definitions.

What the SDK does

Four signals:

Errors — JS exceptions, unhandled promise rejections, and pageNotFound go out as OTLP logs, following the OTel exception.* semantic conventions (exception.type, exception.stacktrace). Anything downstream that speaks OTLP — SkyWalking, OTel Collector, Grafana — recognizes them.
Performance — the metrics listed above. OTLP gauge.
Requests — wx.request / my.request / downloadFile / uploadFile are reported as OTLP delta histograms, one batch per flushInterval (default 5s). The le bucket labels are already in milliseconds, and the MAL rule explicitly declares MILLISECONDS to disable the default SECONDS→MS rescale. Failed requests (4xx / 5xx / timeout) additionally emit an error log so you can pivot from a dashboard to a concrete failure.
Tracing (opt-in) — when enabled, outbound requests get sw8 header injection, and the resulting segments stitch together with backend traces into one end-to-end view. Trace data goes out as SkyWalking SegmentObject, not OTLP traces.

Two reliability and cardinality details worth calling out:

Persisting events on app hide. Mini programs get killed by the framework after some time in background, and weak networks make in-flight events easy to lose. The SDK writes unsent events to wx.setStorage / my.setStorage on onAppHide and restores them on the next launch.

Avoiding cardinality explosions. Set serviceInstance to the app version (e.g. 1.4.2), not a device ID — at a million DAU the device-ID dimension blows up the OAP instance index. For request paths, the SDK exposes urlGroupRules regex patterns to fold parameterized URLs like /api/user/12345 into /api/user/{id} so the endpoint dimension doesn’t blow up either.

What OAP needs

If you’re on main or a release ≥ 10.5.0, the following are already shipped:

config/component-libraries.yml registers WeChat-MiniProgram: 10002 and AliPay-MiniProgram: 10003
config/otel-rules/miniprogram/ holds four MAL rules — service-scoped and instance-scoped for each platform
config/ui-initialized-templates/wechat_mini_program/ and alipay_mini_program/ carry root / service / instance / endpoint dashboards
config/ui-initialized-templates/menu.yaml registers both layers under the Mobile menu group

The only thing left is enabling the OTel receiver and giving the SDK an OTLP HTTP port it can reach. SkyWalking OAP binds its OTLP HTTP handler onto the receiver-sharing-server port, and that port defaults to 0 — meaning it’s folded into the core REST port (12800). If you want the SDK to use the standard OTLP HTTP port 4318, set the sharing port to 4318:

docker run -d --name sw-oap \
  -p 11800:11800 -p 12800:12800 -p 4318:4318 \
  -e SW_STORAGE=banyandb \
  -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \
  -e SW_OTEL_RECEIVER=default \
  -e SW_RECEIVER_SHARING_REST_PORT=4318 \
  apache/skywalking-oap-server:latest

All receivers (OTLP, native segment, browser perf, log report) move to 4318 together, while GraphQL stays on 12800 for the UI.

Minimal SDK config:

import MiniProgramMonitor from 'mini-program-monitor';

MiniProgramMonitor.init({
  service: 'mini-program-demo',
  serviceInstance: '1.4.2',          // Recommended: app version
  collector: 'http://your-oap:4318',
  enable: {
    error: true,
    perf: true,
    request: true,
    tracing: false,                  // Off by default; enable as needed
  },
});

WeChat and Alipay use the same config — the SDK detects the platform at runtime and tags the data accordingly.

Compatibility

WeChat base library ≥ 2.11
Alipay base library ≥ 2.0
Apache SkyWalking OAP main or ≥ 10.5.0, with the OTLP HTTP receiver enabled
Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) also works, but you won’t get the SkyWalking-specific cross-platform dashboards

What’s next

To get involved, head over to SkyAPM/mini-program-monitor and open an issue or PR. The repo also ships a make preview target that boots OAP, the UI, and both platform simulators locally — handy if you want to play with it end-to-end.

Android end-user experience monitoring is still a gap in the SkyWalking ecosystem; contributors interested in closing that one are very welcome.

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Wed, 08 Apr 2026 00:00:00 +0000

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Apache SkyWalking introduced TraceQL support in version 10.4.0, implementing Grafana Tempo’s HTTP query APIs so that Grafana can query and visualize traces stored in SkyWalking without any additional plugins. This means you can now use the familiar Grafana Tempo data source to search, filter, and drill into both SkyWalking native traces and Zipkin-compatible traces — all served by your existing SkyWalking OAP server.

Architecture Overview

┌────────────────────┐         Tempo HTTP API           ┌─────────────────────────────┐
│                    │  ──── /skywalking/api/search ──► │  SkyWalking Native Backend  │
│      Grafana       │                                  │  (Query Traces V2 API)      │
│  (Tempo Data Src)  │                                  ├─────────────────────────────┤
│                    │  ──── /zipkin/api/search ──────► │  Zipkin-Compatible Backend  │
└────────────────────┘                                  └──────────┬──────────────────┘
                                                                   │
                                                        ┌──────────▼──────────────────┐
                                                        │    SkyWalking OAP Server    │
                                                        │  ┌───────────────────────┐  │
                                                        │  │   TraceQL Service     │  │
                                                        │  │  (port 3200)          │  │
                                                        │  └───────────────────────┘  │
                                                        │  ┌───────────────────────┐  │
                                                        │  │  Storage (BanyanDB /  │  │
                                                        │  │  Elasticsearch / …)   │  │
                                                        │  └───────────────────────┘  │
                                                        └─────────────────────────────┘

The TraceQL Service sits inside the OAP server and exposes the Tempo-compatible HTTP API on port 3200 (default). It converts traces from their native format into Tempo’s format, where the trace detail part (Trace message) reuses OTLP Trace definitions.

Limitations and Supported TraceQL Features

TraceQL is a rich query language, but SkyWalking currently implements a practical subset. The following features are supported:

Feature	Examples
Spanset filter	`{resource.service.name="frontend"}`
Resource attributes	`resource.service.name`
Span attributes	`span.http.method`, `span.http.status_code`
Intrinsic fields	`duration`, `name`, `status`
Comparison operators	`=`, `>`, `>=`, `<`, `<=`
Compound conditions	`{resource.service.name="frontend" && duration>100ms}`
Duration units	`us`/`µs`, `ms`, `s`, `m`, `h`

The following features are not yet supported:

Spanset logical operations ({...} AND {...}, {...} OR {...})
Pipeline operations (| operator)
Aggregate functions (count(), avg(), max(), min(), sum())
Regular expression matching (=~, !~)
event and link scopes
kind intrinsic field
Streaming mode (must be disabled in the Grafana Tempo data source settings)

Important: SkyWalking native trace support in TraceQL is based on the Query Traces V2 API. Currently, only BanyanDB storage implements this API. Other storage backends (e.g., Elasticsearch, MySQL, PostgreSQL) do not support SkyWalking native trace queries via TraceQL. Zipkin-compatible traces are not subject to this restriction.

Trace Format Conversion

Since the trace detail part of Tempo’s format reuses OTLP Trace definitions, the conversion descriptions below refer to OTLP field names (e.g., span kind, status code).

SkyWalking Native Trace

Trace ID Encoding

SkyWalking native trace IDs are arbitrary strings (e.g., 2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539), while Grafana Tempo requires pure hex-encoded trace IDs. The TraceQL Service encodes each UTF-8 byte of the original trace ID as two lowercase hex characters:

Original:  2a2e04e8d1114b14925c04a6321ca26c.38.17739924187687539
Encoded:   32613265303465386431313134623134393235633034613633323163613236632e33382e3137373339393234313837363837353339

This encoded hex trace ID is what appears in all API responses and in Grafana. When you click a trace ID in Grafana, the TraceQL Service automatically decodes it back to the original SkyWalking trace ID for the internal query.

Span Kind Mapping

SkyWalking Span Type	OTLP Span Kind
`Entry`	`SPAN_KIND_SERVER`
`Exit`	`SPAN_KIND_CLIENT`
`Local`	`SPAN_KIND_INTERNAL`

Status Mapping

SkyWalking `isError`	OTLP Status Code
`true`	`STATUS_CODE_ERROR`
`false`	`STATUS_CODE_OK`

SpanAttachedEvents

SkyWalking SpanAttachedEvents are converted to OTLP span events, with tags mapped as string attributes and summary mapped as numeric attributes (serialized as strings).

Zipkin Trace

Span Kind Mapping

Zipkin Span Kind	OTLP Span Kind
`CLIENT`	`SPAN_KIND_CLIENT`
`SERVER`	`SPAN_KIND_SERVER`
`PRODUCER`	`SPAN_KIND_PRODUCER`
`CONSUMER`	`SPAN_KIND_CONSUMER`

Status Mapping

If the otel.status_code tag is present, it is used directly.
Otherwise, if the error tag equals true, the status is STATUS_CODE_ERROR.
If neither tag is present, the status defaults to STATUS_CODE_UNSET.

Endpoint and Annotation Mapping

Zipkin endpoint fields are mapped to OTLP attributes (e.g., localEndpoint.ipv4 → net.host.ip), and Zipkin annotations are converted to OTLP span events.

For the full conversion details, see the TraceQL Service documentation.

How to Enable TraceQL

Step 1: Enable the TraceQL Module

By default, the TraceQL module is disabled (selector: ${SW_TRACEQL:-}). To enable it, set the selector to default:

# In application.yml
traceQL:
  selector: ${SW_TRACEQL:default}
  default:
    enableDatasourceSkywalking: ${SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING:true}
    enableDatasourceZipkin: ${SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN:true}

Or via environment variables:

export SW_TRACEQL=default
export SW_TRACEQL_ENABLE_DATASOURCE_SKYWALKING=true
export SW_TRACEQL_ENABLE_DATASOURCE_ZIPKIN=true

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

If you want to query Zipkin traces, you also need to enable the Zipkin receiver so that SkyWalking can ingest Zipkin trace data:

# In application.yml
receiver-zipkin:
  selector: ${SW_RECEIVER_ZIPKIN:default}
  default:
    searchableTracesTags: ${SW_ZIPKIN_SEARCHABLE_TAG_KEYS:http.method}
    sampleRate: ${SW_ZIPKIN_SAMPLE_RATE:10000}
    restHost: ${SW_RECEIVER_ZIPKIN_REST_HOST:0.0.0.0}
    restPort: ${SW_RECEIVER_ZIPKIN_REST_PORT:9411}

Or via environment variable:

export SW_RECEIVER_ZIPKIN=default

Full Configuration Reference

For the complete list of all configuration options and their default values, see the Configuration section of the TraceQL Service documentation.

Configuring Grafana Tempo Data Source

Prerequisite: Grafana 12 or later is required.

Each trace backend (SkyWalking native / Zipkin) needs its own Tempo data source in Grafana, because each is served under a different context path.

Context Paths

The two backends are served under separate context paths on the same port:

Backend	Default Context Path	Env Variable	Full Default URL
SkyWalking native	`/skywalking`	`SW_TRACEQL_REST_CONTEXT_PATH_SKYWALKING`	`http://<oap-host>:3200/skywalking`
Zipkin	`/zipkin`	`SW_TRACEQL_REST_CONTEXT_PATH_ZIPKIN`	`http://<oap-host>:3200/zipkin`

Setting Up the SkyWalking Data Source

In Grafana, go to Configuration → Data Sources → Add data source.
Choose Tempo.
Set the URL to http://<oap-host>:3200/skywalking.
Disable the Streaming option (SkyWalking does not support streaming mode).

Save and test the data source.

Setting Up the Zipkin Data Source

Same as above, but set the URL to http://<oap-host>:3200/zipkin.

Configuring Trace List Result Tags

When you search for traces in Grafana, the trace list panel shows a summary of each trace. The tracesListResultTags configuration controls which span tags are included in the search result and displayed as columns in the trace list.

Env Variable	Default Value	Purpose
`SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS`	`http.method,error`	Tags shown for Zipkin traces
`SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS`	`http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker`	Tags shown for SkyWalking traces

Note that service.name and span.kind are always included regardless of this setting.

These tags appear as attribute columns in the Grafana Tempo trace search results, making it easier to identify and group traces at a glance:

SkyWalking native trace list:

Zipkin trace list:

You can customize these tags based on your application’s instrumentation. For example, if your services heavily use messaging, you might add mq.destination or messaging.system to the list.

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Go to the Explore page in Grafana.
Select the Tempo data source you configured for SkyWalking (e.g., SkyWalkingTraceQL).
Run a test query, then click Add to dashboard and save it as SkyWalking Trace.

Step 2: Configure Variables

Add dashboard variables so users can filter traces dynamically (e.g., by service name):

Step 3: Add a Trace Panel

Choose a Table chart (or edit the panel you saved).
Set Query type to Search.
Set the Service Name query condition to the variable $Service.
Add other query conditions as needed (e.g., duration, span name, tags).
Test and save.

Step 4: View Trace Details

Click any trace ID in the trace panel to jump to the Explore page showing the full trace waterfall view with all spans, tags, and events:

Zipkin Trace Dashboard

The setup for Zipkin traces is identical to SkyWalking native traces — just use the Zipkin Tempo data source you configured (e.g., ZipkinTraceQL).

Zipkin trace detail view:

Summary

With TraceQL support in SkyWalking 10.4.0, you can now leverage Grafana’s powerful Tempo data source to query and visualize both SkyWalking native traces and Zipkin-compatible traces. The key points to remember:

Enable the TraceQL module by setting SW_TRACEQL=default and enabling the desired backends.
Configure separate Tempo data sources in Grafana for each backend (/skywalking and /zipkin).
Disable the Streaming option in the Grafana Tempo data source settings.
Customize result tags via SW_TRACEQL_SKYWALKING_TRACES_LIST_RESULT_TAGS and SW_TRACEQL_ZIPKIN_TRACES_LIST_RESULT_TAGS to control what’s shown in search results.
SkyWalking native trace queries require BanyanDB storage (Zipkin traces work with all storage backends).

For the complete API reference and conversion details, see the TraceQL Service documentation. For Grafana integration details, see Use Grafana As The UI.

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

Sun, 05 Apr 2026 00:00:00 +0000

With the deep penetration of Generative AI (GenAI) into enterprise workflows, developers face a challenging paradox: while powerful LLM capabilities are easily integrated via Spring AI or OpenAI SDKs, the actual performance and reliability of these calls remain largely invisible.

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

Facing high LLM bills, organizations often only see a total sum paid to a provider, but cannot calculate the “ROI” within the application.

Blind Upgrades: You might switch to a premium flagship model for a better experience. But in your specific business scenario, does paying several times more per token actually yield lower latency or a faster TTFT (Time to First Token)?
Lack of Real-World Benchmarks: Official benchmarks mean little without your real-world business requests. You need to know which model achieves the perfect balance between “Token/Cost Consumption” and “Response Speed” under your actual prompt lengths and concurrency levels.

2. The Vanishing “Golden Timeout”

Many teams set timeouts for LLM calls arbitrarily (e.g., 30s or 60s).

Too Short: During peak periods or long-text generation, requests are frequently interrupted, causing business failure rates to soar.
Too Long: If a provider hangs, requests pile up in memory, blocking execution threads and potentially leading to the collapse of the entire Java application or microservice cluster. Only by mastering the P99/P95 Latency can you set rational timeout policies based on data rather than intuition.

3. The Overlooked Experience Killer: TTFT

In GenAI scenarios, a user’s perception of speed depends less on the total duration of the conversation and more on “when the first word appears.” * A streaming response with a 10s total duration but a 500ms TTFT feels instantaneous.

A non-streaming response with a 5s total duration but a 4s TTFT feels “frozen.” If your observability system only tracks total latency, you miss the core UX metric that explains why users complain about “AI slowness.”

SkyWalking 10.4: A “Digital Dashboard”
From the Application Perspective The Virtual GenAI capability introduced in Apache SkyWalking 10.4 fills this “observability vacuum.” It avoids reliance on external gateways by using application-side probes (like the Java Agent) to collect the most authentic data from the client’s perspective.

Precise Latency Distribution: Multi-dimensional metrics (P50, P90, P99) help visualize LLM fluctuations to inform dynamic timeout strategies.
Core UX Metric — TTFT Monitoring: Native support for first-token latency in streaming calls.
Multi-dimensional Model Profiling: Aligns token usage, estimated cost, and performance across Providers and Models, helping you choose the most cost-effective solution for your specific needs.

Virtual GenAI Observability

Virtual GenAI represents Generative AI service nodes detected by probe plugins. All performance metrics are based on the GenAI Client Perspective.

For instance, the Spring AI plugin in the Java Agent detects the response latency of a Chat Completion request. SkyWalking then visualizes these in the dashboard:

Traffic & Success Rate (CPM & SLA)
Latency & TTFT
Token Usage (Input/Output)
Estimated Cost

Screenshots:

How It Works

When the SkyWalking Java Agent or OTLP probes intercept calls to mainstream AI frameworks (e.g., Spring AI, OpenAI SDK), they report Trace data to the SkyWalking OAP. The OAP aggregates and computes this data to generate performance metrics for both Providers and Models, which are then rendered in the built-in Virtual-GenAI dashboards.

Installation & Configuration

Requirements

SkyWalking Java Agent: >= 9.7
SkyWalking OAP: >= 10.4

Semantic Conventions & Compatibility

SkyWalking Virtual GenAI follows OpenTelemetry GenAI Semantic Conventions. OAP identifies GenAI-related Spans based on:

SkyWalking Java Agent

Spans must be of type Exit, have the SpanLayer attribute set to GENAI, and contain the gen_ai.response.model tag.

OTLP / Zipkin Probes

Spans must contain the gen_ai.response.model tag.

For details, refer to the E2E configurations:

GenAI Estimated Cost Configuration

Overview

SkyWalking provides a built-in GenAI Billing Configuration File.

This file defines how SkyWalking maps model names from Trace data to their corresponding providers and estimates the token cost for each LLM call. The estimated cost is displayed in the SkyWalking UI alongside trace and metric data, helping users intuitively understand the financial impact of their GenAI usage.

Important: The pricing in this file is intended for cost estimation only and must not be treated as actual billing or invoice amounts. Users are advised to regularly verify the latest rates on the providers’ official pricing pages.

Configuration Structure

Top-level Fields

Field	Type	Description
`last-updated`	`date`	The last update date of the pricing data. All prices are based on public billing standards announced by providers prior to this date.
`providers`	`list`	List of GenAI provider definitions. Each entry contains matching rules and specific model pricing information.

Provider Definition

Each entry under providers defines a GenAI provider:

providers:
- provider: <provider-name>
  prefix-match:
    - <prefix-1>
    - <prefix-2>
  models:
    - name: <model-name>
      aliases: [<alias-1>, <alias-2>]
      input-estimated-cost-per-m: <cost>
      output-estimated-cost-per-m: <cost>

Field	Type	Required	Description
`provider`	`string`	Yes	The provider identifier (e.g., `openai`, `anthropic`, `gemini`). It is displayed as the Virtual GenAI service name in SkyWalking.
`prefix-match`	`list[string]`	Yes	A list of prefixes used to match model names to this provider. If a model name in the Trace data starts with any of these prefixes, it will be mapped to this provider.
`models`	`list[model]`	No	A list of model definitions containing pricing information. If omitted, the system can still identify the provider but will not perform cost estimation.

Model Definition

Each entry under models defines the pricing for a specific model:

Field	Type	Required	Description
`name`	`string`	Yes	The standard model name used for matching.
`aliases`	`list[string]`	No	Alternative names that should resolve to the same billing entry. This is useful when providers use different naming conventions (see the “Model Aliases” section).
`input-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) input (Prompt) tokens. The default unit is USD.
`output-estimated-cost-per-m`	`float`	No	Estimated cost per 1,000,000 (one million) output (Completion) tokens. The default unit is USD.

Model Matching Mechanism

Provider-Level Prefix Matching

When SkyWalking receives a Trace containing a GenAI call, it determines the Provider based on the following priority order:

gen_ai.provider.name tag: This tag is retrieved first. It follows the latest OpenTelemetry GenAI semantic conventions.
gen_ai.system tag: If the above tag is missing, the system falls back to this legacy tag. Note: This tag is only parsed when processing OTLP or Zipkin format data, primarily for compatibility with older versions of libraries like the Python auto-instrumentation.
Prefix Matching: If neither of the above tags exists, SkyWalking reads the prefix-match rules defined in gen-ai-config.yml and attempts to identify the provider by matching the Model Name.

- provider: openai
  prefix-match:
    - gpt

Any model name starting with gpt (such as gpt-4o, gpt-4.1-mini, or gpt-5-nano) will be mapped to the openai provider. A single provider can have multiple prefixes:

- provider: tencent
  prefix-match:
    - hunyuan
    - Tencent

Model-level Longest-Prefix Matching

Once the provider is determined, SkyWalking uses a Trie-based longest-prefix matching algorithm to find the best billing entry. This is crucial because model names returned in provider API responses often include version numbers or timestamps, differing from the base model name in the config. Example OpenAI config:

models:
- name: gpt-4o
  input-estimated-cost-per-m: 2.5
  output-estimated-cost-per-m: 10.0
- name: gpt-4o-mini
  input-estimated-cost-per-m: 0.15
  output-estimated-cost-per-m: 0.6

Matching behavior:

Model Name in Trace	Matched Configuration Entry	Reason
`gpt-4o`	`gpt-4o`	Exact match
`gpt-4o-2024-08-06`	`gpt-4o`	Longest prefix is `gpt-4o`
`gpt-4o-mini`	`gpt-4o-mini`	Exact match (Longer prefix `gpt-4o-mini` takes priority over `gpt-4o`)
`gpt-4o-mini-2024-07-18`	`gpt-4o-mini`	Longest prefix is `gpt-4o-mini`

This mechanism ensures versioned API model names map to the correct pricing tier without requiring exact full names in the configuration file.

Model Aliases

Some providers use different naming conventions across API responses and documentation. For example, Anthropic’s model might appear as claude-4-sonnet or claude-sonnet-4. The aliases field supports both formats under a single billing entry:

- name: claude-4-sonnet
  aliases: [claude-sonnet-4]
  input-estimated-cost-per-m: 3.0
  output-estimated-cost-per-m: 15.0

Under this configuration, claude-4-sonnet and claude-sonnet-4 (as well as any versioned variants, such as claude-sonnet-4-20250514) will resolve to the same billing entry.
Note: Aliases also participate in longest prefix matching. Therefore, claude-sonnet-4-20250514 will match the alias claude-sonnet-4, which in turn resolves to the pricing information for claude-4-sonnet.

Custom Configuration

Adding a New Provider

To add a provider that is not included in the default configuration:

providers:
# ... Existing providers ...

- provider: ollama
  prefix-match:
    - mymodel
  models:
    - name: mymodel-large
      input-estimated-cost-per-m: 1.0
      output-estimated-cost-per-m: 5.0
    - name: mymodel-small
      input-estimated-cost-per-m: 0.1
      output-estimated-cost-per-m: 0.5

For OTLP/Zipkin data, a dedicated estimated tag has been added. You can now view the cost of each GenAI call directly on the UI.

Main Metrics

1.Provider Level

Metric ID	Description	Meaning
`gen_ai_provider_cpm`	Calls Per Minute	Requests per minute (Throughput)
`gen_ai_provider_sla`	Success Rate	Request success rate
`gen_ai_provider_resp_time`	Avg Response Time	Average response time
`gen_ai_provider_latency_percentile`	Latency Percentiles	Response time percentiles (P50, P75, P90, P95, P99)
`gen_ai_provider_input_tokens_sum/avg`	Input Token Usage	Total and average input token usage
`gen_ai_provider_output_tokens_sum/avg`	Output Token Usage	Total and average output token usage
`gen_ai_provider_total_estimated_cost/avg`	Estimated Cost	Total estimated cost and average cost per call

2. Model Level

Metric ID	Description	Meaning
`gen_ai_model_call_cpm`	Calls Per Minute	Requests per minute for this specific model
`gen_ai_model_sla`	Success Rate	Model-specific request success rate
`gen_ai_model_latency_avg/percentile`	Latency	Average and percentiles of model response duration
`gen_ai_model_ttft_avg/percentile`	TTFT	Time to First Token (Streaming only)
`gen_ai_model_input_tokens_sum/avg`	Input Token Usage	Detailed input token consumption for the model
`gen_ai_model_output_tokens_sum/avg`	Output Token Usage	Detailed output token consumption for the model
`gen_ai_model_total_estimated_cost/avg`	Estimated Cost	Estimated total cost and average cost for the model

Recommended Usage Scenarios

Performance Evaluation: Use Latency and Time to First Token (TTFT) metrics to analyze model inference efficiency and the end-user interaction experience.
Token Monitoring: Real-time monitoring of Input and Output token consumption to analyze resource utilization across different business scenarios.
Cost Alerting: Set alert thresholds based on Estimated Cost or token consumption to promptly detect abnormal calls and prevent budget overruns.

Blog: Monitoring Flink with SkyWalking

Fri, 25 Apr 2025 00:00:00 +0000

Background

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures.

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Since SkyWalking 10.3, a new out-of-the-box feature has been introduced that enables Flink monitoring data to be visualized on the SkyWalking UI via the OpenTelemetry Collector, which gathers metrics from Flink endpoints.

Development

Preparation

Process

Set up SkyWalking oap and UI.
Set up the Flink cluster By configuring jobmanager and taskmanager to expose prometheus http endpoints.
Set up OpenTelemetry-collector.
Run your job.

Data flow

Configuration

docker-compose

version: "3"

services:
  oap:
    extends:
      file: ../../script/docker-compose/base-compose.yml
      service: oap
    ports:
      - "12800:12800"
    networks:
      - e2e

  banyandb:
    extends:
      file: ../../script/docker-compose/base-compose.yml
      service: banyandb
    ports:
      - 17912

  jobmanager:
    image: flink:2.0-preview1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
        metrics.reporter.prom.port: 9260
    ports:
      - "8081:8081"
      - "9260:9260"
    command: jobmanager
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8081"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - e2e

  taskmanager:
    image: flink:2.0-preview1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
        metrics.reporter.prom.port: 9261
    depends_on:
      jobmanager:
        condition: service_healthy
    ports:
      - "9261:9261"
    command: taskmanager
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9261/metrics"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - e2e

  executeJob:
    image: flink:2.0-preview1
    depends_on:
      taskmanager:
        condition: service_healthy
    command: >
      bash -c "
      ./bin/flink run -m jobmanager:8081 examples/streaming/WindowJoin.jar"
    networks:
      - e2e

  otel-collector:
    image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION}
    networks:
      - e2e
    command: [ "--config=/etc/otel-collector-config.yaml" ]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    expose:
      - 55678
    depends_on:
      oap:
        condition: service_healthy

networks:
  e2e:

If you plan to expose metrics data using the pushGateway pattern, please refer to the documentation.

OpenTelemetry-collector

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "flink-jobManager-monitoring"
          scrape_interval: 30s
          static_configs:
            - targets: ['jobmanager:9260']
              labels:
                cluster: flink-cluster
          relabel_configs:
            - source_labels: [ __address__ ]
              target_label: jobManager_node
              replacement: $$1
          metric_relabel_configs:
            - source_labels: [ job_name ]
              action: replace
              target_label: flink_job_name
              replacement: $$1
            - source_labels: [ ]
              target_label: job_name
              replacement: flink-jobManager-monitoring

        - job_name: "flink-taskManager-monitoring"
          scrape_interval: 30s
          static_configs:
            - targets: [ "taskmanager:9261" ]
              labels:
                cluster: flink-cluster
          relabel_configs:
            - source_labels: [ __address__ ]
              regex: (.+)
              target_label: taskManager_node
              replacement: $$1
          metric_relabel_configs:
            - source_labels: [ job_name ]
              action: replace
              target_label: flink_job_name
              replacement: $$1
            - source_labels: [ ]
              target_label: job_name
              replacement: flink-taskManager-monitoring

exporters:
  otlp:
    endpoint: oap:11800
    tls:
      insecure: true

processors:
  batch:
service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - batch
      exporters:
        - otlp

Warning:
Please do not edit the value of the job_name configuration, otherwise SkyWalking will not handle these data.
oap means the address of your SkyWalking oap address,please replace it accordingly.
Since the original Flink metrics contain the job_name labels, and SkyWalking relies on the job_name label to handle OpenTelemetry data, to avoid conflicts, we use metric_relabel_configs to rename the original job_name label to flink_job_name.

Metrics Definition

Monitoring metrics involve in Cluster Metrics, TaskManager Metrics, and Job Metrics.

Cluster Metrics

Cluster Metrics mainly focuses on statistics from the perspective of the entire cluster, as well as displaying JVM-related metrics of the JobManager, such as:

Running Jobs：The number of currently running jobs.
TaskManagers：The number of TaskManagers.
Task Managers Slots Total：The total number of TaskManager slots.
Task Managers Slots Available：The number of available TaskManager slots.
JVM CPU Load：The CPU load of the JobManager’s JVM.

TaskManager Metrics

TaskManager Metrics mainly focuses on statistics from the perspective of individual TaskManager nodes, such as:

JVM Memory Heap Used：The amount of JVM heap memory used on the TaskManager node.
JVM Memory Heap Available：The amount of JVM heap memory available on the TaskManager node.
NumRecordsIn：The number of records received per minute by the TaskManager.
NumBytesInPerSecond：The number of bytes received per second by the TaskManager.
IsBackPressured：Indicates whether the TaskManager node is under backpressure.
IdleTimeMsPerSecond：The idle time per second of the TaskManager node.

Job Metrics

Job Metricsmainly focuses on statistics from the perspective of running jobs, such as:

Job RunningTime：The duration for which the job has been running.
Job Restart Number：The number of times the job has been restarted.
Checkpoints Failed：The number of failed checkpoints.
NumBytesInPerSecond：The number of bytes received per second by the job.

You can find explanations for each metric in the tip of the corresponding chart.

References

Blog: New Features of SkyWalking Go 0.2.0

Mon, 31 Jul 2023 00:00:00 +0000

Announcing Apache SkyWalking Go 0.2.0

I’m excited to announce the release of Apache SkyWalking Go 0.2.0! This version packs several awesome new features that I’ll overview below.

Log Reporting

The log reporting feature allows the Go agent to automatically collect log content from supported logging frameworks like logrus and zap. The logs are organized and sent to the SkyWalking backend for visualization. You can see how the logs appear for each service in the SkyWalking UI:

Making Logs Searchable

You can configure certain log fields to make them searchable in SkyWalking. Set the SW_AGENT_LOG_REPORTER_LABEL_KEYS environment variable to include additional fields beyond the default log level.

For example, with logrus:

# define log with fields
logrus.WithField("module", "test-service").Info("test log")

Metrics Reporting

The agent can now collect and report custom metrics data from runtime/metrics to the backend. Supported metrics are documented here.

Automatic Instrumentation

In 0.1.0, you had to manually integrate the agent into your apps. Now, the new commands can automatically analyze and instrument projects at a specified path, no code changes needed! Try using the following command to import skywalking-go into your project:

# inject to project at current path
skywalking-go-agent -inject=./ -all

Or you can still use the original manual approach if preferred.

Get It Now!

Check out the CHANGELOG for the full list of additions and fixes. I encourage you to try out SkyWalking Go 0.2.0 today! Let me know if you have any feedback.

Blog: Build Grafana dashboards for Apache SkyWalking -- Native PromQL Support

Thu, 16 Mar 2023 00:00:00 +0000

Background

As an application performance monitoring tool for distributed systems, Apache SkyWalking provides monitoring, tracing, diagnosing capabilities for distributed system in Cloud Native architecture. Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem. Especially Prometheus metrics receive widespread support through exporters and integrations. PromQL as Prometheus Querying Language containing a set of expressions and expose HTTP APIs to read metrics.

SkyWalking supports to ingest Prometheus metrics through OpenTelemetry collector and through the aggregate calculation of these metrics to provide a variety of systems monitoring, such as Linux Monitoring and Kubernetes monitoring. SkyWalking already provides native UI and GraphQL API for users. But as designed to provide wider ecological integration capabilities, since 9.4.0, it provides PromQL Service, the third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through it. SkyWalking users will benefit from it when they integrate with different systems.

What is PromQL Service in SkyWalking?

PromQL Service is a query engine on the top of SkyWalking native GraphQL query, with additional query stage calculation capabilities powered by Prometheus expressions. It can accept PromQL HTTP API requests, parse Prometheus expressions, and transform between Prometheus metrics and SkyWalking metrics.

The PromQL Service follows all PromQL’s protocols and grammar and users can use it as they would with PromQL. As SkyWalking is fundamentally different from Prometheus in terms of metric classification, format, storage, etc. PromQL Service doesn’t have to implement the full PromQL feature. Refer to the documentation for the detail.

SkyWalking Basic Concepts

Here are some basic concepts and differences from Prometheus that users need to understand in order to use the PromQL service: Prometheus metrics specify the naming format and structure, the actual metric names and labels are determined by the client provider, and the details are stored. The user aggregates and calculates the metrics using the expression in PromQL. Unlike Prometheus, SkyWalking’s metric mechanism is built around the following core concepts with a hierarchical structure:

Layer: represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). This layer would be the owner of different services detected from different technologies. All Layers definitions can be found here.
Service: Represents a set/group of workloads which provides the same behaviors for incoming requests.
Service Instance: An individual workload in the Service group.
Endpoint: A path in a service for incoming requests.
Process: An operating system process. In some scenarios, a service instance is not a process, such as a pod Kubernetes could contain multiple processes.

The metric name and properties (labels) are configured by the SkyWalking OAP server based on the data source as well as OAL and MAL. SkyWalking provides the ability to down-sampling time series metrics, and generate different time bucket data (minute, hour, day).

The SkyWalking metric stream is as follows:

Traffic

The metadata of the Service/ServiceRelation/Instance/ServiceInstanceRelation/Endpoint/EndpointRelation/Process/ProcessRelation. Include names, layers, properties, relations between them, etc.

Metric

Name: metric name, configuration from OAL and MAL.
Entity: represents the metrics’ belonging and used for the query. An Entity will contain the following information depending on the Scope： Scope represents the metrics level and in query stage represents the Scope catalog, Scope catalog provides high-dimension classifications for all scopes as a hierarchy structure.

Scope	Entity Info
Service	Service(include layer info)
ServiceInstance	Service, ServiceInstance
Endpoint	Service, Endpoint
ServiceRelation	Service, DestService
ServiceInstanceRelation	ServiceInstance, DestServiceInstance
EndpointRelation	Endpoint, DestEndpoint
Process	Service, ServiceInstance, Process
ProcessRelation	Process, ServiceInstance, DestProcess

Value:

single value: long.
labeled value: text, label1,value1|label2,value2|..., such as L2 aggregation,5000 | L1 aggregation,8000.

TimeBucket: the time is accurate to minute, hour, day.

How to use PromQL Service

Setup

PromQL Service is enabled by default after v9.4.0, so no additional configuration is required. The default ports, for example, can be configured by using OAP environment variables:

restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
restPort: ${SW_PROMQL_REST_PORT:9090}
restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
restMaxThreads: ${SW_PROMQL_REST_MAX_THREADS:200}
restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}

Use Prometheus expression

PromQL matches metric through the Prometheus expression. Here is a typical Prometheus metric.

To match the metric, the Prometheus expression is as follows:

In the PromQL Service, these reserved labels would be parsed as the metric name and entity info fields with other labels for the query. The mappings are as follows.

SkyWalking Concepts	Prometheus expression
Metric name	Metric name
Layer	Label
Service	Label
ServiceInstance	Label<service_instance>
Endpoint	Label
…	…

For example, the following expressions are used to match query metrics: service_cpm, service_instance_cpm, endpoint_cpm

service_cpm{service='agent::songs', layer='GENERAL'}
service_instance_cpm{service='agent::songs', service_instance='agent::songs_instance_1', layer='GENERAL'}
endpoint_cpm{service='agent::songs', endpoint='GET:/songs', layer='GENERAL'}

Typical Query Example

At here, we take the SkyWalking Showcase deployment as the playground to demonstrate how to use PromQL for SkyWalking metrics.

The following examples can be used to query the metadata and metrics of services through PromQL Service.

Get metrics names

Query:

http://localhost:9099/api/v1/label/__name__/values

Result:

{
    "status": "success",
    "data": [
        "meter_mysql_instance_qps",
        "service_cpm",
        "envoy_cluster_up_rq_active",
        "instance_jvm_class_loaded_class_count",
        "k8s_cluster_memory_requests",
        "meter_vm_memory_used",
        "meter_apisix_sv_bandwidth_unmatched",
        "meter_vm_memory_total",
        ...
    ]
}

Select a metric and get the labels

Query:

http://localhost:9099/api/v1/labels?match[]=service_cpm

Result:

{
  "status": "success",
  "data": [
    "layer",
    "service",
    "top_n",
    "order"
  ]
}

Get services from a specific layer

Query:

http://127.0.0.1:9099/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

Query specific metric for a service

Query:

http://127.0.0.1:9099/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}

Result:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_cpm",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1679559960,
          "6"
        ]
      }
    ]
  }
}

About the range query and different metrics type for query can refer to the document here.

Build Grafana Dashboard

From the above, we know the mechanism and how to query from PromQL Service, now we can build the Grafana Dashboard for the above service example. Note: All the following configurations are based on Grafana version 9.1.0.

SkyWalking Showcase provides dashboards files such as services of General and Service Mesh layers， we can quickly create a dashboard for the General layer service by importing the dashboard JSON file.

After the Grafana application is deployed, follow the steps below:

Configure Data Source

First, we need to create a data source: In the data source config panel, chose Prometheus and set the URL to the OAP server address, the default port is 9090. Here set the data source name SkyWalking in case there are multiple Prometheus data sources.

Import Dashboard File

Create a dashboard folder named SkyWalking.
Import the dashboard file into Grafana, there are two ways to get the file:
1. From SkyWalking Showcase.
2. Go to SkyWaking Demo: Preview metrics on Grafana, and export it from the General Service dashboard.
Done! Now we can see the dashboard is working, the services are in the drop-down list and the metrics are displayed on the panels.

This is an easy way to build, but we need to know how it works if we want to customize it.

How the dashboard works

Dashboard Settings

Open the Settings-Variables we can see the following variables:

Let’s look at what each variable does:

$DS_SkyWalking

This is a data source ty variable that specifies the Prometheus data source which was defined earlier as SkyWalking.
$layer

This is a constant type because in the ‘General Service’ dashboard, all services belong to the ‘GENERAL’ layer, so they can be used directly in each query Note When you customize other layers, this value must be defined in the Layer mentioned above.
$service

Query type variable, to get all service names under this layer for the drop-down list.

Query expression:
```
label_values(service_traffic{layer='$layer'}, service)
```
The query expression will query HTTP API /api/v1/series for service metadata in $layer and fetch the service name according to the label(service).
$service_instance

Same as the $service is a query variable that is used to select all instances of the service in the drop-down list.

Query expression:
```
label_values(instance_traffic{layer='$layer', service='$service'}, service_instance)
```
The query expression here not only specifies the $layer but also contains the variable $service, which is used to correlate with the services for the drop-down list.
$endpoint

Same as the $service is a query variable that is used to select all endpoints of the service in the drop-down list.

Query expression:
```
label_values(endpoint_traffic{layer='$layer', service='$service', keyword='$endpoint_keyword', limit='$endpoint_limit'}, endpoint)
```
The query expression here specifies the $layer and $service which are used to correlate with the services for the drop-down list. And also accept variables $endpoint_keyword and $endpoint_limit as filtering condition.
$endpoint_keyword

A text type variable that the user can input to filter the return value of $endpoint.
$endpoint_limit

Custom type, which the user can select to limit the maximum number of returned endpoints.

Panel Configurations

There are several typical metrics panels on this dashboard, let’s see how it’s configured.

Common Value Metrics

Select Time series chart panel Service Apdex and click edit.

Query expression
```
service_apdex{service='$service', layer='$layer'} / 10000
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. The calculation Divided by 10000 is used for matching the result units. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.

Labeled Value Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_percentile{service='$service', layer='$layer', labels='0,1,2,3,4', relabels='P50,P75,P90,P95,P99'}
```
The metric scope is Service, add labels service and layer for the match, and the label value used the variables configured above. Add labels='0,1,2,3,4' filter the result label, and addrelabels='P50,P75,P90,P95,P99' rename the result label. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set Connect null values --> Always and Show points --> Always because when the query interval > 1 hour or 1 day SkyWalking returns the hour/day step metrics values.
Set Legend to {{label}} for show up.

Sort Metrics

Select Time series chart panel Service Response Time Percentile and click edit.

Query expression
```
service_instance_cpm{parent_service='$service', layer='$layer', top_n='10', order='DES'}
```
The expression is used for query the sore metrics under service, so add labels parent_service and layer for the match. Add top_n='10' and order='DES' filter the result. The document for the query can refer to here.
Set Query options --> Min interval = 1m, because the metrics min time bucket in SkyWalking is 1m.
Set the Calculation --> Latest*.
Set Legend to {{service_instance}} for show up.

Conclusion

In this article, we introduced what is the PromQL Service in SkyWalking and its background. Detailed how to use PromQL Service and the basic concepts related to SkyWalking, and show how to use PromQL Service to build Grafana dashboards for SkyWalking.

In the future, there will be more integrations by leveraging this protocol, such as CI/CD, HPA (scaling), etc.

Blog: eBPF enhanced HTTP observability - L7 metrics and tracing

Thu, 12 Jan 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source Application Performance Management system that helps users collect and aggregate logs, traces, metrics, and events for display on a UI. In the previous article, we introduced how to use Apache SkyWalking Rover to analyze the network performance issue in the service mesh environment. However, in business scenarios, users often rely on mature layer 7 protocols, such as HTTP, for interactions between systems. In this article, we will discuss how to use eBPF techniques to analyze performance bottlenecks of layer 7 protocols and how to enhance the tracing system using network sampling.

This article will show how to use Apache SkyWalking with eBPF to enhance metrics and traces in HTTP observability.

HTTP Protocol Analysis

HTTP is one of the most common Layer 7 protocols and is usually used to provide services to external parties and for inter-system communication. In the following sections, we will show how to identify and analyze HTTP/1.x protocols.

Protocol Identification

In HTTP/1.x, the client and server communicate through a single file descriptor (FD) on each side. Figure 1 shows the process of communication involving the following steps:

Connect/accept: The client establishes a connection with the HTTP server, or the server accepts a connection from the client.
Read/write (multiple times): The client or server reads and writes HTTPS requests and responses. A single request-response pair occurs within the same connection on each side.
Close: The client and server close the connection.

To obtain HTTP content, it’s necessary to read it from the second step of this process. As defined in the RFC, the content is contained within the data of the Layer 4 protocol and can be obtained by parsing the data. The request and response pair can be correlated because they both occur within the same connection on each side.

Figure 1: HTTP communication timeline.

HTTP Pipeline

HTTP pipelining is a feature of HTTP/1.1 that enables multiple HTTP requests to be sent over a single TCP connection without waiting for the corresponding responses. This feature is important because it ensures that the order of the responses on the server side matches the order of the requests.

Figure 2 illustrates how this works. Consider the following scenario: an HTTP client sends multiple requests to a server, and the server responds by sending the HTTP responses in the same order as the requests. This means that the first request sent by the client will receive the first response from the server, the second request will receive the second response, and so on.

When designing HTTP parsing, we should follow this principle by adding request data to a list and removing the first item when parsing a response. This ensures that the responses are processed in the correct order.

Figure 2: HTTP/1.1 pipeline.

Metrics

Based on the identification of the HTTP content and process topology diagram mentioned in the previous article, we can combine these two to generate process-to-process metrics data.

Figure 3 shows the metrics that currently support the analysis between the two processes. Based on the HTTP request and response data, we can analyze the following data:

Metrics Name	Type	Unit	Description
Request CPM(Call Per Minute)	Counter	count	The HTTP request count
Response Status CPM(Call Per Minute)	Counter	count	The count of per HTTP response status code
Request Package Size	Counter/Histogram	Byte	The request package size
Response Package Size	Counter/Histogram	Byte	The response package size
Client Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the client side
Server Duration	Counter/Histogram	Millisecond	The duration of single HTTP response on the server side

Figure 3: Process-to-process metrics.

HTTP and Trace

During the HTTP process, if we unpack the HTTP requests and responses from raw data, we can use this data to correlate with the existing tracing system.

Trace Context Identification

In order to track the flow of requests between multiple services, the trace system usually creates a trace context when a request enters a service and passes it along to other services during the request-response process. For example, when an HTTP request is sent to another server, the trace context is included in the request header.

Figure 4 displays the raw content of an HTTP request intercepted by Wireshark. The trace context information generated by the Zipkin Tracing system can be identified by the “X-B3” prefix in the header. By using eBPF to intercept the trace context in the HTTP header, we can connect the current request with the trace system.

Figure 4: View of HTTP headers in Wireshark.

Trace Event

We have added the concept of an event to traces. An event can be attached to a span and consists of start and end times, tags, and summaries, allowing us to attach any desired information to the Trace.

When performing eBPF network profiling, two events can be generated based on the request-response data. Figure 5 illustrates what happens when a service performs an HTTP request with profiling. The trace system generates trace context information and sends it in the request. When the service executes in the kernel, we can generate an event for the corresponding trace span by interacting with the request-response data and execution time in the kernel space.

Previously, we could only observe the execution status in the user space. However, by combining traces and eBPF technologies, we can now also get more information about the current trace in the kernel space, which would impact less performance for the target service if we do similar things in the tracing SDK and agent.

Figure 5: Logical view of profiling an HTTP request and response.

Sampling

To ensure efficient data storage and minimize unnecessary data sampling, we use a sampling mechanism for traces in our system. This mechanism triggers sampling only when certain conditions are met. We also provide a list of the top N traces, which allows users to quickly access the relevant request information for a specific trace.

To help users easily identify and analyze relevant events, we offer three different sampling rules:

Slow Traces: Sampling is triggered when the response time for a request exceeds a specified threshold.
Response Status [400, 500): Sampling is triggered when the response status code is greater than or equal to 400 and less than 500.
Response Status [500, 600): Sampling is triggered when the response status code is greater than or equal to 500 and less than 600.

In addition, we recognize that not all request or response raw data may be necessary for analysis. For example, users may be more interested in requesting data when trying to identify performance issues, while they may be more interested in response data when troubleshooting errors. As such, we also provide configuration options for request or response events to allow users to specify which type of data they would like to sample.

Profiling in a Service Mesh

The SkyWalking and SkyWalking Rover projects have already implemented the HTTP protocol analyze and trace associations. How do they perform when running in a service mesh environment?

Deployment

Figure 6 demonstrates the deployment of SkyWalking and SkyWalking Rover in a service mesh environment. SkyWalking Rover is deployed as a DaemonSet on each machine where a service is located and communicates with the SkyWalking backend cluster. It automatically recognizes the services on the machine and reports metadata information to the SkyWalking backend cluster. When a new network profiling task arises, SkyWalking Rover senses the task and analyzes the designated processes, collecting and aggregating network data before ultimately reporting it back to the SkyWalking backend service.

Figure 6: SkyWalking rover deployment topology in a service mesh.

Tracing Systems

Starting from version 9.3.0, the SkyWalking backend fully supports all functions in the Zipkin server. Therefore, the SkyWalking backend can collect traces from both the SkyWalking and Zipkin protocols. Similarly, SkyWalking Rover can identify and analyze trace context in both the SkyWalking and Zipkin trace systems. In the following two sections, network analysis results will be displayed in the SkyWalking and Zipkin UI respectively.

SkyWalking

When SkyWalking performs network profiling, similar to the TCP metrics in the previous article, the SkyWalking UI will first display the topology between processes. When you open the dashboard of the line representing the traffic metrics between processes, you can see the metrics of HTTP traffic from the “HTTP/1.x” tab and the sampled HTTP requests with tracing in the “HTTP Requests” tab.

As shown in Figure 7, there are three lists in the tab, each corresponding to a condition in the event sampling rules. Each list displays the traces that meet the pre-specified conditions. When you click on an item in the trace list, you can view the complete trace.

Figure 7: Sampled HTTP requests within tracing context.

When you click on an item in the trace list, you can quickly view the specified trace. In Figure 8, we can see that in the current service-related span, there is a tag with a number indicating how many HTTP events are related to that trace span.

Since we are in a service mesh environment, each service involves interacting with Envoy. Therefore, the current span includes Envoy’s request and response information. Additionally, since the current service has both incoming and outgoing requests, there are events in the corresponding span.

Figure 8: Events in the trace detail.

When the span is clicked, the details of the span will be displayed. If there are events in the current span, the relevant event information will be displayed on a time axis. As shown in Figure 9, there are a total of 6 related events in the current Span. Each event represents a data sample of an HTTP request/response. One of the events spans multiple time ranges, indicating a longer system call time. It may be due to a blocked system call, depending on the implementation details of the HTTP request in different languages. This can also help us query the possible causes of errors.

Figure 9: Events in one trace span.

Finally, we can click on a specific event to see its complete information. As shown in Figure 10, it displays the sampling information of a request, including the SkyWalking trace context protocol contained in the request header from the HTTP raw data. The raw request data allows you to quickly re-request the request to solve any issues.

Figure 10: The detail of the event.

Zipkin

Zipkin is one of the most widely used distributed tracing systems in the world. SkyWalking can function as an alternative server to provide advanced features for Zipkin users. Here, we use this way to bring the feature into the Zipkin ecosystem out-of-box. The new events would also be treated as a kind of Zipkin’s tags and annotations.

To add events to a Zipkin span, we need to do the following:

Split the start and end times of each event into two annotations with a canonical name.
Add the sampled HTTP raw data from the event to the Zipkin span tags, using the same event name for corresponding purposes.

Figures 11 and 12 show annotations and tags in the same span. In these figures, we can see that the span includes at least two events with the same event name and sequence suffix (e.g., “Start/Finished HTTP Request/Response Sampling-x” in the figure). Both events have separate timestamps to represent their relative times within the span. In the tags, the data content of the corresponding event is represented by the event name and sequence number, respectively.

Figure 11: Event timestamp in the Zipkin span annotation.

Figure 12: Event raw data in the Zipkin span tag.

Demo

In this section, we demonstrate how to perform network profiling in a service mesh and complete metrics collection and HTTP raw data sampling. To follow along, you will need a running Kubernetes environment.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy service, the latest released SkyWalking OAP, and UI.

export SW_OAP_IMAGE=apache/skywalking-oap-server:9.3.0
export SW_UI_IMAGE=apache/skywalking-ui:9.3.0
export SW_ROVER_IMAGE=apache/skywalking-rover:0.4.0

export FEATURE_FLAGS=mesh-with-agent,single-node,elasticsearch,rover
make deploy.kubernetes

After deployment is complete, please run the following script to open SkyWalking UI: http://localhost:8080/.

kubectl port-forward svc/ui 8080:8080 --namespace default

Start Network Profiling Task

Currently, we can select the specific instances that we wish to monitor by clicking the Data Plane item in the Service Mesh panel and the Service item in the Kubernetes panel.

In figure 13, we have selected an instance with a list of tasks in the network profiling tab.

Figure 13: Network Profiling tab in the Data Plane.

When we click the Start button, as shown in Figure 14, we need to specify the sampling rules for the profiling task. The sampling rules consist of one or more rules, each of which is distinguished by a different URI regular expression. When the HTTP request URI matches the regular expression, the rule is used. If the URI regular expression is empty, the default rule is used. Using multiple rules can help us make different sampling configurations for different requests.

Each rule has three parameters to determine if sampling is needed:

Minimal Request Duration (ms): requests with a response time exceeding the specified time will be sampled.
Sampling response status code between 400 and 499: all status codes in the range [400-499) will be sampled.
Sampling response status code between 500 and 599: all status codes in the range [500-599) will be sampled.

Once the sampling configuration is complete, we can create the task.

Figure 14: Create network profiling task page.

Done!

After a few seconds, you will see the process topology appear on the right side of the page.

When you click on the line between processes, you can view the data between the two processes, which is divided into three tabs:

TCP: displays TCP-related metrics.
HTTP/1.x: displays metrics in the HTTP 1 protocol.
HTTP Requests: displays the analyzed request and saves it to a list according to the sampling rule.

Figure 16: TCP metrics in a network profiling task.

Figure 17: HTTP/1.x metrics in a network profiling task.

Figure 18: HTTP sampled requests in a network profiling task.

Conclusion

In this article, we detailed the overview of how to analyze the Layer 7 HTTP/1.x protocol in network analysis, and how to associate it with existing trace systems. This allows us to extend the scope of data we can observe from just user space to also include kernel-space data.

In the future, we will delve further into the analysis of kernel data, such as collecting information on TCP packet size, transmission frequency, network card, and help on enhancing distributed tracing from another perspective.

Additional Resources

Blog: Boost Root Cause Analysis Quickly With SkyWalking’s New Trace-Metrics Association Feature

Mon, 19 Dec 2022 00:00:00 +0000

Observability for modern distributed applications work is critical for understanding how they behave under a variety of conditions and for troubleshooting and resolving issues when they arise. Traces, metrics, and logs are regarded as fundamental parts of the observability stack. Traces are the footprints of distributed system executions, meanwhile, metrics measure system performance with numbers in the timeline. Essentially, they measure the performance from two dimensions. Being able to quickly visualize the connection between traces and corresponding metrics makes it possible to quickly diagnose which process flows are correlated to potentially pathological behavior. This powerful new capability is now available in SkyWalking 9.3.0.

The SkyWalking project started only with tracing, with a focus on 100% sampling-based metrics and topology analysis since 2018. When users face anomaly trends of time-series metrics, like a peak on the line chart, or histogram shows a larger gap between p95 and p95, the immediate question is, why is this happening? One of SkyWalking’s latest features, the trace-metric association, makes it much easier to answer that question and to address the root cause.

How Are Metrics Generated?

SkyWalking provides three ways to calculate metrics:

Metrics built from trace spans, depending on the span’s layer, kind, and tags.
Metrics extracted from logs—a kind of keyword and tags-based metrics extraction.
Metrics reported from mature and mainstream metrics/meter systems, such as OpenTelemetry, Prometheus, and Zabbix.

Tracing tracks the processes of requests between an application’s services. Most systems that generate traffic and performance-related metrics also generate tracing data, either from server-side trace-based aggregations or through client SDKs.

Use SkyWalking to Reduce the Traditional Cost of Trace Indexing

Tracing data and visualization are critical troubleshooting tools for both developers and operators alike because of how helpful they are in locating issue boundaries. But, because it has traditionally been difficult to find associations between metrics and traces, teams have added increasingly more tags into the spans, and search through various combinations. This trend of increased instrumentation and searching has required increased infrastructure investment to support this kind of search. SkyWalking’s metrics and tracing association capabilities can help reduce the cost of indexing and searching that data.

Find the Associated Trace

When looking for association between metrics and traces, the kind of metrics we’re dealing with determines their relationships to traces. Let’s review the standard request rate, error, and duration (RED) metrics to see how it works.

Success Rate Metrics

The success rate is determined by the return code, RPC response code, or exceptions of the process. When the success rate decreases, looking for errors in the traces of this service or pod are the first place to look to find clues.

Figure 1: The success rate graph from SkyWalking’s 9.3.0 dashboard with the option to view related traces at a particular time.

Drilling down from the peak of the success rate, SkyWalking lists all traces and their error status that were collected in this particular minute (Figure 2):

Figure 2: SkyWalking shows related traces with an error status.

Requests to /test can be located from the trace, and the span’s tag indicates a 404 response code of the HTTP request.

Figure 3: A detail view of a request to http://frontend/test showing that the URI doesn’t exist.

By looking at the trace data, it becomes immediately clear that the drop in success rate is caused by requests to a nonexistent URI.

Average Response Time

The average response time metric provides a general overview of service performance. When average response time is unstable, this usually means that the system is facing serious performance impacts.

Figure 4: SkyWalking’s query UI for searching for related traces showing traces for requests that exceed a particular duration threshold.

When you drill down from this metric, this query condition (Figure 4) will reveal the slowest traces of the service in this specific minute. Notice, at least 168ms is added as a condition automatically, to avoid scanning a large number of rows in the Database.

Apdex

Apdex—the Application Performance Index—is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times (Figure 5). The response time is measured from an asset request to completed delivery back to the requestor.

Figure 5: The Apdex formula

A user defines a response time tolerating threshold T. All responses handled in T or less time satisfy the user.

For example, if T is 1.2 seconds and a response completes in 0.5 seconds, then the user is satisfied. All responses greater than 1.2 seconds dissatisfy the user. Responses greater than 4.8 seconds frustrate the user.

When the Apdex score decreases, we need to find related traces from two perspectives: slow traces and error status traces. SkyWalking’s new related tracing features offers a quick way to view both (Figure 6) directly from the Apdex graph.

Figure 6: Show slow trace and error status traces from the Apdex graph

Service Response Time

Percentile MetricThe percentile graph (Figure 7) provides p50, p75, p90, p95, and p99 latency ranks to measure the long-tail issues of service performance.

Figure 7: The service response time percentile graph helps to highlight long-tail issues of service performance.

This percentile graph shows a typical long-tail issue. P99 latency is four times slower than the P95. When we use the association, we see the traces with latency between P95 - P99 and P99 - Infinity.

The traces of requests causing this kind of long-tail phenomena are automatically listing from there.

Figure 8: Query parameters to search for traces based on latency.

Are More Associations Available?

SkyWalking provides more than just associations between between traces and metrics to help you find possible causal relationships and to avoid looking for the proverbial needle in a haystack.

Currently, SkyWalking 9.3.0 offers two more associations: metric-to-metric associations and event-to-metric associations.

Metric-to-metric Associations

There are dozens of metrics on the dashboard—which is great for getting a complete picture of application behavior. During a typical performance issue, the peaks of multiple metrics are affected simultaneously. But, trying to correlate peaks across all of these graphs can be difficult…

Now in SkyWalking 9.3.0, when you click the peak of one graph, the pop-out box lets you see associated metrics.

Figure 9: SkyWalking’s option to view associated metrics.

When you choose that option, all associated metrics graphs will show axis pointers (the dotted vertical lines) in all associated graphs like in Figure 10. This makes it easier to correlate the peaks in different graphs with each other. Often, these correlated peaks with have the same root cause.

Figure 10: Axis pointers (vertical dotted lines) show associations between peaks across multiple metrics graphs.

Event-to-Metric Associations

SkyWalking provides the event concept to associate possible service performance impacted by the infrastructure, such as new deployment even from k8s. Or, the anomaly had been detected by alerting or integrated AIOps engine.

The event to metrics association is also automatically, it could cover the time range of the event on the metric graphs(blue areas). If the area of event and peaks are matched, most likely this event covered this anomaly.

Figure 11: SkyWalking’s event to metric association view.

SkyWalking Makes it Easier and Faster to Find Root Causes

SkyWalking now makes it easy to find associations between metrics, events, and traces, ultimately making it possible to identify root causes and fix problems fast. The associations we’ve discussed in this article are available out-of-box in the SkyWalking 9.3.0 release.

Figure 12: Just click on the dots to see related traces and metrics associations.

Click the dots on any metric graph, and you will see a View Related Traces item pop-out if this metric has logical mapping traces.

Conclusion

In this blog, we took a look at the newly-added association feature between metrics and traces. With this new visualization, it’s now much easier to find key traces to identify root cause of issues.Associations in SkyWalking can go even deeper. Associations from metrics to traces is not the end of diagnosing system bottleneck. In the next post, we will introduce an eBPF powered trace enhancement where you’ll be able to see HTTP request and response details associated with tracing spans from network profiling. Stay tuned.

Blog: Apache ShenYu(incubating) plugin implementation principles and observability practices

Sun, 08 May 2022 00:00:00 +0000

Content

1. Introduction of SkyWalking and ShenYu

1.1 SkyWalking

SkyWalking is an Application Performance Monitoring (APM) and Observability Analysis Platform (OAP) for microservices, distributed systems, and cloud natives, Has powerful features that provide a multi-dimensional means of application performance analysis, including distributed topology diagrams, application performance metrics, distributed link tracing, log correlation analysis and alerts. Also has a very rich ecology. Widely used in various companies and open source projects.

1.2 Apache ShenYu (incubating)

Apache ShenYu (incubating) High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes. Rich protocol support: Http, Spring Cloud, gRPC, Dubbo, SOFARPC, Motan, Tars, etc.

2. Apache ShenYu plugin implementation principle

ShenYu’s asynchrony is a little different from previous exposure to asynchrony, it is a full-link asynchrony, the execution of each plug-in is asynchronous, and thread switching is not a single fixed situation (and the individual plug-in implementation is related). The gateway initiates service calls of various protocol types, and the existing SkyWalking plugins create ExitSpan (synchronous or asynchronous) when they initiate service calls. The gateway receives the request and creates an asynchronous EntrySpan. The asynchronous EntrySpan needs to be concatenated with the synchronous or asynchronous ExitSpan, otherwise the link will be broken.

There are 2 types of tandem solutions：

Snapshot Delivery:
Pass the snapshot after creating the EntrySpan to the thread that created the ExitSpan in some way.
Currently this approach is used in the asynchronous WebClient plugin, which can receive asynchronous snapshots. shenYu proxy Http service or SpringCloud service is to achieve span concatenation through snapshot passing.
LocalSpan transit:
Other RPC class plugins do not receive snapshots for concatenation like Asynchronous WebClient. Although you can modify other RPC plugins to receive snapshots for concatenation, it is not recommended or necessary to do so. This can be achieved by creating a LocalSpan in the thread where the ExitSpan is created, and then connecting the asynchronous EntrySpan and LocalSpan by snapshot passing. This can be done without changing the original plugin code.

The span connection is shown below:

You may ask if it is possible to create LocalSpan inside a generic plugin, instead of creating one separately for ShenYu RPC plugin? The answer is no, because you need to ensure that LocalSpan and ExitSpan are in the same thread, and ShenYu is fully linked asynchronously. The code to create LocalSpan is reused in the implementation.

3. Adding generalized call tracking to the gRPC plugin and keeping it compatible

The existing SkyWalking gRPC plugin only supports calls initiated by way of stubs. For the gateway there is no proto file, the gateway takes generalized calls (not through stubs), so tracing RPC requests, you will find that the link will break at the gateway node. In this case, it is necessary to make the gRPC plugin support generalized calls, while at the same time needing to remain compatible and not affect the original tracing method. This is achieved by determining whether the request parameter is a DynamicMessage, and if it is not, then the original tracing logic through the stub is used. If not, then the original tracing logic via stubs is used, and if not, then the generalized call tracing logic is used. The other compatibility is the difference between the old and new versions of gRPC, as well as the compatibility of various cases of obtaining server-side IP, for those interested in the source code.

4. ShenYu Gateway Observability Practice

The above explains the principle of SkyWalking ShenYu plug-in implementation, the following deployment application to see the effect. SkyWalking powerful, in addition to the link tracking requires the development of plug-ins, other powerful features out of the box. Here only describe the link tracking and application performance analysis part, if you want to experience the power of SkyWalking features, please refer to the SkyWalking official documentation.
Version description:

skywalking-java: 8.11.0-SNAPSHOT source code build. Note: The shenyu plugin will be released in version 8.11.0, and will probably release it initially in May or June. the Java agent is in the regular release phase.
skywalking: 9.0.0 V9 version

Usage instructions:
SkyWalking is designed to be very easy to use. Please refer to the official documentation for configuring and activating the shenyu plugin.

4.1 Sending requests to the gateway

Initiate various service requests to the gateway via the postman client or other means.

4.2 Request Topology Diagram

4.3 Request Trace (in the case of gRPC)

Normal Trace：

Abnormal Trace：

Click on the link node to see the corresponding node information and exception information

Service Provider Span

Gateway request span

4.4 Service Metrics Monitoring

4.5 Gateway background metrics monitoring

Database Monitoring:

Thread pool and connection pool monitoring:

4.6 JVM Monitoring

4.7 Endpoint Analysis

4.8 Exception log and exception link analysis

See official documentation for log configuration

Log monitoring

Distributed link trace details corresponding to exception logs

5. Summary

SkyWalking has very comprehensive support for metrics, link tracing, and logging in observability, and is powerful, easy to use, and designed for large distributed systems, microservices, cloud-native, container architectures, and has a rich ecosystem. Using SkyWalking to provide powerful observability support for Apache ShenYu (incubating) gives ShenYu a boost. Finally, if you are interested in high-performance responsive gateways, you can follow Apache ShenYu (incubating). Also, thanks to SkyWalking such an excellent open source software to the industry contributions.

Blog: SkyWalking 8.4 provides infrastructure monitoring

Mon, 08 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

Background

Apache SkyWalking– the APM tool for distributed systems– has historically focused on providing observability around tracing and metrics, but service performance is often affected by the host. The newest release, SkyWalking 8.4.0, introduces a new feature for monitoring virtual machines. Users can easily detect possible problems from the dashboard– for example, when CPU usage is overloaded, when there’s not enough memory or disk space, or when the network status is unhealthy, etc.

How it works

SkyWalking leverages Prometheus and OpenTelemetry for collecting metrics data as we did for Istio control panel metrics; Prometheus is mature and widely used, and we expect to see increased adoption of the new CNCF project, OpenTelemetry. The SkyWalking OAP Server receives these metrics data of OpenCensus format from OpenTelemetry. The process is as follows:

Prometheus Node Exporter collects metrics data from the VMs.
OpenTelemetry Collector fetches metrics from Node Exporters via Prometheus Receiver, and pushes metrics to SkyWalking OAP Server via the OpenCensus GRPC Exporter.
The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results. The expression rules are in /config/otel-oc-rules/vm.yaml.
We can now see the data on the SkyWalking WebUI dashboard.

What to monitor

SkyWalking provides default monitoring metrics including:

CPU Usage (%)
Memory RAM Usage (MB)
Memory Swap Usage (MB)
CPU Average Used
CPU Load
Memory RAM (total/available/used MB)
Memory Swap (total/free MB)
File System Mount point Usage (%)
Disk R/W (KB/s)
Network Bandwidth Usage (receive/transmit KB/s)
Network Status (tcp_curr_estab/tcp_tw/tcp_alloc/sockets_used/udp_inuse)
File fd Allocated

The following is how it looks when we monitor Linux:

How to use

To enable this feature, we need to install Prometheus Node Exporter and OpenTelemetry Collector and activate the VM monitoring rules in SkyWalking OAP Server.

Install Prometheus Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.0.1.linux-amd64.tar.gz
cd node_exporter-1.0.1.linux-amd64
./node_exporter

In linux Node Exporter exposes metrics on port 9100 by default. When it is running, we can get the metrics from the /metrics endpoint. Use a web browser or command curl to verify.

curl http://localhost:9100/metrics

We should see all the metrics from the output like:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.7777e-05
go_gc_duration_seconds{quantile="0.25"} 0.000113756
go_gc_duration_seconds{quantile="0.5"} 0.000127199
go_gc_duration_seconds{quantile="0.75"} 0.000147778
go_gc_duration_seconds{quantile="1"} 0.000371894
go_gc_duration_seconds_sum 0.292994058
go_gc_duration_seconds_count 2029
...

Note: We only need to install Node Exporter, rather than Prometheus server. If you want to get more information about Prometheus Node Exporter see: https://prometheus.io/docs/guides/node-exporter/

Install OpenTelemetry Collector

We can quickly install a OpenTelemetry Collector instance by using docker-compose with the following steps:

Create a directory to store the configuration files, like /usr/local/otel.
Create docker-compose.yaml and otel-collector-config.yaml in this directory represented below:

docker-compose.yaml

version: "2"
services:
  # Collector
  otel-collector:
    # Specify the image to start the container from
    image: otel/opentelemetry-collector:0.19.0
    # Set the  otel-collector configfile 
    command: ["--config=/etc/otel-collector-config.yaml"]
    # Mapping the configfile to host directory
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "13133:13133" # health_check extension
      - "55678"       # OpenCensus receiver

otel-collector-config.yaml

extensions:
  health_check:
# A receiver is how data gets into the OpenTelemetry Collector
receivers:
  # Set Prometheus Receiver to collects metrics from targets
  # It’s supports the full set of Prometheus configuration
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs: 
              # Replace the IP to your VMs‘s IP which has installed Node Exporter
            - targets: [ 'vm1:9100' ] 
            - targets: [ 'vm2:9100' ]
            - targets: [ ‘vm3:9100' ]

processors:
  batch:

# An exporter is how data gets sent to different systems/back-ends
exporters:
  # Exports metrics via gRPC using OpenCensus format
  opencensus:
    endpoint: "docker.for.mac.host.internal:11800" # The OAP Server address
    insecure: true
  logging:
    logLevel: debug

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [logging, opencensus]

  extensions: [health_check]

In this directory use command docker-compose to start up the container:

docker-compose up -d

After the container is up and running, you should see metrics already exported in the logs:

...
Metric #165
Descriptor:
-> Name: node_network_receive_compressed_total
-> Description: Network device statistic receive_compressed.
-> Unit:
-> DataType: DoubleSum
-> IsMonotonic: true
-> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
DoubleDataPoints #0
Data point labels:
-> device: ens4
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
DoubleDataPoints #1
Data point labels:
-> device: lo
StartTime: 1612234754364000000
Timestamp: 1612235563448000000
Value: 0.000000
...

If you want to get more information about OpenTelemetry Collector see: https://opentelemetry.io/docs/collector/

Set up SkyWalking OAP Server

To activate the oc handler and vm relevant rules, set your environment variables:

SW_OTEL_RECEIVER=default
SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm

Note: If there are other rules already activated , you can add vm with use , as a separator.

SW_OTEL_RECEIVER_ENABLED_OC_RULES=vm,oap

Start the SkyWalking OAP Server.

Done!

After all of the above steps are completed, check out the SkyWalking WebUI. Dashboard VM provides the default metrics of all observed virtual machines. Note: Clear the browser local cache if you used it to access deployments of previous SkyWalking versions.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Apache SkyWalking – Metrics

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Data path

Two platforms, two independent Layers and dashboards

Asymmetric metric semantics

What the SDK does

What OAP needs

Compatibility

What’s next

Blog: Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Query SkyWalking and Zipkin Traces with TraceQL and Visualize in Grafana

Architecture Overview

Limitations and Supported TraceQL Features

Trace Format Conversion

SkyWalking Native Trace

Trace ID Encoding

Span Kind Mapping

Status Mapping

SpanAttachedEvents

Zipkin Trace

Span Kind Mapping

Status Mapping

Endpoint and Annotation Mapping

How to Enable TraceQL

Step 1: Enable the TraceQL Module

Step 2: Enable the Zipkin Receiver (for Zipkin traces only)

Full Configuration Reference

Configuring Grafana Tempo Data Source

Context Paths

Setting Up the SkyWalking Data Source

Setting Up the Zipkin Data Source

Configuring Trace List Result Tags

Building a Trace Dashboard in Grafana

SkyWalking Native Trace Dashboard

Step 1: Explore and Save

Step 2: Configure Variables

Step 3: Add a Trace Panel

Step 4: View Trace Details

Zipkin Trace Dashboard

Summary

Blog: Monitoring LLM Applications with SkyWalking 10.4: Insights into Performance and Cost

The Problem: As Applications “Consume” LLMs, Monitoring Leaves a Blind Spot

1. The “Black Box” of Cost and Performance: Is the Expensive Model Worth It?

2. The Vanishing “Golden Timeout”

3. The Overlooked Experience Killer: TTFT

Virtual GenAI Observability

How It Works

Installation & Configuration

Requirements

Semantic Conventions & Compatibility

SkyWalking Java Agent

OTLP / Zipkin Probes

GenAI Estimated Cost Configuration

Overview

Configuration Structure

Top-level Fields

Provider Definition

Model Definition

Model Matching Mechanism

Provider-Level Prefix Matching

Model-level Longest-Prefix Matching

Model Aliases

Custom Configuration

Adding a New Provider

Main Metrics

1.Provider Level

2. Model Level

Recommended Usage Scenarios

Blog: Monitoring Flink with SkyWalking

Background

Development

Preparation

Process

Data flow

Configuration

docker-compose

OpenTelemetry-collector

Metrics Definition

Cluster Metrics

TaskManager Metrics