Apache SkyWalking – Community

Blog: Monitoring WeChat and Alipay Mini Programs with SkyWalking

Thu, 30 Apr 2026 00:00:00 +0000

Mini programs are a major part of the mobile experience in China, but the open-source observability ecosystem has long focused on web browsers and native apps. SkyWalking already covers browser (client-js), iOS, and the server side; mini programs and Android were the remaining gaps. With SkyAPM/mini-program-monitor joining the SkyWalking ecosystem, the mini-program half of that gap is closed — one SDK supports both WeChat and Alipay, and the matching OAP-side component IDs, MAL rules, and UI templates are merged on main and will ship with 10.5.0.

This post is for teams that already run a SkyWalking backend and want to bring their mini programs into the same observability stack. The interesting parts aren’t that the project exists — they are how the data flows from a mini program to a SkyWalking dashboard, how the two platforms coexist, and what design trade-offs you should know about before rolling this out.

Data path

The SDK uses two protocols:

OTLP HTTP (error logs, performance metrics, request metrics) → OAP /v1/logs, /v1/metrics
SkyWalking native (distributed tracing segments, optional) → OAP /v3/segments

Why not a single protocol? OTLP already covers logs and metrics, so there’s no point reinventing native endpoints for those. But for tracing, OAP’s native SegmentObject maps more cleanly onto SkyWalking’s trace model, and sw8 header propagation to the backend works without any conversion. So traces go native, everything else goes OTLP, and neither side has to translate.

OTLP defaults to protobuf; JSON is available for debugging. The SDK has zero runtime dependencies.

Two platforms, two independent Layers and dashboards

Many teams maintain a WeChat mini program and an Alipay mini program against a shared backend. Rather than collapsing them into a single tagged service, the design promotes each platform to its own Layer — WECHAT_MINI_PROGRAM and ALIPAY_MINI_PROGRAM — with its own dashboard set. The SDK tags every signal with a resource attribute miniprogram.platform = wechat | alipay and assigns each platform its own component ID (WeChat = 10002, Alipay = 10003).

On the OAP side, the MAL rule’s filter routes data into the right Layer at ingest:

metricPrefix: meter_wechat_mp
filter: "{ tags -> tags.miniprogram_platform == 'wechat' }"

The Alipay rule mirrors this with 'alipay'. The two rules are mutually exclusive — no double counting — and produce distinct metric prefixes (meter_wechat_mp_* vs meter_alipay_mp_*) that feed each Layer’s dashboards. Even when both platforms use the same service.name (e.g. mini-program-demo), the UI exposes two completely separate entry points.

Asymmetric metric semantics

This is the design choice I want to highlight. WeChat’s base library exposes PerformanceObserver, which gives you renderer-authoritative timings: app launch, first render, route navigation, script execution, sub-package load — all real measurements. Alipay’s base library doesn’t offer an equivalent, so the SDK falls back to lifecycle hooks: the App.onLaunch → App.onShow delta is used as an approximation of launch time, and renderer-level timings simply aren’t available.

So the two MAL rule sets are deliberately not the same:

WeChat: app_launch_duration, first_render_duration, route_duration, script_duration, package_load_duration, request_duration_percentile, request_cpm
Alipay: app_launch_duration, first_render_duration, request_duration_percentile, request_cpm

The Alipay app_launch_duration is a lifecycle approximation and is not directly comparable to WeChat’s renderer timing — the dashboard tooltip says so explicitly. Putting the two numbers side by side is comparing two different measurement definitions.

What the SDK does

Four signals:

Errors — JS exceptions, unhandled promise rejections, and pageNotFound go out as OTLP logs, following the OTel exception.* semantic conventions (exception.type, exception.stacktrace). Anything downstream that speaks OTLP — SkyWalking, OTel Collector, Grafana — recognizes them.
Performance — the metrics listed above. OTLP gauge.
Requests — wx.request / my.request / downloadFile / uploadFile are reported as OTLP delta histograms, one batch per flushInterval (default 5s). The le bucket labels are already in milliseconds, and the MAL rule explicitly declares MILLISECONDS to disable the default SECONDS→MS rescale. Failed requests (4xx / 5xx / timeout) additionally emit an error log so you can pivot from a dashboard to a concrete failure.
Tracing (opt-in) — when enabled, outbound requests get sw8 header injection, and the resulting segments stitch together with backend traces into one end-to-end view. Trace data goes out as SkyWalking SegmentObject, not OTLP traces.

Two reliability and cardinality details worth calling out:

Persisting events on app hide. Mini programs get killed by the framework after some time in background, and weak networks make in-flight events easy to lose. The SDK writes unsent events to wx.setStorage / my.setStorage on onAppHide and restores them on the next launch.

Avoiding cardinality explosions. Set serviceInstance to the app version (e.g. 1.4.2), not a device ID — at a million DAU the device-ID dimension blows up the OAP instance index. For request paths, the SDK exposes urlGroupRules regex patterns to fold parameterized URLs like /api/user/12345 into /api/user/{id} so the endpoint dimension doesn’t blow up either.

What OAP needs

If you’re on main or a release ≥ 10.5.0, the following are already shipped:

config/component-libraries.yml registers WeChat-MiniProgram: 10002 and AliPay-MiniProgram: 10003
config/otel-rules/miniprogram/ holds four MAL rules — service-scoped and instance-scoped for each platform
config/ui-initialized-templates/wechat_mini_program/ and alipay_mini_program/ carry root / service / instance / endpoint dashboards
config/ui-initialized-templates/menu.yaml registers both layers under the Mobile menu group

The only thing left is enabling the OTel receiver and giving the SDK an OTLP HTTP port it can reach. SkyWalking OAP binds its OTLP HTTP handler onto the receiver-sharing-server port, and that port defaults to 0 — meaning it’s folded into the core REST port (12800). If you want the SDK to use the standard OTLP HTTP port 4318, set the sharing port to 4318:

docker run -d --name sw-oap \
  -p 11800:11800 -p 12800:12800 -p 4318:4318 \
  -e SW_STORAGE=banyandb \
  -e SW_STORAGE_BANYANDB_TARGETS=banyandb:17912 \
  -e SW_OTEL_RECEIVER=default \
  -e SW_RECEIVER_SHARING_REST_PORT=4318 \
  apache/skywalking-oap-server:latest

All receivers (OTLP, native segment, browser perf, log report) move to 4318 together, while GraphQL stays on 12800 for the UI.

Minimal SDK config:

import MiniProgramMonitor from 'mini-program-monitor';

MiniProgramMonitor.init({
  service: 'mini-program-demo',
  serviceInstance: '1.4.2',          // Recommended: app version
  collector: 'http://your-oap:4318',
  enable: {
    error: true,
    perf: true,
    request: true,
    tracing: false,                  // Off by default; enable as needed
  },
});

WeChat and Alipay use the same config — the SDK detects the platform at runtime and tags the data accordingly.

Compatibility

WeChat base library ≥ 2.11
Alipay base library ≥ 2.0
Apache SkyWalking OAP main or ≥ 10.5.0, with the OTLP HTTP receiver enabled
Any other OTLP-compatible backend (OpenTelemetry Collector, Grafana, etc.) also works, but you won’t get the SkyWalking-specific cross-platform dashboards

What’s next

To get involved, head over to SkyAPM/mini-program-monitor and open an issue or PR. The repo also ships a make preview target that boots OAP, the UI, and both platform simulators locally — handy if you want to play with it end-to-end.

Android end-user experience monitoring is still a gap in the SkyWalking ecosystem; contributors interested in closing that one are very welcome.

Blog: Agentic Vibe Coding in a Mature OSS Project: What Worked, What Didn't

Sun, 08 Mar 2026 00:00:00 +0000

Most “vibe coding” stories start with a greenfield project. This one doesn’t.

Apache SkyWalking is a 9-year-old observability platform with hundreds of production deployments, a complex DSL stack, and an external API surface that users have built dashboards, alerting rules, and automation scripts against. When I decided to replace the core scripting engine — purging the Groovy runtime from four DSL compilers — the constraint wasn’t “can AI write the code?” It was: “can AI write the code without breaking anything for existing users?”

The answer turned out to be yes — ~77,000 lines changed across 10 major PRs in about 5 weeks — but only because the AI was tightly guided by a human who understood the project’s architecture, its compatibility contracts, and its users. This post is about the methodology: what worked, what didn’t, and what mature open-source maintainers should know before handing their codebase to AI agents.

The Project in Brief

The task was to replace SkyWalking’s Groovy-based scripting engines (MAL, LAL, Hierarchy) with a unified ANTLR4 + Javassist bytecode compilation pipeline, matching the architecture already proven by the OAL compiler. The internal tech stack was completely overhauled; the external interface had to remain identical.

Beyond the compiler rewrites, the scope included a new queue infrastructure (threads dropped from 36 to 15), virtual thread support for JDK 25+, and E2E test modernization. By conventional estimates, this was 5-8 months of senior engineer work.

For the full technical details on the compiler architecture, see the Groovy elimination discussion.

What is Agentic Vibe Coding?

“Vibe coding” — a term coined by Andrej Karpathy — describes a style of programming where you describe intent and let AI write the code. It’s powerful for prototyping, but on its own, it’s risky for production systems.

Agentic vibe coding takes this further: instead of a single AI autocomplete, you orchestrate multiple AI agents — each with different strengths — under your architectural direction, with automated tests as the safety net. In my workflow:

Claude Code (plan mode): Primary coding agent. Plan mode lets me review the approach before any code is generated. This is critical for architectural decisions — I steer the design, Claude handles the implementation.
Gemini: Code review, concurrency analysis, and verification reports. Gemini reviewed every major PR for thread-safety, feature parity, and edge cases.
Codex: Autonomous task execution for well-defined, bounded work items.

The key insight: AI writes the code, but the architect owns the design. Without deep domain knowledge of SkyWalking’s internals, no AI could have planned these changes. Without AI, I couldn’t have executed them in 5 weeks.

How TDD Made AI Coding Safe

The reason I could move this fast without breaking things comes down to one principle: never let AI code without a test harness.

My workflow for each major change:

Plan mode first: Describe the goal to Claude, review the plan, iterate on architecture before any code is written.
Write the test contract: Define what “correct” means — for the compiler rewrites, this meant cross-version comparison tests that run every expression through both the old and new engines, asserting identical results across 1,290+ expressions.
Let AI implement: With the test contract in place, Claude can write thousands of lines of implementation code. If it’s wrong, the tests catch it immediately.
E2E as the final gate: Every PR must pass the full E2E test suite — Docker-based integration tests that boot the entire server with real storage backends.
AI code review: Gemini reviewed each PR for concurrency issues, thread-safety, and feature parity — catching things that unit tests alone wouldn’t find.

This is the opposite of “hope it works” vibe coding. The AI writes fast, the tests verify fast, and I steer the architecture. The feedback loop is tight enough that I can iterate on complex compiler code in minutes instead of days.

Lessons Learned

AI is a force multiplier, not a replacement. Before any AI agent wrote a single line, a human had to define the replacement solution: what gets replaced, how it gets replaced, and — critically — where the boundaries are. Which APIs could break? The internal compilation pipeline was fair game for a complete overhaul. Which APIs must stay aligned? Every external-facing DSL syntax, every YAML configuration key, every metrics name and tag structure had to remain byte-for-byte identical — because hundreds of deployed dashboards, alerting rules, and user scripts depend on them. Drawing these boundaries required deep knowledge of the codebase and its users. AI executed the plan at extraordinary speed, but the plan itself — the scope, the invariants, the compatibility contract — had to come from a human who understood the blast radius of every change.

Plan mode is non-negotiable for architectural work. Letting AI jump straight to code on a compiler rewrite would be a disaster. Plan mode’s strength is that it collects code context — scanning imports, tracing call chains, mapping class hierarchies — and uses that context to help me fill in implementation details I’d otherwise have to look up manually. But it can’t tell you the design principles. That direction had to come from me, stated clearly upfront, so the AI’s planning stayed on the right track instead of optimizing toward a locally reasonable but architecturally wrong solution.

Know when to hit ESC. Claude has a clear tendency to dive deep into solution code writing once it starts — and it won’t stop on its own when it encounters something that conflicts with the original plan’s concept. Instead of pausing to flag the conflict, it will push forward, improvising around the obstacle in ways that silently violate the design intent. I had to learn to watch for this: when Claude’s output started drifting from the plan, I’d manually cancel the task (ESC), call it off, identify where the plan and reality diverged, adjust the plan, and restart. This interrupt-replan cycle was a regular part of the workflow, not an exception. The architect has to stay in the loop — not just at planning time, but during execution — because AI agents don’t yet know when to stop and ask.

Spec-driven testing is necessary but not sufficient — the logic workflow matters more. It’s tempting to think that if you define the input/output spec clearly enough, AI can fill in the implementation and tests will catch any mistakes. I tried this. It doesn’t work for anything non-trivial. During the expression compiler rewrite, Claude would sometimes change code in unreasonable ways just to make the spec tests pass — the inputs went in, the expected outputs came out, and everything looked green. But the internal logic was wrong: inconsistent with the design patterns the rest of the codebase relied on, impossible to extend, or solving the specific test case through a hack rather than a general mechanism. A spec only checks what the code produces; it says nothing about how the code produces it. For a mature project, the “how” matters enormously — the solution needs to be consistent with the existing architecture, widely adoptable by contributors, and maintainable long-term. That’s why I needed cross-version testing and human review of the implementation path, not just the results.

Testing at two levels kept the rewrite honest. Cross-version testing was part of my design plan from the start — I architected the dual-path comparison framework so that every production DSL expression runs through both the old and new engines, asserting identical results across 1,290+ expressions. This gave me confidence no human review could match, and it was a deliberate planning decision: I knew AI-generated compiler code needed a mechanical proof of behavioral equivalence, not just eyeball review. On top of that, E2E tests served as the project’s existing infrastructure safety net — Docker-based integration tests that boot the entire server with real storage backends. Unit tests and cross-version tests verify logic in isolation; E2E tests verify the system actually works end-to-end. For infrastructure-level changes like queue replacement and thread model changes, E2E is the only gate that truly matters. Together, the two layers — designed-for-this-rewrite cross-version tests and pre-existing E2E infrastructure — caught different classes of bugs and made shipping with confidence possible.

Multiple AIs have different strengths. Claude excels at large-scale code generation with plan mode. Gemini is exceptional at logic review — it can mentally trace code branches with given input data, simulating execution without actually running the code. This is significant for reviewing AI-generated code: Gemini would walk through a generated compiler method step by step, flagging where a null check was missing or where a branch would produce wrong output for a specific edge case. Codex proved most valuable as a test reviewer and honesty checker. AI-generated code has a subtle failure mode: the coding agent can make wrong assumptions and then write tests that pass by setting expected values to match the wrong behavior — effectively bypassing the test safety net. Codex caught cases where Claude had set unreasonable expected values that happened to make tests green, masking logic errors that would have surfaced in production. Using all three as checks on each other was far more effective than relying on any single one.

The Mythical Man-Month still applies — and so does the Mythical Token-Month. Brooks taught us that a task requiring 12 person-months does not mean 12 people can finish it in one month. The same law applies to AI: you cannot simply throw more tokens, more agents, or more parallel sessions at a problem and expect it to converge faster. Communication costs, coordination overhead, requirements analysis, and conceptual integrity — these software engineering fundamentals do not disappear just because your workforce is artificial. Worse, when the direction is wrong — when there’s a conceptual error in the design or an unreasonable architectural choice — AI will not recognize it. It will charge down the wrong path at extraordinary speed, burning tokens furiously while trapped in a vortex of self-justification: patching code to make failing tests pass, adjusting expected values to match wrong behavior, adding workarounds on top of workarounds — each iteration making the codebase look more “complete” while drifting further from correctness. AI vibe coding cannot break out of this spiral on its own. Only a human who understands the domain can recognize “this is fundamentally wrong, stop,” discard the work, and redirect. Speed without direction is just expensive chaos.

The Bigger Picture

The agentic vibe coding approach worked because it combined AI’s speed with human architectural judgment and automated test discipline. It’s not magic — it’s engineering, accelerated.

Brooks also gave us “No Silver Bullet,” and its core distinction matters more than ever: software complexity comes in two kinds. Essential complexity comes from the problem itself — the domain semantics, the behavioral contracts, the concurrency invariants. No tool can eliminate this; it must be understood, modeled, and reasoned about by someone who knows the domain. Accidental complexity comes from the tools and implementation — boilerplate code, manual refactoring across hundreds of files, the mechanical work of translating a design into compilable source. This is exactly where AI excels. What made this project work was recognizing which complexity was which: I owned the essential complexity (architecture, API boundaries, correctness invariants), and AI demolished the accidental complexity (generating 77K lines of implementation, scaffolding test harnesses, rewriting repetitive patterns across dozens of config files). Confuse the two — let AI make essential decisions, or waste human time on accidental work — and you get the worst of both worlds.

Qian Xuesen(Tsien Hsue-shen)’s Engineering Cybernetics offers another lens that proved surprisingly relevant. His core framework — feedback, control, optimization — describes how to keep complex systems running toward their target. AI vibe coding at full speed is like a hypersonic missile: extraordinarily fast, but without a guidance system it just creates a bigger crater in the wrong place. The feedback loop in my workflow was the test harness — cross-version tests and E2E tests providing continuous signal on whether the system was still on course. Control was the human architect deciding when to intervene: reviewing plans before execution, hitting ESC when the direction drifted, choosing which AI to trust for which task. Optimization was iterative: each interrupt-replan cycle refined the approach, each Gemini review tightened the logic, each Codex audit caught assumptions the coding agent had smuggled past the tests. Without all three — feedback to detect deviation, control to correct course, optimization to converge — the speed of AI coding would be not an advantage but a liability. The faster the missile, the more precise the guidance must be.

For more details or to share your own experience with agentic coding on production systems, feel free to reach me on GitHub.

Blog: Apache SkyWalking 2025 in Review: Making BanyanDB Ready for Production

Thu, 01 Jan 2026 00:00:00 +0000

2025 was a very focused year for the Apache SkyWalking community: moving BanyanDB from “native storage” to a “production-ready default”, and making SkyWalking APM fully benefit from that foundation.

This post summarizes the key milestones, with an emphasis on BanyanDB.

Storage strategy: saying goodbye to H2

We started 2025 with a clear direction: the H2 storage option is permanently removed. This change reduced long-term maintenance burden and removed a storage choice that was not aligned with production and cloud-native deployments.

BanyanDB: from 0.8.0 foundations to 0.9.0 production features

BanyanDB 0.8.0 delivered the “day-2 operations” foundation that a default storage backend needs. The community put a lot of effort into making queries faster and more predictable (for example index_mode, numeric index types, and multiple query-path optimizations), while also making the system safer under real production pressure. Disk-usage thresholds and a query memory protector were added as guardrails, and the operational toolbox matured with snapshot/backup/restore utilities and improved metadata synchronization.

Just as importantly, 0.8.0 started filling in the missing pieces of a full platform: native property storage and lifecycle-related capabilities that later enabled stronger HA and stage-based deployment patterns.

BanyanDB 0.9.0 was the “production features” milestone. It introduced the Trace data model as a first-class citizen, which unlocked much deeper trace storage and query capabilities. On the reliability and scaling side, the release brought configurable replicas, liaison-side improvements (including load balancing and moving some TopN flow), and broader correctness work such as migrations, version compatibility checks, and access logs.

It also made long-term operations more cloud-friendly with backup/restore support for AWS S3, GCS, and Azure Blob Storage, and added authentication primitives needed in shared environments. In short, 0.9.0 is where BanyanDB clearly moved beyond a “fast storage engine” into a “production platform”.

SkyWalking APM: BanyanDB becomes the default path

With APM 10.2.0, the project made the strategic shift official: H2 was removed permanently, and BanyanDB 0.8.0 became the default path that SkyWalking invests in. A lot of the work here was not flashy, but essential — refining OAP behavior (group settings, index model changes, Progressive TTL, query limits, and more) so running BanyanDB in production felt stable and predictable.

With APM 10.3.0, SkyWalking and BanyanDB moved forward together: BanyanDB 0.9.0’s new trace model was adopted end-to-end, reducing inefficient query round-trips and enabling new query views that significantly lowered page latency. The integration also expanded into lifecycle-aware operations with hot/warm/cold stage configuration (including TTL and query support), and added BanyanDB self-monitoring through OAP and the UI — the kind of end-to-end polish that turns a storage backend into a truly native solution.

If you’d like this review to cover APM 10.4.x as well, please point me to the corresponding release content in this repo (I didn’t find an APM 10.4.0 release announcement in the current checkout).

Packaging and deployment ecosystem (Helm)

BanyanDB’s production readiness is not only server features — it also depends on deployment maturity.

Helm charts:
- SkyWalking Kubernetes Helm Chart 4.8.0 improved BanyanDB deployment defaults by updating the bundled BanyanDB Helm dependency, fixing an init-job volume-mount mismatch, and aligning OAP/UI images with the APM 10.3.0 line.
- BanyanDB Helm 0.4.0 added backup/restore sidecars and a default volume for property storage.
- BanyanDB Helm 0.5.0 introduced stage-aware patterns (hot/warm/cold), improved lifecycle-sidecar scheduling, moved liaison to StatefulSet, refined internal networking, and expanded configuration options.
- BanyanDB Helm 0.5.1 refined liaison configuration and fixed restore-init environment issues.
- BanyanDB Helm 0.5.3 fixed a liaison/data-node port issue.

The rest of the community: agents and tooling kept moving

While storage was the “main storyline”, the community shipped releases across agents, clients, and surrounding components throughout 2025.

Below is a consolidated view of the other releases, grouped by project, with the most important notes.

SkyWalking Java Agent
- 9.4.0: agent self-observability; async-profiler support; broader plugin improvements.
- 9.5.0: virtual thread executor plugin; compatibility and stability fixes; dependency upgrades.
SkyWalking Go
- 0.6.0: richer manual APIs (events/logs/metrics, set span error); goframev2 plugin; bug fixes including Redis cluster mode.
SkyWalking for NodeJS
- 0.8.0: Express 4/5 compatibility, keep-alive HTTP trace fix, and test/dependency maintenance.
SkyWalking Python
- 1.2.0: sampling service, sw_grpc plugin, async/profiling stability fixes, Python 3.13 support, and dropping Python 3.7.
SkyWalking PHP
- 1.0.0: reach 1.0; add PSR-3 log reporting; upgrade toolchain/dependencies.
SkyWalking Rust
- 0.9.0: migrate to Rust edition 2024 and upgrade dependencies.
- 0.10.0: Kafka client configuration refactor, rdkafka upgrade, CI maintenance.
SkyWalking Ruby
- 0.1.0: initialize agent core and e2e tests; add plugins for Sinatra, redis-rb, net-http, memcached, and Elasticsearch.
SkyWalking Client JS
- 1.0.0: add Core Web Vitals and static resource metrics; fix fetch/resource error handling; dependency and e2e/test improvements.
SkyWalking Satellite
- 1.3.0: support native eBPF Access Log protocol and async-profiler protocol; upgrade Go toolchain.
SkyWalking Eyes
- 0.7.0: improve installation/docs, respect gitignore behavior, upgrade Go, and simplify release steps.
- 0.8.0: add Elixir support and stronger dependency-license scanning (notably Ruby via Gemfile.lock), plus stability fixes.

Looking ahead: possible directions in 2026

2025 was about making BanyanDB ready for production. In 2026, the community is exploring the next set of improvements that could make the whole stack simpler to operate, more stable under stress, and easier to integrate into broader observability ecosystems.

Possible areas include:

BanyanDB: remove the etcd dependency: the direction under discussion is to move away from etcd (given ecosystem activity and maintenance concerns) and rely more on DNS-based discovery plus BanyanDB’s native property capabilities.
BanyanDB: stronger stability testing: more systematic testing, including chaos testing, to validate behavior under failures and noisy conditions.
BanyanDB: better observability export: introducing First Occurrence Data Collection (FODC) as a sidecar and proxy server to provide a unified stream of observability data to third-party systems.
SkyWalking APM: broader runtime and query capabilities: cold-stage data query support, a newer Java runtime (Java 25), and consideration of TraceQL protocol (Temper) support.

Closing

Thanks to everyone who contributed to SkyWalking in 2025. Every contribution is high-value — code, documentation, reviews, testing, issue triage, and operational experience — and each of them helped move the project forward.

We also want to say a special thank you to the countless end users across global companies. Many of the most valuable improvements don’t start from a pull request: they start from real-world use cases, performance investigations, production feedback, bug reports, and the patience to help us reproduce and validate fixes.

As another milestone, SkyWalking reached 968 GitHub contributors globally, and we expect the 1000th contributor milestone to arrive soon in 2026. But the community is much larger than the number suggests, and SkyWalking’s progress has always been driven by collaboration between contributors, adopters, and maintainers.

Apache SkyWalking was originally created by Sheng Wu as a personal project in May 2015. It would never have grown into what it is today without the whole community — and it will keep moving forward because of the community.

Blog: Improving Alert Accuracy with Dynamic Baselines

Mon, 24 Feb 2025 00:00:00 +0000

Background

Apache SkyWalking is an open-source application performance monitoring (APM) system that collects various data from business applications, including metrics, logs, and distributed tracing information, and visualizes them through its UI. It also allows users to configure alerting rules by setting threshold values for specific metrics in the configuration file. When a metric associated with a particular service exceeds the predefined threshold within a given period, an alert is triggered.

However, in real-world scenarios, traffic patterns and invocation behaviors vary across different time periods. For example, in a shopping system, the number of purchases is significantly lower during late-night hours compared to daytime. As a result, system metrics fluctuate within different ranges depending on the time of day. This makes it challenging to rely solely on static threshold values for accurate alerting.

Therefore, dynamically generating thresholds for each time period based on historical data becomes crucial.

Introduce SkyAPM SkyPredictor

Based on the above scenario, we developed the SkyAPM SkyPredictor project to fix this issue. SkyAPM SkyPredictor periodically collects data from SkyWalking and generates dynamic baselines. Meanwhile, SkyWalking queries from SkyPredictor to obtain predicted metric values for the recent period, enabling more precise and adaptive alerting.

NOTE: SkyWalking does not have a hard dependency on the SkyPredictor service. If SkyPredictor is not configured, no predicted values would be retrieved, and not cause any failures in SkyWalking. Additionally, you can use your own AI engine to build a custom prediction system. Simply implement the required protocol as outlined in the official documentation: https://skywalking.apache.org/docs/main/next/en/setup/ai-pipeline/metrics-baseline-integration/

Architecture diagram

As shown in the diagram, the process consists of two steps:

Data Collection & Prediction: The Predictor queries history metrics from SkyWalking’s OAP via its HTTP service. Then processes this data to generate dynamic predicted values for a future time period.
Baseline Query & Alerting: The OAP periodically sends queries to the Predictor to fetch the predicted dynamic baseline. Then evaluates the current metric values with prediction result using MQE. If the deviation exceeds a certain threshold, an alert is triggered.

Data Collection

The Predictor utilizes the following three APIs to query data:

Status API: Retrieves the TTL (Time-to-Live) of history data stored in OAP, helping to determine the available time range for exporting all history metrics.
Metadata API: Fetches the list of services within a specified Layer from OAP, providing insights into which services are generating data.
MQE API: Iterates through the required metrics and the list of services to fetch all history metrics values for each metric associated with each service.

These APIs collectively enable the Predictor to gather history metrics data, which is then used to compute dynamic baselines for future alerting.

Prediction

Once the Prediction service collects data from OAP, it proceeds with forecasting using the open-source Prophet library. The prediction process consists of the following steps:

Data Preparation: The collected metric data is split into multiple DataFrames, each corresponding to a unique combination of service + metric name.
Data Sufficiency Check: If a DataFrame contains less than two days (configurable) of data, the prediction is skipped. This is to ensure accuracy, as an insufficient data volume may lead to unreliable forecasts.
Forecasting: Using Prophet, the Predictor estimates the metric values for each hour over the next day (configurable).
Result Storage: The generated predictions are stored in local files, enabling querying from external services.

Predicted Value

The Prediction service supports calculating the following two types of values:

Predicted Value: Computes the expected metric value for the next hour based on history metrics data.
Prediction Range: Determines the possible upper and lower bounds for the metric in the next hour, representing its expected fluctuation range.

These values help establish a dynamic baseline, allowing the alerting system to account for natural variations while accurately detecting anomalies.

Baseline MQE with Alarm

In OAP, predicted values can be queried directly using an MQE within MQE operation. This operation enables retrieving forecasted values for a future time period.

Since SkyWalking’s alerting system already supports query through MQE expressions, users can configure alerts directly in the alerting configuration file using MQE.

For more details, please refer to the official documentation.

Impact of Data Collection on Prediction Accuracy

The Predictor service supports two different data collection and prediction granularity, each with its own trade-offs in accuracy and resource consumption.

Minute Level: Collects minutes level metrics data.
1. More effective for metrics with high fluctuations, as it captures finer details.
2. Consumes more resources (OAP, DB CPU and System Load resources, Predictor CPU and Memory resources).
3. Alerts are configured based on current value comparisons.
Hour Level: Collects hourly metrics data.
1. Less resource-intensive compared to minute-level collection.
2. Less data volume, resource, and processing cost.
3. Alerts are configured based on predicted range values.

Granularity	Data Fluctuation	Data Volume	Current Value Prediction Accuracy	Range Prediction Accuracy	Best Use Case
Minute	Higher fluctuation	Large	Less accurate	More accurate	Ideal for highly fluctuating metrics, using range-based alerting rules
Hour	Lower fluctuation	Small	More accurate	Relatively accurate	Suitable for stable metrics, using current value-based predictions

Choosing the appropriate granularity depends on the nature of the metric and the desired alerting method. For metrics with high volatility, minute-level collection provides better accuracy when using range-based alerts. For stable metrics, hourly aggregation is sufficient and allows for efficient predictions using current-value comparisons.

Predict use Hourly level by default.

OAP and Predictor Scheduling & Caching

Both SkyWalking OAP and SkyAPM Predictor implement caching strategies to prevent excessive execution and optimize resource usage.

By default, Predictor runs at 00:10, 08:10, and 16:10 every day. It forecasts the next 24 hours and stores the results locally. Updating predictions every 8 hours balances resource efficiency and real-time accuracy. The 10-minute delay (instead of running at exactly 00:00, 08:00, etc.) ensures historical data is fully written to the database before querying.

OAP queries Predictor for all required predicted metrics of a single service. The query covers a ±24-hour time range from the current moment. Results are cached for one hour to reduce redundant queries and improve efficiency.

These mechanisms ensure that predictions remain up-to-date, while minimizing unnecessary processing and system load.

Demo

In this section, I will demonstrate how to preview the predicted values of a metric by deploying a SkyWalking cluster along with the Predictor service in a Kubernetes cluster. This hands-on example will help you understand how to use these components effectively.

Deploy SkyWalking Showcase

SkyWalking Showcase contains a complete set of example services and can be monitored using SkyWalking. For more information, please check the official documentation.

In this demo, we only deploy the predictor service, SkyWalking OAP, and UI.

export FEATURE_FLAGS=single-node,banyandb,baseline
make deploy.kubernetes

Import History Data

Since a newly deployed cluster does not contain history data, I have created a Python script to simulate data. This allows the Predictor service to import data and generate baseline predictions for a future period.

Before importing data, you must expose the 11800 port of the OAP service in your Kubernetes cluster. You can achieve this using kubectl by running the following command:

kubectl port-forward -n skywalking-showcase   service/demo-oap 11800:11800

Then, you can download and run the demo script using the following command:

# clone and get into the demo repository
git clone https://github.com/mrproliu/SkyPredictorDemo && cd SkyPredictorDemo
# installing dependencies
make install
# import data(7 days)
python3 -m client.generate localhost:11800 7

Finally, you can see the output in the console: Metrics send success!.

Prediction metrics

Since the Predictor service runs based on a cron schedule, it does not automatically execute immediately after data import. To force it to collect data and perform a prediction, you can manually delete the Predictor pod, prompting Kubernetes to restart it:

kubectl delete pod -n skywalking-showcase $(kubectl get pods -n skywalking-showcase --no-headers -o custom-columns=":metadata.name" | grep "skywalking-predictor")

Once the Predictor pod restarts, you can check its logs to confirm that the prediction process has been completed.

Predicted for e2e-test-dest-service of service_xxx to xxxx-xx-xx xx:xx:xx.

View in SkyWalking UI

Once the prediction process is complete, you can visualize the predicted values in the SkyWalking UI by configuring the appropriate metric widgets.

First, Run the following command to forward the UI service port to your local machine:

kubectl port-forward svc/demo-ui 8080:80 --namespace skywalking-showcase

Then, you can access this page to view the service traffic that was generated using the Python script earlier: http://localhost:8080/dashboard/MESH/Service/ZTJlLXRlc3QtZGVzdC1zZXJ2aWNl.1/Mesh-Service

To display predicted values, edit the Service Avg Resp Time Widget and add the following MQE:

# The maximum predicted response time.
baseline(service_resp_time, upper)
# The predicted response time.
baseline(service_resp_time, value)
# The minimum predicted response time.
baseline(service_resp_time, lower)

Finally, you can see the predicted values displayed in the widget.

Since the default data collection is hourly and the metric has significant fluctuations, the predicted values are derived from hourly averages rather than minute-level granularity. This approach smooths out fluctuations and provides a more stable baseline for monitoring.

Now, you should see the predicted response times visualized alongside actual values, helping you analyze trends and configure dynamic alerting thresholds effectively.

Conclusion

SkyAPM SkyPredictor enhances alert accuracy by using dynamic baselines instead of static thresholds. It collects history metrics data, forecasts future values with Prophet, and supports minute or hour-level collection for better precision. By integrating predictions into SkyWalking UI, users can optimize alerting and improve system observability.

By integrating dynamic thresholds, SkyWalking can adapt to traffic patterns and detect anomalies more effectively, reducing false positives and improving system observability.

Blog: [Video] Coffee + Software with Josh Long - Apache SkyWalking with Sheng Wu and Apache ShardingSphere with Trista Pan

Mon, 04 Dec 2023 00:00:00 +0000

🚀 Dive into the World of Cutting-Edge Technology with Apache’s Finest! 🌐 Join me today as we embark on an exhilarating journey with two of Apache’s most brilliant minds - Sheng Wu and Trista Pan. We’re exploring the realms of Apache SkyWalking and Apache ShardingSphere, two groundbreaking initiatives that are reshaping the landscape of open-source technology. 🌟

In this exclusive session, we delve deep into Apache SkyWalking - an innovative observability platform that’s revolutionizing how we monitor and manage distributed systems in the cloud. Witness firsthand how SkyWalking is empowering developers and organizations to gain unparalleled insights into their applications, ensuring performance, reliability, and efficient troubleshooting. 🛰️🔍

But there’s more! We’re also unveiling the secrets of Apache ShardingSphere, a dynamic distributed database ecosystem. Learn how ShardingSphere is making waves in the world of big data, offering scalable, high-performance solutions for data sharding, encryption, and more. This is your gateway to understanding how these technologies are pivotal in handling massive data sets across various industries. 🌐💾

Whether you’re a developer, tech enthusiast, or just curious about the future of open-source technology, this is a conversation you don’t want to miss! Get ready to be inspired and informed as we unlock new possibilities and applications of Apache SkyWalking and ShardingSphere. 🚀🌟

Join us, and let’s decode the future together!

Please join and follow Josh’s 龙之春 Youtube Coffee + Software with Josh Long Channel to learn more about technology and open source from telanted engineers and industry leads.

Blog: The Statement for SkyWalking users on HashiCorp license changes

Sun, 13 Aug 2023 00:00:00 +0000

On Aug. 10th, 2023, HashiCorp announced to adopt the Business Source License (BSL) from Mozilla Public License v2.0 (MPL 2.0), here is their post. They officially annouced they have changed the license for the ALL of their open-source products from the previous MPL 2.0 to a source-available license, BSL 1.1. Meanwhile, HashiCorp APIs, SDKs, and almost all other libraries will remain MPL 2.0.

HashiCorp Inc. is one of the most important vendors in the cloud-native landscape, as well as Golang ecosystem. This kind of changes would have potential implications for SkyWalking, which is closely integrated with cloud-native technology stacks.

Conclusion First

What does that mean for SkyWalking users?

SkyWalking community has evaluated our dependencies from HashiCorp products and libraries, the current conclusion is

SkyWalking users would NOT suffer any implication. All components of SkyWalking don’t have hard-dependency on BSL license affected codes.

SkyWalking community have found out all following dependencies of all relative repositories, all licenses are TRUELY stayed unchanged, and compatible with Apache 2.0 License.

OAP Server @kezhenxu94 @wu-sheng
- consul-client Apache 2.0 Repo archived on Jul 27, 2023
BanyanDB @hanahmily @lujiajing1126
- Server @hanahmily
  - hashicorp/golang-lru MPL-2.0
  - hashicorp/hcl MPL-2.0
- CLI @hanahmily No HashiCorp Dependency
SkyWalking OAP CLI @kezhenxu94
- github.com/hashicorp/hcl v1.0.0 MPL-2.0
- All under swck as transitive dependencies
SWCK @hanahmily
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go.net BSD-3
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
Go agent @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-hclog MIT
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-kms-wrapping/entropy MPL-2.0
- hashicorp/go-kms-wrapping/entropy/v2 MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-plugin MPL-2.0
- hashicorp/go-retryablehttp MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-secure-stdlib/base62 MPL-2.0
- hashicorp/go-secure-stdlib/mlock MPL-2.0
- hashicorp/go-secure-stdlib/parseutil MPL-2.0
- hashicorp/go-secure-stdlib/password MPL-2.0
- hashicorp/go-secure-stdlib/tlsutil MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go-version MPL-2.0
- hashicorp/go.net BSD-3-Clause
- hashicorp/golang-lru MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
- hashicorp/vault/api MPL-2.0
- hashicorp/vault/sdk MPL-2.0
- hashicorp/yamux MPL-2.0
SkyWalking eyes @kezhenxu94
- none
SkyWalking Infra e2e @kezhenxu94
- all under swck as transitive dependencies
SkyWalking rover(ebpf agent) @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-hclog MIT
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-retryablehttp MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
SkyWalking satellite @mrproliu
- hashicorp/consul/api MPL-2.0
- hashicorp/consul/sdk MPL-2.0
- hashicorp/errwrap MPL-2.0
- hashicorp/go-cleanhttp MPL-2.0
- hashicorp/go-immutable-radix MPL-2.0
- hashicorp/go-msgpack MIT
- hashicorp/go-multierror MPL-2.0
- hashicorp/go-rootcerts MPL-2.0
- hashicorp/go-sockaddr MPL-2.0
- hashicorp/go-syslog MIT
- hashicorp/go-uuid MPL-2.0
- hashicorp/go.net BSD-3-Clause
- hashicorp/golang-lru MPL-2.0
- hashicorp/hcl MPL-2.0
- hashicorp/logutils MPL-2.0
- hashicorp/mdns MIT
- hashicorp/memberlist MPL-2.0
- hashicorp/serf MPL-2.0
SkyWalking Terraform (scripts) @kezhenxu94
- No HashiCorp Dependency
- The scripts for Terraform users only. No hard requirement.

The GitHub ID is listed about the PMC members did the evaluations.

FAQ

If I am using Consul to manage SkyWalking Cluster or configurations, does this license change bring an implication?

YES, anyone using their server sides would be affected once you upgrade to later released versions after Aug. 10th, 2023.

This is HashiCorp’s statement

End users can continue to copy, modify, and redistribute the code for all non-commercial and commercial use, except where providing a competitive offering to HashiCorp. Partners can continue to build integrations for our joint customers. We will continue to work closely with the cloud service providers to ensure deep support for our mutual technologies. Customers of enterprise and cloud-managed HashiCorp products will see no change as well. Vendors who provide competitive services built on our community products will no longer be able to incorporate future releases, bug fixes, or security patches contributed to our products.

So, notice that, the implication about whether voilating BSL 1.1 is determined by the HashiCorp Inc about the status of the identified competitive relationship. We can’t provide any suggestions. Please refer to FAQs and contacts for the official explanations.

Will SkyWalking continoue to use HashiCorp Consul as an optional cluster coordinator and/or an optional dynamic configuration server?

For short term, YES, we will keep that part of codes, as the licenses of the SDK and the APIs are still in the MPL 2.0.

But, during the evaluation, we noticed the consul client we are using is rickfast/consul-client which had been archived by the owner on Jul 27, 2023. So, we are facing the issues that no maintaining and no version to upgrade. If there is not a new consul Java client lib available, we may have to remove this to avoid CVEs or version incompatible with new released servers.

Blog: Celebrate 22k stars

Tue, 13 Jun 2023 00:00:00 +0000

Celebrating 22k Stars!

The Apache SkyWalking community is thrilled to reach the milestone of 22k stars on GitHub! This showcases its popularity and impact as an APM and observability tool.

Since launching in 2016 to provide an open source APM solution, SkyWalking has evolved into a full stack observability platform with distributed tracing, metrics monitoring and alerting. It’s seeing widespread adoption globally, especially in Asia where APM needs are expanding rapidly.

The growing user base has enabled SkyWalking to achieve massive deployments demonstrating its ability to scale to extreme levels. There have been reported deployments collecting over 100TB of data from companies’ complex distributed applications, monitoring over 8000 microservices and analyzing 100 billion distributed traces - providing end-to-end visibility, performance monitoring and issue troubleshooting for some of the largest distributed systems in the world.

This success and widespread adoption has attracted an active community of nearly 800 contributors, thanks in part to programs like GSoC and OSPP(Open Source Promotion Plan) that bring in university contributors. The SkyWalking team remains focused on building a reliable, performant platform to observe complex distributed systems. We’ll continue innovating with features like service mesh monitoring and metric analytics.Your ongoing support, feedback and contributions inspire us!

Thank you for helping SkyWalking reach 22k stars on GitHub! This is just the beginning - we have ambitious plans and can’t wait to have you along our journey!

Blog: Monitoring DynamoDB with SkyWalking

Mon, 13 Mar 2023 00:00:00 +0000

Background

Apache SkyWalking is an open-source application performance management system that helps users collect and aggregate logs, traces, metrics, and events, and display them on the UI. Starting from OAP 9.4.0, SkyWalking has added AWS Firehose receiver, which is used to receive and calculate the data of CloudWatch metrics. In this article, we will take DynamoDB as an example to show how to use SkyWalking to receive and calculate CloudWatch metrics data for monitoring Amazon Web Services.

What are Amazon CloudWatch and Amazon Kinesis Data Firehose？

Amazon CloudWatch is a metrics repository, this tool can collect raw data from AWS (e.g. DynamoDB) and process it into readable metrics in near real-time. Also, we can use Metric Stream to continuously stream CloudWatch metrics to a selected target location for near real-time delivery and low latency. SkyWalking takes advantage of this feature to create metric streams and direct them to Amazon Kinesis Data Firehose transport streams for further transport processing.

Amazon Kinesis Data Firehoseis an extract, transform, and load (ETL) service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. SkyWalking takes advantage of this feature to eventually direct the metrics stream to the aws-firehose-receiver for OAP to calculate and ultimately display the metrics.

The flow chart is as follows.

Notice

Due to Kinesis Data Firehose specifications, the URL of the HTTP endpoint must use the HTTPS protocol and must use port 443. Also, this URL must be proxied by Gateway and forwarded to the real aws-firehose-receiver.
The TLS certificate must be signed by a CA and the self-signed certificate will not be trusted by Kinesis Data Firehose.

Setting up DynamoDB monitoring

Next, let’s take DynamoDB as an example to illustrate the necessary settings in aws before using OAP to collect CloudWatch metrics:

Go to Kinesis Console, create a data stream, and select Direct PUT for Source and HTTP Endpoint for Destination. And set HTTP Endpoint URL to Gateway URL. The rest of the configuration options can be configured as needed.

Go to the CloudWatch Console, select Metrics-Stream in the left control panel, and click Create metric stream. Select AWS/DynamoDB for namespace. Also, you can add other namespaces as needed. Kinesis Data Firehose selects the data stream created in the first step. Finally, set the output format to opentelemetry0.7. The rest of the configuration options can be configured as needed.

At this point, the AWS side of DynamoDB monitoring configuration is set up.

SkyWalking OAP metrics processing analysis

SkyWalking uses aws-firehose-receiver to receive and decode AWS metrics streams forwarded by Gateway, and send it to Opentelemetry-receiver for processing and transforming into SkyWalking metrics. Then, the metrics are analyzed and aggregated by Meter Analysis Language (MAL) and finally presented on the UI.

The MAL part and the UI part of SkyWalking support users’ customization, to display the metrics data in a more diversified way. For details, please refer to MAL doc and UI doc.

Typical metrics analysis

Scope

In SkyWalking, there is the concept of scope. By using scopes, we can classify and aggregate metrics more rationally. In the monitoring of DynamoDB, two of these scopes are used - Service and Endpoint.

Service represents a set of workloads that provide the same behavior for incoming requests. Commonly used as cluster-level scopes for services, user accounts are closer to the concept of clusters in AWS. So SkyWalking uses AWS account id as a key to map AWS accounts to Service types.

Similarly, Endpoint represents a logical concept, often used in services for the path of incoming requests, such as HTTP URI path or gRPC service class + method signature, and can also represent the table structure in the database. So SkyWalking maps DynamoDB tables to Endpoint type.

Metrics

Metric Name	Meaning
AccountMaxReads / AccountMaxWrites	The maximum number of read/write capacity units that can be used by an account.
AccountMaxTableLevelReads / AccountMaxTableLevelWrites	The maximum number of read/write capacity units that can be used by a table or global secondary index of an account.
AccountProvisionedReadCapacityUtilization / AccountProvisionedWriteCapacityUtilization	The percentage of provisioned read/write capacity units utilized by an account.
MaxProvisionedTableReadCapacityUtilization / MaxProvisionedTableWriteCapacityUtilization	The percentage of provisioned read/write capacity utilized by the highest provisioned read table or global secondary index of an account.

Above are some common account metrics (Serivce scope). They are various configuration information in DynamoDB, and SkyWalking can show a complete picture of the database configuration changes by monitoring these metrics.

Metric Name	Meaning
ConsumedReadCapacityUnits / ConsumedWriteCapacityUnits	The number of read/write capacity units consumed over the specified time period.
ReturnedItemCount	The number of items returned by Query, Scan or ExecuteStatement (select) operations during the specified time period.
SuccessfulRequestLatency	The latency of successful requests to DynamoDB or Amazon DynamoDB Streams during the specified time period.
TimeToLiveDeletedItemCount	The number of items deleted by Time to Live (TTL) during the specified time period.

The above are some common table metrics (Endpoint scope), which will also be aggregated into account metrics. These metrics are generally used to analyze the performance of the database, and users can use them to determine the reasonable level of database configuration. For example, users can track how much of their provisioned throughput is used through ConsumedReadCapicityUnits / ConsumedReadCapicityUnits to determine the reasonableness of the preconfigured throughput of a table or account. For more information about provisioned throughput, see Provisioned Throughput Intro.

Metric Name	Meaning
UserErrors	Requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 400 status code during the specified time period.
SystemErrors	The requests to DynamoDB or Amazon DynamoDB Streams that generate an HTTP 500 status code during the specified time period.
ThrottledRequests	Requests to DynamoDB that exceed the provisioned throughput limits on a resource.
TransactionConflict	Rejected item-level requests due to transactional conflicts between concurrent requests on the same items.

The above are some common error metrics, among which UserErrors are account-level metrics and the rest are table-level metrics. Users can set alarms on these metrics, and if warnings appear, then it may indicate that there are some problems with the use of the database, and users need to check and verify by themselves.

Notice

SkyWalking’s metrics selection for DynamoDB comes directly from CloudWatch metrics, which can also be found at CloudWatch metrics doc to get metrics details.

Demo

In this section, we will demonstrate how to use terraform to create a DynamoDB table and other AWS services that can generate metrics streams, and deploy Skywalking to complete the metrics collection.

First, you need a running gateway instance, such as NGINX, which is responsible for receiving metrics streams from AWS and forwarding them to the aws-firehose-receiver. Note that the gateway needs to be configured with certificates to accept HTTPS protocol requests.

Below is an example configuration for NGINX. The configuration does not need to be identical, as long as it can send incoming HTTPS requests to oap host:12801/aws/firehose/metrics.

server {
        listen       443 ssl;
        
        ssl_certificate     /crt/test.pem;
        ssl_certificate_key  /crt/test.key; 
        
        ssl_session_timeout  5m;   
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; 
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;  
        ssl_prefer_server_ciphers on; 
    
        location /aws/firehose/metrics {
            proxy_pass http://test.xyz:12801/aws/firehose/metrics;
        }
    }

Deploying SkyWalking

There are various ways to deploy SkyWalking, and you can get them directly from the release page.

Of course, if you are more comfortable with Kubernetes, you can also find the appropriate deployment method from SkyWalking-kubernetes.

Please note that no matter which deployment method you use, please make sure that the OAP and UI version is 9.4.0 or higher and that port 12801 needs to be open.

The following is an example of a deployment using the helm command.

export SKYWALKING_RELEASE_VERSION=4.3.0 
export SKYWALKING_RELEASE_NAME=skywalking  
export SKYWALKING_RELEASE_NAMESPACE=default

helm install "${SKYWALKING_RELEASE_NAME}" \
  oci://registry-1.docker.io/apache/skywalking-helm \
  --version "${SKYWALKING_RELEASE_VERSION}" \
  -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.image.tag=9.4.0 \
  --set oap.storageType=elasticsearch \
  --set ui.image.tag=9.4.0 \
  --set oap.ports.firehose=12801

Start the corresponding AWS service

The terraform configuration file is as follows (example modified inTerraform Registry - kinesis_firehose_delivery_stream）：

terraform configuration file

provider "aws" {
  region = "ap-northeast-1"
  access_key = "[need change]your access_key"
  secret_key = "[need change]your secret_key"
}
resource "aws_dynamodb_table" "basic-dynamodb-table" {
  name           = "GameScores"
  billing_mode   = "PROVISIONED"
  read_capacity  = 20
  write_capacity = 20
  hash_key       = "UserId"
  range_key      = "GameTitle"

  attribute {
    name = "UserId"
    type = "S"
  }

  attribute {
    name = "GameTitle"
    type = "S"
  }

  attribute {
    name = "TopScore"
    type = "N"
  }

  ttl {
    attribute_name = "TimeToExist"
    enabled        = true
  }

  global_secondary_index {
    name               = "GameTitleIndex"
    hash_key           = "GameTitle"
    range_key          = "TopScore"
    write_capacity     = 10
    read_capacity      = 10
    projection_type    = "INCLUDE"
    non_key_attributes = ["UserId"]
  }

  tags = {
    Name        = "dynamodb-table-1"
    Environment = "production"
  }
}

resource "aws_cloudwatch_metric_stream" "main" {
  name          = "my-metric-stream"
  role_arn      = aws_iam_role.metric_stream_to_firehose.arn
  firehose_arn  = aws_kinesis_firehose_delivery_stream.http_stream.arn
  output_format = "opentelemetry0.7"

  include_filter {
    namespace = "AWS/DynamoDB"
  }
}

# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html
data "aws_iam_policy_document" "streams_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["streams.metrics.cloudwatch.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "metric_stream_to_firehose" {
  name               = "metric_stream_to_firehose_role"
  assume_role_policy = data.aws_iam_policy_document.streams_assume_role.json
}

# https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-trustpolicy.html
data "aws_iam_policy_document" "metric_stream_to_firehose" {
  statement {
    effect = "Allow"

    actions = [
      "firehose:PutRecord",
      "firehose:PutRecordBatch",
    ]

    resources = [aws_kinesis_firehose_delivery_stream.http_stream.arn]
  }
}
resource "aws_iam_role_policy" "metric_stream_to_firehose" {
  name   = "default"
  role   = aws_iam_role.metric_stream_to_firehose.id
  policy = data.aws_iam_policy_document.metric_stream_to_firehose.json
}

resource "aws_s3_bucket" "bucket" {
  bucket = "metric-stream-test-bucket"
}

resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.bucket.id
  acl    = "private"
}

data "aws_iam_policy_document" "firehose_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["firehose.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "firehose_to_s3" {
  assume_role_policy = data.aws_iam_policy_document.firehose_assume_role.json
}

data "aws_iam_policy_document" "firehose_to_s3" {
  statement {
    effect = "Allow"

    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
    ]

    resources = [
      aws_s3_bucket.bucket.arn,
      "${aws_s3_bucket.bucket.arn}/*",
    ]
  }
}

resource "aws_iam_role_policy" "firehose_to_s3" {
  name   = "default"
  role   = aws_iam_role.firehose_to_s3.id
  policy = data.aws_iam_policy_document.firehose_to_s3.json
}

resource "aws_kinesis_firehose_delivery_stream" "http_stream" {
  name        = "metric-stream-test-stream"
  destination = "http_endpoint"

  http_endpoint_configuration {
    name       = "test_http_endpoint"
    url        = "[need change]Gateway url"
    role_arn   = aws_iam_role.firehose_to_s3.arn
  }
  s3_configuration {
    role_arn   = aws_iam_role.firehose_to_s3.arn
    bucket_arn = aws_s3_bucket.bucket.arn
  }
}

Steps to use.

Get the access_key and secret_key of the AWS account.( For how to get them, please refer to create-access-key )
Fill in the access_key and secret_key you got in the previous step, and fill in the corresponding URL of your gateway in the corresponding location of aws_kinesis_firehose_delivery_stream configuration.
Copy the above content and save it to the main.tf file.
Execute the following code in the corresponding path.

terraform init
terraform apply

At this point, all the required AWS services have been successfully created, and you can check your console to see if the services were successfully created.

Done!

If all the above steps were successful, please wait for about five minutes. After that, you can visit the SkyWalking UI to see the metrics.

Currently, the metrics collected by SkyWalking by default are displayed as follows.

account metrics:

table metrics：

Other services

Currently, SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services and display them through a custom dashboard.

Material

Blog: Monitoring AWS EKS and S3 with SkyWalking

Sun, 12 Mar 2023 00:00:00 +0000

SKyWalking OAP’s existing OpenTelemetry receiver can receive metrics through the OTLP protocol, and use MAL to analyze related metrics in real time. Starting from OAP 9.4.0, SkyWalking has added an AWS Firehose receiver to receive and analyze CloudWatch metrics data. This article will take EKS and S3 as examples to introduce the process of SkyWalking OAP receiving and analyzing the indicator data of AWS services.

EKS

OpenTelemetry Collector

OpenTelemetry (OTel) is a series of tools, APIs, and SDKs that can generate, collect, and export telemetry data, such as metrics, logs, and traces. OTel Collector is mainly responsible for collecting, processing, and exporting. For telemetry data, Collector consists of the following main components:

Receiver: Responsible for obtaining telemetry data, different receivers support different data sources, such as prometheus, kafka, otlp.
Processor: Process data between receiver and exporter, such as adding or deleting attributes.
Exporter: Responsible for sending data to different backends, such as kafka, SkyWalking OAP (via OTLP).
Service: Components enabled as a unit configuration, only configured components will be enabled.

OpenTelemetry Protocol Specification(OTLP)

OTLP mainly describes how to receive (pull) indicator data through gRPC and HTTP protocols. The OpenTelemetry receiver of SKyWalking OAP implements the OTLP/gRPC protocol, and the indicator data can be exported to OAP through the OTLP/gRPC exporter. Usually the data flow of a Collector is as follows:

Monitor EKS with OTel

EKS monitoring is realized through OTel. You only need to deploy OpenTelemetry Collector in the EKS cluster in the way of DaemonSet – use AWS Container Insights Receiver as the receiver, and set the address of otlp exporter to the address of OAP. In addition, it should be noted that OAP is used job_name : aws-cloud-eks-monitoring as the identifier of EKS metrics according to the attribute, so it is necessary to configure a processor in the collector to add this attribute.

OTel Collector configuration demo


extensions:
  health_check:
receivers:
  awscontainerinsightreceiver:
processors:
# To enable OAP to correctly identify EKS metrics, add the job_name attribute
  resource/job-name:
    attributes:
    - key: job_name   
      value: aws-cloud-eks-monitoring
      action: insert     

# Specify OAP as exporters
exporters:
  otlp:
    endpoint: oap-service:11800 
    tls:
      insecure: true
  logging:
      loglevel: debug          
service:
  pipelines:
    metrics:
      receivers: [awscontainerinsightreceiver]
      processors: [resource/job-name]
      exporters: [otlp,logging]
  extensions: [health_check]

By default, SkyWalking OAP counts the network, disk, CPU and other related indicator data in the three dimensions of Node, Pod, and Service. Only part of the content is shown here.

Pod dimensions

Service dimensions

EKS monitoring complete configuration

Click here to view complete k8s resource configuration


apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-otel-sa
  namespace: aws-otel-eks

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: aoc-agent-role
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "endpoints"]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["nodes/stats", "configmaps", "events"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["otel-container-insight-clusterleader"]
    verbs: ["get","update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create","get","update"]    

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: aoc-agent-role-binding
subjects:
  - kind: ServiceAccount
    name: aws-otel-sa
    namespace: aws-otel-eks
roleRef:
  kind: ClusterRole
  name: aoc-agent-role
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-conf
  namespace: aws-otel-eks
  labels:
    app: opentelemetry
    component: otel-agent-conf
data:
  otel-agent-config: |
    extensions:
      health_check:

    receivers:
      awscontainerinsightreceiver:

    processors:
      resource/job-name:
        attributes:
        - key: job_name   
          value: aws-cloud-eks-monitoring
          action: insert     

    exporters:
      otlp:
        endpoint: oap-service:11800
        tls:
          insecure: true
      logging:
          loglevel: debug          

    service:
      pipelines:
        metrics:
          receivers: [awscontainerinsightreceiver]
          processors: [resource/job-name]
          exporters: [otlp,logging]
      extensions: [health_check]

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-otel-eks-ci
  namespace: aws-otel-eks
spec:
  selector:
    matchLabels:
      name: aws-otel-eks-ci
  template:
    metadata:
      labels:
        name: aws-otel-eks-ci
    spec:
      containers:
        - name: aws-otel-collector
          image: amazon/aws-otel-collector:v0.23.0
          env:
         	  # Specify region
            - name: AWS_REGION
              value: "ap-northeast-1"
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: K8S_NAMESPACE
              valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
          imagePullPolicy: Always
          command:
            - "/awscollector"
            - "--config=/conf/otel-agent-config.yaml"
          volumeMounts:
            - name: rootfs
              mountPath: /rootfs
              readOnly: true
            - name: dockersock
              mountPath: /var/run/docker.sock
              readOnly: true
            - name: varlibdocker
              mountPath: /var/lib/docker
              readOnly: true
            - name: containerdsock
              mountPath: /run/containerd/containerd.sock
              readOnly: true
            - name: sys
              mountPath: /sys
              readOnly: true
            - name: devdisk
              mountPath: /dev/disk
              readOnly: true
            - name: otel-agent-config-vol
              mountPath: /conf
            - name: otel-output-vol  
              mountPath: /otel-output
          resources:
            limits:
              cpu:  200m
              memory: 200Mi
            requests:
              cpu: 200m
              memory: 200Mi
      volumes:
        - configMap:
            name: otel-agent-conf
            items:
              - key: otel-agent-config
                path: otel-agent-config.yaml
          name: otel-agent-config-vol
        - name: rootfs
          hostPath:
            path: /
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
        - name: varlibdocker
          hostPath:
            path: /var/lib/docker
        - name: containerdsock
          hostPath:
            path: /run/containerd/containerd.sock
        - name: sys
          hostPath:
            path: /sys
        - name: devdisk
          hostPath:
            path: /dev/disk/
        - name: otel-output-vol  
          hostPath:
            path: /otel-output
      serviceAccountName: aws-otel-sa

S3

Amazon CloudWatch

Amazon CloudWatch is a monitoring service provided by AWS. It is responsible for collecting indicator data of AWS services and resources. CloudWatch metrics stream is responsible for converting indicator data into stream processing data, and supports output in two formats: json and OTel v0.7.0.

Amazon Kinesis Data Firehose (Firehose)

Firehose is an extract, transform, load (ETL) service that reliably captures, transforms, and serves streaming data into data lakes, data stores (such as S3), and analytics services.

To ensure that external services can correctly receive indicator data, AWS provides Kinesis Data Firehose HTTP Endpoint Delivery Request and Response Specifications (Firehose Specifications) . Firhose pushes Json data by POST

Json data example

{
  "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "timestamp": 1578090901599
  "records": [
    {
      "data": "aGVsbG8="
    },
    {
      "data": "aGVsbG8gd29ybGQ="
    }
  ]
}

requestId: Request id, which can achieve deduplication and debugging purposes.
timestamp: Firehose generated the timestamp of the request (in milliseconds).
records: Actual delivery records
1. data: The delivered data, encoded in base64, can be in json or OTel v0.7.0 format, depending on the format of CloudWatch data (described later). Skywalking currently supports OTel v0.7.0 format.

aws-firehose-receiver

aws-firehose-receiver provides an HTTP Endpoint that implements Firehose Specifications: /aws/firehose/metrics. The figure below shows the data flow of monitoring DynamoDB, S3 and other services through CloudWatch, and using Firehose to send indicator data to SKywalking OAP.

Step-by-step setup of S3 monitoring

Enter the S3 console and create a filter forRequest metrics: Amazon S3 >> Buckets >> (Your Bucket) >> Metrics >> metrics >> View additional charts >> Request metrics
Enter the Amazon Kinesis console, create a delivery stream, Source select Direct PUT, Destination select HTTP Endpoint. And set HTTP endpoint URL to https://your_domain/aws/firehose/metrics. Other configuration items:

Buffer hints: Set the size and period of the cache
Access key just matches the AccessKey in aws-firehose-receiver
Retry duration: Retry period
Backup settings: Backup settings, optionally backup the posted data to S3 at the same time.

Enter the CloudWatch console Streams and click Create CloudWatch Stream. And Select your Kinesis Data Firehose stream configure the delivery stream created in the second step in the item. Note that it needs to be set Change output format to OpenTelemetry v0.7.0.

At this point, the S3 monitoring configuration settings are complete. The S3 metrics currently collected by SkyWalking by default are shown below:

Other service

Currently SkyWalking officially supports EKS, S3, DynamoDB monitoring. Users also refer to the OpenTelemetry receiver to configure OTel rules to collect and analyze CloudWatch metrics of other AWS services, and display them through a custom dashboard.

Material

Blog: [License Issue] Aliyun(阿里云)'s trace analysis service copied SkyWalking's trace profiling page.

Tue, 03 Jan 2023 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 3rd, 2023, we received reports about Aliyun Trace Analysis Service. It provides a cloud service compatible with SkyWalking trace APIs and agents.

On their product page, there is a best-practice document describing about their service is not SkyWalking OAP, but can work with SkyWalking agents to support SkyWalking’s In-Process(Trace) Profiling.

BUT, they copied the whole page of SkyWalking’s profiling UI, including page layout, words, and profiling task setup. The only difference is the color schemes.

SkyWalking UI

Aliyun Trace Analysis UI on their document page

The UI visualization is a part of the copyright. Aliyun declared their backend is NOT a re-distribution of SkyWalking repeatedly on their website, and they never mentioned this page is actually copied from upstream.

This is a LICENSE issue, violating SkyWalking’s copyright and Apache 2.0 License. They don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

Blog: [Video] Distributed tracing demo using Apache SkyWalking and Kong API Gateway

Thu, 11 Aug 2022 00:00:00 +0000

Observability essential when working with distributed systems. Built on 3 pillars of metrics, logging and tracing, having the right tools in place to quickly identify and determine the root cause of an issue in production is imperative. In this Kongcast interview, we explore the benefits of having observability and demo the use of Apache SkyWalking. We walk through the capabilities that SkyWalking offers out of the box and debug a common HTTP 500 error using the tool.

Andrew Kew is interviewed by Viktor Gamov, a developer advocate at Kong Inc

Andrew is a highly passionate technologist with over 16 valuable years experience in building server side and cloud applications. Having spent the majority of his time in the Financial Services domain, his meritocratic rise to CTO of an Algorithmic Trading firm allowed him to not only steer the business from a technology standpoint, but build robust and scalable trading algorithms. His mantra is “right first time”, thus ensuring the projects or clients he is involved in are left in a better place than they were before he arrived.

He is the founder of a boutique software consultancy in the United Kingdom, QuadCorps Ltd, working in the API and Integration Ecosystem space and is currently on a residency programme at Kong Inc as a senior field engineer and technical account manager working across many of their enterprise strategic accounts.

Blog: Integrating Apache SkyWalking with source code

Thu, 14 Apr 2022 00:00:00 +0000

Introduction

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it. - Mark Weiser

Mark Weiser prophetically argued in the late 1980s, that the most far-reaching technologies are those which vanish into thin air. According to Weiser, “Whenever people learn something sufficiently well, they cease to be aware of it.” This disappearing act, as Weiser claimed, is not limited to technology but rather human psychology. It is this very experience that allows us to escape lower-level thinking into higher-level thinking. For once we are no longer impeded by mundane details, we are then free to focus on new goals.

This realization becomes more relevant as APMs become increasingly popular. As more applications are deployed with APMs, the number of abstract representations of the underlying source code also increases. While this provides great value to many non-development roles within an organization, it does pose additional challenges to those in development roles who must translate these representations into concepts they can work with (i.e. source code). Weiser sums this difficultly up rather succinctly when he states that “Programmers should no more be asked to work without access to source code than auto-mechanics should be asked to work without looking at the engine.”

Still, APMs collect more information only to produce a plethora of new abstract representations. In this article, we will introduce a new concept in Source++, the open-source live-coding platform, specifically designed to allow developers to monitor production applications more intuitively.

Live Views

And we really don’t understand even yet, hundreds of metrics later, what make a program easier to understand or modify or reuse or borrow. I don’t think we’ll find out by looking away from programs to their abstract interfaces. The answers are in the source code. - Mark Weiser

As APMs move from the “nice to have” category to the “must-have” category, there is a fundamental feature holding them back from ubiquity. They must disappear from consciousness. As developers, we should feel no impulse to open our browsers to better understand the underlying source code. The answers are literally in the source code. Instead, we should improve our tools so the source code conveniently tells us what we need to know. Think of how simple life could be if failing code always indicated how and why it failed. This is the idea behind Source++.

In our last blog post, we discussed Extending Apache SkyWalking with non-breaking breakpoints. In that post, we introduced a concept called Live Instruments, which developers can use to easily debug live production applications without leaving their IDE. Today, we will discuss how existing SkyWalking installations can be integrated into your IDE via a new concept called Live Views. Unlike Live Instruments, which are designed for debugging live applications, Live Views are designed for increasing application comprehension and awareness. This is accomplished through a variety of commands which are input into the Live Command Palette.

Live Command Palette

The Live Command Palette (LCP) is a contextual command prompt, included in the Source++ JetBrains Plugin, that allows developers to control and query live applications from their IDE. Opened via keyboard shortcut (Ctrl+Shift+S), the LCP allows developers to easily view metrics relevant to the source code they’re currently viewing. The following Live View commands are currently supported:

Command: view (overview/activity/traces/logs)

The view commands display contextual popups with live operational data of the current source code. These commands allow developers to view traditional SkyWalking operational data filtered down to the relevant metrics.

Command: watch log

The watch log command allows developers to follow individual log statements of a running application in real-time. This command allows developers to negate the need for manually scrolling through the logs to find instances of a specific log statement.

Command: (show/hide) quick stats

The show quick stats command displays live endpoint metrics for a quick idea of an endpoint’s activity. Using this command, developers can quickly assess the status of an endpoint and determine if the endpoint is performing as expected.

Future Work

A good tool is an invisible tool. By invisible, I mean that the tool does not intrude on your consciousness; you focus on the task, not the tool. Eyeglasses are a good tool – you look at the world, not the eyeglasses. - Mark Weiser

Source++ aims to extend SkyWalking in such a way that SkyWalking itself becomes invisible. To accomplish this, we plan to support custom developer commands. Developers will be able to build customized commands for themselves, as well as commands to share with their team. These commands will recognize context, types, and conditions allowing for a wide possibility of operations. As more commands are added, developers will be able to expose everything SkyWalking has to offer while focusing on what matters most, the source code.

If you find these features useful, please consider giving Source++ a try. You can install the plugin directly from your JetBrains IDE, or through the JetBrains Marketplace. If you have any issues or questions, please open an issue. Feedback is always welcome!

Blog: [Resolved][License Issue] Volcengine Inc.(火山引擎) violates the Apache 2.0 License when using SkyWalking.

Fri, 28 Jan 2022 00:00:00 +0000

Apache SkyWalking is an open-source APM for a distributed system, Apache Software Foundation top-level project.

On Jan. 28th, we received a License violation report from one of the committers (anonymously). They have a cloud service called Application Performance Monitoring - Distributed Tracing (应用性能监控全链路版). At the Java service monitoring section, it provides this agent download link

wget https://datarangers.com.cn/apminsight/repo/v2/download/java-agent/apminsight-java-agent_latest.tar.gz

We downloaded it at 23:15 Jan. 28th UTC+8(Beijing), and archived it at here

We have confirmed this is a distribution of SkyWalking Java agent.

We listed several pieces of evidence to prove this here, every reader could compare with the official SkyWalking source codes

The first and the easiest one is agent.config file, which is using the same config keys, and the same config format.

This is the Volcengine’s version, and check SkyWalking agent.config

In the apmplus-agent.jar, Volcengine’s agent core jar, you could easily find several core classes exactly as same as SkyWalking’s.

The ComponentsDefine class is unchanged, even with component ID and name. This is Volcengine’s version, and check SkyWalking’s version

The whole code names, package names, and hierarchy structure are all as same as SkyWalking 6.x version.

This is the Volcengine package hierarchy structure, and check the SkyWalking’s version

Volcengine Inc.’s team changed all package names, removed the Apache Software Foundation’s header, and don’t keep Apache Software Foundation and Apache SkyWalking’s LICENSE and NOTICE file in their redistribution.

Also, we can’t find anything on their website to declare they are distributing SkyWalking.

All above have proved they are violating the Apache 2.0 License, and don’t respect Apache Software Foundation and Apache SkyWalking’s IP and Branding.

We have contacted their legal team, and wait for their official response.

Resolution

On Jan. 30th night, UTC+8, 2022. We received a response from Volcengine’s APMPlus team. They admitted their violation behaviors, and made the following changes.

Volcengine’s APMPlus service page was updated on January 30th and stated that the agent is a fork version(re-distribution) of Apache SkyWalking agent. Below is the screenshot of Volcengine’s APMPlus product page.

Volcengine’s APMPlus agent distributions were also updated and include SkyWalking’s License and NOTICE now. Below is the screenshot of Volcengine’s APMPlus latest agent, you could download from the product page. We keep a copy of their Jan. 30th 2022 at here.

Volcengine’s APMPlus team had restored all license headers of SkyWalking in the agent, and the modifications of the project files are also listed in “SkyWalking-NOTICE”, which you could download from the product page.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

Appendix

Inquiries of committers

Q: I hope Volcengine Inc. can give a reason for this license issue, not just an afterthought PR. This will not only let us know where the issue is but also avoid similar problems in the future.

A(apmplus apmplus@volcengine.com):

The developers neglected this repository during submitting compliance
assessment.
Currently, APMPlus team had introduced advanced tools provided by the
company for compliance assessment, and we also strengthened training for
our developers.
In the future, the compliance assessment process will be further improved
from tool assessment and manual assessment.

Blog: Extending Apache SkyWalking with non-breaking breakpoints

Mon, 06 Dec 2021 00:00:00 +0000

Non-breaking breakpoints are breakpoints specifically designed for live production environments. With non-breaking breakpoints, reproducing production bugs locally or in staging is conveniently replaced with capturing them directly in production.

Like regular breakpoints, non-breaking breakpoints can be:

placed almost anywhere
added and removed at will
set to fire on specific conditions
expose internal application state
persist as long as desired (even between application reboots)

The last feature is especially useful given non-breaking breakpoints can be left in production for days, weeks, and even months at a time while waiting to capture behavior that happens rarely and unpredictably.

How do non-breaking breakpoints work?

If you’re familiar with general distributed tracing concepts, such as “traces” and “spans”, then you’re already broadly familiar with how non-breaking breakpoints work. Put simply, non-breaking breakpoints are small fragments of code added during runtime that, upon the proper conditions, save a portion of the application’s current state, and resume normal execution. In SkyWalking, this can be implemented by simply opening a new local span, adding some tags, and closing the local span.

While this process is relatively simple, the range of functionality that can be achieved through this technique is quite impressive. Save the current and global variables to create a non-breaking breakpoint; add the ability to format log messages to create just-in-time logging; add the ability to trigger metric telemetry to create real-time KPI monitoring. If you keep moving in this direction, you eventually enter the realm of live debugging/coding, and this is where Source++ comes in.

Live Coding Platform

Source++ is an open-source live coding platform designed for production environments, powered by Apache SkyWalking. Using Source++, developers can add breakpoints, logs, metrics, and distributed tracing to live production software in real-time on-demand, right from their IDE or CLI. While capable of stand-alone deployment, the latest version of Source++ makes it easier than ever to integrate into existing Apache SkyWalking installations. This process can be completed in a few minutes and is easy to customize for your specific needs.

For a better idea of how Source++ works, take a look at the following diagram:

In this diagram, blue components represent existing SkyWalking architecture, black components represent new Source++ architecture, and the red arrows show how non-breaking breakpoints make their way from production to IDEs. A process that is facilitated by Source++ components: Live Probe, Live Processors, Live Platform, and Live Interface.

Live Probe

The Live Probe is currently available for JVM and Python applications. It runs alongside the SkyWalking agent and is responsible for dynamically adding and removing code fragments based on valid instrumentation requests from developers. These code fragments in turn make use of the SkyWalking agent’s internal APIs to facilitate production instrumentation.

Live Processors

Live Processors are responsible for finding, extracting, and transforming data found in distributed traces produced via live probes. They run alongside SkyWalking collectors and implement additional post-processing logic, such as PII redaction. Live processors work via uniquely identifiable tags (prefix spp.) added previously by live probes.

One could easily view a non-breaking breakpoint ready for processing using Rocketbot, however, it will look like this:

Even though the above does not resemble what’s normally thought of as a breakpoint, the necessary information is there. With live processors added to your SkyWalking installation, this data is refined and may be viewed more traditionally via live interfaces.

Live Platform

The Live Platform is the core part of the Source++ architecture. Unlike the live probe and processors, the live platform does not have a direct correlation with SkyWalking components. It is a standalone server responsible for validating and distributing production breakpoints, logs, metrics, and traces. Each component of the Source++ architecture (probes, processors, interfaces) communicates with each other through the live platform. It is important to ensure the live platform is accessible to all of these components.

Live Interface

Finally, with all the previous parts installed, we’re now at the component software developers will find the most useful. A Live Interface is what developers use to create, manage, and view non-breaking breakpoints, and so on. There are a few live interfaces available:

With the Live Instrument Processor enabled, and the JetBrains Plugin installed, non-breaking breakpoints appear as such:

The above should be a sight far more familiar to software developers. Beyond the fact that you can’t step through execution, non-breaking breakpoints look and feel just like regular breakpoints.

For more details and complete setup instructions, please visit:

https://github.com/sourceplusplus/deploy-skywalking

Blog: [Webinar] SkyWalking 8.x Introduction

Wed, 04 Aug 2021 00:00:00 +0000

GOUP hosted a webinar, and invited Sheng Wu to introduce Apache SkyWalking. This is a 1.5 hours presentation including the full landscape of Apache SkyWalking 8.x.

Chapter04 Session10 - Apache Skywalking by Sheng Wu

Blog: [Community win] SkyWalking achieved 500 contributors milestone.

Sat, 10 Jul 2021 00:00:00 +0000

SkyWalking is an open source APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based architectures.

From 2020, it has dominated the open source APM market in China, and expanded aggressively in North American, Europe and Asia’s other countries.

With over 6 years (2015-2021) of development, driven by the global open source community, SkyWalking now provides full stack observability covering metrics, tracing and logging, plus event detector, which are built based on various native and ecosystem solutions.

Language agent-based(Java, Dot Net, Golang, PHP, NodeJS, Python, C++, LUA) in-process monitoring, is as powerful as commercial APM vendors’ agents. Mostly auto-instrumentation, and good interactivity.
Service Mesh Observability, working closely with Envoy and Istio teams.
Transparent integration of popular metrics ecosystem. Accept metrics from Prometheus SDK, OpenTelemetry collectors, Zabbix agents, etc.
Log collection with analysis capability from FluentD, Fluent-bit, Filebeat, etc. agents.
Infrastructure monitoring, such as Linux and k8s, is out of the box.

The SkyWalking ecosystem was started by very few people. The community drives the project to cover real scenarios, from tracing to the whole APM field. Even today, more professional open source developers, powered by the vendors behind them, are bringing the project to a different level.

Typically and most attractively, SkyWalking is going to build the first known open source APM specific database in the world, at least providing

Time series-based database engine.
Support traces/logs and metrics in the database core level.
High performance with cluster mode and HPA.
Reasonable resource cost.

We nearly doubled the number of contributors in the last year, from ~300 to over 500. The whole community is very energetic. Here, we want to thank our 47 committers(28 PMC members included), listed here, and over 400 other contributors.

We together built this humongous Apache Top Level project, and proved the stronge competitiveness of an open-source project.

This is a hard-won and impressive achievement. We won’t stop here. The trend is there, the ground is solid. We are going to build the top-level APM system relying on our open-source community.

500 Contributors List

GitHub
1095071913	182148432**	295198088**	394102339**	437376068**
50168383	55846420**	826245622**	844067874	Ahoo-Wang
AirTrioa	AlexanderWert	AlseinX	AngryMills	Ax1an
BFergerson	BZFYS	CalvinKirs	CharlesMaster	ChaunceyLin5152
CommissarXia	Cvimer	DeadLion	Doublemine	Du-fei
ElderJames	EvanLjp	FatihErdem	FeynmanZhou	Fine0830
FingerLiu	FrankyXu	Gallardot	GerryYuan	HackerRookie
HarryFQ	Heguoya	Hen1ng	HendSame	Humbertzhang
IanCao	IluckySi	Indifer	J-Cod3r	JaredTan95
Jargon96	Jijun	JoeKerouac	JohnNiang	Johor03
Jozdortraz	Jtrust	Just-maple	KangZhiDong	LazyLei
LiWenGu	Lin1997	Linda-pan	LiteSun	Liu-XinYuan
MiracleDx	Miss-you	MoGuGuai-hzr	MrYzys	O-ll-O
Patrick0308	QHWG67	Qiliang	QuanjieDeng	RandyAbernethy
RedzRedz	Runrioter	SataQiu	ScienJus	SevenBlue2018
ShaoHans	Shikugawa	SoberChina	SummerOfServenteen	Switch-vov
TJ666	Technoboy-	TerrellChen	TeslaCN	TheRealHaui
TinyAllen	TomMD	ViberW	Videl	WALL-E
WeihanLi	WildWolfBang	WillemJiang	Wooo0	XhangUeiJong
Xlinlin	YczYanchengzhe	Yebemeto	YoungHu	YunaiV
YunfengGao	Z-Beatles	ZS-Oliver	ZhHong	ZhuoSiChen
a198720	a1vin-tian	a526672351	acurtain	adamni135
adermxzs	adriancole**	aeolusheath	agile6v	aix3
aiyanbo	ajanthan	alexkarezin	alonelaval	amogege
amwyyyy	andyliyuze	andyzzl	aoxls	arugal
ascrutae	ascrutae**	augustowebd	aviaviavi	bai-yang
beckhampu	beckjin	beiwangnull	bigflybrother	bootsrc
bostin	brucewu-fly	buxingzhe	buzuotaxuan	bwh12398**
c feng	c1ay	candyleer	carllhw	carlvine500
carrypann	cheenursn	cheetah012	chenbeitang	chenglei**
chengshiwen	chenmudu	chenpengfei	chenvista	chess-equality
chestarss	chidaodezhongsheng	chopin-d	clevertension	clk1st
cngdkxw	cnlangzi	codeglzhang	codelipenghui	coder-yqj
coki230	compilerduck	constanine	coolbeevip	crystaldust
cui-liqiang	cuiweiwei	cutePanda123	cyberdak	cyejing
cyhii	dafu-wu	dagmom	dalekliuhan**	darcydai
dengliming	devkanro	devon-ye	dickens7	dimaaan
dingdongnigetou	dio	divyakumarjain	dmsolr	dominicqi
donbing007	dsc6636926	dvsv2	dzx2018	echooymxq
efekaptan	elk-g	emschu	eoeac	evanljp**
evanxuhe	feelwing1314	fgksgf	fredster33	fuhuo
fulmicoton	fushiqinghuan111	geektcp	geomonlin	ggndnn
gitter-badger	givingwu	glongzh	gnr163	gonedays
grissom-grissom	grissomsh	guodongq	guyukou	gxthrj
gy09535	gzshilu	hailin0	hanahmily	haotian2015
haoyann	hardzhang	harvies	heihaozi	hepyu
heyanlong	hi-sb	honganan	horber	hsoftxl
huangyoje	huliangdream	huohuanhuan	iluckysi	innerpeacez
itsvse	jasper-zsh	jbampton	jialong121	jinlongwang
jjlu521016	jjtyro	jmjoy	jsbxyyx	justeene
juzhiyuan	jy00464346	kaanid	kagaya85	karott
kayleyang	kevinyyyy	kezhenxu94	kikupotter	kilingzhang
killGC	kkl129	klboke	ksewen	kuaikuai
kun-song	kylixs	landonzeng	langke93	langyan1022
langyizhao	lazycathome	leemove	leizhiyuan	libinglong
lijial	lilien1010	limfriend	linkinshi	linliaoy
liqiangz	liu-junchi	liufei**	liuhaoXD	liuhaoyang
liuweiyi**	liuyanggithup	liuzhengyang	liweiv	lixin40**
lizl9**	lkxiaolou	llissery	louis-zhou	lpcy
lpf32	lsyf	lucperkins	lujiajing1126	lunamagic1978
lunchboxav	lxin96**	lxliuxuankb	lytscu	lyzhang1999
mage3k	makefriend8	makingtime	mantuliu	maolie
margauxcabrera	masterxxo	maxiaoguang64	me**	membphis
mestarshine	mgsheng	michaelsembwever	mikkeschiren	ming_flycash**
minquan.chen**	misaya	momo0313	moonming	mrproliu
mrproliu**	muyun12	nacx	neatlife	neeuq
nic-chen	nickwongwong	nikitap492	nileblack	nisiyong
novayoung	oatiz	oflebbe	olzhy	onecloud360
osiriswd	panniyuyu	peng-yongsheng	pengweiqhca	potiuk
probeyang	purgeyao	qijianbo010	qinhang3	qiuyu-d
qjgszzx	qq362220083	qqeasonchen	qxo	ralphgj
raybi-asus	refactor2	remicollet	rlenferink	rootsongjc
rovast	ruibaby	s00373198	scolia	sdanzo
seifeHu	sergicastro	shiluo34	sikelangya	simonlei
sk163	snakorse	songzhendong	songzhian	songzhian**
sonxy	spacewander	stalary	stenio2011	stevehu
stone-wlg	sungitly	surechen	swartz-k	sxzaihua
tangxqa	tanjunchen	tankilo	tanzhen**	taskmgr
tbdpmi	terranhu	terrymanu	tevahp	thanq
thebouv	tianyk	tianyuak	tincopper	tinyu0
tom-pytel	tristaZero	tristan-tsl	trustin	tsuilouis
tuohai666	tzsword-2020	tzy1316106836	vcjmhg	viktoryi
vision-ken	viswaramamoorthy	wallezhang	wang-yeliang	wang_weihan**
wangrzneu	wankai123	wbpcode	web-xiaxia	webb2019
weiqiang-w	weiqiang333	wendal	wengangJi	wenjianzhang
whfjam	whl12345	willseeyou	wilsonwu	wind2008hxy
wingwong-knh	withlin	wl4g	wqr2016	wu-sheng
wuguangkuo	wujun8	wuwen5	wuxingye	x22x22
xbkaishui	xcaspar	xdRight	xiaoweiyu**	xiaoxiangmoe
xiaoy00	xinfeingxia85	xingren23	xinzhuxiansheng	xonze
xuanyu66	xuchangjunjx	xudianyang	yanbw	yanfch
yang-xiaodong	yangxb2010000	yanickxia	yanmaipian	yanmingbi
yantaowu	yaojingguo	yaowenqiang	yazong	ychandu
ycoe	yimeng	yu199195	yuqichou	yushuqiang**
yuyujulin	yxudong	yymoth	zaunist	zaygrzx
zcai2	zeaposs	zhang98722	zhanghao001	zhangjianweibj
zhangkewei	zhangsean	zhangxin**	zhaoyuguang	zhe1926
zhentaoJin	zhongjianno1**	zhousiliang163	zhuCheer	zhyyu
zifeihan	zijin-m	zkscpqm	zoidbergwill	zoumingzm
zouyx	zpf1989	zshit	zxbu	zygfengyuwuzu

Blog: OpenSearch, a new storage option to avoid ElasticSearch's SSPL

Sun, 09 May 2021 00:00:00 +0000

We posted our Response to Elastic 2021 License Change blog 4 months ago. It doesn’t have a big impact in the short term, but because of the incompatibility between SSPL and Apache 2.0, we lost the chance of upgrading the storage server, which concerns the community and our users. So, we have to keep looking for a new option as a replacement.

There was an open source project, Open Distro for Elasticsearch, maintained by the AWS team. It is an Apache 2.0-licensed distribution of Elasticsearch enhanced with enterprise security, alerting, SQL, and more. After Elastic relicensed its projects, we talked with their team, and they have an agenda to take over the community leadship and keep maintaining Elasticsearch, as it was licensed by Apache 2.0. So, they are good to fork and continue.

On April 12th, 2021, AWS announced the new project, OpenSearch, driven by the community, which is initialized from people of AWS, Red Hat, SAP, Capital One, and Logz.io. Read this Introducing OpenSearch blog for more detail.

Once we had this news in public, we begin to plan the process of evaluating and testing OpenSearch as SkyWalking’s storage option. Read our issue.

Today, we are glad to ANNOUNCE, OpenSearch could replace ElastcSearch as the storage, and it is still licensed under Apache 2.0.

This has been merged in the main stream, and you can find it in the dev doc already.

OpenSearch

OpenSearch storage shares the same configurations as Elasticsearch 7. In order to activate Elasticsearch 7 as storage, set storage provider to elasticsearch7. Please download the apache-skywalking-bin-es7.tar.gz if you want to use OpenSearch as storage.

SkyWalking community will keep our eyes on the OpenSearch project, and look forward to their first GA release.

NOTE: we have to add a warning NOTICE to the Elasticsearch storage doc:

NOTICE: Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch version 7.11. So please choose the suitable Elasticsearch version according to your usage.

Blog: [Resolved][License Issue] Tencent Cloud TSW service violates the Apache 2.0 License when using SkyWalking.

Fri, 22 Jan 2021 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system, Apache Software Foundation top-level project.

At Jan. 11th, 2021, we noticed the Tencent Cloud Service, Tencent Service Watcher - TSW, for first time. Due to the similar short name, which SkyWalking is also called SW in the community, we connected with the service team of Tencent Cloud, and kindly asked.

They used to replay, TSW is purely developed by Tencent team itself, which doesn’t have any code dependency on SkyWalking.. We didn’t push harder.

But one week later, Jan 18th, 2021, our V.P., Sheng got the report again from Haoyang SkyWalking PMC member, through WeChat DM(direct message),. He provided complete evidence to prove TSW actually re-distributed the SkyWalking’s Java agent. We keep one copy of their agent’s distribution(at Jan. 18th), you could be downloaded here.

Some typically evidences are here

ServiceManager is copied and package-name changed in the TSW’s agent.
ContextManager is copied and ackage-name changed in the TSW’s agent.

At the same time, we checked their tsw-client-package.zip, it didn’t include the SkyWalking’s LICENSE and NOTICE. Also, they didn’t mention TSW agent is the re-ditribution SkyWalking on their website.

With all above information, we had enough reason to believe, from the tech perspective, they were violating the Apache 2.0 License.

From the 18th Jan., 2021, we sent mail [Apache 2.0 License Violation] Tencent Cloud TSW service doesn't follow the Apache 2.0 License to brief the SkyWalking PMC, and took the following actions to connect with Tencent.

Made direct call to Tencent Open Source Office.
Connected with Tencent Cloud TVP program committee, as Sheng Wu(Our VP) is a Tencent Cloud TVP.
Talked with the Tencent Cloud team lead.

In all above channels, we provided the evidences of copy-redistribution hebaviors, requested them to revaluate their statements on the website, and follow the License’s requirements.

Resolution

At Jan. 19th night, UTC+8, 2021. We received response from the Tencent cloud team. They admited their violation behaviors, and did following changes

Tencent Cloud TSW service page states, the agent is the fork version(re-distribution) of Apache SkyWalking agent.
TSW agent distributions include the SkyWalking’s License and NOTICE. Below is the screenshot, you could download from their product page. We keep a copy of their Jan. 19th 2021 at here.

We have updated the status to the PMC mail list. This license violation issue has been resolved for now.

The SkyWalking community and program management committee will keep our eyes on Tencent TSW.

Blog: Response to Elastic 2021 License Change

Sun, 17 Jan 2021 00:00:00 +0000

Elastic announced their license change, Upcoming licensing changes to Elasticsearch and Kibana.

We are moving our Apache 2.0-licensed source code in Elasticsearch and Kibana to be dual licensed under Server Side Public License (SSPL) and the Elastic License, giving users the choice of which license to apply. This license change ensures our community and customers have free and open access to use, modify, redistribute, and collaborate on the code. It also protects our continued investment in developing products that we distribute for free and in the open by restricting cloud service providers from offering Elasticsearch and Kibana as a service without contributing back. This will apply to all maintained branches of these two products and will take place before our upcoming 7.11 release. Our releases will continue to be under the Elastic License as they have been for the last three years.

Also, they provide the FAQ page for more information about the impact for the users, developers, and vendors.

In the perspective of Apache Software Foundation, SSPL has been confirmed as a Catalog X LICENSE(https://www.apache.org/legal/resolved.html#category-x), which means hard-dependency as a part of the core is not allowed. With that, we can’t only focus on it anymore. We need to consider other storage options. Right now, we still have InfluxDB, TiDB, H2 server still in Apache 2.0 licensed. Right now, we still have InfluxDB, TiDB, H2 server as storage options still in Apache 2.0 licensed.

As one optional plugin, we need to focus on the client driver license. Right now, we are only using ElasticSearch 7.5.0 and 6.3.2 drivers, which are both Apache 2.0 licensed. So, we are safe. For further upgrade, here is their announcement. They answer these typical cases in the FAQ page.

I build a SaaS application using Elasticsearch as the backend, how does this affect me?

This source code license change should not affect you - you can use our default distribution or develop applications on top of it for free, under the Elastic License. This source-available license does not contain any copyleft provisions and the default functionality is free of charge. For a specific example, you can see our response to a question around this at Magento.

Our users still could use, redistribute, sale the products/services, based on SkyWalking, even they are using self hosting Elastic Search unmodified server.

I’m using Elasticsearch via APIs, how does this change affect me?

This change does not affect how you use client libraries to access Elasticsearch. Our client libraries remain licensed under Apache 2.0, with the exception of our Java High Level Rest Client (Java HLRC). The Java HLRC has dependencies on the core of Elasticsearch, and as a result this client library will be licensed under the Elastic License. Over time, we will eliminate this dependency and move the Java HLRC to be licensed under Apache 2.0. Until that time, for the avoidance of doubt, we do not consider using the Java HLRC as a client library in development of an application or library used to access Elasticsearch to constitute a derivative work under the Elastic License, and this will not have any impact on how you license the source code of your application using this client library or how you distribute it.

The client driver license incompatible issue will exist, we can’t upgrade the driver(s) until they release the Apache 2.0 licensed driver jars. But users are still safe to upgrade the drivers by themselves.

Apache SkyWalking will discuss the further actions here. If you have any question, welcome to ask. In the later 2021, we will begin to invest the posibility of creating SkyWalking’s observability database implementation.

Blog: Celebrate SkyWalking single repository hits the 300 contributors mark

Tue, 22 Dec 2020 00:00:00 +0000

Apache SkyWalking is an open source APM for distributed system. Provide tracing, service mesh observability, metrics analysis, alarm and visualization.

Just 11 months ago, on Jan. 20th, 2020, SkyWalking hit the 200 contributors mark. With the growth of the project and the community, SkyWalking now includes over 20 sub(ecosystem) projects covering multiple language agents and service mesh, integration with mature open source projects, like Prometheus, Spring(Sleuth), hundreds of libraries to support all tracing/metrics/logs fields. In the past year, the number of contributors grows super astoundingly , and all its metrics point to its community vibrancy. Many corporate titans are already using SkyWalking in a large-scale production environment, including, Alibaba, Huawei, Baidu, Tencent, etc.

Recently, our SkyWalking main repository overs 300 contributors.

Our website has thousands of views from most countries in the world every week.

Although we know that, the metrics like GitHub stars and the numbers of open users and contributors, are not a determinant of vibrancy, they do show the trend, we are very proud to share the increased numbers here, too.

We double those numbers and are honored with the development of our community.

Thank you, all of our contributors. Not just these 300 contributors of the main repository, or nearly 400 contributors in all repositories, counted by GitHub. There are countless people contributing codes to SkyWalking’s subprojects, ecosystem projects, and private fork versions; writing blogs and guidances, translating documents, books, and presentations; setting up learning sessions for new users; convincing friends to join the community as end-users, contributors, even committers. Companies behinds those contributors support their employees to work with the community to provide feedback and contribute the improvements and features upstream. Conference organizers share the stages with speakers from the SkyWalking community.

SkyWalking can’t make this happen without your help. You made this community extraordinary.

At this crazy distributed computing and cloud native age, we as a community could make DEV, OPS, and SRE teams’ work easier by locating the issue(s) in the haystack quicker than before, like why we named the project as SkyWalking, we will have a clear site line when you stand on the glass bridge Skywalk at Grand Canyon West.

376 Contributors counted by GitHub account are following. Dec. 22st, 2020. Generated by a tool deveoped by Yousa

1095071913
50168383
Ahoo-Wang
AirTrioa
AlexanderWert
AlseinX
Ax1an
BFergerson
BZFYS
CharlesMaster
ChaunceyLin5152
CommissarXia
Cvimer
Doublemine
ElderJames
EvanLjp
FatihErdem
FeynmanZhou
Fine0830
FingerLiu
Gallardot
GerryYuan
HackerRookie
Heguoya
Hen1ng
Humbertzhang
IanCao
IluckySi
Indifer
J-Cod3r
JaredTan95
Jargon96
Jijun
JohnNiang
Jozdortraz
Jtrust
Just-maple
KangZhiDong
LazyLei
LiWenGu
Liu-XinYuan
Miss-you
O-ll-O
Patrick0308
QHWG67
Qiliang
RandyAbernethy
RedzRedz
Runrioter
SataQiu
ScienJus
SevenPointOld
ShaoHans
Shikugawa
SoberChina
SummerOfServenteen
TJ666
TerrellChen
TheRealHaui
TinyAllen
TomMD
ViberW
Videl
WALL-E
WeihanLi
WildWolfBang
WillemJiang
Wooo0
XhangUeiJong
Xlinlin
YczYanchengzhe
YoungHu
YunaiV
ZhHong
ZhuoSiChen
ZS-Oliver
a198720
a526672351
acurtain
adamni135
adermxzs
adriancole
aeolusheath
agile6v
aix3
aiyanbo
ajanthan
alexkarezin
alonelaval
amogege
amwyyyy
arugal
ascrutae
augustowebd
bai-yang
beckhampu
beckjin
beiwangnull
bigflybrother
bostin
brucewu-fly
c1ay
candyleer
carlvine500
carrypann
cheenursn
cheetah012
chenpengfei
chenvista
chess-equality
chestarss
chidaodezhongsheng
chopin-d
clevertension
clk1st
cngdkxw
codeglzhang
codelipenghui
coder-yqj
coki230
coolbeevip
crystaldust
cui-liqiang
cuiweiwei
cyberdak
cyejing
dagmom
dengliming
devkanro
devon-ye
dimaaan
dingdongnigetou
dio
dmsolr
dominicqi
donbing007
dsc6636926
duotai
dvsv2
dzx2018
echooymxq
efekaptan
eoeac
evanxuhe
feelwing1314
fgksgf
fuhuo
geektcp
geomonlin
ggndnn
gitter-badger
glongzh
gnr163
gonedays
grissom-grissom
grissomsh
guodongq
guyukou
gxthrj
gzshilu
hailin0
hanahmily
haotian2015
haoyann
hardzhang
harvies
hepyu
heyanlong
hi-sb
honganan
hsoftxl
huangyoje
huliangdream
huohuanhuan
innerpeacez
itsvse
jasonz93
jialong121
jinlongwang
jjlu521016
jjtyro
jmjoy
jsbxyyx
justeene
juzhiyuan
jy00464346
kaanid
karott
kayleyang
kevinyyyy
kezhenxu94
kikupotter
kilingzhang
killGC
klboke
ksewen
kuaikuai
kun-song
kylixs
landonzeng
langke93
langyan1022
langyizhao
lazycathome
leemove
leizhiyuan
libinglong
lilien1010
limfriend
linkinshi
linliaoy
liuhaoXD
liuhaoyang
liuyanggithup
liuzhengyang
liweiv
lkxiaolou
llissery
louis-zhou
lpf32
lsyf
lucperkins
lujiajing1126
lunamagic1978
lunchboxav
lxliuxuankb
lytscu
lyzhang1999
magic-akari
makingtime
maolie
masterxxo
maxiaoguang64
membphis
mestarshine
mgsheng
michaelsembwever
mikkeschiren
mm23504570
momo0313
moonming
mrproliu
muyun12
nacx
neatlife
neeuq
nic-chen
nikitap492
nileblack
nisiyong
novayoung
oatiz
oflebbe
olzhy
onecloud360
osiriswd
peng-yongsheng
pengweiqhca
potiuk
purgeyao
qijianbo010
qinhang3
qiuyu-d
qqeasonchen
qxo
raybi-asus
refactor2
remicollet
rlenferink
rootsongjc
rovast
scolia
sdanzo
seifeHu
shiluo34
sikelangya
simonlei
sk163
snakorse
songzhendong
songzhian
sonxy
spacewander
stalary
stenio2011
stevehu
stone-wlg
sungitly
surechen
swartz-k
sxzaihua
tanjunchen
tankilo
taskmgr
tbdpmi
terranhu
terrymanu
tevahp
thanq
thebouv
tianyuak
tincopper
tinyu0
tom-pytel
tristaZero
tristan-tsl
trustin
tsuilouis
tuohai666
tzsword-2020
tzy1316106836
vcjmhg
vision-ken
viswaramamoorthy
wankai123
wbpcode
web-xiaxia
webb2019
weiqiang333
wendal
wengangJi
wenjianzhang
whfjam
wind2008hxy
withlin
wqr2016
wu-sheng
wuguangkuo
wujun8
wuxingye
x22x22
xbkaishui
xcaspar
xiaoxiangmoe
xiaoy00
xinfeingxia85
xinzhuxiansheng
xudianyang
yanbw
yanfch
yang-xiaodong
yangxb2010000
yanickxia
yanmaipian
yanmingbi
yantaowu
yaowenqiang
yazong
ychandu
ycoe
yimeng
yu199195
yuqichou
yuyujulin
yymoth
zaunist
zaygrzx
zcai2
zeaposs
zhang98722
zhanghao001
zhangjianweibj
zhangkewei
zhangsean
zhaoyuguang
zhentaoJin
zhousiliang163
zhuCheer
zifeihan
zkscpqm
zoidbergwill
zoumingzm
zouyx
zshit
zxbu
zygfengyuwuzu

Blog: [Video] Another backend storage solution for the APM system

Sat, 21 Nov 2020 00:00:00 +0000

The APM system provides the tracing or metrics for distributed systems or microservice architectures. Back to APM themselves, they always need backend storage to store the necessary massive data. What are the features required for backend storage? Simple, fewer dependencies, widely used query language, and the efficiency could be into your consideration. Based on that, traditional SQL databases (like MySQL) or NoSQL databases would be better choices. However, this topic will present another backend storage solution for the APM system viewing from NewSQL. Taking Apache Skywalking for instance, this talking will share how to make use of Apache ShardingSphere, a distributed database middleware ecosystem to extend the APM system’s storage capability.

As a senior DBA worked at JD.com, the responsibility is to develop the distributed database and middleware, and the automated management platform for database clusters. As a PMC of Apache ShardingSphere, I am willing to contribute to the OS community and explore the area of distributed databases and NewSQL.

Blog: [Video] Improve Apache APISIX observability with Apache SkyWalking

Sat, 21 Nov 2020 00:00:00 +0000

Apache APISIX is a cloud-native microservices API gateway, delivering the ultimate performance, security, open-source and scalable platform for all your APIs and microservices. Apache SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud-native and container-based (Docker, Kubernetes, Mesos) architectures. Through the powerful plug-in mechanism of Apache APISIX, Apache Skywalking is quickly supported, so that we can see the complete life cycle of requests from the edge to the internal service. Monitor and manage each request in a visual way, and improve the observability of the service.

Blog: [Video] SourceMarker - Continuous Feedback for Developers

Sat, 21 Nov 2020 00:00:00 +0000

Today’s monitoring solutions are geared towards operational tasks, displaying behavior as time-series graphs inside dashboards and other abstractions. These abstractions are immensely useful but are largely designed for software operators, whose responsibilities require them to think in systems, rather than the underlying source code. This is problematic given that an ongoing trend of software development is the blurring boundaries between building and operating software. This trend makes it increasingly necessary for programming environments to not just support development-centric activities, but operation-centric activities as well. Such is the goal of the feedback-driven development approach. By combining IDE and APM technology, software developers can intuitively explore multiple dimensions of their software simultaneously with continuous feedback about their software from inception to production.

Brandon Fergerson is an open-source software developer who does not regard himself as a specialist in the field of programming, but rather as someone who is a devoted admirer. He discovered the beauty of programming at a young age and views programming as an art and those who do it well to be artists. He has an affinity towards getting meta and combining that with admiration of programming, has found source code analysis to be exceptionally interesting. Lately, his primary focus involves researching and building AI-based pair programming technology.

Blog: [Video] The history of distributed tracing storage

Sat, 21 Nov 2020 00:00:00 +0000

Over the past few years, and coupled with the growing adoption of microservices, distributed tracing has emerged as one of the most commonly used monitoring and troubleshooting methodologies. New tracing tools are increasingly being introduced, driving adoption even further. One of these tools is Apache SkyWalking, a popular open-source tracing, and APM platform. This talk explores the history of the SkyWalking storage module, shows the evolution of distributed tracing storage layers, from the traditional relational database to document-based search engine. I hope that this talk contributes to the understanding of history and also that it helps to clarify the different types of storage that are available to organizations today.

Hongtao Gao is the engineer of tetrate.io and the former Huawei Cloud expert. One of PMC members of Apache SkyWalking and participates in some popular open-source projects such as Apache ShardingSphere and Elastic-Job. He has an in-depth understanding of distributed databases, container scheduling, microservices, ServicMesh, and other technologies.

Blog: [Video] Apache grows in China

Fri, 20 Nov 2020 00:00:00 +0000

Sheng Wu is a founding engineer at tetrate.io, leads the observability for service mesh and hybrid cloud. A searcher, evangelist, and developer in the observability, distributed tracing, and APM. He is a member of the Apache Software Foundation. Love open source software and culture. Created the Apache SkyWalking project and being its VP and PMC member. Co-founder and PMC member of Apache ShardingSphere. Also as a PMC member of Apache Incubator and APISIX. He is awarded as Microsoft MVP, Alibaba Cloud MVP, Tencent Cloud TVP.

In the Apache FY2020 report, China is on the top of the download statistics. More China initiated projects joined the incubator, and some of them graduated as the Apache TLP. Sheng joined the Apache community since 2017, in the past 3 years, he witnessed the growth of the open-source culture and Apache way in China. Many developers have joined the ASF as new contributors, committers, foundation members. Chinese enterprises and companies paid more attention to open source contributions, rather than simply using the project like before. In the keynote, he would share the progress about China embracing the Apache culture, and willing of enhancing the whole Apache community.

Blog: Observability at Scale: SkyWalking it is

Tue, 11 Aug 2020 00:00:00 +0000

Author: Sheng Wu
Original link, Tetrate.io blog

SkyWalking, a top-level Apache project, is the open source APM and observability analysis platform that is solving the problems of 21st-century systems that are increasingly large, distributed, and heterogenous. It’s built for the struggles system admins face today: To identify and locate needles in a haystack of interdependent services, to get apples-to-apples metrics across polyglot apps, and to get a complete and meaningful view of performance.

SkyWalking is a holistic platform that can observe microservices on or off a mesh, and can provide consistent monitoring with a lightweight payload.

Let’s take a look at how SkyWalking evolved to address the problem of observability at scale, and grew from a pure tracing system to a feature-rich observability platform that is now used to analyze deployments that collect tens of billions of traces per day.

Designing for scale

When SkyWalking was first initialized back in 2015, its primary use case was monitoring the first-generation distributed core system of China Top Telecom companies, China Unicom and China Mobile. In 2013-2014, the telecom companies planned to replace their old traditional monolithic applications with a distributed system. Supporting a super-large distributed system and scaleablity were the high-priority design goals from Day one. So, what matters at scale?

Pull vs. push

Pull and push modes relate to the direction of data flow. If the agent collects data and pushes them to the backend for further analysis, we call it “push” mode. Debate over pull vs. push has gone on for a long time. The key for an observability system is to minimize the cost of the agent, and to be generally suitable for different kinds of observability data.

The agent would send the data out a short period after it is collected. Then, we would have less concern about overloading the local cache. One typical case would be endpoint (URI of HTTP, service of gRPC) metrics. Any service could easily have hundreds, even thousands of endpoints. An APM system must have these metrics analysis capabilities.

Furthermore, metrics aren’t the only thing in the observability landscape; traces and logs are important too. SkyWalking is designed to provide a 100% sampling rate tracing capability in the production environment. Clearly, push mode is the only solution.

At the same time, using push mode natively doesn’t mean SkyWalking can’t do data pulling. In recent 8.x releases, SkyWalking supports fetching data from Prometheus-instrumented services for reducing the Non-Recurring Engineering of the end users. Also, pull mode is popular in the MQ based transport, typically as a Kafka consumer. The SkyWalking agent side uses the push mode, and the OAP server uses the pull mode.

The conclusion: push mode is the native way, but pull mode works in some special cases too.

Metrics analysis isn’t just mathematical calculation

Metrics rely on mathematical theories and calculations. Percentile is a good measure for identifying the long tail issue, and reasonable average response time and successful rate are good SLO(s). But those are not all. Distributed tracing provides not just traces with detailed information, but high values metrics that can be analyzed.

The service topology map is required from Ops and SRE teams for the NOC dashboard and confirmation of system data flow. SkyWalking uses the STAM (Streaming Topology Analysis Method) to analyze topology from the traces, or based on ALS (Envoy Access Log Service) in the service mesh environment. This topology and metrics of nodes (services) and lines (service relationships) can’t be pulled from simple metrics SDKs.

As with fixing the limitation of endpoint metrics collection, SkyWalking needs to do endpoint dependency analysis from trace data too. Endpoint dependency analysis provides more important and specific information, including upstream and downstream. Those dependency relationships and metrics help the developer team to locate the boundaries of a performance issue, to specific code blocks.

Pre-calculation vs. query stage calculation?

Query stage calculation provides flexibility. Pre-calculation, in the analysis stage, provides better and much more stable performance. Recall our design principle: SkyWalking targets a large-scale distributed system. Query stage calculation was very limited in scope, and most metrics calculations need to be pre-defined and pre-calculated. The key of supporting large datasets is reducing the size of datasets in the design level. Pre-calculation allows the original data to be merged into aggregated results downstream, to be used in a query or even for an alert check.

TTL of metrics is another important business enabler. With the near linear performance offered by queries because of pre-calculation, with a similar query infrastructure, organizations can offer higher TTL, thereby providing extended visibility of performance.

Speaking of alerts, query-stage calculation also means the alerting query is required to be based on the query engine. But in this case, when the dataset increasing, the query performance could be inconsistent. The same thing happens in a different metrics query.

Cases today

Today, SkyWalking is monitoring super large-scale distributed systems in many large enterprises, including Alibaba, Huawei, Tencent, Baidu, China Telecom, and various banks and insurance companies. The online service companies have more traffic than the traditional companies, like banks and telecom suppliers.

SkyWalking is the observability platform used for a variety of use cases for distributed systems that are super-large by many measures:

Lagou.com, an online job recruitment platform
- SkyWalking is observing >100 services, 500+ JVM instances
- SkyWalking collects and analyzes 4+ billion traces per day to analyze performance data, including metrics of 300k+ endpoints and dependencies
- Monitoring >50k traffic per second in the whole cluster
Yonghui SuperMarket, online service
- SkyWalking analyzes at least 10+ billion (3B) traces with metrics per day
- SkyWalking’s second, smaller deployment, analyzes 200+ million traces per day
Baidu, internet and AI company, Kubernetes deployment
- SkyWalking collects 1T+ traces a day from 1,400+ pods of 120+ services
- Continues to scale out as more services are added
Beike Zhaofang(ke.com), a Chinese online property brokerage backed by Tencent Holdings and SoftBank Group
- Has used SkyWalking from its very beginning, and has two members in the PMC team.
- Deployments collect 16+ billion traces per day
Ali Yunxiao, DevOps service on the Alibaba Cloud,
- SkyWalking collects and analyzes billions of spans per day
- SkyWalking keeps AliCloud’s 45 services and ~300 instances stable
A department of Alibaba TMall, one of the largest business-to-consumer online retailers, spun off from Taobao
- A customized version of SkyWalking monitors billions of traces per day
- At the same time, they are building a load testing platform based on SkyWalking’s agent tech stack, leveraging its tracing and context propagation cabilities

Conclusion

SkyWalking’s approach to observability follows these principles:

Understand the logic model: don’t treat observability as a mathematical tool.
Identify dependencies first, then their metrics.
Scaling should be accomplished easily and natively.
Maintain consistency across different architectures, and in the performance of APM itself.

Resources

Read about the SkyWalking 8.1 release highlights.
Get more SkyWalking updates on Twitter.
Sign up to hear more about SkyWalking and observability from Tetrate.

Blog: SkyWalking hits 200 contributors mark

Mon, 20 Jan 2020 00:00:00 +0000

Author: Wu Sheng, tetrate.io, SkyWalking original creator, SkyWalking V.P.
GitHub, Twitter, Linkedin

The SkyWalking project provides distributed tracing, topology map analysis, service mesh telemetry analysis, metrics analysis and a super cool visualization targeting distributed systems in k8s or traditional VM deployments.

The project is widely used in Alibaba, Huawei, Tencent, DiDi, xiaomi, Pingan, China’s top 3 telecom companies (China Mobile, China telecom, China Unicom), airlines, banks and more. It has over 140 company users listed on our powered by page.

Today, we welcome and celebrate reaching 200 code contributors on our main repo. We hereby mark this milestone as official today, : Jan. 20th 2020.

At this great moment, I would like to share SkyWalking’s 4-year open source journey.

I wrote the first line on Nov. 1st, 2015, guiding people to understand a distributed system just as micro-services and distributed architecture were becoming popular. In the first 2 years, I never thought it would become such a big and active community. I didn’t even expect it would be an open source project. Initially, the goal was primarily to teach others about distributed tracing and analysis.

It was a typical open source project in obscurity in its first two years. But people still showed up, asked questions, and tried to improve the project. I got several invitations to share the project at local meetups.All these made me realize people really needed a good open source APM project.

In 2017, I decided to dedicate myself as much as possible to make the project successful, and it became my day job. To be honest, I had no clue about how to do that; at that time in China, it was rare to have this kind of job. So, I began to ask friends around me, “Do you want to collaborate on the open source APM with me?” Most people were busy and gave a clear NO, but two of them agreed to help: Xin Zhang and Yongsheng Peng. We built SkyWalking 3.x and shared the 3.2 release at GOPS Shanghai, China.

It became the first adoption version used in production

Compared to today’s SkyWalking, it was a toy prototype, but it had the same tracing design, protocol and analysis method.

That year the contributor team was 15-20, and the project had obvious potential to expand. I began to consider bringing the project into a worldwide, top-level open source foundation. Thanks to our initial incubator mentors, Michael Semb Wever, William Jiang, and Luke Han, this really worked. At the end of 2017, SkyWalking joined the Apache Incubator, and kept following the Apache Way to build community. More contributors joined the community.

With more people spending time on the project collaborations, including codes, tests, blogs, conference talks, books and uses of the project, a chemical reaction happens. New developers begin to provide bug fixes, new feature requirements and new proposals. At the moment of graduation in spring 2019, the project had 100 contributors. Now, only 9 months later, it’s surged to 200 super quickly. They enhance the project and extend it to frontiers we never imaged: 5 popular language agents, service mesh adoption, CLI tool, super cool visualization. We are even moving on thread profiling, browser performance and Nginx tracing NOW.

Over the whole 4+ years open source journey, we have had supports from leaders in the tracing open source community around the world, including Adrian Cole, William Jiang, Luke Han, Michael Semb Wever, Ben Sigelman, and Jonah Kowall. And we’ve had critical foundations’ help, especially Apache Software Foundation and the Cloud Native Computing Foundation.

Our contributors also have their support from their employers, including, to the best of my knowledge, Alibaba, Huawei, China Mobile, ke.com, DaoCloud, Lizhi.fm, Yonghui Supermarket, and dangdang.com. I also have support from my employers, tetrate.io, Huawei, and OneAPM.

Thanks to our 200+ contributors and the companies behind them. You make this magic happen.