![]() ![]() What caught our interest was that the total CPU time taken for the Go regular expression engine was around 37 percent of the CPU, which was twice the system calls. This makes sense, as every UDP socket operation involves a system call. The largest accumulation of events was system calls. Pprof gave us the percentage of CPU time each functions took in descending order: Then, we used the perf_data_converter tool to convert perf file perf.data to profile.proto and used pprof to analyze the results. This gave us a rough idea of functions taking up most of the CPU time. To take a closer look, we started doing some profiling upon the StatsD exporter and used the perf tool to sample stacks. When the client request rate climbed up to several thousand requests per second, we spotted a high CPU usage of the StatsD exporter that took one and a half cores on an AWS m4.large instance. On Kong Cloud, various StatsD events are generated for each request. In the example below, two StatsD events are translated according to the mapping config to the left. The exporter has a mapping config that maps the StatsD metric to a Prometheus metric. The StatsD Prometheus exporter is a daemon that listens for StatsD events and exports them in Prometheus exposition formats. On Kong Cloud, we use the StatsD Prometheus exporter in our metrics pipeline to measure KPIs (Key Performance Indicator) of our service. The above example represents a metric called with the type of counter and the value of 123. A typical StatsD event looks like:Įvery StatsD event is a string in a format of :|. StatsD is a metrics server that accepts events from UDP or TCP protocol and export them to various backends. In this blog post we discuss the use case of StatsD and Prometheus on Kong Cloud, the performance problem we found, and the way we proposed to solve it. Kong Cloud has been using StatsD and Prometheus heavily in monitoring and metrics collecting.
0 Comments
Leave a Reply. |