Telemetry of Metrics using Prometheus

Infrastructure backends are designed to provide support functionality such as telemetry capturing systems. These services traditionally directly integrate with the backend systems, creating a hard coupling. Istio provides a flexible model to collect telemetry for the services in a mesh. Mixer is the Istio component responsible for providing the telemetry collection.

The Envoy sidecar (proxy) logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only calls Mixer infrequently.

You can find a list of Mixer adapters at https://istio.io/docs/reference/config/policy-and-telemetry/adapters/. Mixer templates are used to send data to individual adapters.

Mixer is in essence an attribute processing machine. An attribute is a small bit of data that describes a single property of a specific service request or the environment for the request. Each attribute has a name and a type. The Envoy is the main producer of attributes in Istio. Here is a list of common attributes in Istio: https://istio.io/docs/reference/config/policy-and-telemetry/attribute-vocabulary/.

An Instance configuration is used to map request attributes to adapter inputs for a handler. Rules specify when a particular handler is invoked with a specific instance configuration.

In the following part, you will add configuration with three pieces of the Mixer functionality to collect metrics:

  1. Generation of an instance from Istio attributes,

  2. Creation of a handler (a configured Mixer adapter) for processing generated instances,

  3. A rule to dispatch instances to handlers,

  • Review the file samples/bookinfo/telemetry/metrics.yaml,

$ cat samples/bookinfo/telemetry/metrics.yaml
# Configuration for metric instances
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
name: doublerequestcount
namespace: istio-system
spec:
compiledTemplate: metric
params:
value: "2" # count each request twice
dimensions:
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "client", "server")
source: source.workload.name | "unknown"
destination: destination.workload.name | "unknown"
message: '"twice the fun!"'
monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
name: doublehandler
namespace: istio-system
spec:
compiledAdapter: prometheus
params:
metrics:
- name: double_request_count # Prometheus metric name
instance_name: doublerequestcount.instance.istio-system # Mixer instance name (fully-qualified)
kind: COUNTER
label_names:
- reporter
- source
- destination
- message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
name: doubleprom
namespace: istio-system
spec:
actions:
- handler: doublehandler
instances: [ doublerequestcount ]
  • The metrics specification adds an instance configuration, a handler for the instance input and a rule to map the input from the instance to the handler,

  • This will display the following metric in Prometheus:

istio_double_request_count{destination="reviews-v3",instance="123.45.678.901:42422",
job="istio-mesh",message="twice the fun!",reporter="server",source="productpage-v1"}
  • Apply the metrics specification,

$ kubectl apply -f samples/bookinfo/telemetry/metrics.yaml
instance.config.istio.io/doublerequestcount created
handler.config.istio.io/doublehandler created
rule.config.istio.io/doubleprom created
  • Make sure that a Prometheus deployment is running,

$ kubectl -n istio-system get deployments -l app=prometheus
NAME READY UP-TO-DATE AVAILABLE AGE
prometheus 1/1 1 1 14h
$ kubectl -n istio-system get pod -l app=prometheus
NAME READY STATUS RESTARTS AGE
prometheus-588dd7d7c5-96qhr 1/1 Running 0 14h
  • Next you can send requests to your Bookinfo application and see the integration with Prometheus,

  • If you need the external IP address again for the ingress Gateway, use the following command,

$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 172.21.123.4 123.45.678.90 15020:32461/TCP,
80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31599/TCP,15030:32349/TCP,
15031:32405/TCP,15032:30296/TCP,15443:30522/TCP 111s
  • Setup port-forwarding for Prometheus to your localhost,

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l \
app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &
[1] 36820
(.virtualenvs) Remkos-MBP:istio-1.3.2 user1$ Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
$ curl -X GET http://$GATEWAY_URL/productpage