Telemetry of Metrics using Prometheus

Infrastructure backends are designed to provide support functionality such as telemetry capturing systems. These services traditionally directly integrate with the backend systems, creating a hard coupling. Istio provides a flexible model to collect telemetry for the services in a mesh. Mixer is the Istio component responsible for providing the telemetry collection.

The Envoy sidecar (proxy) logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only calls Mixer infrequently.

You can find a list of Mixer adapters at https://istio.io/docs/reference/config/policy-and-telemetry/adapters/. Mixer templates are used to send data to individual adapters.

Mixer is in essence an attribute processing machine. An attribute is a small bit of data that describes a single property of a specific service request or the environment for the request. Each attribute has a name and a type. The Envoy is the main producer of attributes in Istio. Here is a list of common attributes in Istio: https://istio.io/docs/reference/config/policy-and-telemetry/attribute-vocabulary/.

An Instance configuration is used to map request attributes to adapter inputs for a handler. Rules specify when a particular handler is invoked with a specific instance configuration.

In the following part, you will add configuration with three pieces of the Mixer functionality to collect metrics:

1. Generation of an instance from Istio attributes,

2. Creation of a handler (a configured Mixer adapter) for processing generated instances,

3. A rule to dispatch instances to handlers,

• Review the file samples/bookinfo/telemetry/metrics.yaml,

$ cat samples/bookinfo/telemetry/metrics.yaml
# Configuration for metric instances
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: doublerequestcount
  namespace: istio-system
spec:
  compiledTemplate: metric
  params:
    value: "2" # count each request twice
    dimensions:
      reporter: conditional((context.reporter.kind | "inbound") == "outbound", "client", "server")
      source: source.workload.name | "unknown"
      destination: destination.workload.name | "unknown"
      message: '"twice the fun!"'
    monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: doublehandler
  namespace: istio-system
spec:
  compiledAdapter: prometheus
  params:
    metrics:
    - name: double_request_count # Prometheus metric name
      instance_name: doublerequestcount.instance.istio-system # Mixer instance name (fully-qualified)
      kind: COUNTER
      label_names:
      - reporter
      - source
      - destination
      - message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: doubleprom
  namespace: istio-system
spec:
  actions:
  - handler: doublehandler
    instances: [ doublerequestcount ]

• The metrics specification adds an instance configuration, a handler for the instance input and a rule to map the input from the instance to the handler,

• This will display the following metric in Prometheus:

istio_double_request_count{destination="reviews-v3",instance="123.45.678.901:42422",
job="istio-mesh",message="twice the fun!",reporter="server",source="productpage-v1"}

• Apply the metrics specification,

$ kubectl apply -f samples/bookinfo/telemetry/metrics.yaml
instance.config.istio.io/doublerequestcount created
handler.config.istio.io/doublehandler created
rule.config.istio.io/doubleprom created

• Make sure that a Prometheus deployment is running,

$ kubectl -n istio-system get deployments -l app=prometheus
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
prometheus   1/1     1            1           14h
$ kubectl -n istio-system get pod -l app=prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-588dd7d7c5-96qhr   1/1     Running   0          14h

• Next you can send requests to your Bookinfo application and see the integration with Prometheus,

• If you need the external IP address again for the ingress Gateway, use the following command,

$ kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)   AGE
istio-ingressgateway   LoadBalancer   172.21.123.4   123.45.678.90   15020:32461/TCP,
80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31599/TCP,15030:32349/TCP,
15031:32405/TCP,15032:30296/TCP,15443:30522/TCP   111s

• Setup port-forwarding for Prometheus to your localhost,

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l \
app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &
[1] 36820
(.virtualenvs) Remkos-MBP:istio-1.3.2 user1$ Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

• Open Prometheus in a browser on your localhost via http://localhost:9090/graph?g0.range_input=1h&g0.expr=istio_double_request_count&g0.tab=1,

• Send requests to the Bookinfo application, which will show up immediately in the Prometheus dashboard,

$ curl -X GET http://$GATEWAY_URL/productpage