remkohdev
  • Learn to Code
  • About Me
  • CI/CD
    • DevOps101
      • Welcome
  • OpenShift
    • Setup OpenShift
      • Setup Minishift
    • Builds
      • Source-to-Image (S2I)
        • Setup S2I
        • Build, Run, Deploy from Source
      • Jenkins Pipeline
    • Jenkins as a Service
      • Setup Jenkins on Openshift
      • Create a Pipeline for Java Spring Boot
  • Istio
    • Setup Istio on IKS
      • Login to IKS
    • Setup Istio on Openshift 3.11
    • Traffic Shifts with a VirtualService
    • Telemetry of Metrics using Prometheus
    • Telemetry of Distributed Tracing using Jaeger
    • Security with Mutual TLS (mTLS)
  • Apache Kafka
    • Setup Apache Kafka on IBM Cloud
    • Setup Apache Kafka on OpenShift
    • Produce and Consume Streams with Kafka Console Tools
    • Produce and Consume Streams with Spring Boot
    • Using the Event Streams CLI
    • Kafka Admin API
  • API Connect
    • APIC CLI
      • Manage API Lifecycle with apic
    • Securing your API
      • Setup AppID
      • Setup API Connect
      • Optional: Add Node-RED Test Server
      • Add 3rd Party OAuth OIDC
        • Create a Custom AppID API
        • Add a Security Definition to your API
Powered by GitBook
On this page

Was this helpful?

  1. Istio

Telemetry of Metrics using Prometheus

PreviousTraffic Shifts with a VirtualServiceNextTelemetry of Distributed Tracing using Jaeger

Last updated 5 years ago

Was this helpful?

Infrastructure backends are designed to provide support functionality such as telemetry capturing systems. These services traditionally directly integrate with the backend systems, creating a hard coupling. Istio provides a flexible model to collect telemetry for the services in a mesh. Mixer is the Istio component responsible for providing the telemetry collection.

The Envoy sidecar (proxy) logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only calls Mixer infrequently.

You can find a list of Mixer adapters at . Mixer templates are used to send data to individual adapters.

An Instance configuration is used to map request attributes to adapter inputs for a handler. Rules specify when a particular handler is invoked with a specific instance configuration.

In the following part, you will add configuration with three pieces of the Mixer functionality to collect metrics:

  1. Generation of an instance from Istio attributes,

  2. Creation of a handler (a configured Mixer adapter) for processing generated instances,

  3. A rule to dispatch instances to handlers,

  • Review the file samples/bookinfo/telemetry/metrics.yaml,

$ cat samples/bookinfo/telemetry/metrics.yaml
# Configuration for metric instances
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: doublerequestcount
  namespace: istio-system
spec:
  compiledTemplate: metric
  params:
    value: "2" # count each request twice
    dimensions:
      reporter: conditional((context.reporter.kind | "inbound") == "outbound", "client", "server")
      source: source.workload.name | "unknown"
      destination: destination.workload.name | "unknown"
      message: '"twice the fun!"'
    monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: doublehandler
  namespace: istio-system
spec:
  compiledAdapter: prometheus
  params:
    metrics:
    - name: double_request_count # Prometheus metric name
      instance_name: doublerequestcount.instance.istio-system # Mixer instance name (fully-qualified)
      kind: COUNTER
      label_names:
      - reporter
      - source
      - destination
      - message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: doubleprom
  namespace: istio-system
spec:
  actions:
  - handler: doublehandler
    instances: [ doublerequestcount ]
  • The metrics specification adds an instance configuration, a handler for the instance input and a rule to map the input from the instance to the handler,

  • This will display the following metric in Prometheus:

istio_double_request_count{destination="reviews-v3",instance="123.45.678.901:42422",
job="istio-mesh",message="twice the fun!",reporter="server",source="productpage-v1"}
  • Apply the metrics specification,

$ kubectl apply -f samples/bookinfo/telemetry/metrics.yaml
instance.config.istio.io/doublerequestcount created
handler.config.istio.io/doublehandler created
rule.config.istio.io/doubleprom created
  • Make sure that a Prometheus deployment is running,

$ kubectl -n istio-system get deployments -l app=prometheus
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
prometheus   1/1     1            1           14h
$ kubectl -n istio-system get pod -l app=prometheus
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-588dd7d7c5-96qhr   1/1     Running   0          14h
  • Next you can send requests to your Bookinfo application and see the integration with Prometheus,

  • If you need the external IP address again for the ingress Gateway, use the following command,

$ kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)   AGE
istio-ingressgateway   LoadBalancer   172.21.123.4   123.45.678.90   15020:32461/TCP,
80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31599/TCP,15030:32349/TCP,
15031:32405/TCP,15032:30296/TCP,15443:30522/TCP   111s
  • Setup port-forwarding for Prometheus to your localhost,

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l \
app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &
[1] 36820
(.virtualenvs) Remkos-MBP:istio-1.3.2 user1$ Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
  • Send requests to the Bookinfo application, which will show up immediately in the Prometheus dashboard,

$ curl -X GET http://$GATEWAY_URL/productpage

Mixer is in essence an attribute processing machine. An attribute is a small bit of data that describes a single property of a specific service request or the environment for the request. Each attribute has a name and a type. The Envoy is the main producer of attributes in Istio. Here is a list of common attributes in Istio: .

Open Prometheus in a browser on your localhost via ,

https://istio.io/docs/reference/config/policy-and-telemetry/attribute-vocabulary/
http://localhost:9090/graph?g0.range_input=1h&g0.expr=istio_double_request_count&g0.tab=1
https://istio.io/docs/reference/config/policy-and-telemetry/adapters/