3rd Karrot SRE Meetup

2023.6.15

1. What has Karrot built in the last 2 years? (Byun Junghoon) #

Karrot Developer Platform! #

Reasons for operating Kontrol (Developer Platform)

Service Development && SRE

Service Development → Request | Infrastructure ← SRE

Engineer ticket request → SRE task (automation, script) → Infrastructure

Direction pursued by the Service Development Team

• Rapid changes
• Diverse attempts

Direction pursued by the SRE Team

• Stable operations
• Policy application
• Consistent form
• Approval

Growing organization and service scale #

+200 Engineers

4 Regions

8 Clusters

+200 Namespaces (Each namespace can be seen as one project)

Developer Self-Service (Engineers directly manage infrastructure)

Kontrol #

Project control plane provided by the SRE team

Deployment: November 2021

Usage: Create a Dockerfile as a base, then manually register the project initially → a guide appears. It abstracts Kubernetes to define service routing for main deployment testing → it also provides actions, so deployment is possible → when you click deploy (it fetches the latest commit from the main branch of the repository and starts the deployment pipeline).

There's a feature within Kontrol to create Docker images and push them.

yml → converts to Kubernetes yml for use.

Clicking the deployed server URL immediately brings up the server in the Karrot server cluster.

(It addressed the drawback of Argo CD where projects were scattered, making it hard to distinguish which project was which. It allows deployment per employee, not per GitHub ID, and visualizes the history of who deployed what and when).

Real-time visualization of main deployment testing (Kubernetes)

Kontrol → GoCD → Kubernetes Cluster

Environment Variables

When creating Kontrol, secrets were integrated into AWS Secret Manager.

Previously, Argo CD managed everything separately with yml files (from a Kubernetes resource perspective).

Created yml files for each type of config to set environment variables.

CronJob → Schedule, want to trigger manually for debugging... enabled manual execution (provides log records).

Expanded Kontrol by attaching a DB, implemented global deployment Kontrol, replaced GoCD with Argo Workflows.

Projects complementing Kontrol #

• katalog: A DevOps project that facilitates project ownership management. It's integrated with Kontrol in the backend.
• kost: A cost visualization project. There was a cost issue. Since resource ownership began to be managed, cost visualization started. (Graphs, daily, hourly visualization, not prioritizing discount rates... checking if costs unexpectedly increased). This is also integrated with Kontrol in the backend.
• krp: A project that helps easily create resources.

Benefits gained from the platform

1. Reduced infrastructure task requests
2. Reduced configuration debugging time
3. Batch application for configuration changes
4. Unified infrastructure.

2. Things I wish I had known before building an enterprise-wide deployment system (Kim Gyuhwan) #

Experiences and insights gained while building Kontrol

Table of Contents #

1. Deployment System Characteristics and Current Status
2. Relationship between Developers and SREs
3. Providing Operational Context
4. Summary

Deployment System Characteristics and Current Status #

Container application

Dockerfile and Jib build support

Low threshold

Self Service

Kontrol Deployment Status #

Kontrol 45 → 129

ArgoCD 122 → 134

Average 76 deployments per day

Relationship between Developers and SREs #

SRE: Configuration constraints for stability

Developer: Productivity improvement

Developer perspective ≠ SRE perspective

Developer

• Fast deployment
• Immediate rollback in case of issues
• Quick problem identification

SRE

• Stable operations
• Consistent management
• Provide low threshold

Conflict between Kontrol concept and development team's unique culture

Kontrol concept

• Images are built only once, regardless of environment.

Development team situation

• Development teams building applications that require separate images for each environment.

Low adoption rate due to conflict

Loss of Trust #

A system nobody wants → potential risk

If this spreads, it won't be used company-wide.

Incentives are needed to align interests.

Utilize key operational metrics.

Causes of deployment request failures

Ratio of non-deployment user task types

Number of deployments per period

Deployment time per deployment

Deployment failure rate

→ The earlier you collect, the better. → Collect as early as possible.

Providing Operational Context #

As much context as possible is needed to understand what will happen next.

If appropriate context is provided

• Reduced communication costs between developers and SREs
• Faster identification of problem causes

Difficulty in checking config and Secret change history.

Must navigate between platforms to check.

Problem of context not being properly shared during outages → communication difficulties.

Deployment

Instance restart

Replica scale in/out

Provide necessary context for each situation in the appropriate place.

Display how much traffic is coming in.

Display what events are occurring.

Summary #

Operate with the concept of paved roads vs. unpaved roads.

Prepare key metrics as quickly as possible.

Provide meaningful context in appropriate places.

3. Click! Click! The Story of Multi-Cluster Support for Global Expansion (Yoo Byunghwa) #

June 2022

Argo CD vs Kontrol

ArgoCD

• Requires good knowledge of K8S
• Requests via JIRA, handled by people
• Central control is difficult, configuration fragmentation issues
• High degree of freedom
• Global region deployment possible outside Korea

Kontrol

• Low entry barrier
• Self-service
• Easy central control, standardization = free lunch, win-win
• Not a high degree of freedom
• Only Korean region deployment possible

Kontrol was not used because global service deployment was not supported ㅜㅜ

→ Delivery Part OKR START from Q2 2022!

Kontrol will support global deployment by Q1 2023!

Where to set the region list?

Option A. Example of values.yaml file in project repository

 regions:
	 - kr
	 - ca

Problem: If 'ca' is deleted and deployed, should the project in the 'ca' service region be taken down?

Deployment Pipeline

Prepare

• Receives which project and Git revision to deploy
• Clones the Git project, checks out to that revision

Kontrol creates 3 project pipelines in stages when a project is created.

→ Moves to Stateless pipeline

• Kontrol generates a deployment ID, uses Auto Increment value in DB
• Kontrol uses DB for optimistic locking to implement mutex

Environment Variable Management

• One Config references a maximum of 8 Secrets (AWS Secret Manager)
• Started managing these Secrets in DB for atomic state management

The Big 3 that enhanced the development experience #

1. TypeScript

Few bugs were found during large-scale work, and it allowed for confident major changes.

1. Prisma

A next-generation ORM (Object Relational Mapping) framework gaining attention in the JavaScript and TypeScript communities.

1. Graphite

Enabled high-quality reviews from a code reviewer's perspective (allowed for stack change-based development).

GoCD is a good tool, but it didn't fit Kontrol.

4. Building a DevOps Foundation through Project Ownership Management #

Cloud Resource Ownership Management #

What if there's no resource owner? If there's no owner for cloud resources, you can't ask questions about various issues.

Attach tags? Tagging work.

Problem

• What if chatA's name changes to ChatA Team? → Change everything one by one?!
• What if the ownership of that resource transfers to ChatB Team? → Change everything one by one?!
• What if Terraform sync also needs to be considered when changing that tag?

Kontrol needed ownership...

Need for Ownership in the Platform #

There is one Kontrol, and many services exist on top of it (chat, delivery, etc.). It was difficult to distinguish ownership just by project name.

We need a platform suitable for DevOps...

Let's integrate and manage reliable ownership information in one place!

katalog service description, architecture, future #

1. Service

By providing project ownership and creating a service that manages various metadata based on ownership, we can build a DevOps foundation that can directly/indirectly provide insights. → Developed the katalog service (karrot + service catalog).

katalog

Deployed from each Kontrol.

Cloud resource ownership managed in katalog.

After introducing katalog

Manage resource ownership using the Team's code value from the internal HR API → Even if team/organization names change, there's no issue because they have unique values.

1. Architecture

Adopted Go.

AWS manages using the Tag Editor API.

By utilizing the AWS Resource Groups Tag Editor API, resources can be queried or ownership tags can be changed using the catalog ID for each resource.

GCP manages resource ownership by querying with service-specific APIs and labels.

GCP Cloud Asset API filters based on specific APIs and labels.

Drawback → AWS has a label update feature, but GCP does not ㅠ.

However, each service's API could fetch more resources at once compared to the Cloud Asset API.

1. Future

2. Expansion of Katalog: Metadata managed by Katalog, language, service tiering, project SLO, etc.

   1. e.g., Expansion of data managed by Katalog such as Datadog's language, service tiering, project SLO.

3. Expansion of services using Katalog: Expansion of programs that use the Katalog API → ownership-based notifications, cost dashboards, self-service cloud resource operations, ownership-based ... services will expand.

5. Cloud Resources Managed Directly by Developers (Kim Seung-ho) #

Manual (startup) → Code (development team grows) → Manual (infrastructure team formed) → Code (infrastructure team grows)

Karrot Story #

Karrot adopted Terraform surprisingly early.

Manual -? Terraform (startup phase)

• Expected to be able to isolate cloud resources by country.
  • If one country is well-built with Terraform, the rest can be done easily?!
• Development teams can directly create and manage resources (Terraform code PR or console operations possible).
• Acknowledged the reality that not every development team can have Terraform developers (= can request the infrastructure team).

Terraform operations started to become a burden.

1000 Terraform states...

What if I need to add a resource to a project created 3 years ago and never modified?

Matching Terraform versions, provider versions, and states can take a whole day.

Apply only to specific resources → Why use Terraform if it's like this ㅠㅠ

Solution

→ Give each development team a cloud account and entrust infrastructure management:

• Burden on development team personnel
• Infrastructure inefficiency

vs

→ Take away all developer team permissions and let only the infrastructure team manage:

• Burden on infrastructure personnel, reduced developer team efficiency
• Doesn't fit Karrot's development culture

Real solution

• Not as difficult as Terraform
• Without waiting for the infrastructure team
• While adhering to internal management rules
• What if developers could directly manage cloud resources? → Kontrol is here!

Introducing Krp

Krp #

Goal: Maximize value delivery to users.

• Adopted Python/FastAPI
  • Postpone DB adoption as much as possible.
  • If a DB was needed from the start, Django would have been used.
• AWS
  • Most resources exist on AWS.
• Boto3 library for AWS API calls
  • Decided not to rely on Terraform breaking up with Terraform ㅠㅠ
  • Anticipated pain when plan breaks or state doesn't match.

Goals

1. Reduce recurring AWS resource creation Jira issues to 0.
2. Reduce time complexity of resource creation across multi-environment, multi-region from O(n) to O(1).
3. Eliminate missing resource creations.

Karrot rules applied behind the UI

• Create subnet groups per Redis instance.
• Specify account environment variable values.
  • Subnets or security groups
  • AZ, snapshot window, maintenance window
• Apply ownership tags received from Katalog.
• Specify Parameters Group per engine version.
  • Control keys command usage.

Redis feature additions

• Create Datadog dashboard for monitoring.
• Send messages to Slack.
• Adjust replica count.
• Adjust shard count.
• Modify node type.
• Delete request.
• Management history.

Retrospective #

• Postpone DB adoption as much as possible.
  • Initially, no DB allowed rapid feature development.
  • Felt the need after about 10 months.
• Boto3 AWS API is more complex than expected, maybe Terraform isn't so bad after all.
  • Excessively fragmented API client.
  • Internally asynchronous, so responses are fast but cumbersome.
  • If creation fails midway, need to delete resources created in between.
  • But there's no alternative...

Additional resources being handled

• StaticSite (S3 + Cloudfront + Route53)
• DynamoDB

6. How Karrot Monitors Services (Bae Sangik) #

1. Karrot's Past and Present #

August 26, 2021 (about 1.5 years ago)

HTTP 75k rps, gRPC 60k rps

→ HTTP 145k rps, gRPC 344k rps (June 10, 2023)

MAU from 12 million to 18 million!

Organizational structure and scalability

Before building a tall building, a solid foundation is paramount!

Refining Infrastructure Policies #

If there's no alignment within the infrastructure team, developers can also be confused.

Always be wary of technical debt, fragmentation, and costs.

Policies are very important in defining and streamlining metadata.

By taking good care of common monitoring areas, we support developers in areas they might miss.

2. Monitoring Methodologies #

USE method

• utilization
• saturation
• errors

RED method

• rate
• errors
• duration

Four golden Signals

Karrot Monitoring Infrastructure

Mainly internal monitoring infrastructure composed of Prometheus, Cortex, Loki, etc.

Sub-assistance from Datadog (effectively utilizing Watchdog, APM, Cloud Integration).

Implementing monitoring methodologies tailored to our situation.

• Service Dashboard
  • Provided to help developers better understand their services based on namespaces. Consistent dashboards foster discussion and growth.
• Service Overall Dashboard
  • Goes beyond the namespace standard.

K8s group designed to effectively show common Kubernetes events/status.

Operating a service overall dashboard (global query based on region).

Global monitoring and broken window management.

After anomaly detection based on monitoring, and narrowing down the debugging scope based on alerts,

Root causes are often found using logs accumulated in Loki, separate data pipelines, APM, etc.

Dashboard accumulation according to changing situations

Outage log and dashboard enhancement

• Writing 5 Whys and lessons learned in outage logs.
• Sharing with each other at weekly developer team meetings.
• Follow-up actions to reinforce the infrastructure team.

Q1 OKR Cost Reduction

Node monitoring enhancement due to Container CPU limit release.

• CPU, Memory, Disk, Network... etc.

Pod eviction list and dashboard enhancement due to Spot Instance production adoption.

Region skew monitor added due to Locality Load Balancing application.

Building an Infrastructure Doctor Dashboard

• Istio Doctor Dashboard
• HAproxy Doctor Dashboard
• Kube-system Doctor Dashboard

This can reduce unnecessary debugging time for engineers by providing a high-level judgment of whether an issue is infrastructure-related or not.

Infra Planning

• Used for capacity planning with the help of metrics and dashboards.
• Judging work priority and issue level by considering impact.
• Analyzing until the next upgrade based on issues and dashboard data.

Accumulation of Capabilities

• Sharing knowledge and turning the wheel further by enhancing dashboards and alert systems.
• This leads to Karrot service stability and SRE capability enhancement.
• We believe this builds predictable infrastructure.

7. Cost Savings Case Study Using Locality LB in a Multi-AZ EKS Cluster (Heo Jinsu) #

Overview #

Which pods generate a lot of cross-zone traffic?

Cross-zone (Redis, EKS...etc)

→ We looked at cross-zone traffic by combining VPC Flow Logs and Pod Metadata.

• Istio Ingress Gateway was the main culprit.
• A lot also occurred within the Istio Mesh.
• Istiod consumed a lot of cost.
• Monitoring also consumed a lot...

Istio Locality LB

How can Istio Locality LB be implemented?

EKS Worker Node information contains Topology information.

Istio OutlierDetection is required for Istio Locality LB.

Outlier Detection is essential in the Destination Rule attached to the Virtual Service.

Topology Aware Hints

What about pods that don't use Istio?

service.kubernetes.to/topology-aware-hints: auto

Cost

How much did costs actually decrease?

Karrot saved over $70,000. Win-win!

Considerations for Availability and Load Balancing

What if an AZ fails or balanced distribution breaks?

• If an AZ fails, it automatically fails over.
• The real problem is when balanced distribution breaks.
• Not all pods distribute traffic evenly.
• Active-Active configuration of a single-zone EKS.

How it works?

What is Istio?

An open-source framework that implements a service mesh, providing features to control and manage communication between microservices. This includes load balancing between services, which is implemented through Istio's Envoy proxy.

8. Building an Efficient Alert System with Grafana (Lee Sieun) #

Introducing Grafana Alerting #

Alerting (legacy)

• Grafana v10 no longer supports it ㅠㅠ

There are 3 states for an alert rule:

1. OK (healthy state)
2. PENDING: The alert condition has been met, and an alert set has been created. If this state is maintained for the "for" duration defined in the Alert Rule, it transitions to the "FIRING" state.
3. FIRING: If the "PENDING" state is maintained for the given duration, it transitions to this state. When this happens, Prometheus sends the alert's "Notification" to Alertmanager.

Rule

• Evaluate Every: How often to evaluate (e.g., every few minutes)
• Evaluate For: How long to observe (e.g., for a few minutes)

Conditions

• Alert evaluation conditions (thresholds)

No data and error handling

Notifications

Notification channels

• Configure various types of notification channels.

Limitations #

1. All evaluated time series are sent as a single message.
2. Unable to respond to alerts on a per-namespace basis.
3. Readability degradation due to limitations in message content customization.
4. Unable to adjust reminder times for different alert types.
5. Need to view 담당자 (responsible person) information per project.

The advent of alert-delivery #

• Let's separate alerts by namespace.
• Improve message readability.
• Individual alert control.
• Include 담당자 (responsible person) information in alerts.
• View overall alert status at once!

→ Final message format

• Title includes namespace and alert type.
• Metric labels added for improved readability.
• Kontrol project link added.
• 담당자 (responsible person) information added using Katalog API.
• Service dashboard & log dashboard links added.
• Direct mention of 담당자 (responsible person) if needed (alert occurs N times or more).
• Snooze possible within Slack with an "Mute Alert" button.

Individual alert control feature - Team mention

Detailed control example 1 (

For ingress 5xx errors, do not mention the team for 503 codes.

But if it's 503, but the responseFlag is UC or the metric value is over 1000, then mention.

If the 5xx error value exceeds 1200, immediately mention the entire SRE team.

)

Individual alert control feature 2 - Snooze

Mute alerts from Slack or web → Slack alerts are not sent even if Grafana is in an alerting state.

Efficacy and Challenges #

• Reduced verbose alerts through individual alert control.
• Hourly reporting of alert progress.
• Easy confirmation of alert history per namespace.
• Visualization of overall alert status.

Review from a Backend Developer's Perspective (Me) #

• Even if backend development and SRE are separate, it seems important to always pay attention to infrastructure.
• Backend developers shouldn't just handle frameworks and create APIs, but also contribute to infrastructure construction.
  • That is, they need to pay a lot of attention to DevOps.
• It's really good to know how to use tools like Docker, Kubernetes, and Terraform.
• MSA is the best.
• In larger companies, managing environmental issues in various aspects is evident, and I think it's necessary to delve deeper into cost and traffic analysis.
• By creating such a stable environment, a more robust infrastructure can be built, leading to a better environment.
• Tasks like using Grafana to display server status alerts seem simple, but applying requirements to display them is difficult... Still, if implemented, it would be really useful!