Optimizing performance across the stack

Database

Chose your weapon

Of course, the choice of which database to use already has a big impact regarding performance. So you really need to know the data you are dealing with and ideally the data access patterns. This is no simple task and as usual, there is no single, golden choice but rather a long list of considerations and compromises. Are we dealing with time-series like data? How is the data structured? Is it even structured? What are the relationships between data? How will the service evolve? What are the consumers of the data and what are their needs?

It’s good if the architect makes conscious decisions taking into account the above questions. And in my opinion also important not to change the DB every half a year just because there is yet another new great database that is «so much better than everything else on the market».

In our case we took a Aurora PostgreSQL DB as our «weapon». No graph DB or No-SQL DB, just a «classical» relational database. Each database has its own characteristic and challenges, so not all of the following aspects may apply.

Indexing

We faced some bottlenecks when we started to analyze our queries. And most issues were related to missing indexes. Its really crucial to have indexes e.g. for join columns.

CREATE INDEX idx_locations_is_part_of ON locations(is_part_of);

Let’s move up to the application layer, which in our case is a Spring Boot Java application. Performance responsibilities are typically shared across different layers rather than being confined to a single one. Optimal performance often requires proper configuration and adjustment across multiple layers working together!

Lazy vs Eager Loading

@Entity
public class User {
    @OneToMany(fetch = FetchType.EAGER)
    private List<Order> orders;  // Related Orders fetched immediately with User
}

N+1 Problem

During our performance analysis, we discovered an N+1 query problem. Here’s what this means:

• One query fetches a list of locations
• For each location, another query fetches the locations address
• Result: 1 initial query + N queries for each address
• Example: 1000 locations = 1 + 1000 = 1001 database queries

This is really bad because performance degrades as the data grows!

To solve such an issue, there are multiple approaches like e.g. using a custom query with a JOIN FETCH:

Or what we did, which was having custom repository methods that preload the referenced entities (e.g. addresses) in bulk for a set of locations in one query. The preloaded data is passed to the «mapper» which maps the entity classes to DTOs. Due to lazy fetch type, hibernate was executing the additional queries exactly in the mapper where the actual references were accessed.

Why not generally use eager loading then to avoid the N+1 problem?

With eager loading everything is fetched upfront, the mentioned approaches above (and others) give you more control over what references to load and also when to load them!

Pagination

Unwanted columns

Transaction management was one point, we did not handle great in the beginning. We defined write operations as transactional on Controller level and read operations were not marked as such. What did we change then?

First its not recommended at all to use transactional on Controllers, it even may not work functionally correct (check for «proxy» and «AOP» to get into the details).

Second, the goal is to keep transactions as short as possible since we don’t want too lock DB tables too long. In our case, the method annotated with transactional contained logic that is not really required in a transaction. Putting the logic into a service method helps to split concerns. In our architecture, changes of business objects also need to emit Domain Events. Ideally this is not inside the DB transaction. You can use the Outbox Pattern to tackle such a scenario.

Connection Pooling

Additional Performance Considerations

There are many things to consider if you really want to fine tune your service! Besides the above mentioned points, things like caching or batch fetching could be beneficial. For our service we did not apply any of these on database or application level.

Load Balancer

• The `Kubernetes Service` configuration creates a load balancer (NLB since `aws-load-balancer-nlb-target-type` is used) for the application running on EKS. The selector must match the PODs labels (here «my-app»). The following configuration is not complete but shows the essential parts for the load balancer:

  • resource "kubernetes_service" "my_service" {
      metadata {
        annotations = {
          "service.beta.kubernetes.io/aws-load-balancer-type"                     = "external"
          "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type"          = "ip"
        }
        name      = "${var.resource_prefix}-nlb"
        namespace = var.namespace
      }
      spec {
        selector = {
          app = "my-app"
        }
        type = "LoadBalancer"
      }
    }

• The network NLB is mainly used to distribute the traffic. For the actual scaling on EKS additional scaling layers can be used. Since we are using `EKS Fargate` which handles

  underlying infrastructure itself, this is done by a so called `Horizontal Pod Autoscaler (HPA)`. Scaling is always based on a metric. In our case with `Horizontal Pod Autoscaler`, we can define this in our deployment.yaml with the following values:

  • autoscaling:
      enabled: true          # enables HPA
      minReplicas: 1
      maxReplicas: 10
      targetCPUUtilizationPercentage: 80
      targetMemoryUtilizationPercentage: 80

  • So scaling is based on cpu and memory utilization. To have this metrics available for HPA: Be aware that you need to additionally deploy a metrics server on EKS!

    Ok nice, we are able to scale our application pods now if CPU or memory usage gets too high. But wait is this enough? No you are right! If we just have one DB instance, we would still have a bottleneck. In addition, for having a fast reaction for the POD scaling, its important to have fast container startup (readiness/liveness). For our tests against the required NFRs, using spring boot with our setup was sufficient but if needed it could be worth looking at a framework based on graal VM and native images.
• For making the DB also scalable we apply the following configuration (besides the settings already shown in multi-az & read-replicas):

  • # Cluster configuration with serverless v2 scaling
    resource "aws_rds_cluster" "aurora_cluster" {
      cluster_identifier = "my-aurora-cluster"
      serverlessv2_scaling_configuration {
        min_capacity = var.serverless_min_capacity
        max_capacity = var.serverless_max_capacity
      }
    }

  • `serverlessv2_scaling_configuration` defines a range for so called `Aurora Capacity Units` (ACU = memory and cpu bundle). If there is little workload, the DB releases its resources and increases ACU again in steps if workload increases. As usual in the cloud, you only pay the capacity used.
• Aurora automatically sets up a reader and a writer endpoint. The writer endpoint connects to the primary instance and the reader endpoint serves for read operations using the

  available read replicas. To really make use of this setup, the application needs to route traffic accordingly (read/write separation). We defined two custom annotations

  `TransactionalReader` and `TransactionalWriter` that are then used in the Service classes conceptually like this:

  • @Service
    public class LocationService {
        @TransactionalReader
        public Location findById(Long id) { ... }    // uses reader endpoint
    
        @TransactionalWriter
        public Location create(Location location) { ... }  // uses writer endpoint
    }

     

API

Caching and payload size

resource "aws_api_gateway_stage" "example" {
  ...

  cache_cluster_enabled = true # Enable cluster cache 
  cache_cluster_size    = "0.5" # Cache size in GB
}

resource "aws_api_gateway_method_settings" "method_cache_settings" {
  ...
  method_path = "GET/my-resource"

  settings {
    caching_enabled       = true
    cache_ttl_in_seconds  = 3600 # Time-to-live for the cache
    cache_data_encrypted  = true # Encrypt cached data
  }
}

Bulk operations and other media types

Our API is using the `application/vnd.api+json` media type for our `json:api` compliant API. Some clients had the need for creating, deleting or patching multiple entries at once. For that we introduced a dedicated media type for bulk operation, similar to `application/vnd.yourapp.bulk+json`. That’s a bit of effort for implementing it but at the end you may save quite a bit due to reduced round trips and network overhead.

I like json:api and its good to have a standard for APIs especially when used in platform like in our case. But if performance is very crucial you could consider using yet another media type. You could have a try with `application/x-protobuf` which is very compact, binary format.

Additional interfaces

Monitoring and Testing

Even though its the last chapter of this blog, its probably the most important aspect. You should not apply things blindly but rather continuously test your performance and monitor your key metrics.

Especially the initially mentioned NFR requirements, we wanted to test and be sure our service can fulfill it.

The motivation behind the load tests was of course to prove that our service can fulfill the requested load (NFRs), e.g. p99 response time should be lower than 500ms while the platform has 500’000 locations and so on.

There are many tools available for testing a REST API.

We had hands on with postman, insomnia, artillery, k6, locust and were also looking into gatling and jmeter.

As usual there is not the best tool but at one point we decided to go on with k6 which seemed to serve our needs best.

K6 is an open source performance tool written in go but the actual tests are written in java script. You can run it e.g. locally in a docker container, in your CICD pipeline or you can run it even as a «Kubernetes Operator» (k6 operator) which distributes the tests across multiple K8s pods.

We initially started with a simple k6 test but later decided to extend the setup and use the k6 operator in order to be able to test higher expected loads (future expected RPS (Requests Per Second) and virtual users).

A bit a challenging task was to prefill our database with the required amount of data especially in a reasonable time within our CICD pipelines. Also we wanted to have the data randomized and are using `javafaker` to get some variations of the data.

At the end, the large setup creates more than 15 million db entities which took still more than an hour in our CICD setup. Hence we decided to only repopulate the DB from time to time.

Our actual k6 tests run 4 typical client scenarios in parallel for 10 minutes (e.g. services reading locations, services retrieving whole building structure and devices, and other scenarios also performing write operations). The k6 test looks conceptually like below:

export let options = {
    teardownTimeout: '2m',
    summaryTrendStats: ['avg', 'min', 'max', 'p(95)', 'p(99)'],
    scenarios: {
        scenario1: {
            executor: 'constant-arrival-rate',
            duration: testDuration,
            preAllocatedVUs: 50,
            rate: 50, 
            timeUnit: '1s',
            exec: 'scenario1',
        },
        scenario2: {
            // settings
        }
        ...
    }
};

export function setup() {
    // retrieve data base setup and make it available in the executed scenarios
    return getDbSetup();
}

export function scenario1(data) {
    group('scenario1-read-locations', function () {
        const response = readLocations(baseUrl, partitionId, filters);
        check(response, {
            'status is 2xx': (r) => r.status >= 200 && r.status < 300,
        });      
}

export function scenario2(data) {
    group('scenario2-retrieve-building-structure', function () {
      // run scenario 2
}

Monitoring & Alerting

locals {
  access_log_format = {
    "requestTime" : "$context.requestTime", # Captures the request time
    "requestID"   : "$context.requestId",
    "httpMethod"  : "$context.httpMethod",
    "resourcePath": "$context.resourcePath",
    "httpStatus"  : "$context.status",
    "elapsedTimeMs": "$context.responseLatency"
  }
}

# enable access logs
access_log_settings {
  destination_arn = aws_cloudwatch_log_group.my_log_group.arn
  format          = jsonencode(local.access_log_format)
}

Such dashboards are helpful during development and in case of issues to analyze the system. I am personally a big fan of good alerts, that actively inform developers if certain metrics go up. So please add alerts for your most critical metrics for a production setup.

conclusions