• Home
• Performance

Performance

Performance is a critical aspect of any real-time application. Providing low latency with high throughput is essential for delivering a seamless user experience.

Our Performance Tests

These test is designed to set a performance benchmark by simulating a simple real-world senario, taking factors such as network conditions and location into account. We have chosen Artillery as our load testing tool, a powerful and flexible solution that allows you to simulate a large number of concurrent users. All tests are run from AWS Fargate containers across various regions, as documented below.

The Test Scenario

We aimed to simulate a simple real-world scenario where a virtual user would perform the following actions:

• Establish an authenticated connection
• Join a room (randomly selected from a list of 100 room IDs)
• Subscribe to a single event
• Publish 10 messages, with a 2-second wait time between each message
• Stay connected for an additional 20 seconds, during which time the user would receive all messages
• Disconnect

The test configuration is as follows:

config:
  phases:
    - duration: 10
      arrivalRate: 5
      name: warmup
    - duration: 10
      arrivalRate: 25
      name: rampup
    - duration: 60
      arrivalRate: 100
      name: sustained
    - duration: 10
      arrivalRate: 20
      name: warmdown
  processor: './processorFunctions.js'
  variables:
    expiresIn: 3600

scenarios:
  - engine: ws
    name: client_pubsub
    flow:
      - connect:
          function: connectHandler
      - function: generateRoomId
      - function: generateRoomJoinEvent
      - send: '{{ roomJoinEvent }}'
      - function: generateRoomSubscribeEvent
      - send: '{{ roomSubscribeEvent }}'
      - loop:
          - function: generateClientMessageEvent
          - send: '{{ clientMessageEvent }}'
          - think: 2
        count: 10
      - think: 20

Hardware Configuration

Our hardware resources are horizontally scaled, reacting dynamically to changes in capacity based on CPU and memory usage. During the test we wanted to test the minimum spec using our absolute base configuration. As such, the hardware involved is as follows:

• Websocket Servers: 2 x t3a.medium instances (2 vCPUs, 4 GiB memory)
• Message Brokers: 1 x mq.t3.micro AmazonMQ broker (1 vCPUs, 1 GiB memory)
• Caching and internal queueing: 1 x cache.t3.small Elasticache instance
• Other Micro-Services: 1 x t3a.small instance (1 vCPU, 2 GiB memory)
• Peristant Storage: 1 x db.t4g.small instance (1 vCPU, 2 GiB memory)

This is an extremely modest setup, but it is sufficient to simulate our basic real-world scenario. We've found that the minimum hardware configuration is necessary to ensure the service can handle the load and provide a stable and reliable experience for our users.

Each of the services listed above react to memory and cpu uasge alarms and will scale out based on breaches, during the test a scaling event was not triggered.

Understanding the Configuration

Each phase represents a different part of the test.

The warm-up phase is used to establish a baseline for the test, allowing us to determine the performance of the service before the sustained phase begins. This lasts for 10 seconds with 5 new connections being established every second.

The ramp-up phase is used to increase the number of concurrent users during the test, allowing us to simulate a higher load. This lasts for 10 seconds with 25 new connections being established every second.

The sustained phase is used to maintain a steady stream of users during the test, allowing us to measure the performance of the service under heavy load. In this case, the phase lasts for 60 seconds with an additional 100 virtual users connecting and performing each step every second.

The warm-down phase is used to gradually decrease the number of concurrent users during the test, allowing us to measure the performance of the service after the sustained phase has ended. This lasts for 10 seconds with 20 new connections being established every second.

The Performance Evaluation

To illustrate global performance, the test was carried out from two locations: eu-west-2 (London) and us-east-1 (N. Virginia). The results are as follows:

During the test, manual testers intermittently reloaded and published messages to a test application for visual feedback. This interaction accounts for the additional sessions and connections established during each test run.

If you're interested to know more about the test or have specific performance requirements or enquiries, please contact us at support@relaybox.net.