Create a Runner to Run Applications

Similar to Application Nodes in Canvas, the process of creating a Runner is divided into Steps, focusing on Cost Management, Resource Configuration, Runner Type, Allocation, Availability, and Security, ensuring a comprehensive setup for your Runner.

To create a Runner in Cortex, you have two options:

• You can click the Create Runner button on the Runners Page (cf. Creating New Runners)

• Or you can the cilck the Run button under Actions on the Applications Page, where you'll need to activate the Create Runner toggle. (cf. Run Applications Using Runners)

Using the latter option requires entering Personal Encryption Keys for any Application-level Vaults used in the Application.

Step 1 - Runner Name and Deployment Type

In this Step your initial task is to assign a distinctive name to your Runner. After naming your runner, you should select Deployment Type. Cortex offers versatile deployment options to fit your specific needs.

• You can leverage Cortex as a SaaS solution by configuring Runners directly within the platform.

• For those preferring to keep their data and operations within their own infrastructure, Cortex supports On-prem deployment.

• Additionally, the Bring-your-own-cloud (BYOC) option allows you to integrate Cortex seamlessly with your existing cloud environment, offering flexibility and control over your data processing landscape.

For detailed guidance on On-prem and BYOC deployment options, please reach out to us at [e-mail]. Our team is ready to assist you with any queries or setup requirements.

Step 2 - Deployment Configuration

Select Cloud Provider

To ensure Cortex Runners operate in your preferred cloud environment, please select the cloud provider you intend to use. This selection enables Cortex to tailor its services to align with the specific features and capabilities of your chosen cloud platform, optimizing performance and integration.

Select Region

For optimal performance and compliance with data residency regulations, please select the region where you wish to create your Runner. This choice ensures that your data processing activities are geographically aligned with your operational requirements and legal obligations.

If the region you require is not currently available or if you wish to activate new regions, please reach out to us at [e-mail]. Coordinating with you we will start enabling additional regions to meet your deployment needs.

Step 3 - Billing Method

During the Billing Method Step, you will be prompted to select a cost management method for your Runners.

Type of Cost Management influences various aspects of your Runner, including the resource limits set in the resource limits (cf. #step-2-resource-configuration), the scalability of the Runner (cf. #step-3-runner-type), and the availability configuration (cf. #step-5-availability).

Your options include:

• Opting for the $150 Free Trial, available if you have not yet exhausted the one-time $150 free trial credits allocated to your account upon creation and initial login.

Free Trial permits the creation of a single XS-size Runner only. It restricts the ability to opt for a Scalable Runner and excludes the possibility of enabling High Availability features.

• The Pay-as-you-go option becomes available once you have provided at least one verified Payment Account and a Billing Address. (cf. Payment Accounts, Billing Addresses)

Pay-as-you-go method offers significant flexibility in terms of Runner Configuration, Type, and Availability, allowing for a more tailored deployment strategy.

For the Pay-as-you-go option you can either impose a monthly Soft Budget Cap with a non-recurring Surge Budget or a monthly Budget Cap to control your expenses. (cf. Hard Budget Cap, Soft Budget Cap)

• Selecting either a Monthly Subscription or a Yearly Subscription with Monthly Limits enables the creation of a Runner under the terms of one of your Subscriptions.

The specific configuration of your Runner's Configuration, Type, and Availability depends on the details of your Monthly or Yearly Subscription.

If you have multiple subscriptions, you can create the corresponding number of Runners, each aligned with the limits and capabilities defined by your subscriptions.

Step 4 - Resource Configuration

During the Resource Configuration Step, you will establish the resource limits for your Runner, specified via the CPU, RAM, and Egress limits it should have. There are five size options to choose from: XS, S, M, L, and XL. To inform your decision, consult the table provided below, which details the resource limits associated with each Runner size.

Size	CPU	RAM (GB)	Egress (GB)
XS	2	4	200
S	4	8	400
M	6	12	600
L	8	16	800
XL	10	20	1000

Step 5 - Runner Type

In the Runner Type Step, you have the choice between opting for a Fixed Capacity Runner or a Scalable Runner.

• Fixed Capacity Runners operate your Applications using a singular Node. However, should there be an increase in event volume or modifications made to an already running Application, it's possible for the Runner to hit its resource limits. This scenario could lead to a decline in your Applications' performance, identifiable through Application Metrics such as Latency, Throughput, and Total Events Ingested.

Fixed Capacity Runners have unchanging resource allocation, which does not adapt to fluctuations in workload or event volume.

• Scalable Runners are designed to dynamically adjust the number of Nodes to alter its resource allocation in response to the needs of your Applications. This type of Runner is Horizontally Scalable, meaning it can automatically add additional Nodes to accommodate your Applications if the current Runner resource limits are nearing depletion or are exceeded.

Scalable Runners add/remove new Nodes to your Runner to alter the overall limit of the runner resources.

The primary advantage of Scalable Runners is their ability to maintain optimal Application performance by ensuring that resources are scaled up as needed to handle increased workloads, thus preventing performance degradation.

Recommendations and Additional Considerations

Choosing between a Fixed Capacity Runner and a Scalable Runner for deploying your Applications involves considering several key factors related to your project requirements, workload variability, and performance expectations. Here are some recommendations to guide your decision:

Opt for a Fixed Capacity Runner if:

• Predictable Workloads: Your Applications experience consistent and predictable workloads that do not fluctuate significantly over time.

• Cost Control: You are focused on minimizing costs and can accurately predict your resource needs, thus avoiding the expense of underutilized resources.

Scalable Runner

Consider a Scalable Runner if:

• Variable Workloads: Your Application(s) face variable or unpredictable workloads, including spikes in traffic or processing demands that can quickly exceed fixed resource limits.

• Performance Priority: Maintaining optimal performance and avoiding degradation, especially during peak demand periods, is critical for your user experience and operational efficiency.

• Growth and Flexibility: You anticipate growth or changes in your application usage and want the flexibility to automatically scale resources without manual intervention or downtime.

Additional Considerations

• Cost vs. Performance: Scalable Runners may incur higher costs due to the dynamic allocation of resources. Balance the need for scalability with budget constraints, considering both short-term and long-term perspectives.

• Monitoring and Management: Deploying Scalable Runners requires effective monitoring tools to track Application Metrics (Latency, Throughput, Total Events Ingested) and ensure resources are scaled appropriately. Ensure you have the tools and processes in place to manage this dynamic environment.

• Trial and Error: If uncertain, consider starting with a Fixed Capacity Runner for initial deployment and closely monitor performance. If you encounter performance issues or anticipate growth, transitioning to a Scalable Runner might be warranted.

Step 6 - Runner Allocation

In the Runner Allocation Step, you will determine the Allocation Type for your Runner.

You have two options for Runner allocation:

1. Reserved Runners are dedicated to running a single Application at a time. It's not possible to deploy more than one Application on Reserved Runners. This option is ideal for applications that require dedicated resources or when you want to isolate an Application's resources for security or performance reasons.

2. Shared Runners have the capability to run multiple Applications simultaneously. This option provides flexibility and efficient resource utilization, especially suitable for applications that can share resources without compromising performance or security.

It's important to note that within Cortex, all Runners are exclusive to your organization; they are not accessible to users from other organizations.

Recommendations and Additional Considerations

When deciding between Reserved Runners and Shared Runners for deploying your applications within Cortex, the choice should be guided by your specific needs, including application performance requirements, resource utilization efficiency, and operational context. Here are some recommendations to help you make an informed decision:

Reserved Runners are particularly beneficial for critical applications where the predictability, reliability and security of resource availability are paramount.

Choose Reserved Runners if:

• Dedicated Resources: Your Application demands dedicated resources to ensure consistent performance, without the risk of resource contention with other applications.

• Compliance and Security: You have strict compliance or security requirements that necessitate isolating your application from others to mitigate risks.

• Predictable Performance: You prioritize predictable performance, where the resource allocation and performance metrics must be consistent and not influenced by the load of other applications.

Shared Runners offer a flexible and efficient solution for running multiple applications, ideal for non-critical, development, or test environments where resource isolation is less of a concern.

Opt for Shared Runners if:

• Cost Efficiency: You aim to optimize resource utilization and cost by distributing resources across multiple Applications, especially when they have variable or low usage patterns.

• Scalability: Your Applications can benefit from dynamically shared resources, scaling efficiently within the capacity provided by Shared Runners.

• Flexibility: You are managing multiple smaller Applications or services that do not require dedicated resources and can coexist without impacting each other's performance.

Additional Considerations

• Resource Planning: Assess the resource demands of your Applications, considering peak loads and potential growth.

• Budget Constraints: Consider your budgetary constraints, as Reserved Runners might increase costs due to their exclusive nature, while Shared Runners can be more cost-effective.

• Application Compatibility: Evaluate whether your applications can coexist on Shared Runners without adverse effects, or if they require the isolation that Reserved Runners provide.

Step 7 - Availability

In the Availability Step, you're presented with the decision to activate Minimum High Availability for your Application(s). Opting for Minimum High Availability guarantees the uninterrupted availability of your Applications.

Enabling Minimum High Availability ensures that your Applications remain operational, even in the face of individual Runner failures, by automatically switching to a backup Runner with no noticeable disruption to users.

The process is fully automated and can be enabled with a simple toggle, offering peace of mind and continuous operation without manual intervention.

How does Minimum High Availability work?

Minimum High Availiability is achieved by deploying an additional Runner alongside your primary Runner, operating on an Active-Passive configuration. The secondary (Passive) Runner remains unseen on the Runners Page, poised to take over as the Active Runner should the current Active Runner encounter an error and become unavailable.

Upon this switch from Passive to Active, the system automatically initiates the creation of a new Passive Runner to ensure the newly active one also has a backup.

This proactive cycle ensures resilience, as a new Passive Runner is always ready to step in if the Active Runner fails for any reason.

This seamless transition maintains a consistent user experience, with Minimum High Availability ensuring that your Application(s) face no downtime and that data loss is prevented.

Recommendations and Additional Considerations

When considering whether to enable Minimum High Availability (HA) for your applications, it’s essential to weigh the benefits against your specific needs, operational requirements, and potential impact on cost. Here are some recommendations to guide your decision:

Enable Minimum High Availability if:

• Critical Operations: Your Applications are critical to your business operations, and any downtime would result in significant operational disruption or financial loss.

• Data Integrity: Maintaining continuous data integrity and avoiding any data loss is paramount for your Application's functionality and your business's requirements.

• User Experience: You aim to provide a seamless user experience, where even brief periods of unavailability could negatively impact user satisfaction or trust.

Considerations Before Enabling Minimum High Availability:

• Cost Implications: Implementing HA involves additional costs due to the deployment of extra Runners to serve as backups. Assess whether the added expense aligns with your budget and the criticality of your application.

Step 8 - Security

During the Security Step, you will establish the security configurations for your Runner. It is essential for each Runner to have JVM Secrets and Connection Secrets to function correctly. These secrets are integral for enabling Sink, Source, Table, and Aggregation Nodes to establish connections and facilitate data transfer within and outside of Cortex. You have two primary options for managing these secrets:

• Auto-Generated Secrets: Opt for automatically assigned JVM Secrets and Connection Secrets. This choice simplifies the initial setup by providing you with ready-to-use credentials. These auto-generated credentials can be accessed via the Runner Details option found under the Actions column on the Runners Page.
• Runner-Level Vault: Choose to integrate a Runner-level Vaults for managing your Runner's security settings. After you select the Runner-level Vault to utilize, you must enter its Personal Encryption Key. This option allows for a centralized management point for all your Runner's secrets, providing an added layer of control and customization. The Runner-level Vault can be accessed using the Vaults Page.

It is crucial to note that once a Runner is created, its security settings, including both the JVM and Connection Secrets, cannot be altered.

This means you cannot switch between Auto-Generated Credentials and a Runner-level Vault or change the specifics within the chosen method after the Runner's creation. This decision is final and plays a critical role in how your Runner connects with and secures data across Nodes.

Step 9- Preview

In Preview Step, you're provided with a concise summary of all the configurations of your Runner. This step is pivotal for reviewing and ensuring that your configurations are as intended before completing node setup.

• Viewing Configurations: Preview Step presents a consolidated view of your node Runner setup.

• Saving and Exiting: Use the Complete button to save your changes and exit the node and return back to Canvas.

• Revisions: Use the Back button to return to any Step of modify node setup.

The Preview Step offers a user-friendly summary to manage and finalize node settings in Cortex.