Lecture on NA10 Scaling

Introduction

• Speakers: Omar and the host from Berlin.
• Topic: Scaling NA10 instance, step by step.
• Goal: Understand how NA10 works and how to scale it.

Fun Fact by Omar

• Omar, as a developer, drinks less coffee more milk to avoid stomach issues.
• He can fall asleep very quickly (~30 seconds), which amazes his wife.

What is Scaling?

• Definition: Scaling is the capability of a system to handle a growing amount of work by adding resources to it.
• Types of Scaling:
  • Vertical Scaling: Adding more power to a single machine (CPU, RAM).
  • Horizontal Scaling: Adding more machines to handle tasks (Distributed computing).
• NA10 initially only supported vertical scaling, but it's now transitioning to support horizontal scaling.

NA10 Structure Overview

• Editor Interface: User Interface for drawing workflows, saving them, viewing executions, and adding credentials.
• Internal API: Saves actions (e.g., saving workflows, activating them) to the database.
• Webhook Registration Service: Registers and deregisters webhooks as needed.
• Initiators: Components that determine when a workflow should be executed.
  • Webhooks: Triggered by HTTP requests from external services like Trello, Asana.
  • Pollers: Request data from services periodically to determine workflow execution.
  • Trigger Nodes: Time-based/event-based triggers.
• Workers: Execute the actual tasks and can now be scaled horizontally.

Basic NA10 Setup and Demonstration

• Initial Setup: Basic NA10 setup using SQLite for the database.
• Scaling With Docker: Using Docker to set up containers for Redis, PostgreSQL, and multiple NA10 instances.
• Commands: Provided Docker commands to set up Redis, PostgreSQL, and to start NA10.
• Environment Variables: Explained how to configure NA10 to use PostgreSQL and Redis with environment variables.
• Horizontal Scaling: Demonstrated starting multiple worker instances using Docker.
• Separate Worker and Webhook Processes: Running specific worker processes and webhook handling in separate containers.
• Load Distribution: Showed how load is distributed among multiple workers.

Important Considerations

• Encryption Key: Must share the same encryption key for credentials among all instances.
• Access to Resources: Database and Redis need to be accessible to NA10 core, workers, and webhooks.
• Sharing Environment Variables File: Makes it easy to keep configurations consistent.

Q&A Highlights

• Lots of Executions: ~1 million executions per month is considered a lot, performance tuning might be needed based on specific needs.
• Environment Variables: Can share via environment files; also discussed advanced configuration options available in the docs.
• Number of Workers: Tested up to 7-8 workers, no hardcoded upper limit; scaling depends on infrastructure (Redis, PostgreSQL load handling).
• Connections and Security: Discussed how to handle private data and firewall settings to ensure security while scaling.
• Concurrency Settings: Explained the concept of concurrency settings for workers.
• Horizontal Scaling: Methods for horizontally scaling across servers, including cloud setups (Amazon EC2, ECS).
• Worker-Specific Jobs: Currently, jobs are not routed to specific workers but are equally distributed in a round-robin fashion.

Conclusion

• Costume Deployment: Scaling NA10 involves thoughtful deployment of resources (Redis, Postgres, Workers, and Webhooks).

References

• Documentation: Provided links and mentioned where to find additional configuration settings and other details on NA10's documentation site.
• Community Forum: Encouraged participants to use the community forum for additional questions and support.

• Overall, emphasized that NA10 can be effectively scaled horizontally for performance and reliability.