How to enable queue mode in n8n?

To enable queue mode in n8n, you need to update your environment variables or configuration file to set the execution mode to 'queue'. This allows n8n to handle workflows through a queue system, making it more reliable for processing larger workloads and preventing data loss during server restarts or crashes.

 

Step 1: Understanding Queue Mode in n8n

 

Queue mode is an execution mode in n8n that processes workflows through a queue system. In the default mode (regular), workflows are executed directly. However, in queue mode, executions are added to a queue first and then processed one after another.

Queue mode offers several advantages:

• Better reliability for handling larger workloads
• Prevention of data loss during server restarts or crashes
• Improved management of system resources
• Ability to continue processing workflows after server restarts

 

Step 2: Check Prerequisites

 

Before enabling queue mode, ensure you have:

• n8n installed (version 0.147.0 or later recommended for best queue implementation)
• Access to configure environment variables for your n8n instance
• Admin access to restart the n8n service after configuration changes

 

Step 3: Configure Queue Mode via Environment Variables

 

The simplest way to enable queue mode is by setting the appropriate environment variable. You'll need to set EXECUTIONS_MODE to queue.

For Linux/macOS systems, you can set this temporarily by running:

export EXECUTIONS\_MODE=queue
n8n start

For a permanent configuration, add this to your .env file or system environment variables.

 

Step 4: Configure Queue Mode in Configuration File

 

If you're using a configuration file (like config.json), you can enable queue mode by adding or modifying the following setting:

{
  "executions": {
    "mode": "queue",
    "type": "own" 
  }
}

Note: The "type" parameter can be set to either "own" (uses n8n's internal queue) or "bull" (uses BullMQ, which requires Redis).

 

Step 5: Configure Queue Mode with Redis (Recommended for Production)

 

For production environments, it's recommended to use Redis with queue mode for better performance and reliability. To configure n8n with Redis:

1. First, ensure Redis is installed and running on your system or accessible from your n8n instance.

2. Set the following environment variables:

EXECUTIONS\_MODE=queue
QUEUE_BULL_REDIS\_HOST=localhost  # Replace with your Redis host
QUEUE_BULL_REDIS\_PORT=6379       # Replace with your Redis port

If your Redis instance requires authentication, also set:

QUEUE_BULL_REDIS_PASSWORD=your_password

 

Step 6: Additional Queue Configuration Options

 

You can further customize your queue settings with these optional environment variables:

# Set the maximum number of jobs to process concurrently
EXECUTIONS\_PROCESS=5

# Configure retry mechanism for failed jobs
EXECUTIONS_RETRY_MODE=true
EXECUTIONS_RETRY_MAX=3

# Set data pruning options
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX\_AGE=336  # Maximum age in hours (336 = 14 days)

# Set timeout for executions (in seconds)
EXECUTIONS\_TIMEOUT=3600  # 1 hour

 

Step 7: Restart n8n to Apply Changes

 

After configuring the environment variables or configuration file, restart your n8n instance to apply the changes:

For npm installations:

n8n stop
n8n start

For Docker installations:

docker restart n8n\_container

For systemd service:

sudo systemctl restart n8n

 

Step 8: Verify Queue Mode is Enabled

 

To verify that queue mode is correctly enabled:

1. Log in to your n8n interface.
2. Create a simple test workflow with a few nodes.
3. Execute the workflow and monitor its execution.
4. Check the execution logs for any queue-related messages.

You can also check the system logs to see if n8n started with queue mode enabled:

grep "queue" /path/to/n8n/logs

Or, if using Docker:

docker logs n8n\_container | grep "queue"

 

Step 9: Monitoring and Managing the Queue

 

Once queue mode is enabled, you can monitor and manage executions in the queue:

1. The n8n UI shows active executions and their status.

2. For advanced monitoring, consider setting up:

   • Redis Commander if using Redis for visualization of the queue
   • Prometheus and Grafana for metrics collection and visualization

3. To clear the queue in case of issues, you can use Redis CLI commands (if using Redis):

redis-cli
KEYS bull:n8n-job:\*
DEL [key names from above]

 

Step 10: Troubleshooting Common Issues

 

If you encounter issues with queue mode, try the following:

• Check n8n logs for error messages related to the queue configuration
• Verify Redis connection if you're using Redis as the queue backend
• Ensure you have sufficient system resources (memory, CPU) for your workload
• For workflows stuck in the queue, try restarting n8n or clearing the queue

Common error resolution:

# If Redis connection fails, check Redis status
redis-cli ping

# If workflows get stuck, increase the timeout
export EXECUTIONS\_TIMEOUT=7200  # 2 hours

# If you experience memory issues, adjust concurrency
export EXECUTIONS\_PROCESS=2  # Lower the concurrent execution count

 

Step 11: Best Practices for Queue Mode

 

To get the most out of queue mode, follow these best practices:

• Use Redis as the queue backend for production environments
• Configure appropriate concurrency based on your server resources
• Set up proper retry mechanisms for workflows that might fail
• Implement monitoring to keep track of queue size and execution times
• Regularly backup your Redis data if using Redis as the queue backend
• Optimize your workflows to prevent long-running executions from blocking the queue

 

Step 12: Scaling Queue Mode for Larger Deployments

 

For larger deployments, consider:

• Setting up multiple n8n workers pointing to the same Redis instance
• Implementing Redis clustering for high availability
• Using dedicated hardware or scalable cloud services for your queue backend
• Separating different types of workflows into different queues for better resource management

Example configuration for multiple workers:

# On each worker
EXECUTIONS\_MODE=queue
QUEUE_BULL_REDIS\_HOST=shared-redis-server
QUEUE_BULL_REDIS\_PORT=6379

With these steps, you should have queue mode successfully enabled and optimized for your n8n instance, providing better reliability and performance for your workflow executions.