Each flow is separately configured to use the firehose. This API allows firehose configuration to be tested before it is set on a flow.

The Firehose feature supports multiple cloud storage services for event data export. This document describes the configuration options for both AWS S3 and Azure Blob Storage services.

Legacy Configuration (Backward Compatible)

For existing flows, the legacy configuration continues to work:

{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-s3-bucket",
    "prefix": "events/production"
  }
}

New Service-Based Configuration

The new service-based configuration allows multiple cloud storage providers:

{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-s3-bucket",
        "prefix": "events/aws"
      },
      "azure": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439012", 
        "bucket": "my-azure-container",
        "prefix": "events/azure"
      }
    }
  }
}

AWS S3 Validation

The /firehose endpoint validates AWS credentials by:

1. Creating a test file with unique name: leadconduit_verification_[flow_id_]YYYYMMDDHHMMSSMS.txt
2. Uploading it to the specified bucket (with optional prefix) when verification_file=true (default)
3. Alternatively, validating bucket access without creating file when verification_file=false
4. Returning the S3 response with ETag or validation confirmation

Example Requests:

Basic validation (creates file):

curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&prefix=test"

With flow ID (includes flow identifier in filename):

curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&flow_id=507f1f77bcf86cd799439011"

Validation only (no file created):

curl -X GET "https://app.leadconduit.com/firehose?service=aws&access_key_id=AKIA...&secret_access_key=wJal...&bucket=my-bucket&verification_file=false"

Example Responses:

File created:

{
  "validated": true,
  "verification_file": true
}

Validation only:

{
  "validated": true,
  "verification_file": false
}

Azure Blob Storage Validation

The /firehose endpoint validates Azure credentials by:

1. Creating a BlobServiceClient from the connection string
2. Checking if the specified container exists and is accessible
3. Optionally creating verification file based on verification_file parameter
4. Returning container validation information

Example Request:

curl -X GET "https://app.leadconduit.com/firehose?service=azure&connection_string=DefaultEndpointsProtocol=https;AccountName=test;AccountKey=key;EndpointSuffix=core.windows.net&bucket=my-container"

Example Responses:

With verification file:

{
  "validated": true,
  "verification_file": true
}

Validation only (verification_file=false):

{
  "validated": true,
  "verification_file": false
}

Single Service Configuration

When only one service is configured, events are sent to that service. If the service fails, events are spooled for retry.

Multi-Service Configuration

When multiple services are configured:

1. Each enabled service receives the event independently
2. If a service fails, it generates its own spool message for retry
3. Each service can have different bucket/container and prefix configurations

Spooling and Retry

• Each service failure generates a separate SQS message for retry
• Spooled events contain service-specific metadata (bucket, prefix, credentials)
• The unspooler processes each service's failed events independently
• File-based spooling creates separate paths for each service configuration

From Legacy to Service-Based Configuration

1. Keep existing configuration: Legacy configuration continues to work
2. Add services object: Gradually migrate to service-based configuration
3. Test thoroughly: Validate both configurations work as expected

Example migration:

Before:

{
  "firehose": {
    "enabled": true,
    "credential_id": "507f1f77bcf86cd799439011",
    "bucket": "my-bucket"
  }
}

After:

{
  "firehose": {
    "enabled": true,
    "services": {
      "aws": {
        "enabled": true,
        "credential_id": "507f1f77bcf86cd799439011",
        "bucket": "my-bucket"
      }
    }
  }
}