neeto-email-pipeline-nano

A Rails engine that consolidates all outbound email infrastructure for neeto apps: email prefixing, interception, rate limiting, delivery logging, SparkPost webhook handling, and email open tracking.

Contents

1. Architecture Overview
2. Development with Host Application
   • Engine
     • Installation
     • Configuration
   • Frontend Package
     • Installation
     • Components
3. Features
   • Email Prefixer
   • Email Interception
   • Rate Limiting
   • Email Logging
   • SparkPost Webhooks
   • Email Open Tracking
   • Email Link Click Tracking
4. Testing Locally
5. Instructions for Publishing

Architecture Overview

Every outbound email passes through the following pipeline, whether sent via deliver_later (Sidekiq) or deliver_now (inline). The engine detects the delivery path automatically and captures organization context from Sidekiq middleware or custom mail headers accordingly.

flowchart TD
    A[Mailer.deliver_later] --> B[Sidekiq Queue]
    B --> C["EmailLogMiddleware
    Generates UUID, sets Thread.current context"]
    C --> D["MailThrottleMiddleware
    Checks hourly rate limit per org"]

    D -->|Rate limited| E["Reschedule job with backoff
    Preserves email_log_uuid in job hash"]
    E --> R["EmailLogMiddleware handle_post_yield
    Creates rate_limited placeholder
    (email: nil, rate_limited_at set)"]
    R -->|"Job re-enters queue
    with same UUID"| B

    D -->|Allowed| F[Build Mail Object]

    DN[Mailer.deliver_now] --> DN1["after_action callback
    Sets X-Neeto-Organization-Id,
    X-Mailer-Class, X-Mailer-Action headers"]
    DN1 --> F

    F --> T{"EmailOpenTrackable
    included & trackable present?"}
    T -->|Yes| U["after_action injects 1×1 pixel
    EmailOpenReceipt created with HMAC token"]
    U --> G
    T -->|No| G

    G["EmailPrefixer
    Adds [STAGING]/[DEV] prefix to subject"]
    G --> H["Interceptor
    Drops/forwards based on environment"]
    H -->|Dropped| I["LoggingInterceptor
    Mark as dropped with reason"]
    H -->|Allowed| J["LoggingInterceptor
    Deletes rate_limited placeholder (if any),
    creates per-recipient log records
    with rate_limited_at preserved"]
    J --> K[Mail Delivery]
    K --> L["LoggingObserver
    Mark queued records as handed_over"]
    L --> M{Delivery Method?}
    M -->|SparkPost| N[SparkPost Delivers Email]
    N --> O[SparkPost Webhook]
    O --> P["SparkpostWebhookJob
    Match via email_log_uuid in rcpt_meta"]
    P --> Q2[Update Status: Delivered / Bounced / Spam]
    M -->|Gmail / Outlook / SMTP| S["Final status: handed_over
    No further tracking available"]

    C -->|Exception| Q[Mark as Failed]

    subgraph "Asynchronous — triggered when recipient opens email"
        W[Recipient Opens Email] --> X["GET /neeto_email_pipeline/
        email_open_receipts/:token.png"]
        X --> Y["EmailOpenReceiptsController
        Updates first_opened_at, open_count"]
        Y --> Z[Returns 1×1 transparent PNG]
    end

    style DN fill:#60a5fa,color:#000
    style DN1 fill:#60a5fa,color:#000
    style E fill:#fbbf24,color:#000
    style R fill:#fbbf24,color:#000
    style I fill:#f87171,color:#000
    style Q2 fill:#34d399,color:#000
    style S fill:#94a3b8,color:#000
    style Q fill:#f87171,color:#000
    style U fill:#a78bfa,color:#000
    style Y fill:#a78bfa,color:#000
    style Z fill:#a78bfa,color:#000

deliver_now vs deliver_later: The deliver_later path (yellow nodes) goes through Sidekiq middleware for rate limiting and context setup. The deliver_now path (blue nodes) skips Sidekiq entirely — organization context is provided via custom mail headers (X-Neeto-Organization-Id, X-Mailer-Class, X-Mailer-Action) set by an after_action callback. These headers are stripped before the email leaves the system.

Email Lifecycle Statuses

Development with Host Application

Engine

Installation

1. Add the gem to your application's Gemfile:

   source "NEETO_GEM_SERVER_URL" do
     # ..existing gems
   
     gem 'neeto-email-pipeline-engine'
   end
   

2. Install the gem:

   bundle install
   

3. Mount the engine in config/routes.rb:

   mount NeetoEmailPipelineEngine::Engine, at: "/neeto_email_pipeline"
   

4. Copy and run migrations:

   bundle exec rails neeto_email_pipeline_engine:install:migrations
   bundle exec rails db:migrate
   

Configuration

constants.yml

Add the following to your config/constants.yml:

defaults: &defaults
  email_pipeline:
    email_logging_enabled: true
    retention_days: 30

development:
  <<: *defaults

test:
  <<: *defaults

staging:
  <<: *defaults

production:
  <<: *defaults

secrets.yml (via Vault)

Add SparkPost webhook credentials to config/secrets.yml:

defaults: &defaults
  email_pipeline:
    sparkpost_webhook_username: <%= ENV['...'] %>
    sparkpost_webhook_password: <%= ENV['...'] %>

Scheduled Jobs

Add the purge job to config/scheduled_jobs.yml:

purge_old_email_logs:
  cron: "0 3 * * *"
  class: NeetoEmailPipelineEngine::PurgeOldEmailLogsJob
  description: "Purge email logs older than retention period"

Frontend Package

Installation

1. Add the package:

   yarn add @bigbinary/neeto-email-pipeline-frontend
   

2. Install peer dependencies (neetoui, neeto-molecules, neeto-commons-frontend, neeto-filters-frontend, etc.) if not already present.

Components

The package exports the EmailLogs component:

import { EmailLogs } from "@bigbinary/neeto-email-pipeline-frontend";

<EmailLogs
  canViewEmailLogs={true}
  breadcrumbs={[
    { text: "Admin panel", link: "/admin" },
    { text: "Email Logs" },
  ]}
  helpUrl="https://neetocalhelp.neetokb.com/p/a-8ed45325"
  pageTitle="Email Logs"
/>;

Props:

Features

Email Prefixer

Adds environment prefix to email subjects in non-production environments (e.g., [STAGING], [DEVELOPMENT]). Also supports per-organization custom prefixes via the email_prefix field on the Organization model.

Requires: Every mailer must be called with organization_id in params:

UserMailer.with(organization_id: org.id).welcome_email.deliver_later

Email Interception

The Interceptor controls which emails are actually sent based on the environment:

Staging/Development:

• Forwards emails to configured addresses (mail_interceptor.forward_emails_to in secrets)
• Whitelists specific addresses (mail_interceptor.whitelisted_emails in secrets)
• Intercepts emails matching ^cpt.*bigbinary\.com$

Production:

• Filters out emails to @example.com and @example.net

All environments — emails are dropped when:

• More than 10 unique recipients across To, CC, and BCC
• Subject or body contains blocked content patterns (depop, dep0p)
• All recipients have bounced 2+ times (checked via NeetoTower API)

The drop reason is recorded in the email log's status_detail field.

Note: If the NeetoTower API is unavailable (timeout, connection refused, 5xx error), the engine treats the bounce count as 0 and allows the email to be sent. Errors are reported to Honeybadger for visibility.

Rate Limiting

The MailThrottleMiddleware enforces per-organization hourly email send limits:

• Configured via organization.max_hourly_emails.
• Can be globally overridden with GLOBAL_HOURLY_EMAIL_LIMIT_FOR_ALL_ORGS env var
• Rate limiting is enabled by default in production, controlled by RATE_LIMIT_ENGINE_ENABLED env var
• When rate limited, the job is rescheduled with exponential backoff (60s to 30min)
• NeetoTower is told about the hit (deduplicated per 10-minute window): the limit, the emails waiting behind it, the emails the spam screening dropped today and the mailers that sent. When NeetoTower can't take the report, the notice goes to the rate-limit Slack channel instead. Products need NEETO_TOWER_AUTH_TOKEN set to NeetoTower's server token

Email Logging

When email_logging_enabled is true, every outbound email is tracked with:

• Per-recipient records: One row per recipient (to/cc/bcc) with email, subject, from, delivery method
• Email body capture: HTML body stored for preview in the admin UI
• Full lifecycle tracking: queued → rate_limited → handed_over → delivered/bounced/spam_complaint
• Rate limit awareness: Records carry rate_limited_at timestamp when the email was throttled
• Automatic purge: PurgeOldEmailLogsJob deletes records older than retention_days

deliver_now support: Emails sent via deliver_now are also logged. Since these emails bypass Sidekiq middleware, the engine reads organization context from custom mail headers (X-Neeto-Organization-Id, X-Mailer-Class, X-Mailer-Action) that are set by an after_action callback in the mailer. The headers are stripped before delivery so they never reach the recipient.

SparkPost Webhooks

The engine provides a webhook endpoint for SparkPost delivery events:

Endpoint: POST /neeto_email_pipeline/webhooks/sparkpost/events

Authentication: HTTP Basic Auth using credentials from vault.

Handled events:

• delivery → status: delivered
• bounce, out_of_band, policy_rejection → status: bounced
• spam_complaint → status: spam_complaint

Correlation: The LoggingInterceptor injects email_log_uuid into the X-MSYS-API header metadata. SparkPost echoes this back in webhook events via rcpt_meta.email_log_uuid, which the SparkpostWebhookJob uses to find the exact recipient record.

SparkPost Setup

1. Go to SparkPost → Webhooks → Create Webhook
2. Target URL: https://app.yourproduct.com/neeto_email_pipeline/webhooks/sparkpost/events
3. Authentication: HTTP Basic with username/password from your vault config
4. Events to subscribe:
   • Delivery: delivery
   • Bounce: bounce, out_of_band, policy_rejection
   • Spam: spam_complaint

Email Open Tracking

Tracks when recipients open emails by injecting a 1×1 transparent tracking pixel into outgoing emails. When the recipient's mail client loads the pixel, the engine records the open event.

Setup

1. Include the EmailOpenTrackable concern in your mailer:

   class CommentMailer < ApplicationMailer
     include NeetoEmailPipelineEngine::EmailOpenTrackable
   end
   

2. Pass trackable in your mailer params:

   CommentMailer.with(
     organization_id: org.id,
     trackable: { id: comment.id, klass: "Comment" }
   ).created_email.deliver_later
   

3. Override email_open_tracking_enabled? on your Organization model:

   def email_open_tracking_enabled?
     setting&.read_receipts_enabled?
   end
   

   The default (defined in the engine's Helpers concern) returns false.

How It Works

• The EmailOpenTrackable concern registers an after_action that runs after the mailer action builds the message.
• If params[:trackable] is present and organization.email_open_tracking_enabled? returns true, the concern:
  1. Creates an EmailOpenReceipt with an HMAC-SHA256 hashed token
  2. Determines attribution_scope (single_recipient or shared_envelope based on CC/BCC presence)
  3. Appends a 1×1 transparent PNG <img> tag before the closing </body> tag
• When the recipient opens the email, their mail client loads the pixel image, hitting GET /neeto_email_pipeline/email_open_receipts/:token.png.
• The controller atomically updates first_opened_at (set only on the first open), last_opened_at, and open_count.
• The endpoint always returns a valid 1×1 transparent PNG regardless of token validity — no information is leaked about whether a token exists.

Querying Open Data

Host apps query the engine's model directly:

receipt = NeetoEmailPipelineEngine::EmailOpenReceipt.find_by(
  trackable_type: "Comment", trackable_id: comment.id
)
receipt&.first_opened_at  # => 2026-06-22 14:30:00 UTC
receipt&.open_count       # => 3
receipt&.attribution_scope # => "single_recipient"

Security

• Tokens are 256-bit (SecureRandom.urlsafe_base64(32)) and stored as HMAC-SHA256 digests — a database leak does not expose valid pixel URLs.
• The pixel endpoint skips all authentication (CSRF, session, org loading).
• Responses never reveal token validity (always returns a valid PNG).
• If tracking is disabled after an email is sent, subsequent pixel hits are ignored.

Known Limitations

• Email clients that block remote images (Apple Mail Privacy Protection, Outlook defaults) will not trigger the pixel — open counts may undercount.
• Email proxying (Gmail image proxy, Apple MPP) can trigger false positives.

Email Link Click Tracking

Tracks when recipients click links inside outgoing emails by rewriting each tracked <a> tag's href into a signed redirect URL. When the recipient clicks the link, the engine records the click event and forwards them to the original destination.

Setup

1. Include the EmailLinkClickTrackable concern in your mailer:

   class CommentMailer < ApplicationMailer
     include NeetoEmailPipelineEngine::EmailLinkClickTrackable
   end
   

2. Pass trackable in your mailer params:

   CommentMailer.with(
     organization_id: org.id,
     trackable: { id: comment.id, klass: "Comment" }
   ).created_email.deliver_later
   

3. Tag each trackable link in the email body with a data-link-index attribute :

   <a href="https://example.com/pricing" data-link-index="0">See pricing</a>
   

4. Override email_link_click_tracking_enabled? on your Organization model:

   def email_link_click_tracking_enabled?
     setting&.link_click_tracking_enabled?
   end
   

   The default (defined in the engine's Helpers concern) returns false.

How It Works

• The EmailLinkClickTrackable concern registers an after_action that runs after the mailer action builds the message.
• If params[:trackable] is present and organization.email_link_click_tracking_enabled? returns true, the concern scans the HTML body for <a href> tags containing a data-link-index attribute and an HTTP or HTTPS URL. Links without a data-link-index attribute or with unsupported URL schemes, such as mailto:, tel:, or #, are left untouched.
• For each trackable link, the concern:
  1. Creates an EmailLinkClick record up front, storing the destination URL, link name/index, recipient, organization, and attribution_scope (single_recipient or shared_envelope, based on CC/BCC presence)
  2. Generates a signed token (via ActiveSupport::MessageVerifier) encoding the record's id and destination url
  3. Rewrites the link's href to point at the redirect endpoint with that token
• When the recipient clicks the link, their mail client requests GET /neeto_email_pipeline/email_link_clicks/:token.
• The controller verifies the token, atomically increments click_count, and redirects the browser to the original destination URL.

Querying Click Data

Host apps query the engine's model directly:

clicks = NeetoEmailPipelineEngine::EmailLinkClick.where(
  trackable_type: "Comment", trackable_id: comment.id
)
clicks.sum(:click_count)          # => 5
clicks.pluck(:link_name, :click_count)
# => [["See pricing", 3], ["Watch demo", 2]]

Security

• Tokens are cryptographically signed (via ActiveSupport::MessageVerifier), not encrypted. Anyone can decode a token to see the destination URL and record id, but any attempt to alter them, or to construct a fake token, will fail signature verification without knowledge of the app's secret key.
• Invalid or tampered tokens return a 404 rather than redirecting anywhere, preventing the endpoint from being used as an open redirect.
• If tracking is disabled after an email is sent, links already rewritten in that email will still redirect and record clicks, since the token is self-contained and independent of the current tracking setting.

Testing Locally

Email Interception

To test email forwarding in development, add the following to config/secrets.yml:

development:
  mail_interceptor:
    forward_emails_to: "[email protected]"
    whitelisted_emails: ""

Emails will be forwarded to the configured address instead of the original recipient.

Rate Limiting

Rate limiting is disabled in development by default. To enable:

1. Set RATE_LIMIT_ENGINE_ENABLED=true in your environment

2. Set the org's hourly limit:

   Organization.first.update!(max_hourly_emails: 5)
   

3. Send more emails than the limit to trigger throttling

Email Logging

1. Enable logging in config/constants.yml:

   development:
     email_pipeline:
       email_logging_enabled: true
       retention_days: 30
   

2. Restart Sidekiq (the engine registers interceptors/observers at boot)

3. Send an email from the app and check the Email Logs page in admin panel

SparkPost Webhooks (Local)

To test webhooks locally, use a tunneling service:

1. Start a tunnel to your local server (e.g., tunn.dev, ngrok)

2. Use the tunnel URL as the webhook target in SparkPost

3. Add webhook credentials to config/secrets.yml:

   development:
     email_pipeline:
       sparkpost_webhook_username: "your-username"
       sparkpost_webhook_password: "your-password"
   

4. Use a subdomain that maps to a valid organization (e.g., spinkart.tunn.dev)

Email Open Tracking

1. Enable tracking for the organization:

   # In the host app, override email_open_tracking_enabled? to return true
   # e.g., in neeto-desk: toggle Read Receipts in Admin Panel > Tickets
   

2. Send an email from a mailer that includes EmailOpenTrackable and passes trackable in params.

3. Check the intercepted email in /mailbox — the HTML source should contain a 1×1 <img> tag pointing to /neeto_email_pipeline/email_open_receipts/<token>.png.

4. Open the image URL in a browser to simulate an email open. Verify the receipt was recorded:

   NeetoEmailPipelineEngine::EmailOpenReceipt.last
   # => #<EmailOpenReceipt open_count: 1, first_opened_at: "...">
   

To send actual emails from your local, add the following lines to your development.rb

config.action_mailer.delivery_method = :smtp
config.action_mailer.perform_deliveries = true
config.action_mailer.raise_delivery_errors = true

You will also need to add the following env variables:

SPARKPOST_USERNAME=xxx
SPARKPOST_PASSWORD=xxx
SPARKPOST_DOMAIN=xxx # a verified Sparkpost domain. Unverified domains will raise errors.

Instructions for Publishing

Consult the building and releasing packages guide for details on how to publish.