Shipping an AI Content Platform on Someone Else's Rate Limits

SocialPro AI generates and schedules social content across multiple platforms using OpenAI's API — ChatGPT for text generation, DALL·E for accompanying images. When people ask about the interesting engineering problem in building it, they usually expect the answer to be about prompt engineering or model selection. It wasn't. The interesting problem was building a reliable platform on top of a third party's rate limits and per-token cost, while running concurrently with a separate 23-person job-marketplace engagement competing for the same engineering attention inside the same company.

The Naive Version Would Have Failed

A straightforward first implementation — call OpenAI synchronously every time a user requests content generation — works perfectly in a demo. It falls over the very first time a marketing team decides to schedule a week's worth of posts in one sitting, which is exactly the workflow the product exists to support. You either eat a wall of HTTP 429 responses from OpenAI, or you eat a very large bill from firing off redundant generations nobody actually reads.

Both failure modes look the same to a user: the button spins, then errors. Neither failure mode is acceptable in a product whose entire pitch is "schedule a month of content in twenty minutes."

The Fix: Queue, Batch, Throttle

Every generation request goes into a Redis-backed job queue, consumed by a pool of Node.js workers. This decouples "the user asked for content" from "OpenAI is currently processing that request," which is the single change that made everything else possible.

Token-Bucket Rate Limiting

The worker pool sits behind a token-bucket rate limiter tuned to OpenAI's actual published limits for our account tier, not a guess. Bursts — a user scheduling fifty posts at once — get smoothed into a steady stream of requests instead of being rejected outright. The user still sees their content appear progressively rather than all at once, but it appears reliably.

class TokenBucket {
  constructor(capacity, refillRatePerSecond) {
    this.capacity = capacity;
    this.tokens = capacity;
    this.refillRate = refillRatePerSecond;
    this.lastRefill = Date.now();
  }

  tryConsume(cost = 1) {
    this._refill();
    if (this.tokens >= cost) {
      this.tokens -= cost;
      return true;
    }
    return false;
  }

  _refill() {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
    this.lastRefill = now;
  }
}

Caching Repeated Patterns

Prompt/response pairs for recurring content patterns — repeated post templates, common image styles a brand reuses week after week — are cached. This cut a meaningful share of redundant OpenAI calls entirely, because a lot of "content generation" in this domain isn't actually novel; it's the same brand voice and visual style applied to a new topic, which means large parts of the prompt structure repeat verbatim across requests.

The cheapest OpenAI call is the one you don't make. Caching wasn't an optimization we added later — it became a first-class part of the generation pipeline once we saw how much repetition existed in real usage patterns.

Catching Bad Output Before It Reaches a Human

AI-generated content isn't uniformly good, and that's true regardless of how good the underlying model is — some generations are simply weaker than others for reasons that are hard to predict in advance. The naive approach is to let every generation flow through to the scheduling queue and rely on a human reviewer to catch the weak ones before publishing. That works, but it puts a manual bottleneck in front of a product whose value proposition is speed.

Instead, a custom NLP scoring pass evaluates each output — checking for things like generic filler language, tone mismatch against the brand's configured voice, and structural incoherence — and flags weak generations for automatic regeneration before they're ever queued for scheduling. A human still reviews content before it's published, but they're reviewing content that's already cleared a quality bar, not triaging raw model output.

What the Scoring Pass Checks

• Similarity to generic, low-specificity phrasing patterns that tend to correlate with weak generations.
• Tone alignment against the brand voice profile configured for that account.
• Basic structural coherence — does the post actually have a clear point, or does it wander.
• Length and formatting fit for the target platform (a post generated for Twitter/X shouldn't need trimming for LinkedIn).

Keeping It Out of the Way of the Other Project

SocialPro AI is architected as a fully separate service from the job-marketplace codebase it shares a company with. That separation wasn't primarily a technical requirement — the two products don't need to talk to each other at all. It was a resourcing one. Keeping the services decoupled meant the two projects could be staffed, scaled, and shipped independently, without either one's release cadence blocking the other's, and without a shared codebase becoming a source of merge conflicts between two teams with completely different priorities.

Security Considerations

A few things mattered more than usual for an AI-content product specifically:

1. API key isolation. The OpenAI API key lives only in the worker service, never in any client-reachable code path, and is rotated on a schedule independent of any single incident.
2. Output sanitization before storage. Generated content is sanitized before it's stored or rendered anywhere, since AI output is, from a security standpoint, untrusted input the same way user input would be.
3. Per-tenant rate limit isolation. One customer scheduling an unusually large batch of content should never be able to starve another customer's queue — the token bucket is partitioned per account, not global.

Performance Tips

Technique	Why it helps
Queue-based decoupling	User-facing requests never block on OpenAI's actual response time
Token-bucket throttling	Smooths bursts instead of triggering hard rate-limit rejections
Prompt/response caching	Eliminates redundant calls for recurring content patterns
Automatic regeneration on low score	Keeps human review focused on genuinely borderline content

Common Mistakes

• Calling a third-party AI API synchronously in the user request path, then being surprised when the product doesn't scale past demo usage.
• Assuming your rate limit is your published tier limit under all conditions — real limits can be affected by account history and usage patterns, so build in headroom.
• Treating AI-generated content as inherently safe to display without sanitization, simply because it came from your own pipeline rather than directly from a user.
• Skipping automated quality scoring because "a human will catch it," which quietly turns your human reviewers into the actual bottleneck in the product.

Where It Landed

Since the rate limiter and caching layer shipped, the platform has had zero platform-wide throttling incidents, and cost per generated post dropped meaningfully thanks to cache hits on repeated patterns. Customers get a unified dashboard for multi-platform scheduling and analytics, built without pulling engineering time away from the concurrent marketplace project — which, given the team's size, was the constraint that mattered just as much as any technical one.

FAQ

Q: Why not just request a higher rate limit from OpenAI instead of building throttling logic?
We did both. A higher limit buys headroom; it doesn't remove the need to handle bursts gracefully, because any fixed limit can still be exceeded by a large enough customer action.

Q: How do you avoid caching stale or repetitive-feeling content for a brand that wants variety?
The cache keys on structural prompt patterns, not literal output — so a cached "template" still generates fresh copy for the actual topic, while skipping redundant calls for the parts of the prompt (brand voice instructions, formatting rules) that are identical across requests.

Handling Partial Failures Gracefully

Rate limiting and caching solve the throughput and cost problem, but they don't eliminate failure entirely — OpenAI's API still has occasional latency spikes and transient errors unrelated to rate limits, and a production content platform has to treat those as a normal, expected condition rather than an exceptional one.

Every generation job carries a retry policy with exponential backoff, capped at a small number of attempts before the job is marked as failed and surfaced to the user with a clear, specific message rather than a generic error. This distinction matters more than it might seem: a user who schedules fifty posts and sees "3 posts failed to generate, click to retry" can act on that information immediately. A user who sees a vague "something went wrong" for the whole batch has no way to know whether zero posts succeeded or forty-seven did.

What We Learned About Retry Budgets

Early on, we set retry limits too aggressively, assuming more retries would always improve reliability. In practice, an OpenAI outage that lasts several minutes doesn't get fixed by retrying faster — it just burns through the retry budget for every queued job simultaneously, and then every job fails at roughly the same moment once budgets are exhausted, creating a thundering-herd of user-facing failures right as the underlying issue resolves. We now jitter retry timing per job specifically to avoid this synchronized-failure pattern, so jobs naturally spread their retry attempts across a window rather than all hitting the API again at the exact same instant.

A Note on Cost Attribution

Because generation is queued and asynchronous, it's tempting to lose track of which customer account is actually driving OpenAI spend. We tag every job with the originating account at enqueue time and roll up token usage per account daily, which turned out to be essential both for internal cost monitoring and, later, for building usage-based pricing tiers into the product — a requirement that came from the business side well after the queueing architecture was already in place, and which the existing per-account tagging made straightforward to support.

Conclusion

Building on top of someone else's API means their operational limits are effectively your architecture's constraints, whether or not you plan for them upfront. Treating OpenAI's rate limits and cost structure as a first-class design input — not an afterthought to patch in once real usage exposed the gaps — is what let SocialPro AI scale past demo-day usage without becoming an unreliable, expensive product to run.