> For the complete documentation index, see [llms.txt](https://docs.uptiq.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.uptiq.ai/document-ai/cookbooks/event-driven-pipeline.md).

# Build an event-driven pipeline

**Goal.** Submit documents and have Document AI call you when each one finishes, replacing the polling loop entirely.

Extraction takes around 90 seconds per page. Polling for that is workable but wasteful; at volume it is a lot of requests that mostly say "not yet". Webhooks invert it — you register an endpoint once and receive a signed POST per completion.

### Before you start

* An API key, exported as `$DOCAI_API_KEY`.
* A **publicly reachable HTTPS endpoint**. It must be reachable from the platform, so `localhost` will not do during development — use a tunnel.
* **Owner or Admin** access to the portal, to configure the endpoint and read the signing key.

### Step 1 — Register your endpoint

Webhook endpoints are configured in the portal, not through the API.

1. Sign in as **Admin** or **Owner**.
2. **Settings → Edit Account Information**.
3. Enter your URL in **Webhook URL** and save.
4. In the same panel, **Reveal** the **Webhook Signing Key** (`whk_…`) and store it as `WEBHOOK_PRIVATE_KEY`.

{% hint style="info" %}
Endpoints are account-wide — every completion goes to every configured URL. There is no per-event subscription, and extraction and classification requests cannot override the destination. [Generation](/document-ai/developer-documentation/integration-guides/generation.md) is the exception: its request accepts a per-job `webhookUrl`.
{% endhint %}

### Step 2 — Verify signatures before trusting anything

Your endpoint is a public URL that receives document data. The `x-signature` header is the only thing separating a genuine delivery from anyone who finds the address.

```javascript
const crypto = require('crypto');
const express = require('express');

const app = express();

// Keep the raw body — see the warning below.
app.use(express.json({
  verify: (req, _res, buf) => { req.rawBody = buf; }
}));

app.post('/hooks/docai', (req, res) => {
  const secret    = process.env.WEBHOOK_PRIVATE_KEY;
  const signature = req.headers['x-signature'];

  if (!signature) return res.status(401).end();

  const digest = crypto.createHmac('sha256', secret)
                       .update(req.rawBody)
                       .digest('hex');

  const a = Buffer.from(signature, 'hex');
  const b = Buffer.from(digest, 'hex');
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
    return res.status(401).end();
  }

  handle(req.body);
  res.status(200).end();   // acknowledge fast — see step 4
});
```

{% hint style="danger" %}
Verify against the **raw** bytes, not a re-serialised object. `JSON.stringify(req.body)` after your framework has parsed the request can reorder keys or change spacing, which invalidates a signature that was actually fine. Capturing `rawBody` as above avoids the whole class of problem.
{% endhint %}

### Step 3 — Make the handler idempotent

Deliveries retry, so the same event will sometimes arrive twice. Every payload carries an `eventId`:

```javascript
async function handle(payload) {
  const { eventId, _id, status } = payload;

  if (await seen(eventId)) return;      // already processed
  await markSeen(eventId);

  if (status === 'Processed') await onSuccess(payload);
  else if (status === 'Failed') await onFailure(payload);
}
```

Design for at-least-once delivery. Deduplicating on `eventId` is far simpler than trying to make the platform deliver exactly once.

### Step 4 — Acknowledge quickly, process afterwards

Respond `200` as soon as you have verified and stored the event. Do the real work on a queue.

Two reasons. A slow endpoint eventually times out and the delivery is treated as failed even though you received it. And **`4xx` responses are terminal** — the platform stops retrying that destination rather than backing off. Returning `4xx` because your database was briefly unavailable permanently discards the event.

| Your response    | Platform behaviour                   |
| ---------------- | ------------------------------------ |
| `2xx`            | Delivered                            |
| `4xx`            | **Permanent failure — retries stop** |
| `5xx` or timeout | Retried with backoff                 |

Retries are 3 attempts with exponential backoff starting at 30 seconds, capped at 8 minutes. Deliveries stuck in processing can be retried after 24 hours. See [Webhooks & Integrations](/document-ai/guides-1/webhooks-and-integrations.md).

### Step 5 — Submit work

Nothing changes about submission. Use the asynchronous endpoints and ignore the returned ID if you like — the webhook will bring it back.

```bash
curl -X POST "https://api-docai-uat.uptiq.ai/extract" \
  -H "X-Api-Key: $DOCAI_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json
```

### Step 6 — Handle grouped completions

When one file yields several documents, you get one delivery **per child**, not one for the set:

```json
{
  "_id": "…",
  "status": "Processed",
  "extractionGroupId": "ext_group_123",
  "extractionGroupIndex": 0,
  "extractionGroupTotal": 5,
  "eventId": "evt_abc123"
}
```

Do not assume they arrive in order. Count distinct `extractionGroupIndex` values against `extractionGroupTotal` to decide when a group is complete, then fetch the whole thing with `GET /document-extractions/group/{id}` — see [Extract many documents at once](/document-ai/cookbooks/bulk-extraction.md).

Batch jobs also emit a summary-level delivery with `webhookType: "batch_extraction"`, which is separate from the per-document events.

### Step 7 — Monitor

The **Webhook Monitor** in the portal lists every delivery with its status and attempt count, and lets you retrigger one by hand after fixing a downstream problem. Use it when something did not arrive — it will tell you whether the platform tried.

{% hint style="warning" %}
An extraction record carries a `webhookStatus` object. On an account with **no webhook URL configured** it reads `{"status": "Failed", "processingAttempts": 0}` — zero attempts, nothing wrong. Confirm an endpoint is actually configured before investigating a "failure".
{% endhint %}

### When it goes wrong

| Symptom                             | Cause                                            | Fix                                                                           |
| ----------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------- |
| No deliveries at all                | No URL configured, or not publicly reachable     | Check **Settings → Edit Account Information**; test from outside your network |
| Every signature mismatches          | Verifying a re-serialised body, or the wrong key | Verify raw bytes; re-reveal the signing key                                   |
| Deliveries stop after one failure   | You returned `4xx`                               | Return `5xx` for transient problems so retries continue                       |
| Same document processed twice       | No idempotency                                   | Deduplicate on `eventId`                                                      |
| `webhookStatus: Failed`, 0 attempts | No endpoint configured                           | Configure one                                                                 |

### Related pages

* [Webhook payloads](/document-ai/developer-documentation/getting-started/webhook-payloads.md) — every payload shape and the full signature contract.
* [Webhooks & Integrations](/document-ai/guides-1/webhooks-and-integrations.md) — the Monitor, retry schedule and retrigger.
* [Extract asynchronously and poll](/document-ai/cookbooks/async-extraction-and-polling.md) — the alternative when you cannot host an endpoint.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.uptiq.ai/document-ai/cookbooks/event-driven-pipeline.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.