> ## Documentation Index
> Fetch the complete documentation index at: https://docs.affinda.com/llms.txt
> Use this file to discover all available pages before exploring further.
## Submitting Feedback
If you encounter incorrect, outdated, or confusing documentation on this page, submit feedback:
POST https://docs.affinda.com/feedback
```json
{
"path": "/academy/straight-through-processing",
"feedback": "Description of the issue"
}
```
Only submit feedback when you have something specific and actionable to report.
# Achieving straight-through processing
> Configure machine validation, confidence thresholds, and automation rules to achieve straight-through processing for high-volume document workflows.
## Purpose
This tutorial guides users on how they can achieve full document automation with straight-through-processing in Affinda. Stacking validation rules, data mapping, and correctly configuring your validation settings in Affinda allows documents to pass through without the need for human intervention.
This guide is designed for advanced users in Affinda looking to achieve straight-through processing. To complete this guide, Users should already have their Document Type(s) configured and hold an Owner or Admin role in Affinda.
## What is straight-through processing?
Straight-through-processing in the context of Affinda refers to a fully automated, end-to-end workflow in which documents that enter the system can be:
* Parsed and classified
* Key data fields extracted and validated
* Enriched or transformed as needed
All without the need for human intervention at any stage.
## Straight-through processing in Affinda
Straight-through processing allows documents to be automatically processed without manual intervention. This can be achieved in Affinda by using three features together:
1. **Data Mapping:** To specify how extracted data should be formatted
2. **Validation Rules:** To define what makes a document valid or invalid
3. **Validation Settings:** To control how documents are processed based on their validation status
When these features are properly configured, valid documents that meet all rules will automatically pass through to your downstream systems, while documents that fail validation will be flagged for manual review. This ensures accuracy while reducing unnecessary manual checks, helping your organization work more efficiently.
## Data Mapping
Data mapping is a powerful capability that enhances the accuracy of data extraction. With data mapping configured, the extracted data from a document can be mapped against a list of known options from your knowledge base to validate that it meets expected values. This enables correct values to bypass review, whilst only incorrect values are flagged.
Data Source mapping can be configured through the Affinda app or the API. To configure in the App:
First, start by creating your Data Source field. Navigate to the Configure document type menu, and create a new field with data type "Data Source".
Data Sources act as \*\*lightweight master-data tables \*\*that you can manage from within Affinda.
Select "+ New Data Source" and upload your data source file. Affinda supports CSV, XLSX, or JSON files as data sources.
Or create via **API**:
```http theme={null}
POST /data-sources # Create
```
A *key column* is the single field in your source table whose values are guaranteed to be unique; it is used by Affinda to reliably identify each record unambiguously.
The *label* is the human-readable name stored for each record in a Mapping Data Source; it’s what users will see in the validation UI and drop-downs.
You can manage your Data Sources at the organizational level.
Select the document field to match eg. Employer Name.
Pick the data source property (the column you want to match it to)
Select the match type:
* Exact - strings must be identical.
* Partial - succeeds when the extracted string appears anywhere in the candidate.
* Fuzzy - uses edit-distance logic to allow minor typos to still be matched.
Tick "Required" if validation should fail when nothing matches for this criterion. Enabling this will flag documents that have failed the matching criteria against your data source, flagging them for review. Documents that have passed the matching criteria won't be flagged and can continue to your downstream workflows with the confidence that their fields were extracted correctly.
Ensure your data source is always up to date to maximise documents that can be STP.
You can update data sources manually or automate it with API calls from your integration.
Call `PUT /data-sources/{id}` for full replacement or `PATCH /data-sources/{id}`for incremental updates.
### Tips for reliable matching
* Trim whitespace and normalise case in your source files.
* Ensure the key column is unique.
* Keep label strings short so they fit comfortably in the UI.
## Validation rules
Within the Affinda platform, validation rules can be configured to automatically check whether extracted data meets predefined business logic. When these rules pass, the data is considered valid and does not require user review, enabling straight-through processing for compliant documents. Validation rules help identify documents that fail to meet business rules, prompting review.
Navigate to the Document Type Configuration interface of the document type that contains the field you want to apply a text transformation to.
Click the pencil icon next to the field of your choice to open up the Field Settings.
Open the "Validation Rule" drop-down.
Or go to the validation tab, and click "Add Validation Rule".
Validation rules can be created on just one or multiple extracted fields. Type "@fieldname" to reference a field. Write the rule in natural language and press generate to apply.
**Some example validation rules you might use for inspiration are:**
* "Check @SupplierVAT is a valid VAT"
* "Check @CustomerAddress has a valid postcode"
* "Check the sum of the @subtotals = @totalamount"
* "@purchaseorder is 7 digits long"
You can also include the pass or fail messages you want to be returned in the Validation Interface (e.g. “Ensure the invoice date is before the due date. If not, show: ‘Due date cannot be earlier than invoice date.").
In the advanced settings of your validation rule, you can configure how the rule should act in the case of missing data.
**Available Options:**
* **Skip** -The rule is skipped entirely—no pass or fail outcome is recorded.
* **Fail** - The rule automatically fails if any required field is missing.
* **Pass** - The rule is marked as passed, regardless of missing fields.
* **Ignore** - The rule runs by treating missing fields as neutral, attempting to return a valid result based on available data.
* For example, useful in cases like subtotal + freight + tax = total, where freight is occasionally absent.
You can see how your document performs against your new validation rule and make changes accordingly.
You can also click 'View Code' to see the code generated for the rule.
It's best to check your validation rules across different documents to ensure they perform as desired. Test both documents that should pass and fail the rule.
The easiest way to manage validation rules is in the Validation tab in the Document Type Configuration Interface. There, you can disable fields when needed.
## Validation Settings
Once you have your data mapping and validation rules configured, the last step to enable straight-through processing is to enable **auto-confirmation**. This allows Affinda to confirm documents that have passed all data mapping and validation rules, sending them downstream automatically without human review. Documents that fail data mapping or validation rules will be flagged and left for human review.
You can enable auto-confirmation in the **Validation Settings** within your **Workflow Settings**.
If 'Auto-Confirmation' is enabled, a document will skip the 'For Review' queue if there are no failed rules on that document (and at least one rule is enabled). It ***does not*** require each individual field to be auto-confirmed for the document to be auto-confirmed.
## User Validation
Documents that aren't correct or don't pass business rules will need to be reviewed. Affinda's Document Validation interface offers a user-friendly platform to review and manage these documents. Setting up straight-through processing ensures these documents are flagged for easy review. To learn how to review documents, see our [Tutorial: Review Extraction. ](/configuration/user-validation)