CSV Import Validator Skill

Catch broken CSVs before they corrupt your database. When a file lands in an inbound folder, this skill scans the headers and a sample of rows for schema mismatches, weird columns, and obvious data quality red flags.

The Problem

The classic data-pipeline failure mode: a vendor sends a CSV with a renamed column, an extra column, or "TBD" instead of a number — and the pipeline either crashes 20 minutes later mid-load, or worse, loads garbage and nobody notices for a week. A 5-second pre-flight check on the first 20 rows catches almost all of these.

How It Works

1. A trigger-file watches an inbound folder. New CSV arrives → the job fires.
2. A runtimeenv task reads the header and first 20 rows of the CSV.
3. The csv-import-validator skill produces a verdict (PASS / PASS_WITH_WARNINGS / REJECT) with specific flags.
4. The message task notifies the data team — and on REJECT, the file is moved to a quarantine folder so the pipeline does not pick it up.

Setup

1. Install this skill
2. Create a trigger-file linked to your inbound folder (e.g. /data/inbox/orders/)
3. Use the YAML below. Customize the expected schema or remove it to run in discover mode

Example YAML Job

name: csv-import-validate
version: 1.0.0
description: Validate inbound CSV files before loading

notifications:
  - userNames: [intouch]
    alertOnError: true

tasks:
  - name: read-csv
    tool: runtimeenv
    properties:
      scriptContent: |
        #!/bin/bash
        FILE="${TRIGGER_FILE:-/data/inbox/orders/latest.csv}"
        ROWCOUNT=$(($(wc -l < "$FILE") - 1))
        echo "filename: $(basename "$FILE")"
        echo "totalRowCount: $ROWCOUNT"
        echo "header:"
        head -n 1 "$FILE"
        echo "sampleRows:"
        sed -n '2,21p' "$FILE"

  - name: validate
    tool: skill
    properties:
      skillName: csv-import-validator
      input: |
        expectedSchema:
          - order_id (string, required)
          - customer_id (string, required)
          - customer_email (string)
          - order_date (date YYYY-MM-DD, required)
          - amount (number, required)
          - currency (string, required)
          - shipping_address (string, required)

        {{read-csv.output}}

  - name: route
    tool: runtimeenv
    properties:
      scriptContent: |
        #!/bin/bash
        VERDICT="{{validate.answer}}"
        if echo "$VERDICT" | grep -q "Verdict: REJECT"; then
          mv "${TRIGGER_FILE}" /data/inbox/quarantine/
          echo "moved to quarantine"
        else
          mv "${TRIGGER_FILE}" /data/inbox/processed/
          echo "moved to processed"
        fi

  - name: notify
    tool: message
    properties:
      subject: "CSV validation result"
      body: "{{validate.answer}}"
      userNames: "intouch"

Customization

• Discover mode — drop the expectedSchema block in the input; the skill won't flag column count mismatches, just bad data
• Sample size — bump sed -n '2,21p' to a larger range for more thorough sampling on suspect feeds
• Auto-route on PASS — chain a sql task after route to do the actual load, gated on the verdict line

Notes

• ${TRIGGER_FILE} is the standard env var the trigger-file engine sets when the job fires; verify your trigger-file engine version sets it (older versions set TRIGGERFILE_PATH)
• For very wide CSVs (>50 columns), consider truncating the sample to the first 25 columns to keep the prompt cheap

Cost

• AI assistant: ~$0.002 per validation with Claude Haiku — pennies per file, regardless of file size