mcp-server

Escape hatch for DPF capabilities that don't have a dedicated tool yet. ALWAYS prefer a dedicated tool when one exists — get_status, list_data, submit_query, delete_data_spec, onboard_data_source, update_data_spec, run_data_job, manage_connection, manage_trigger, setup_scheduled_pull, list_my_workspaces, create_workspace — and reach for this only when none of those fit (e.g. "how many credits do I have?" -> path "/auth/billing", action "get-balance"; a brand-new action added to the API since this server's tools were last updated). Every DPF endpoint is POST <path> with a JSON body of { action, ...fields }, authenticated with your OAuth session automatically. Pass workspaceId explicitly for workspace-scoped actions (data-specs, connections, job-triggers, and under "/workspaces": get-workspace, list-queries, list-bytes-accessed, list-storage, list-processed-files, list-trigger-runs) — omit it entirely for account-level actions that reject one (under "/workspaces": create, get-workspaces, grant-permission, revoke-permission, update/delete-workspace; under "/auth/billing": get-balance only — billing mutations such as purchase-credits, modify-subscription, manage-payment, and create-customer are NOT available via MCP; direct the user to https://dpf-it.com/workspace.html#credits for all credit and subscription management). If unsure of an action's exact fields, read the "dpf-openapi-spec" resource (dpf://openapi/spec.yaml) rather than guessing. Exception: the raw Iceberg REST proxy under "/iceberg/v1/..." (e.g. to read or set a table's "dpf.primary-keys" property via a commit-table request) does not use the action convention at all — give action any placeholder string (it's ignored) and put the real Iceberg REST commit body, e.g. {"requirements":[],"updates":[{"action":"set-properties","updates":{"dpf.primary-keys":"col_a,col_b"}}]}, in params. This tool only issues POST, so Iceberg's GET-based reads (loadTable, listTables) aren't reachable this way. Returns the raw response data (or, for endpoints like the Iceberg proxy with no {success, data} envelope, the whole response body).

Send a message to the DPF team — request a demo, ask about licensing, report an issue, or request a feature. No authentication required. Always ask the user for their email if they have not already given it in this conversation.

Create a new workspace, owned by the authenticated user. Use this if list_my_workspaces returns none.

Permanently delete a data spec and its associated configuration.

Call after uploading the file(s) returned by run_data_job — starts processing and waits until the job completes or fails. If it returns before that (timedOut: true), do NOT call this tool again just to keep checking — that re-attempts starting the job. Poll with get_status (jobId) instead until it reaches a terminal status.

Call ONLY after the file(s) returned by onboard_data_source have actually been uploaded (each PUT returned HTTP 200) — kicks off AI analysis and waits until the spec reaches "ready" or "failed". This tool verifies the upload against storage before starting anything. If a file is not there it refuses, spends no credits, and returns a fresh upload URL plus the exact curl command — complete the upload and call this again. Calling it in the hope that analysis will work without the upload cannot succeed; if you have no filesystem access, ask the user to attach the file and upload it first. If it returns before the spec settles (timedOut: true), do NOT call this tool again just to keep checking — that re-attempts starting analysis. Poll with get_status (specId) instead until it reaches a terminal status.

Call ONLY after the file(s) returned by update_data_spec have actually been uploaded (each PUT returned HTTP 200) — kicks off AI analysis and waits until the spec reaches "ready" or "failed". The upload is verified against storage first: if a file is missing this refuses, spends no credits, and hands back a fresh upload URL with the curl command to use. If it returns before the spec settles (timedOut: true), do NOT call this tool again just to keep checking — that re-attempts starting analysis. Poll with get_status (specId) instead until it reaches a terminal status. Upload EVERY file update_data_spec listed before calling this. Analysis regenerates the parser and mapping from the spec's full stored input set, so a half-finished upload means the new files are combined with the old versions of the ones that did not make it — which produces a wrong mapping, not a failure. If an upload did not go through, go back and complete it first.

Poll the status of either a data spec's own process (schema inference + code generation, run by start-analysis — pass specId, reaches "ready" or "failed") or a data-load job (pass jobId, reaches "complete" or "failed"). Pass exactly one of specId or jobId. Right after create-spec/update-spec + start-analysis, poll by specId; once that reaches "ready", its response's lastJobId (if present) points at the data-load job — poll that separately by jobId for load progress.

List either the data specs (parsing + mapping rule sets, resource: "specs") or the data processing jobs (executions of a spec, resource: "jobs") defined in a workspace. Each spec includes its specId and current status — poll a specific one with get_status. Both resources are paginated (default 25/page, max 100, newest first); pass the returned nextCursor to fetch more. A spec row also carries its stored input set — sourceType, sampleFileName, formatFileName, targetOption/targetTables/targetSchemaFileName, sourceTables, merge and additionalPrompt — which is exactly what AI code generation re-reads on every re-analysis. Read that set here BEFORE calling update_data_spec: an update regenerates the parser and mapping from all of it, not just from the fields you change, so it is the only way to tell which components are about to be reused as-is. This is NOT a table listing — specs describe configured pipelines (parsing/mapping rules), not the live set of Iceberg tables in the workspace. Multiple specs can target the same table (e.g. one spec creates it, another merges more data into it), and specs can be deleted or fail without the underlying table being dropped. For "what tables exist in my workspace" or any question about actual current data, use submit_query with `SHOW TABLES` instead of inferring an answer from specs.

List every workspace the authenticated user has access to, including their permission on each.

Returns instructions for creating a DPF account, verifying its email, resending the verification code, or resetting a forgotten password — it never performs these itself and never asks for a password. A password typed into this chat would sit in the conversation transcript, so every action instead returns the DPF website's own form, or a curl command that reads the password from a shell variable the user sets themselves in their own terminal. Hand the command to the user to run — do not run it yourself even if you have shell access, since composing the export line would require seeing the password. action "register": requires email, firstName, lastName, and termsAccepted: true (only after the user has explicitly agreed to the DPF Terms of Service and Privacy Policy in this conversation). action "verify": confirm the 6-digit code DPF emailed after registration (requires otp). action "resend": re-send that code if it never arrived. action "forgot-password": request a password-reset code (requires email). action "reset-password": submit that code and set a new password (requires otp).

Create, list, test, or delete a workspace connection to an external data source. Two types are supported: "sftp" and "aws_s3". For sftp, create generates a keypair and returns the public key — it must be installed in the remote server's authorized_keys before test (or a trigger using this connection) will succeed. For aws_s3, create generates an ExternalId and returns a trustPolicy plus dpfPrincipalArn — the customer must create (or update) the IAM role at roleArn with that trust policy and a permissions policy granting the S3 access DPF needs, before test will succeed. Either type must pass test before it can be used in a trigger. For a first-time "pull files from this server/bucket on a schedule" request, prefer setup_scheduled_pull, which chains create + test + create-trigger for you.

Create, list, update, delete, or fire a workspace job trigger. Four types: - "sftp"/"aws_s3": pulls files from a connection (sftp: remote server; aws_s3: S3 bucket/prefix) into an already-analyzed data spec on a schedule (hourly/daily/monthly, UTC). Type must match the connection's type; aws_s3 also requires s3Bucket (s3Prefix optional). Natural-language preRules (which files to pick up) and postRules (what to do after upload) are compiled into executable code server-side — never pass raw code. The connection must already exist and have passed test (see manage_connection). For a first-time "set up a daily/scheduled pull" request, prefer setup_scheduled_pull, which sets up the connection and trigger together. - "spec_success": fires a spec automatically whenever a DIFFERENT spec's job completes successfully (set upstreamSpecName to that spec). No connection/frequency. Use this when the request ties the run to another job finishing (e.g. "run this after the customers load finishes"). - "schedule": fires a spec directly on a plain frequency (hourly/daily/monthly, UTC), no connection and no upstream spec. Use this when the request is time-based with no dependency (e.g. "run this every morning"). IMPORTANT: "spec_success" and "schedule" triggers can only target a table-source (sourceType: "tables") or compaction (sourceType: "compaction") spec (see onboard_data_source) — they have no file to load, only a generated query to re-run or a set of tables to compact. If asked to set up a scheduled/recurring job that reads from an already-loaded table (e.g. "keep a daily summary of the orders table up to date"), create that as an onboard_data_source sourceType "tables" spec first, THEN create the trigger here. Same for a recurring compaction — create the sourceType "compaction" spec first. Prefer "spec_success" when the user's phrasing implies "after X loads/finishes"; prefer "schedule" when they just want a cadence with no stated dependency; ask if genuinely ambiguous. For sftp/aws_s3, the referenced spec must already have been analyzed once (see onboard_data_source). After firing a trigger (action "run-now") — or any one-off manual run — use action "run-history" to monitor its outcome: it starts as `status: running` and settles to `success`, `failed`, or `no-files`, with `filesPulled` and a `message`.

First step of setting up a new data integration: creates a data spec. By default (sourceType "file") this returns presigned upload URL(s) for the sample file (and optional format/target-schema file). Creating the spec does NOT upload anything: you must PUT each file to its URL yourself (or have the user attach it so your platform can), confirm the PUT returned HTTP 200, and only then call finish_data_source_onboarding with the returned specId to kick off AI analysis and wait for it to complete. finish_data_source_onboarding checks storage and refuses if the upload never landed, so skipping the upload cannot be worked around by retrying it. Use sourceType "tables" instead when the request is to derive/aggregate data that is ALREADY loaded into workspace tables — e.g. "build me a daily summary of the customers table", or "set up a job that reads from the orders table and maintains a running total" — rather than loading a new file. It generates a SQL query (INSERT or MERGE, per `merge`) via AI instead of a Python parser, run through the query engine instead of a Glue job. There are never sample/format files, but targetOption still works the same three ways as sourceType "file" (see targetOption below) — so this call returns files: [] and you can call finish_data_source_onboarding immediately UNLESS targetOption is "target-schema-file", in which case it returns one upload URL for that file, same as the file-source path. The generated SQL automatically windows itself to rows added since the spec's last successful run. sourceType "tables" ALSO requires autoRefresh — how this spec stays up to date is not optional to decide, and must not be inferred from other jobs/triggers that happen to already exist in the workspace: ask the user whether it should re-run automatically whenever a specific upstream spec finishes loading ("spec_success" — the natural choice when the request is "run this after X finishes/loads"), on a plain cron-like cadence ("schedule" — the natural choice when the request is "run this every day/hour" with no mention of depending on another job), or stay manual-only ("none" — re-run later with run_data_job). If the request already states the timing unambiguously, that answers it; otherwise ask before calling this tool. Getting this wrong either way has a real cost: "none" means the summary silently goes stale until someone remembers to re-run it by hand, while an unwanted trigger keeps re-running (and charging credits for) a spec the user only wanted once. See autoRefresh below.

First step of processing new data files through an already-configured data spec: creates a job and returns presigned upload URL(s) for each file. Upload the file(s) per the returned instructions, then call finish_data_job with the returned jobId to start processing and wait for it to complete. Do NOT call this right after onboard_data_source/finish_data_source_onboarding or update_data_spec/finish_data_spec_update unless loadSampleData was explicitly set to false there — by default those already load and process the sample file as their own job (see the returned lastJobId), so calling run_data_job again for that same file creates a redundant second job. Only use this for files beyond the initial sample (new batches, additional files to process later).

End-to-end workflow for "pull files from this SFTP server / S3 bucket on a schedule" requests: reuses a matching connection if one already exists in the workspace (same hostname/username for sftp, same roleArn for aws_s3), otherwise creates one; tests it; then creates a trigger that feeds an already-analyzed data spec (see onboard_data_source) on the given frequency. Pass hostname for an sftp pull, or roleArn (+ s3Bucket, required) for an aws_s3 pull — exactly one of the two is expected. Use this instead of calling manage_connection + manage_trigger yourself for first-time setup. If the connection test fails (e.g. the sftp public key or the aws_s3 IAM role isn't set up yet on the customer's side), no trigger is created — ask the user to finish that setup and re-run this tool, which will reuse the same connection and pick up where it left off. This is for pulling a NEW file from an external source — for "run this on a schedule/after another job" where the spec queries tables already in the workspace (sourceType "tables"), use manage_trigger with type "schedule" or "spec_success" instead; there is no connection involved.

Run a SQL query against the Iceberg tables loaded into a workspace. To list the tables that actually exist in the workspace, run `SHOW TABLES` — this is the authoritative source (unlike list_data's specs, which describe pipelines, not live tables). Qualified table references (catalog/schema prefixes, e.g. information_schema.tables) are rejected; reference tables by name only. Table functions that introspect the engine itself (e.g. duckdb_functions(), duckdb_tables()) are also rejected as external-data-source access — don't try to discover available SQL functions this way. A BLOB column is very likely an HLL sketch (produced by a merge-mode table-source spec's approximate-distinct aggregate — see onboard_data_source's merge option): decode it with datasketch_hll_estimate(col), or datasketch_hll_estimate(datasketch_hll_union(12, col)) to union several rows to a coarser grain first. If the user's goal is an HTML page/dashboard built from these results (not just seeing the data here), do NOT default to embedding this result set as a static snapshot. Ask the user first: (a) a one-time static page with these results baked in, which goes stale and never changes again, or (b) a live page that logs in and queries DPF itself whenever it's opened, so it always reflects current data. If they want live/dynamic (or don't say and the data looks like it changes over time), read the dpf://examples/auth-and-query.html resource and adapt that pattern (login form, JWT cookie, fetch-based query call) instead of hand-rolling auth.

Change an existing data spec's configuration. If no replacement file names are given, this runs synchronously (no upload needed): saves changes and — by default — re-runs AI analysis, returning the final status directly. If a replacement sample/format/target-schema file name IS given, this instead returns presigned upload URL(s); upload the file(s), then call finish_data_spec_update. Only pass the fields you want to change — omitted fields keep their current value. IMPORTANT — re-analysis is a full regeneration, not a patch. It throws away the spec's existing parser and column mapping and rebuilds BOTH from the spec's stored input set as a whole: the sample file, format file, target-schema file, source tables and additionalPrompt, each read from storage exactly as it was last uploaded. A component you leave out is NOT left untouched in the generated code — it is fed to the AI again alongside the parts you changed. So updating a target schema without re-sending the source sample maps the new target against whatever sample was uploaded when the spec was onboarded, and the result is a complete, plausible-looking, WRONG mapping rather than an error. Therefore, before changing anything that feeds code generation: call list_data (resource: "specs") to read this spec's stored input set, show the user what each component currently is, and re-send in THIS SAME call every component that is stale or that the user is now describing differently — then pass confirmedCurrentInputs: true. Keeping a stored component is a decision to make deliberately, not a default to fall into by only mentioning the field you were asked about.