Overview

The Pipelines API provides a flexible way to capture CI data in DX where a native data connector does not exist.

To use the Pipelines API, instrument your pipelines to send requests to our endpoint with updates on the status of builds.

These requests are summed up to create a database record that includes the duration, user attribution, and associated code change.

For example, given the following requests:

Name	Identifier	Started	Finished	Status
test_run	step 1	11:25:18		running
test_run	step 2	11:25:21		success
test_run	step 3	11:24:40		failure
test_run	step 4	11:25:21		running
test_run	step 5	11:26:07	11:46:67	success

DX will collect them into one record that represents the whole run like this:

Name	Started	Finished	Status
test_run	11:24:40	11:46:67	failure

The final status will be calculated by looking at the notifications sorted by started_at and replacing the status if the newer status is higher on this priority list than the previous status:

failure > cancelled > success > running > unknown

By default the status will be unknown if our API is notified about a run without any other status provided. This might happen if you have a pre-boot startup process that informs us about the run before any steps or tasks have been run.

In addition to collecting the overall result of a run in one place, DX also links the run to to code changes, and attributes it to individuals and teams when an identifier (i.e., email, GitHub username, or GitLab username) is included in request data.

API methods

pipelineRuns.sync

pipelineRuns.sync

Inserts or updates a pipeline run based on the provided reference_id.

Required arguments

Name	Description
pipeline_name	Name of the pipeline that this job is for. Example: `integration_tests`
pipeline_source	Which project source is this? CircleCI, Jenkins, Tekton etc. Example: `Tekton`
reference_id	A globally unique identifier for this run. Example: `AA713E04-1880-4E17-8C49-4A41733DAFCF`
started_at	A string with a unix timestamp. Each notification does not need to have the same started_at value. They can arrive in any sequence and the earliest value will be used. Example: `1702193699`

Optional arguments

Name	Description
status	One of: `failure` `running` `success` `cancelled` Defaults to `unknown`
finished_at	A string with a unix timestamp. Example: `1702213699`
repository	The full name of the repository. Example: `getdx/some_repo`
commit_sha	The commit SHA that the build was run on. If commit SHA is provided, repository is required.
pr_number	The number of the pull request that the build was run on. If PR number is provided, repository is required. Example: `1`
source_url	The web address to view details about the pipeline run.
head_branch	Used to know which branch the pipeline run was run against. Example: `main`
email	Email address of the user who initiated the build. Example: `[email protected]`
github_username	GitHub username of the user who initiated the build. Example: `johndoe`
gitlab_username	GitLab username of the user who initiated the build. Example: `johndoe`

Example

curl -X POST https://yourinstance.getdx.net/api/pipelineRuns.sync \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_bearer_token_here" \
  -d '{
  "pipeline_name": "build_panda_finder", 
  "pipeline_source": "Tekton",
  "reference_id": "AA713E04-1880-4E17-8C49-4A41733DAFCF",
  "started_at": 1702193699,
  "finished_at": 1702213699,
  "status": "success"
  }'

⚠️ Make sure to catch errors in your API requests so that your pipeline success is not dependent on the Data Cloud API uptime.

Error codes

Error Code	Description
`not_authed`	This error occurs if the API request does not include a valid API key for authentication.
`invalid_auth`	The provided API key is invalid. This can happen if the key is expired or does not exist.
`account_inactive`	The user account associated with the API key is deactivated or suspended.
`required_params_missing`	One or more required parameters were not provided in the request.