Data Sources

List Pipeline Data Sources

List<PipelineDataSource> pipelines().dataSources().getDataSources(, )

GET/api/v1/pipelines/{pipeline_id}/data-sources

Add Data Sources To Pipeline

List<PipelineDataSource> pipelines().dataSources().updateDataSources(, )

PUT/api/v1/pipelines/{pipeline_id}/data-sources

Update Pipeline Data Source

PipelineDataSource pipelines().dataSources().update(, )

PUT/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}

Get Pipeline Data Source Status

ManagedIngestionStatusResponse pipelines().dataSources().getStatus(, )

GET/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}/status

Sync Pipeline Data Source

Pipeline pipelines().dataSources().sync(, )

POST/api/v1/pipelines/{pipeline_id}/data-sources/{data_source_id}/sync

ModelsExpand Collapse

class PipelineDataSource:

Schema for a data source in a pipeline.

String id

Unique identifier

formatuuid

Component component

Component that implements the data source

One of the following:

class UnionMember0:

class CloudS3DataSource:

String bucket

The name of the S3 bucket to read from.

Optional<String> awsAccessId

The AWS access ID to use for authentication.

Optional<String> awsAccessSecret

The AWS access secret to use for authentication.

formatpassword

Optional<String> className

Optional<String> prefix

The prefix of the S3 objects to read from.

Optional<String> regexPattern

The regex pattern to filter S3 objects. Must be a valid regex pattern.

Optional<String> s3EndpointUrl

The S3 endpoint URL to use for authentication.

Optional<Boolean> supportsAccessControl

class CloudAzStorageBlobDataSource:

String accountUrl

The Azure Storage Blob account URL to use for authentication.

String containerName

The name of the Azure Storage Blob container to read from.

Optional<String> accountKey

The Azure Storage Blob account key to use for authentication.

formatpassword

Optional<String> accountName

The Azure Storage Blob account name to use for authentication.

Optional<String> blob

The blob name to read from.

Optional<String> className

Optional<String> clientId

The Azure AD client ID to use for authentication.

Optional<String> clientSecret

The Azure AD client secret to use for authentication.

formatpassword

Optional<String> prefix

The prefix of the Azure Storage Blob objects to read from.

Optional<Boolean> supportsAccessControl

Optional<String> tenantId

The Azure AD tenant ID to use for authentication.

class CloudGoogleDriveDataSource:

String folderId

The ID of the Google Drive folder to read from.

Optional<String> className

Optional<ServiceAccountKey> serviceAccountKey

A dictionary containing secret values

Optional<Boolean> supportsAccessControl

class CloudOneDriveDataSource:

String clientId

The client ID to use for authentication.

String clientSecret

The client secret to use for authentication.

formatpassword

String tenantId

The tenant ID to use for authentication.

String userPrincipalName

The user principal name to use for authentication.

Optional<String> className

Optional<String> folderId

The ID of the OneDrive folder to read from.

Optional<String> folderPath

The path of the OneDrive folder to read from.

Optional<List<String>> requiredExts

The list of required file extensions.

Optional<SupportsAccessControl> supportsAccessControl

class CloudSharepointDataSource:

String clientId

The client ID to use for authentication.

String clientSecret

The client secret to use for authentication.

formatpassword

String tenantId

The tenant ID to use for authentication.

Optional<String> className

Optional<String> driveName

The name of the Sharepoint drive to read from.

Optional<List<String>> excludePathPatterns

List of regex patterns for file paths to exclude. Files whose paths (including filename) match any pattern will be excluded. Example: [‘/temp/’, ‘/backup/’, ‘.git/’, ‘.tmp$’, ’^~’]

Optional<String> folderId

The ID of the Sharepoint folder to read from.

Optional<String> folderPath

The path of the Sharepoint folder to read from.

Optional<Boolean> getPermissions

Whether to get permissions for the sharepoint site.

Optional<List<String>> includePathPatterns

List of regex patterns for file paths to include. Full paths (including filename) must match at least one pattern to be included. Example: [‘/reports/’, ‘/docs/..pdf$’, ‘^Report..pdf$’]

Optional<List<String>> requiredExts

The list of required file extensions.

Optional<String> siteId

The ID of the SharePoint site to download from.

Optional<String> siteName

The name of the SharePoint site to download from.

Optional<SupportsAccessControl> supportsAccessControl

class CloudSlackDataSource:

String slackToken

Slack Bot Token.

formatpassword

Optional<String> channelIds

Slack Channel.

Optional<String> channelPatterns

Slack Channel name pattern.

Optional<String> className

Optional<String> earliestDate

Earliest date.

Optional<Double> earliestDateTimestamp

Earliest date timestamp.

Optional<String> latestDate

Latest date.

Optional<Double> latestDateTimestamp

Latest date timestamp.

Optional<Boolean> supportsAccessControl

class CloudNotionPageDataSource:

String integrationToken

The integration token to use for authentication.

formatpassword

Optional<String> className

Optional<String> databaseIds

The Notion Database Id to read content from.

Optional<String> pageIds

The Page ID’s of the Notion to read from.

Optional<Boolean> supportsAccessControl

class CloudConfluenceDataSource:

String authenticationMechanism

Type of Authentication for connecting to Confluence APIs.

String serverUrl

The server URL of the Confluence instance.

Optional<String> apiToken

The API token to use for authentication.

formatpassword

Optional<String> className

Optional<String> cql

The CQL query to use for fetching pages.

Optional<FailureHandlingConfig> failureHandling

Configuration for handling failures during processing. Key-value object controlling failure handling behaviors.

Example: { “skip_list_failures”: true }

Currently supports:

skip_list_failures: Skip failed batches/lists and continue processing

Optional<Boolean> skipListFailures

Whether to skip failed batches/lists and continue processing

Optional<Boolean> indexRestrictedPages

Whether to index restricted pages.

Optional<Boolean> keepMarkdownFormat

Whether to keep the markdown format.

Optional<String> label

The label to use for fetching pages.

Optional<String> pageIds

The page IDs of the Confluence to read from.

Optional<String> spaceKey

The space key to read from.

Optional<Boolean> supportsAccessControl

Optional<String> userName

The username to use for authentication.

class CloudJiraDataSource:

Cloud Jira Data Source integrating JiraReader.

String authenticationMechanism

Type of Authentication for connecting to Jira APIs.

String query

JQL (Jira Query Language) query to search.

Optional<String> apiToken

The API/ Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

Optional<String> className

Optional<String> cloudId

The cloud ID, used in case of OAuth2.

Optional<String> email

The email address to use for authentication.

Optional<String> serverUrl

The server url for Jira Cloud.

Optional<Boolean> supportsAccessControl

class CloudJiraDataSourceV2:

Cloud Jira Data Source integrating JiraReaderV2.

String authenticationMechanism

Type of Authentication for connecting to Jira APIs.

String query

JQL (Jira Query Language) query to search.

String serverUrl

The server url for Jira Cloud.

Optional<String> apiToken

The API Access Token used for Basic, PAT and OAuth2 authentication.

formatpassword

Optional<ApiVersion> apiVersion

Jira REST API version to use (2 or 3). 3 supports Atlassian Document Format (ADF).

One of the following:

_2("2")

_3("3")

Optional<String> className

Optional<String> cloudId

The cloud ID, used in case of OAuth2.

Optional<String> email

The email address to use for authentication.

Optional<String> expand

Fields to expand in the response.

Optional<List<String>> fields

List of fields to retrieve from Jira. If None, retrieves all fields.

Optional<Boolean> getPermissions

Whether to fetch project role permissions and issue-level security

Optional<Long> requestsPerMinute

Rate limit for Jira API requests per minute.

Optional<Boolean> supportsAccessControl

class CloudBoxDataSource:

AuthenticationMechanism authenticationMechanism

The type of authentication to use (Developer Token or CCG)

One of the following:

DEVELOPER_TOKEN("developer_token")

CCG("ccg")

Optional<String> className

Optional<String> clientId

Box API key used for identifying the application the user is authenticating with

Optional<String> clientSecret

Box API secret used for making auth requests.

formatpassword

Optional<String> developerToken

Developer token for authentication if authentication_mechanism is ‘developer_token’.

formatpassword

Optional<String> enterpriseId

Box Enterprise ID, if provided authenticates as service.

Optional<String> folderId

The ID of the Box folder to read from.

Optional<Boolean> supportsAccessControl

Optional<String> userId

Box User ID, if provided authenticates as user.

String dataSourceId

The ID of the data source.

formatuuid

LocalDateTime lastSyncedAt

The last time the data source was automatically synced.

formatdate-time

String name

The name of the data source.

String pipelineId

The ID of the pipeline.

formatuuid

String projectId

SourceType sourceType

One of the following:

S3("S3")

AZURE_STORAGE_BLOB("AZURE_STORAGE_BLOB")

GOOGLE_DRIVE("GOOGLE_DRIVE")

MICROSOFT_ONEDRIVE("MICROSOFT_ONEDRIVE")

MICROSOFT_SHAREPOINT("MICROSOFT_SHAREPOINT")

SLACK("SLACK")

NOTION_PAGE("NOTION_PAGE")

CONFLUENCE("CONFLUENCE")

JIRA("JIRA")

JIRA_V2("JIRA_V2")

BOX("BOX")

Optional<LocalDateTime> createdAt

Creation datetime

formatdate-time

Optional<CustomMetadata> customMetadata

Custom metadata that will be present on all data loaded from the data source

One of the following:

class UnionMember0:

List<JsonValue>

String

double

boolean

Optional<Status> status

The status of the data source in the pipeline.

One of the following:

NOT_STARTED("NOT_STARTED")

IN_PROGRESS("IN_PROGRESS")

SUCCESS("SUCCESS")

ERROR("ERROR")

CANCELLED("CANCELLED")

Optional<LocalDateTime> statusUpdatedAt

The last time the status was updated.

formatdate-time

Optional<Double> syncInterval

The interval at which the data source should be synced.

Optional<String> syncScheduleSetBy

The id of the user who set the sync schedule.

Optional<LocalDateTime> updatedAt

Update datetime

formatdate-time

Optional<DataSourceReaderVersionMetadata> versionMetadata

Version metadata for the data source

Optional<ReaderVersion> readerVersion

The version of the reader to use for this data source.

One of the following:

_1_0("1.0")

_2_0("2.0")

_2_1("2.1")