Overview
Batch export helps you to incorporate the platform-generated enterprise data into your data collection and analysis.
Partner and Client users can able to setup a batch export. The partner can specify different data types by client for export. When a partner uses the API to create an export, the export is created for all clients who have not previously created the same export using the user interface. You can schedule batch exports using the Batch Export API.
You can export the following types of data for each client. It enables you to export data for each client either on demand or at scheduled intervals to Amazon AWS S3 and Microsoft Azure Blob Storage.
- Alert: Each Client alert list will export into the cloud storage.
- Audit: Each Client and All Client audit data will export into the cloud storage.
- Integrations: Each Client / Partner installed integration configuration data will export into the cloud storage.
- Inventory: Each Client level inventory data will be exported into the cloud storage.
- Metrics: Each Client level metric data will be exported into the cloud storage.
- Tickets: Each Client level ticket data will be exported into the cloud storage.
- Usage: The usage collection data(metering usage data for billing purposes) will be exported at the partner level into the cloud storage.
- Tenants: User can configure the export at partner level, and it gives the partner details and all client details under the partner.
Note
- Tenant level means – Partner / Client
- Cloud means – AWS S3 / Azure Blob Container
- Support only Partner level installed integrations in the batch exports.
Types of Supported Batch Export
Navigate to Setup > Export > Batch Export and click Type to see the list of supported batch exports.
The following batch export types for different data types are available:
Export Type | Applicable Data Type(s) | Description |
---|---|---|
Snapshot | Inventory, Integrations, and Tenants | This data export provides a snapshot view of the data at the instance of export generation into the Amazon S3 bucket folder or Azure Blob storage. |
Incremental | Alert, Ticket, Metric, Audit and Usage | The incremental batch export sends the previous three months of data as the first batch export. Subsequent exports are the incremental export of updated records. You can schedule recurring or on-demand data export:
Metric data export occurs every hour by default and cannot be customized. You can get metric data using the metric APIs. |
Data is exported in JSON format. A list of failed exports is also provided if applicable.
The exported data content depends on the type and frequency of the export schedule, which determines the data items and quantity.
After exporting the data to the installed integrations you can view the exported data on the platform, AWS S3, and Azure Blob.
How to Create a Batch Export?
By following these steps, you can successfully create a batch export:
Step 1: Select the Batch Export Add-ons
Follow the steps below to enable the Batch Export Add ons:
- At the partner level, go to Setup > Accounts > Partners.
- Click a Partner Name from the list for which you want to select Batch Exports add-on.
- Click Edit.
- Click the Add ons tab.
- Select the Batch Exports.
- Click Save.
Step 2: Install an Export Integration
Integrate with AWS S3 or Azure Blob to export the data to an AWS S3 bucket or an Azure Blob container.
To create an Export integration from the integration section, follow the steps below:
- Go to Setup > Integrations > Integrations.
- From Available Integrations, go to Exports and then select AWS S3 or Azure blob apps.
- Click the Install button.
- The Install Integration page will appear, enter the following information and click Install.
- Name: Name of the integration
- Upload Logo: Optional logo for the integration.
- It will take you to the configuration page, where you need enter the following details:
- Access Key ID
- Secret access key
- Bucket Name
- Base URI
- Click on Save.
When creating a new batch export, this saved integration will be available in the “Export to” drop-down menu in the Batch Export section.
Refer to the following links for more information on how to create AWS S3 bucket storage or Microsoft Azure Blob storage folders:
- Create AWS S3 data export integrations, refer Data Export to Amazon S3.
- Create Azure Blob Storage data export integrations, refer Azure Blob Storage.
Step 3: Create a Batch Export
Once you have successfully completed the above two steps. Now, follow the steps below to create a batch export:
Select Setup > Exports > Batch Export.
From the BATCH EXPORT screen, click the Add + icon.
Enter the following information:
Property Description Name Unique name of the export. Type Type of data to export: Alerts, Audit, Integrations, Inventory, Metrics, Tickets, Tenants, and Usage
Note: Client-level users will not have access to export Tenants and Usage type of export data.Client Client for whom the data is exported. Schedule - On Demand Only generates exports when a request is created or rerun.
- On-Demand and on Recurring Schedule generates export data when you raise a rerun request and generates an incremental export at scheduled intervals.
Export to Specify integration to AWS S3 or Azure Blob. Failure Export Notification If you enable this option, you will receive a notification if the export fails. Click Save & Run to apply the export and display export details.
Note
During on-demand execution, the data availability of metrics data on the configured buckets (AWS S3 or Azure blob storage) takes an hour from the requested time.For an example: If the request was made at
13:00 GMT
, 13:20 GMT
, or 13:40 GMT
, the data would be available on AWS S3 or Azure blob storage after next an hour (only after 1400 GMT
). Change in the file name format of the file, see the View Metric Type Batch Export
The above note is only applicable if Type is selected to “Metric” from the dropdown.
View Batch Exports
You can view the configured Batch Export details in Setup> Exports > Batch Export. The BATCH EXPORT list page displays the details of the Batch Export.
Attribute | Description |
---|---|
Batch Export Name | Name of a batch export |
Client Name | Selected client name |
Last Run | Time and date of the last export |
Last Export | Status of the last export:
|
Created By | Author |
Action | To export data. The Run Now action is available only for data export with the On Demand Only option. |
View Batch Exports on AWS S3
You can view the generated batch exports in an AWS S3 bucket in the corresponding folders. For example, the alerts export is stored in the alerts folder:
AWS stores the export files in S3 folders in JSON format. The export file name has the following encoding:
- (a) schedule of batch export, recurring or on-demand
- (b) batch export type
- (c) unique client ID
- (d) schedule starting timestamp
- (e) schedule ending timestamp
- (f) recurring export serial number
View Metric Type Batch Export
You can view the latest file format for AWS S3 metric batch export. The export file name has the following encoding:
- (A) schedule of batch export, recurring or on-demand
- (B) batch export type
- (C) unique client ID
- (D) schedule starting timestamp
- (E) unique id of the file
- (F) unique timestamp of the file
Note
Known issue: Enhanced Metric Batch Exports is currently not reporting failure notifications.View Batch Export on Azure Blob
You can view the generated data exports in the Azure Blob container in the corresponding folders:
Azure Blob stores the export files in Azure Blob containers in JSON format. The export file name has the encoding shown in the following figure:
- (A) schedule of batch export, recurring or on-demand
- (B) batch export type
- (C) unique client ID
- (D) schedule starting timestamp
- (E) schedule ending timestamp
- (F) recurring export serial number