Batch Progress¶

The /batch_progress endpoint provides real-time progress tracking for batch processing operations. It allows you to monitor the status of both /batch_process and /batch_supply_shed operations by querying their tracking IDs.

This endpoint is essential for monitoring long-running batch operations, as these processes are asynchronous and can take significant time to complete depending on the number of plots or facilities being processed.

Overview¶

The /batch_progress endpoint returns different response types depending on the type of batch operation:

BatchProcessProgress: For /batch_process operations
BatchSupplyShedProgress: For /batch_supply_shed operations

Both response types provide detailed information about the current status, completion percentage, and any failures that may have occurred.

Parameters¶

Query Parameters¶

Parameter	Type	Required	Default	Description
`tracking_id`	string	Yes*	-	The tracking ID (collection_id) of the batch processing operation
`tracking_ids`	list[string]	Yes*	-	A list of tracking IDs for batch supply shed operations. Only supports batch supply shed tracking IDs and concatenates to a single progress object
`filename`	string	No	-	Deprecated: Use `tracking_id` instead. The original filename submitted during batch_process, or the collection_id/tracking_id of the batch process/supply shed operation
`dummy_data`	boolean	No	`false`	Whether to return dummy data for testing (not exposed in API schema)

* Exactly one of tracking_id, tracking_ids, or filename must be provided.

Getting the Tracking ID¶

The tracking ID is returned in the response when you submit a batch operation:

/batch_process: Returns task IDs and collection metadata
/batch_supply_shed: Returns facility tracking IDs and collection metadata

You can also retrieve tracking IDs from: - The response of the initial batch operation submission - The /get_collections endpoint

Response Types¶

BatchProcessProgress¶

Returned for /batch_process operations. Provides stage-based progress information.

{
  "n_stages": 4,
  "stage": 2,
  "progress": 50,
  "failed": false,
  "complete": false,
  "overall_progress": 37
}

Fields¶

Field	Type	Description
`n_stages`	integer	Total number of stages in the batch process (typically 4)
`stage`	integer	Current stage number (1 to n_stages)
`progress`	integer	Progress percentage for the current stage (0-100)
`failed`	boolean	Whether the batch process has failed
`complete`	boolean	Whether the batch process has completed successfully
`overall_progress`	integer	Overall progress percentage across all stages (0-100)
`skip_reason`	string \| null	Reason for skipping when processing was not run. See fetch_deforestation_check for the full list of values. Included in progress response whether the batch run is complete or not.

Stages¶

The batch process has 4 stages:

export_to_icechunk: Export the assets the statistics read.
icechunk_zonal_stats: Compute the zonal statistics.
check_tables: Validate the result tables.
normalise_collection: Write the results to PostgreSQL. This is the final stage.

BatchSupplyShedProgress¶

Returned for /batch_supply_shed operations. Provides facility-based progress information.

Note: Unlike BatchProcessProgress, BatchSupplyShedProgress does not use the n_stages concept. Instead, it tracks progress by counting facilities in different processing states. Batch supply shed operations go through 5 stages (compared to 4 stages for batch process), but these are tracked as facility counts rather than stage-based progress.

{
  "complete": false,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 3,
  "total_number_of_facilities_completed": 92,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0
}

Fields¶

Field	Type	Description
`complete`	boolean	Whether all facilities have completed processing (including failures)
`total_number_of_facilities`	integer	Total number of facilities submitted for processing
`total_number_of_ungeocoded_facilities`	integer	Number of facilities that failed to geocode and won't be processed
`total_number_of_facilities_failed`	integer	Number of facilities that have failed in any part of the process (including geocoding failures)
`total_number_of_facilities_completed`	integer	Number of facilities that have completed the process successfully (doesn't include failures)
`total_number_of_facilities_queued`	integer	Number of facilities queued to be processed
`total_number_of_facilities_preprocessing`	integer	Number of facilities in the pre-processing stage (export-to-icechunk → detect-plots)
`total_number_of_facilities_processing`	integer	Number of facilities in the processing stage, which starts after detect-plots and ends with normalise-collection
`skip_reason`	string \| null	Reason for skipping when processing was not run. See fetch_deforestation_check for the full list of values. Included in progress response whether the batch run is complete or not.

Facility Processing Flows¶

A facility moves through 5 stages, one more than the batch process:

Queued: The facility waits to start. It has no stage event yet.
Preprocessing (2 stages):
export_to_icechunk: Export the assets the statistics read.
detect_plots: Detect, discretize and finalise the plots.
Processing (3 stages):
icechunk_zonal_stats: Compute the zonal statistics.
check_tables: Validate the result tables.
normalise_collection: Write the results to PostgreSQL.
Completed: The facility reached normalise_collection.
Failed: The facility failed at one of the stages.

A plots-only supply shed stops at detect_plots.

Total: 5 stages for a supply shed, and 4 stages for a batch process.

Usage Examples¶

Python¶

Check Progress for Batch Process¶

check_batch_progress.py
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Using tracking_id (recommended)
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={"tracking_id": "0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7"},
    headers=headers
)

progress = response.json()
print(f"Overall Progress: {progress['overall_progress']}%")
print(f"Stage: {progress['stage']}/{progress['n_stages']}")
print(f"Complete: {progress['complete']}")
print(f"Failed: {progress['failed']}")

Check Progress for Batch Supply Shed¶

check_supply_shed_progress.py
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Single facility tracking ID
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={"tracking_id": "0xabc123..."},
    headers=headers
)

progress = response.json()
print(f"Total Facilities: {progress['total_number_of_facilities']}")
print(f"Completed: {progress['total_number_of_facilities_completed']}")
print(f"Failed: {progress['total_number_of_facilities_failed']}")
print(f"Queued: {progress['total_number_of_facilities_queued']}")
print(f"Preprocessing: {progress['total_number_of_facilities_preprocessing']}")
print(f"Processing: {progress['total_number_of_facilities_processing']}")
print(f"Complete: {progress['complete']}")

Check Progress for Multiple Supply Shed Tracking IDs¶

check_multiple_supply_shed_progress.py
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Multiple facility tracking IDs (only for batch supply shed)
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={
        "tracking_ids": [
            "0xabc123...",
            "0xdef456...",
            "0x789ghi..."
        ]
    },
    headers=headers
)

# Returns combined progress for all facilities
progress = response.json()
print(f"Total Facilities: {progress['total_number_of_facilities']}")
print(f"Completed: {progress['total_number_of_facilities_completed']}")

JavaScript¶

check_batch_progress.js
const axios = require('axios');

const headers = {
  'Authorization': 'Bearer <authorization_token>'
};

// Check progress for batch process
axios.get('https://epoch-sco2-api.com/batch_progress', {
  params: {
    tracking_id: '0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7'
  },
  headers: headers
})
  .then(response => {
    const progress = response.data;
    console.log(`Overall Progress: ${progress.overall_progress}%`);
    console.log(`Stage: ${progress.stage}/${progress.n_stages}`);
    console.log(`Complete: ${progress.complete}`);
    console.log(`Failed: ${progress.failed}`);
  })
  .catch(error => {
    console.error('Error checking progress:', error);
  });

cURL¶

# Check progress using tracking_id
curl -X GET "https://epoch-sco2-api.com/batch_progress?tracking_id=0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7" \
  -H "Authorization: Bearer <authorization_token>"

# Check progress for multiple supply shed tracking IDs
curl -X GET "https://epoch-sco2-api.com/batch_progress?tracking_ids=0xabc123&tracking_ids=0xdef456" \
  -H "Authorization: Bearer <authorization_token>"

Monitoring Best Practices¶

Polling Frequency¶

When monitoring batch operations, consider the following polling strategies:

Initial Phase: Poll every 30-60 seconds for the first few minutes
Active Processing: Poll every 2-5 minutes during active processing
Near Completion: Poll every 1-2 minutes as progress approaches 100%

Error Handling¶

Always check the failed field (for batch process) or total_number_of_facilities_failed (for batch supply shed) to detect failures:

progress = response.json()

if isinstance(progress, dict):
    if 'failed' in progress and progress['failed']:
        print("Batch process has failed!")
    elif 'total_number_of_facilities_failed' in progress:
        if progress['total_number_of_facilities_failed'] > 0:
            print(f"{progress['total_number_of_facilities_failed']} facilities failed")

Completion Detection¶

For batch process operations:

if progress['complete']:
    print("Batch process completed successfully!")
elif progress['failed']:
    print("Batch process failed!")
else:
    print(f"Still processing... {progress['overall_progress']}% complete")

For batch supply shed operations:

if progress['complete']:
    print("All facilities have completed processing!")
    print(f"Successful: {progress['total_number_of_facilities_completed']}")
    print(f"Failed: {progress['total_number_of_facilities_failed']}")
else:
    remaining = (
        progress['total_number_of_facilities'] - 
        progress['total_number_of_facilities_completed'] - 
        progress['total_number_of_facilities_failed']
    )
    print(f"Still processing {remaining} facilities...")

Response Examples¶

Batch Process Progress (In Progress)¶

{
  "n_stages": 4,
  "stage": 2,
  "progress": 45,
  "failed": false,
  "complete": false,
  "overall_progress": 36
}

Batch Process Progress (Complete)¶

{
  "n_stages": 4,
  "stage": 4,
  "progress": 100,
  "failed": false,
  "complete": true,
  "overall_progress": 100,
  "skip_reason": null
}

Batch Process Progress (Skipped)¶

{
  "n_stages": 4,
  "stage": 4,
  "progress": 100,
  "failed": false,
  "complete": true,
  "overall_progress": 100,
  "skip_reason": "not processed: validation confidence too low"
}

Batch Process Progress (Failed)¶

{
  "n_stages": 4,
  "stage": 3,
  "progress": 0,
  "failed": true,
  "complete": false,
  "overall_progress": 50
}

Batch Supply Shed Progress (In Progress)¶

{
  "complete": false,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 2,
  "total_number_of_facilities_completed": 60,
  "total_number_of_facilities_queued": 5,
  "total_number_of_facilities_preprocessing": 8,
  "total_number_of_facilities_processing": 20
}

Batch Supply Shed Progress (Complete)¶

{
  "complete": true,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 3,
  "total_number_of_facilities_completed": 92,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0,
  "skip_reason": null
}

Batch Supply Shed Progress (Skipped)¶

{
  "complete": true,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 0,
  "total_number_of_facilities_completed": 0,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0,
  "skip_reason": "not processed: validation confidence too low"
}

Error Responses¶

400 Bad Request¶

Returned when: - No tracking ID, tracking IDs, or filename is provided - Multiple parameters are provided (only one is allowed) - All tracking IDs in tracking_ids are not for batch supply shed runs - No tracking IDs are provided when using tracking_ids parameter - The filename is not a valid hash

Example error messages:

{
  "detail": "Exactly one of filename, tracking_id, or tracking_ids must be provided"
}

{
  "detail": "All tracking ids must be for batch supply shed runs"
}

{
  "detail": "No tracking ids provided"
}

{
  "detail": "The filename is not a valid hash."
}

400 Bad Request (Not Found)¶

Returned when the tracking ID does not exist or cannot be found.

{
  "detail": "Unable to find any batch runs for tracking_id: <tracking_id>"
}

This can also occur when: - The tracking ID is invalid - The batch run hasn't been created yet - The user doesn't have access to the batch run

403 Forbidden¶

Returned when the user doesn't have permission to access the batch operation.

{
  "detail": "Insufficient permissions"
}

/batch_process: Submit a batch processing operation
/batch_supply_shed: Submit a batch supply shed operation
/get_collections: List all collections and their status

Notes¶

The filename parameter is deprecated. Use tracking_id instead for better reliability.
The tracking_ids parameter only works for batch supply shed operations and will return a combined progress object.
Progress updates are not real-time; there may be a delay of a few seconds to minutes depending on the processing stage.
For batch supply shed operations, facilities are processed asynchronously, so progress may vary significantly between facilities.

Batch Progress¶

Overview¶

Parameters¶

Query Parameters¶

Getting the Tracking ID¶

Response Types¶

BatchProcessProgress¶

Fields¶

Stages¶

BatchSupplyShedProgress¶

Fields¶

Facility Processing Flows¶

Usage Examples¶

Python¶

Check Progress for Batch Process¶

Check Progress for Batch Supply Shed¶

Check Progress for Multiple Supply Shed Tracking IDs¶

JavaScript¶

cURL¶

Monitoring Best Practices¶

Polling Frequency¶

Error Handling¶

Completion Detection¶

Response Examples¶

Batch Process Progress (In Progress)¶

Batch Process Progress (Complete)¶

Batch Process Progress (Skipped)¶

Batch Process Progress (Failed)¶

Batch Supply Shed Progress (In Progress)¶

Batch Supply Shed Progress (Complete)¶

Batch Supply Shed Progress (Skipped)¶

Error Responses¶

400 Bad Request¶

400 Bad Request (Not Found)¶

403 Forbidden¶

Related Endpoints¶

Notes¶