Skip to content

Batch Progress

The /batch_progress endpoint provides real-time progress tracking for batch processing operations. It allows you to monitor the status of both /batch_process and /batch_supply_shed operations by querying their tracking IDs.

This endpoint is essential for monitoring long-running batch operations, as these processes are asynchronous and can take significant time to complete depending on the number of plots or facilities being processed.

Overview

The /batch_progress endpoint returns different response types depending on the type of batch operation:

  • BatchProcessProgress: For /batch_process operations
  • BatchSupplyShedProgress: For /batch_supply_shed operations

Both response types provide detailed information about the current status, completion percentage, and any failures that may have occurred.

Parameters

Query Parameters

Parameter Type Required Default Description
tracking_id string Yes* - The tracking ID (collection_id) of the batch processing operation
tracking_ids list[string] Yes* - A list of tracking IDs for batch supply shed operations. Only supports batch supply shed tracking IDs and concatenates to a single progress object
filename string No - Deprecated: Use tracking_id instead. The original filename submitted during batch_process, or the collection_id/tracking_id of the batch process/supply shed operation
dummy_data boolean No false Whether to return dummy data for testing (not exposed in API schema)

* Exactly one of tracking_id, tracking_ids, or filename must be provided.

Getting the Tracking ID

The tracking ID is returned in the response when you submit a batch operation:

  • /batch_process: Returns task IDs and collection metadata
  • /batch_supply_shed: Returns facility tracking IDs and collection metadata

You can also retrieve tracking IDs from: - The response of the initial batch operation submission - The /get_collections endpoint - The /get_task_status endpoint

Response Types

BatchProcessProgress

Returned for /batch_process operations. Provides stage-based progress information.

{
  "n_stages": 4,
  "stage": 2,
  "progress": 50,
  "failed": false,
  "complete": false,
  "overall_progress": 37
}

Fields

Field Type Description
n_stages integer Total number of stages in the batch process (typically 4)
stage integer Current stage number (1 to n_stages)
progress integer Progress percentage for the current stage (0-100)
failed boolean Whether the batch process has failed
complete boolean Whether the batch process has completed successfully
overall_progress integer Overall progress percentage across all stages (0-100)
skip_reason string | null Optional reason for skipping processing (e.g., "low - not processed" for low confidence). Included in progress response whether the batch run is complete or not.

Stages

The batch process consists of 4 stages:

  1. start_batch_process: Initial batch process setup and initialization
  2. start_zonal_stats: Beginning zonal statistics computation for environmental metrics
  3. ingest_images_orchestration: Orchestrating satellite image ingestion
  4. orchestrate_ingestion: Final orchestration step for completing ingestion and table finalization

BatchSupplyShedProgress

Returned for /batch_supply_shed operations. Provides facility-based progress information.

Note: Unlike BatchProcessProgress, BatchSupplyShedProgress does not use the n_stages concept. Instead, it tracks progress by counting facilities in different processing states. Batch supply shed operations go through 7 processing flows (compared to 4 stages for batch process), but these are tracked as facility counts rather than stage-based progress.

{
  "complete": false,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 3,
  "total_number_of_facilities_completed": 92,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0
}

Fields

Field Type Description
complete boolean Whether all facilities have completed processing (including failures)
total_number_of_facilities integer Total number of facilities submitted for processing
total_number_of_ungeocoded_facilities integer Number of facilities that failed to geocode and won't be processed
total_number_of_facilities_failed integer Number of facilities that have failed in any part of the process (including geocoding failures)
total_number_of_facilities_completed integer Number of facilities that have completed the process successfully (doesn't include failures)
total_number_of_facilities_queued integer Number of facilities queued to be processed
total_number_of_facilities_preprocessing integer Number of facilities in the pre-processing stage (start-batch-supply-shed → detect-plots → export_vectors)
total_number_of_facilities_processing integer Number of facilities in the processing stage (complete-detect-plots → start-zonal-stats → ingest_images → complete-zonal-stats → check-tables)
skip_reason string | null Optional reason for skipping processing (e.g., "low - not processed" for low confidence). Included in progress response whether the batch run is complete or not.

Facility Processing Flows

Facilities progress through 7 processing flows (more than the 4 stages in batch process):

  1. Queued: Facility is waiting to start processing (not yet started any flow)
  2. Preprocessing (4 flows):
  3. start_batch_supply_shed: Initial batch supply shed setup
  4. detect_plots: Plot detection and discretization
  5. export_vectors_orchestration: Exporting plot geometries
  6. complete_detect_plots: Finalizing plot detection
  7. Processing (3 flows):
  8. start_zonal_stats: Beginning zonal statistics computation
  9. ingest_images_orchestration: Orchestrating satellite image ingestion
  10. orchestrate_ingestion: Final orchestration step for completing ingestion
  11. Completed: Facility processing finished successfully (completed the final orchestration step)
  12. Failed: Facility processing encountered an error at any flow

Total: 7 flows for batch supply shed vs 4 stages for batch process

Usage Examples

Python

Check Progress for Batch Process

check_batch_progress.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Using tracking_id (recommended)
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={"tracking_id": "0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7"},
    headers=headers
)

progress = response.json()
print(f"Overall Progress: {progress['overall_progress']}%")
print(f"Stage: {progress['stage']}/{progress['n_stages']}")
print(f"Complete: {progress['complete']}")
print(f"Failed: {progress['failed']}")

Check Progress for Batch Supply Shed

check_supply_shed_progress.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Single facility tracking ID
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={"tracking_id": "0xabc123..."},
    headers=headers
)

progress = response.json()
print(f"Total Facilities: {progress['total_number_of_facilities']}")
print(f"Completed: {progress['total_number_of_facilities_completed']}")
print(f"Failed: {progress['total_number_of_facilities_failed']}")
print(f"Queued: {progress['total_number_of_facilities_queued']}")
print(f"Preprocessing: {progress['total_number_of_facilities_preprocessing']}")
print(f"Processing: {progress['total_number_of_facilities_processing']}")
print(f"Complete: {progress['complete']}")

Check Progress for Multiple Supply Shed Tracking IDs

check_multiple_supply_shed_progress.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import requests

headers = {'Authorization': "Bearer <authorization_token>"}

# Multiple facility tracking IDs (only for batch supply shed)
response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={
        "tracking_ids": [
            "0xabc123...",
            "0xdef456...",
            "0x789ghi..."
        ]
    },
    headers=headers
)

# Returns combined progress for all facilities
progress = response.json()
print(f"Total Facilities: {progress['total_number_of_facilities']}")
print(f"Completed: {progress['total_number_of_facilities_completed']}")

JavaScript

check_batch_progress.js
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
const axios = require('axios');

const headers = {
  'Authorization': 'Bearer <authorization_token>'
};

// Check progress for batch process
axios.get('https://epoch-sco2-api.com/batch_progress', {
  params: {
    tracking_id: '0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7'
  },
  headers: headers
})
  .then(response => {
    const progress = response.data;
    console.log(`Overall Progress: ${progress.overall_progress}%`);
    console.log(`Stage: ${progress.stage}/${progress.n_stages}`);
    console.log(`Complete: ${progress.complete}`);
    console.log(`Failed: ${progress.failed}`);
  })
  .catch(error => {
    console.error('Error checking progress:', error);
  });

cURL

1
2
3
4
5
6
7
# Check progress using tracking_id
curl -X GET "https://epoch-sco2-api.com/batch_progress?tracking_id=0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7" \
  -H "Authorization: Bearer <authorization_token>"

# Check progress for multiple supply shed tracking IDs
curl -X GET "https://epoch-sco2-api.com/batch_progress?tracking_ids=0xabc123&tracking_ids=0xdef456" \
  -H "Authorization: Bearer <authorization_token>"

Monitoring Best Practices

Polling Frequency

When monitoring batch operations, consider the following polling strategies:

  1. Initial Phase: Poll every 30-60 seconds for the first few minutes
  2. Active Processing: Poll every 2-5 minutes during active processing
  3. Near Completion: Poll every 1-2 minutes as progress approaches 100%

Error Handling

Always check the failed field (for batch process) or total_number_of_facilities_failed (for batch supply shed) to detect failures:

progress = response.json()

if isinstance(progress, dict):
    if 'failed' in progress and progress['failed']:
        print("Batch process has failed!")
    elif 'total_number_of_facilities_failed' in progress:
        if progress['total_number_of_facilities_failed'] > 0:
            print(f"{progress['total_number_of_facilities_failed']} facilities failed")

Completion Detection

For batch process operations:

if progress['complete']:
    print("Batch process completed successfully!")
elif progress['failed']:
    print("Batch process failed!")
else:
    print(f"Still processing... {progress['overall_progress']}% complete")

For batch supply shed operations:

if progress['complete']:
    print("All facilities have completed processing!")
    print(f"Successful: {progress['total_number_of_facilities_completed']}")
    print(f"Failed: {progress['total_number_of_facilities_failed']}")
else:
    remaining = (
        progress['total_number_of_facilities'] - 
        progress['total_number_of_facilities_completed'] - 
        progress['total_number_of_facilities_failed']
    )
    print(f"Still processing {remaining} facilities...")

Response Examples

Batch Process Progress (In Progress)

{
  "n_stages": 4,
  "stage": 2,
  "progress": 45,
  "failed": false,
  "complete": false,
  "overall_progress": 36
}

Batch Process Progress (Complete)

{
  "n_stages": 4,
  "stage": 4,
  "progress": 100,
  "failed": false,
  "complete": true,
  "overall_progress": 100,
  "skip_reason": null
}

Batch Process Progress (Skipped)

{
  "n_stages": 4,
  "stage": 4,
  "progress": 100,
  "failed": false,
  "complete": true,
  "overall_progress": 100,
  "skip_reason": "low - not processed"
}

Batch Process Progress (Failed)

{
  "n_stages": 4,
  "stage": 3,
  "progress": 0,
  "failed": true,
  "complete": false,
  "overall_progress": 50
}

Batch Supply Shed Progress (In Progress)

{
  "complete": false,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 2,
  "total_number_of_facilities_completed": 60,
  "total_number_of_facilities_queued": 5,
  "total_number_of_facilities_preprocessing": 8,
  "total_number_of_facilities_processing": 20
}

Batch Supply Shed Progress (Complete)

{
  "complete": true,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 3,
  "total_number_of_facilities_completed": 92,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0,
  "skip_reason": null
}

Batch Supply Shed Progress (Skipped)

{
  "complete": true,
  "total_number_of_facilities": 100,
  "total_number_of_ungeocoded_facilities": 5,
  "total_number_of_facilities_failed": 0,
  "total_number_of_facilities_completed": 0,
  "total_number_of_facilities_queued": 0,
  "total_number_of_facilities_preprocessing": 0,
  "total_number_of_facilities_processing": 0,
  "skip_reason": "low - not processed"
}

Error Responses

400 Bad Request

Returned when: - No tracking ID, tracking IDs, or filename is provided - Multiple parameters are provided (only one is allowed) - All tracking IDs in tracking_ids are not for batch supply shed runs - No tracking IDs are provided when using tracking_ids parameter - The filename is not a valid hash

Example error messages:

{
  "detail": "Exactly one of filename, tracking_id, or tracking_ids must be provided"
}

{
  "detail": "All tracking ids must be for batch supply shed runs"
}

{
  "detail": "No tracking ids provided"
}

{
  "detail": "The filename is not a valid hash."
}

400 Bad Request (Not Found)

Returned when the tracking ID does not exist or cannot be found.

{
  "detail": "Unable to find any batch runs for tracking_id: <tracking_id>"
}

This can also occur when: - The tracking ID is invalid - The batch run hasn't been created yet - The user doesn't have access to the batch run

403 Forbidden

Returned when the user doesn't have permission to access the batch operation.

{
  "detail": "Insufficient permissions"
}
  • /batch_process: Submit a batch processing operation
  • /batch_supply_shed: Submit a batch supply shed operation
  • /get_task_status: Get detailed task status information
  • /get_collections: List all collections and their status

Notes

  • The filename parameter is deprecated. Use tracking_id instead for better reliability.
  • The tracking_ids parameter only works for batch supply shed operations and will return a combined progress object.
  • Progress updates are not real-time; there may be a delay of a few seconds to minutes depending on the processing stage.
  • For batch supply shed operations, facilities are processed asynchronously, so progress may vary significantly between facilities.