Skip to content

Batch Process

Batch Process is a meta-endpoint that sequentially runs all of the following endpoints:

  • /stage_plots

  • /upload_plots

  • /batch_deforestation_check OR /batch_biomass_emissions OR /batch_biodiversity OR /batch_water (depending on the stats parameter)

The /batch_process endpoint is a convenient way to run all of these endpoints in one go, as most of the time, programmatic users will want an easy way to go from staging to processing to results.

It is an asynchronous REST service. This means that data will not be returned immediately in a request response. Results can be retrieved via the Fetch API endpoints once processing completes.

Moreover, /batch_process is not exposed in Epoch's dashboard, only its separate steps are. This meta-endpoint is only available via the API.

Parameters and Headers

Basic Parameters

An example of a basic payload for the /batch_process endpoint is the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
request_data = {
    "collection_name": '<collection_name>',
    'description': '<collection_description>',
    "stats": 'luc_emissions,biodiversity', # (1)!
    "crop_type": "Cocoa",
    "non_biomass": True # (2)!
}   

files = {
    'file': ('<file_name>', open('<local_path_to_file>', 'rb'),
             'application/geo+json') # (4)!
}
  1. Can also be 'eudr_deforestation', 'water', or any combination separated by commas.
  2. A boolean to export non-biomass emissions. This is optional.
  3. Can be application/geo+json, text/csv, application/shapefile, application/geoparquet

Complete Parameter List

Parameter Type Required Default Description
collection_name string Yes - Collection name to assign to the staged file
input_payload object/string No {} Detailed agricultural and supply chain data for emissions calculations
crop_type string/int No - The crop type to assign to the staged file
processing_type string No - The processing type that the commodity undergoes
system_type string No - The system type that the commodity is grown in
description string No - A description of the staged file
stats string No "eudr_deforestation,luc_emissions,biodiversity,water" The zonal statistics to compute, separated by commas
non_biomass boolean No false A boolean to export non-biomass emissions
validate_locations boolean No true A boolean to check the validity of the input geometries

Country Auto-Detection and Override

The endpoint automatically detects the country from file geometries when validate_locations=True:

  • If the detected country differs from the specified country parameter and represents ≥50% of locations, the country is automatically overridden
  • The original and detected countries are returned in the response's country_override_info field
  • This ensures processing uses the correct country even if the user-provided country parameter is incorrect

Example: 

# User specifies country="Ghana" but file contains locations in Nigeria
# If ≥50% of locations are in Nigeria, country is automatically overridden to "Nigeria"
response = {
    "status": "PROCESSING",
    "collection_id": "...",
    "country_override_info": {
        "was_overridden": True,
        "original_country": "Ghana",
        "detected_country": "Nigeria",
        "message": "Country automatically overridden from 'Ghana' to 'Nigeria' based on file locations (75/100 locations in Nigeria)"
    }
}

Input Payload Parameter

The input_payload parameter is a complex nested structure that contains detailed agricultural and supply chain data for accurate emissions calculations. It consists of three main payload types:

  • CFT Payload: Cool Farm Tool data for farm-level emissions
  • LCA Payload: Life Cycle Assessment data for supply chain emissions
  • EUDR Payload: EU Deforestation Regulation compliance data

For detailed documentation on the input_payload structure, see the Input Payload Documentation.

Example with Input Payload

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import json

# Define the input payload with detailed agricultural data
input_payload = {
    "cft_payload": {
        "crop": {
            "type": "cocoa",
            "yield": {"value": 500, "unit": "kg/ha"}
        },
        "farm": {
            "size": {"value": 10, "unit": "ha"},
            "location": {"country": "Ghana"}
        },
        "fertilisers": [
            {
                "type": "NPK",
                "application_rate": {"value": 200, "unit": "kg/ha"}
            }
        ]
    },
    "lca_payload": {
        "country": "Ghana",
        "year": 2024,
        "commodity_type": "cocoa"
    },
    "eudr_payload": {
        "product_type": "cocoa",
        "production_country": "Ghana",
        "production_year": 2024
    }
}

request_data = {
    "collection_name": "cocoa_plots_2024",
    "description": "Cocoa plots with detailed agricultural practices",
    "input_payload": json.dumps(input_payload),  # JSON stringify for form data
    "crop_type": "cocoa",
    "stats": "luc_emissions,eudr_deforestation",
    "non_biomass": True
}

With the headers being:

1
headers = {'Authorization': "Bearer <authorization_token>"}

The Authorization header must be replaced by your user token. Check this page for more information on how to authenticate.

Python

In python, you can submit a request in the following way:

batch_process.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import requests
import json

response = requests.post("https://epoch-sco2-api.com/batch_process", 
                         files=files, 
                         data=request_data, 
                         headers=headers)

json_result = json.loads(response.content)
print(json_result)

Javascript

batch_process.js
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
const axios = require('axios');
const fs = require('fs');
const FormData = require('form-data');

// Create a form-data object
const form = new FormData();

const request_data = {
    "collection_name": '<collection_name>',
    "description": '<collection_description>',
    "stats": 'luc_emissions,biodiversity',
    "date": '2024-08-07', 
    "export_raster": false,
};

// Append form fields
form.append('request_data', JSON.stringify(request_data));

// Append the file
form.append('file', fs.createReadStream('<local_path_to_file>'), {
    filename: '<file_name>',
    contentType: '<file_format>' // e.g., 'application/geo+json', 'text/csv'
});

// Define headers, including the form-data headers
const headers = {
    'Authorization': "Bearer <authorization_token>",
    ...form.getHeaders()
};

// Send the request using axios
axios.post('https://epoch-sco2-api.com/batch_process', form, { headers })
    .then(response => {
        console.log(response.data);
    })
    .catch(error => {
        console.error(error);
    });

Curl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
curl -X POST https://epoch-sco2-api.com/batch_process \
-H "Authorization: Bearer <authorization_token>" \
-F "request_data={
    \"collection_name\": \"'<collection_name>\",
    \"description\": \"'<collection_description>\",
    \"stats\": \"luc_emissions,biodiversity\",
    \"date\": \"2024-08-07\",
    \"export_raster\": false
}" \
-F "file=@<local_path_to_file>;type=<file_format>;filename=<file_name>"

Deforestation Categorization Metrics

The deforestation detection and categorization follows the European Union Deforestation Regulation (EUDR) definition of forest and uses standardized thresholds to classify deforestation events as critical or non-critical.

For detailed information on EUDR forest definitions, deforestation alert thresholds, and confidence levels, see the Deforestation Check documentation.

Response

The response indicates that processing has started in the background, or that processing was skipped:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Normal processing response
response = {
    "status": "PROCESSING",
    "collection_id": "0x44c87929d1ed966d86a844a98744bd7e3d47aa3da48e262150874c2bc8313feb",
    "message": "Batch Process processing started in background"
}

# Processing response with country override
response = {
    "status": "PROCESSING",
    "collection_id": "0x44c87929d1ed966d86a844a98744bd7e3d47aa3da48e262150874c2bc8313feb",
    "message": "Batch Process processing started in background",
    "country_override_info": {
        "was_overridden": True,
        "original_country": "Ghana",
        "detected_country": "Nigeria",
        "message": "Country automatically overridden from 'Ghana' to 'Nigeria' based on file locations (75/100 locations in Nigeria)"
    }
}

# Skipped response (when skip_on_low_confidence=True and confidence is "low")
response = {
    "status": "SKIPPED",
    "collection_id": "0x44c87929d1ed966d86a844a98744bd7e3d47aa3da48e262150874c2bc8313feb",
    "message": "Processing skipped due to low validation confidence"
}

Important: - The collection_id (also known as tracking_id) is used to monitor progress via the /batch_progress endpoint - Processing is asynchronous - results will be available once processing completes - Use the collection_id to track progress and retrieve results - When skip_on_low_confidence=True and validation confidence is "low", processing is skipped immediately and a stat_supply_shed row is created with the skip reason - Validation confidence can be checked in stat_supply_shed.locations_confidence after processing - The country_override_info field is included when country auto-detection overrides the specified country parameter

Check Status

To check the status of your batch process, use the /batch_progress endpoint. It provides high-level progress information with stage-based tracking.

check_batch_progress.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Get the tracking_id from the batch_process response
tracking_id = response.get("collection_id")  # or from response metadata

# Check progress
progress_response = requests.get(
    "https://epoch-sco2-api.com/batch_progress",
    params={"tracking_id": tracking_id},
    headers=headers
)

progress = progress_response.json()
print(f"Overall Progress: {progress['overall_progress']}%")
print(f"Stage: {progress['stage']}/{progress['n_stages']}")
print(f"Complete: {progress['complete']}")

For detailed documentation on the /batch_progress endpoint, see the Batch Progress documentation.

List Uploaded Collections

You can list all collections uploaded to the system using the /get_collections endpoint. This is useful for discovering collections and retrieving their tracking IDs for use with /batch_progress.

get_collections.py
1
2
3
4
response = requests.get("https://epoch-sco2-api.com/get_collections", headers=headers)

json_result = json.loads(response.content)
print(json_result)

An example response looks like this:

{
  'my_first_collection': {
    'collection_id': '0xda47b7f12b109822e2604ebfc1669c514f43d4ae53b85c73e2ff6510127cd2d7', 
    'description': 'my_first_collection_description', 
    'ingestion_time': '2024-09-19T11:03:38.264237+00:00', 
    'plot_number': 97, 
    'batch_jobs': [{
      'outcome': 'CHECK_TABLES', 
      'complete_task_ids': ['G67KNVLR4EQYRZXTOXXOHKPX', '2MO7KULJM6HRWN773ZBXPYMM', 'RAYWZNBXWH5QJRFGJCR5PGDS', 'TANRMNOTUPFWSRVRLQFZZWW5', '6H3SZL2P5K4BMUA6F5HQMTMK', 'RVKROC6E4RJMIWPZN6JEXCMB', 'WROZEPXIQY3NYPXBDQYKGCV6', '6DSKNLUXWG74NR7C7LYLSTZ2', 'CQZTFDK6A7GKIWMFHMNHIRFZ'], 
      'failed_task_ids': [], 
      'start_date': '2017-01-01', 
      'end_date': '2024-09-19', 
      'stats': ['luc_emissions', 'biodiversity', 'eudr_deforestation'], 
      'status': 'in_progress'
    }, {
      'outcome': 'INGEST_IMAGES', 
      'complete_task_ids': ['3DPJ2QRACXWI4KEX7THG5SXH', '3NH6TCJL2QDDPHYKFPEG4GTU', 'QSNNEACBLNW4DNN2ULD3BB77', 'COO7JLFWKRU3LBG6OAVOBOOG', 'CW7LVR5F24F5ZJBROYCM2W22', 'LEM3KLQTUHHOMOG6VIE4QEDB', '2W6VL75XPMF4KB6EWSXIW56T', 'M7VWTIBA3Y3YVEKLMTLFG5ZP', 'J4EK7UPQG75X2BJRTB5TZUDI', 'XRIOOMMW7LJGU3UOTRXG6YYA', 'KEYQVDJVH4IOF2YE3GBNU2V2', 'WHRN5NKJPH3ECIAALDZQKVTD', 'DAN7DRLIHKEA2TEL46HHNCR2'], 
      'failed_task_ids': [], 
      'start_date': '2017-01-01', 
      'end_date': '2024-09-19', 
      'stats': ['luc_emissions', 'biodiversity', 'eudr_deforestation'], 
      'status': 'in_progress'
    }]
  }
}

Note: The collection_id returned by /get_collections can be used as the tracking_id parameter in /batch_progress for more detailed progress tracking.