Building Wireless Firmware Updates on ESP-IDF's OTA Foundation

Our Hoopi Pedal uses a Daisy Seed (STM32H7) for audio DSP and an ESP32 for WiFi-based control.

Overview

The ESP32 OTA system for Hoopi Pedal provides:

• Wireless Updates: Upload new firmware via HTTP API
• CRC32 Verification: Ensures firmware integrity before flashing
• Dual Partition Layout: Two app partitions for seamless updates
• Automatic Rollback: Returns to previous firmware if new version fails to boot
• Version Tracking: Major/minor version numbers for update management

Leveraging ESP-IDF's OTA Infrastructure

This implementation builds on top of ESP-IDF's robust OTA system rather than reinventing the wheel. Here's how the responsibilities are divided:

What ESP-IDF/Bootloader Provides

What We Build On Top

ESP-IDF OTA APIs Used

// Partition management
esp_ota_get_next_update_partition()   // Find the inactive OTA slot
esp_ota_get_running_partition()       // Get current boot partition

// Flash operations
esp_ota_begin()                       // Start OTA, erases target partition
esp_ota_write()                       // Write firmware chunks
esp_ota_end()                         // Finalize and validate
esp_ota_abort()                       // Cancel on error

// Boot management
esp_ota_set_boot_partition()          // Set next boot target
esp_ota_mark_app_valid_cancel_rollback()  // Confirm firmware is good
esp_ota_get_state_partition()         // Check if PENDING_VERIFY

This layered approach means we get battle-tested flash handling and rollback protection from ESP-IDF, while adding the HTTP transport and verification logic specific to our use case.

Architecture

Partition Layout

The ESP32-S3 uses an 8MB flash with dual OTA partitions:

┌─────────────────────────────────────────────────────────┐
│                    8MB Flash Layout                      │
├──────────┬──────────┬───────────────────────────────────┤
│  Offset  │   Size   │            Partition              │
├──────────┼──────────┼───────────────────────────────────┤
│  0x0000  │   16KB   │  Bootloader (second stage)        │
│  0x8000  │    4KB   │  Partition Table                  │
│  0x9000  │   24KB   │  NVS (WiFi credentials, etc.)     │
│  0xF000  │    8KB   │  OTA Data (boot selection)        │
│ 0x11000  │    4KB   │  PHY Init Data                    │
│ 0x20000  │    2MB   │  OTA_0 (App Partition)            │
│ 0x220000 │    2MB   │  OTA_1 (App Partition)            │
│ 0x420000 │   64KB   │  Coredump                         │
│ 0x430000 │  ~3.8MB  │  FAT Storage                      │
└──────────┴──────────┴───────────────────────────────────┘

OTA State Machine

Update Flow Sequence

Rollback Protection

Implementation Details

Firmware Version Definition

Firmware version is defined at compile time in esp32_ota.h:

// ESP32 Firmware Version - update these when releasing new firmware
#define ESP32_FW_VERSION_MAJOR  1
#define ESP32_FW_VERSION_MINOR  0

OTA State Tracking

The OTA module maintains state throughout the update process:

typedef enum {
    ESP32_OTA_STATE_IDLE,
    ESP32_OTA_STATE_RECEIVING,    // Receiving firmware via HTTP
    ESP32_OTA_STATE_VERIFYING,    // Verifying CRC
    ESP32_OTA_STATE_COMPLETE,     // Ready to reboot
    ESP32_OTA_STATE_ERROR
} esp32_ota_state_t;

// State variables
static volatile esp32_ota_state_t esp32_ota_state = ESP32_OTA_STATE_IDLE;
static volatile size_t esp32_ota_received_bytes = 0;
static volatile size_t esp32_ota_total_bytes = 0;
static volatile int esp32_ota_progress_percent = 0;
static volatile uint32_t esp32_ota_expected_crc = 0;
static volatile uint32_t esp32_ota_calculated_crc = 0xFFFFFFFF;
static volatile uint8_t esp32_ota_expected_major = 0;
static volatile uint8_t esp32_ota_expected_minor = 0;

CRC32 Calculation

Firmware integrity is verified using CRC32 (same polynomial as zlib):

static inline uint32_t esp32_crc32_update(uint32_t crc, const uint8_t *data, size_t len) {
    for (size_t i = 0; i < len; i++) {
        crc ^= data[i];
        for (int j = 0; j < 8; j++) {
            crc = (crc >> 1) ^ (0xEDB88320 & -(crc & 1));
        }
    }
    return crc;
}

static inline uint32_t esp32_crc32_finalize(uint32_t crc) {
    return ~crc;
}

Beginning an OTA Update

static esp_err_t esp32_ota_begin(size_t firmware_size, uint32_t expected_crc,
                                  uint8_t expected_major, uint8_t expected_minor) {
    // Validate state
    if (esp32_ota_state != ESP32_OTA_STATE_IDLE) {
        return ESP_ERR_INVALID_STATE;
    }

    // Get the next OTA partition
    esp32_ota_update_partition = esp_ota_get_next_update_partition(NULL);
    if (esp32_ota_update_partition == NULL) {
        return ESP_ERR_NOT_FOUND;
    }

    // Begin OTA - this erases the partition
    esp_err_t err = esp_ota_begin(esp32_ota_update_partition,
                                   firmware_size,
                                   &esp32_ota_handle);
    if (err != ESP_OK) {
        return err;
    }

    // Initialize state
    esp32_ota_total_bytes = firmware_size;
    esp32_ota_received_bytes = 0;
    esp32_ota_expected_crc = expected_crc;
    esp32_ota_calculated_crc = 0xFFFFFFFF;
    esp32_ota_expected_major = expected_major;
    esp32_ota_expected_minor = expected_minor;
    esp32_ota_state = ESP32_OTA_STATE_RECEIVING;

    return ESP_OK;
}

Writing Firmware Chunks

static esp_err_t esp32_ota_write(const void *data, size_t len) {
    if (esp32_ota_state != ESP32_OTA_STATE_RECEIVING) {
        return ESP_ERR_INVALID_STATE;
    }

    // Write to flash
    esp_err_t err = esp_ota_write(esp32_ota_handle, data, len);
    if (err != ESP_OK) {
        esp32_ota_state = ESP32_OTA_STATE_ERROR;
        return err;
    }

    // Update running CRC
    esp32_ota_calculated_crc = esp32_crc32_update(
        esp32_ota_calculated_crc,
        (const uint8_t*)data,
        len
    );

    // Update progress
    esp32_ota_received_bytes += len;
    esp32_ota_progress_percent = (esp32_ota_received_bytes * 100) / esp32_ota_total_bytes;

    return ESP_OK;
}

Finalizing the Update

static esp_err_t esp32_ota_end(void) {
    esp32_ota_state = ESP32_OTA_STATE_VERIFYING;

    // Finalize and verify CRC
    uint32_t final_crc = esp32_crc32_finalize(esp32_ota_calculated_crc);

    if (final_crc != esp32_ota_expected_crc) {
        esp_ota_abort(esp32_ota_handle);
        esp32_ota_state = ESP32_OTA_STATE_ERROR;
        return ESP_ERR_INVALID_CRC;
    }

    // End OTA write
    esp_err_t err = esp_ota_end(esp32_ota_handle);
    if (err != ESP_OK) {
        esp32_ota_state = ESP32_OTA_STATE_ERROR;
        return err;
    }

    // Set the new boot partition
    err = esp_ota_set_boot_partition(esp32_ota_update_partition);
    if (err != ESP_OK) {
        esp32_ota_state = ESP32_OTA_STATE_ERROR;
        return err;
    }

    esp32_ota_state = ESP32_OTA_STATE_COMPLETE;
    return ESP_OK;
}

Boot Validation

On startup, the firmware checks if it's running after an OTA update:

void app_main(void) {
    ESP_ERROR_CHECK(init_nvs());

    // Check OTA rollback status
    if (esp32_ota_is_pending_verify()) {
        ESP_LOGW(TAG, "OTA: Running new firmware, awaiting confirmation");
        ESP_LOGW(TAG, "OTA: If device crashes before confirmation, bootloader will rollback");
    } else {
        const esp_partition_t *running = esp_ota_get_running_partition();
        if (running) {
            ESP_LOGI(TAG, "Running from partition: %s", running->label);
        }
    }

    // ... rest of initialization
}

API Reference

POST /api/esp32-ota

Start an ESP32 firmware update.

Query Parameters:

Request Body: Raw binary firmware data

Headers:

Content-Type: application/octet-stream
Content-Length: <firmware_size>

Success Response:

{
  "status": "ok",
  "message": "OTA complete, rebooting in 3 seconds"
}

Error Response:

{
  "status": "error",
  "message": "CRC mismatch: expected 0xabc123, got 0xdef456"
}

GET /api/esp32-ota/status

Get the current OTA status and firmware version.

Response (idle):

{
  "state": "idle",
  "current_major": 1,
  "current_minor": 0,
  "running_partition": "ota_0",
  "next_partition": "ota_1",
  "firmware_pending_verify": false
}

Response (receiving):

{
  "state": "receiving",
  "current_major": 1,
  "current_minor": 0,
  "expected_major": 1,
  "expected_minor": 1,
  "received_bytes": 524288,
  "total_bytes": 1109520,
  "progress_percent": 47,
  "running_partition": "ota_0",
  "next_partition": "ota_1",
  "firmware_pending_verify": false
}

Response (after reboot, pending confirmation):

{
  "state": "idle",
  "current_major": 1,
  "current_minor": 1,
  "running_partition": "ota_1",
  "next_partition": "ota_0",
  "firmware_pending_verify": true
}

POST /api/esp32-ota/abort

Abort an in-progress OTA update.

Response:

{
  "status": "ok",
  "message": "ESP32 OTA aborted"
}

POST /api/esp32-ota/confirm

Confirm the new firmware is working correctly. This prevents automatic rollback.

Response:

{
  "status": "ok",
  "message": "Firmware marked as valid"
}

Client Implementation Example

Shell Script

#!/bin/bash
# esp32_ota_update.sh - Update ESP32 firmware

DEVICE_IP="${1:-192.168.4.1}"
FIRMWARE="${2:-build/hoopi.bin}"
MAJOR="${3:-1}"
MINOR="${4:-1}"

if [ ! -f "$FIRMWARE" ]; then
    echo "Error: Firmware file not found: $FIRMWARE"
    exit 1
fi

# Calculate CRC32
CRC=$(crc32 "$FIRMWARE" | awk '{print $1}')
SIZE=$(stat -f%z "$FIRMWARE" 2>/dev/null || stat -c%s "$FIRMWARE")

echo "Firmware: $FIRMWARE"
echo "Size: $SIZE bytes"
echo "CRC32: $CRC"
echo "Target version: $MAJOR.$MINOR"
echo ""

# Upload firmware
echo "Uploading firmware..."
curl -X POST \
    -H "Content-Type: application/octet-stream" \
    --data-binary "@$FIRMWARE" \
    "http://$DEVICE_IP/api/esp32-ota?major=$MAJOR&minor=$MINOR&crc=$CRC"

echo ""
echo "Device will reboot. Waiting 10 seconds..."
sleep 10

# Check new version
echo "Checking firmware version..."
curl -s "http://$DEVICE_IP/api/esp32-ota/status" | jq .

# Confirm if pending
PENDING=$(curl -s "http://$DEVICE_IP/api/esp32-ota/status" | jq -r '.firmware_pending_verify')
if [ "$PENDING" = "true" ]; then
    echo "Confirming firmware..."
    curl -X POST "http://$DEVICE_IP/api/esp32-ota/confirm"
fi

Swift (iOS)

class ESP32OTAManager {
    let baseURL: URL

    init(deviceIP: String) {
        self.baseURL = URL(string: "http://\(deviceIP)")!
    }

    func updateFirmware(
        data: Data,
        majorVersion: Int,
        minorVersion: Int,
        progressHandler: @escaping (Double) -> Void,
        completion: @escaping (Result<Void, Error>) -> Void
    ) {
        // Calculate CRC32
        let crc = data.crc32().hexString

        // Build URL with query parameters
        var components = URLComponents(url: baseURL.appendingPathComponent("/api/esp32-ota"),
                                        resolvingAgainstBaseURL: false)!
        components.queryItems = [
            URLQueryItem(name: "major", value: "\(majorVersion)"),
            URLQueryItem(name: "minor", value: "\(minorVersion)"),
            URLQueryItem(name: "crc", value: crc)
        ]

        var request = URLRequest(url: components.url!)
        request.httpMethod = "POST"
        request.setValue("application/octet-stream", forHTTPHeaderField: "Content-Type")
        request.httpBody = data

        let task = URLSession.shared.dataTask(with: request) { data, response, error in
            if let error = error {
                completion(.failure(error))
                return
            }
            completion(.success(()))
        }
        task.resume()

        // Poll for progress
        pollProgress(progressHandler: progressHandler)
    }

    func confirmFirmware(completion: @escaping (Result<Void, Error>) -> Void) {
        var request = URLRequest(url: baseURL.appendingPathComponent("/api/esp32-ota/confirm"))
        request.httpMethod = "POST"

        URLSession.shared.dataTask(with: request) { _, _, error in
            if let error = error {
                completion(.failure(error))
            } else {
                completion(.success(()))
            }
        }.resume()
    }

    private func pollProgress(progressHandler: @escaping (Double) -> Void) {
        Timer.scheduledTimer(withTimeInterval: 0.5, repeats: true) { timer in
            self.getStatus { result in
                switch result {
                case .success(let status):
                    if status.state == "receiving" {
                        progressHandler(Double(status.progressPercent) / 100.0)
                    } else if status.state == "idle" || status.state == "complete" {
                        timer.invalidate()
                    }
                case .failure:
                    timer.invalidate()
                }
            }
        }
    }
}

Configuration

sdkconfig Settings

Enable OTA and rollback support:

CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"
CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y

Partition Table (partitions.csv)

# Name,   Type, SubType,  Offset,   Size,     Flags
nvs,      data, nvs,      0x9000,   0x6000,
otadata,  data, ota,      0xf000,   0x2000,
phy_init, data, phy,      0x11000,  0x1000,
ota_0,    app,  ota_0,    0x20000,  0x200000,
ota_1,    app,  ota_1,    0x220000, 0x200000,
coredump, data, coredump, 0x420000, 0x10000,
storage,  data, fat,      0x430000, 0x3D0000,

First-Time Setup

When first flashing a device with OTA support (or changing partition layout):

# Erase flash completely (required for partition table changes)
idf.py erase-flash

# Flash bootloader, partition table, and app
idf.py flash

After this initial flash, all subsequent updates can be done wirelessly via the OTA API.

Troubleshooting

Common Issues

"No OTA partition available"

• Ensure partition table includes ota_0 and ota_1 partitions
• Verify otadata partition exists
• Re-flash with idf.py erase-flash && idf.py flash

"CRC mismatch"

• Verify the CRC32 calculation matches (use crc32 command on the binary)
• Check for network transmission errors
• Ensure complete file is being sent

"Firmware too large"

• Maximum firmware size is 2MB per partition
• Check build output for actual binary size
• Consider optimizing with -Os compiler flag

Device rolls back after OTA

• Ensure app calls /api/esp32-ota/confirm after successful boot
• Check for crashes in the new firmware during initialization
• Increase the confirmation timeout in the app

Debug Logging

Monitor OTA progress via serial:

idf.py monitor

Example output:

I (1234) ESP32_OTA: Writing to partition 'ota_1' at offset 0x220000
I (1234) ESP32_OTA: OTA started: size=1109520, expected_crc=0x12345678, target_version=1.1
I (5678) ESP32_OTA: CRC verified: 0x12345678
I (5678) ESP32_OTA: OTA complete, new boot partition: ota_1
I (5678) ESP32_OTA: Rebooting in 3 seconds...

Security Considerations

1. Network Security: OTA endpoint is accessible over WiFi. Consider:

   • Using WPA2/WPA3 for the AP
   • Implementing authentication tokens
   • Rate limiting update requests

2. Firmware Validation: CRC32 provides integrity checking but not authenticity. For production:

   • Consider signed firmware images
   • Implement secure boot

3. Rollback Protection: The automatic rollback feature prevents bricked devices but could be exploited to downgrade firmware. Consider:

   • Anti-rollback counters in eFuse
   • Version validation before accepting updates

Interested in Hoopi? We're crowdfunding on https://www.crowdsupply.com/scope-creep-labs/hoopi-pedal — sign up for updates or back the project.

The Daisy's firmware can also be updated via the ESP32. Details: https://scopecreeplabs.com/blog/stm32-esp32-ota/