Skip to content

NOTES

Overview

The offline sensing module provides high-frequency batch sensor data acquisition and storage functionality. It is designed for scenarios requiring precise, time-stamped data collection with configurable sampling parameters and multiple storage options.

Features

  • ✅ High-frequency sampling (default: 100 Hz, range: 1 - 4000 Hz, limit based on ADXL355 sensor's maximum ODR)
  • ✅ Configurable sampling duration (default: 10 seconds, range: 0.1 - 3600 seconds, limited by PSRAM memory)
  • ✅ Memory buffer storage (enabled/disabled)
  • ✅ SD card storage (enabled/disabled)
  • ✅ MQTT report after sampling completion (enabled/disabled)
  • ✅ High-precision timer-based sampling (ESP timer)
  • ✅ Automatic file naming with timestamps and parameters
  • ✅ Thread-safe operation
  • ✅ Support for delayed and scheduled start

Architecture

Timer-Based Sampling

The module uses ESP timer for precise periodic sampling:

  1. Timer Creation: Creates a periodic timer with period = 1 / sampling_frequency
  2. Timer Callback: Executes at each timer tick to read and store sensor data
  3. Data Storage: Stores data in memory buffer and/or SD card
  4. Completion Report: Generates report and optionally sends via MQTT

Data Flow

ESP Timer → Timer Callback → Sensor Read → Memory Buffer → SD Card Storage
                                                                    ↓
                                                            MQTT Report

Implementation Principles

Architecture Overview

The offline sensing module implements a timer-driven, batch collection architecture using ESP-IDF's high-precision timer system. The design prioritizes high-frequency data collection with reliable storage, using a two-phase approach: real-time collection and post-processing storage.

Core Components

  1. ESP Timer (esp_timer)

    • High-precision hardware timer with microsecond accuracy
    • Periodic timer mode: triggers callback at fixed intervals
    • Timer period: period_us = 1,000,000 / sampling_frequency_hz
    • Timer callback executes in timer context (high priority, interrupt-like)

  2. Timer Callback Function

    • Executes synchronously at each timer tick
    • Performs sensor read operations (non-blocking SPI)
    • Stores data to memory buffer with mutex protection
    • Records timestamp for each sample
    • Minimal processing time to avoid missing timer ticks

  3. Memory Buffer

    • Pre-allocated array based on sampling parameters
    • Allocated from PSRAM (external RAM) to support large buffer sizes for high-frequency/long-duration sampling
    • Thread-safe access using FreeRTOS mutex (SemaphoreHandle_t)
    • Stores structured data: {timestamp_us, x, y, z, temp}
    • Index-based sequential writing during sampling

  4. SD Card Storage

    • Post-processing: writes all collected data after sampling completes
    • Blocking file I/O operations (runs in calling task context)
    • CSV format with header row
    • Automatic filename generation with timestamp and parameters

  5. Blocking Execution Model

    • offline_sensing_start() blocks until sampling duration completes
    • Auto-stop mechanism: Timer callback automatically stops when expected number of samples is reached
    • Expected samples = (frequency × duration) + 1 (includes t=0 sample)
    • Uses vTaskDelay() to wait for sampling duration with margin
    • All post-processing (SD write, MQTT report) happens after sampling

Programming Tools & APIs

  • ESP-IDF Timer API: esp_timer_create(), esp_timer_start_periodic(), esp_timer_get_time()
  • ESP-IDF Heap Management: heap_caps_malloc(), heap_caps_free() with MALLOC_CAP_SPIRAM for PSRAM allocation
  • FreeRTOS Synchronization: xSemaphoreCreateMutex() for thread-safe buffer access
  • Standard C File I/O: fopen(), fprintf(), fclose() for SD card operations
  • ESP-IDF MQTT Client: esp_mqtt_client_publish() for report transmission
  • FreeRTOS Task Management: vTaskDelay() for blocking wait

Execution Flow

  1. Initialization: Allocate memory buffer based on frequency × duration
  2. Start:
    • Record start timestamp
    • Create and start periodic timer
    • Execute immediate first sample
  3. Sampling Phase (blocking):
    • Timer callback fires at fixed intervals
    • Read sensor via SPI and store to memory buffer (mutex-protected)
    • Auto-stop when expected samples reached: Callback sets s_is_running = false when s_total_samples >= s_expected_samples
    • Wait for duration using vTaskDelay() with margin to ensure all samples collected
  4. Post-Processing Phase (after sampling):
    • Stop and delete timer
    • Write memory buffer to SD card (if enabled)
    • Generate report with statistics
    • Send MQTT report (if enabled)
    • Return from blocking function

Key Design Decisions

  • Two-Phase Design: Separate real-time collection (timer callback) from storage (post-processing)
  • PSRAM Allocation: Uses external PSRAM for memory buffer to support large buffer sizes (typically 8MB available)
  • Memory Buffer First: Store all data in RAM first, then write to SD card in one batch
  • Auto-Stop Mechanism: Timer callback automatically stops sampling when expected sample count is reached
  • Blocking Model: Simplifies state management, ensures data integrity
  • Mutex Protection: Prevents race conditions between timer callback and main task
  • Immediate First Sample: Ensures exact sample count (N samples for N seconds at N Hz, including t=0 sample)

Configuration

Default Configuration

Default values are defined in tiny_measurement_config.h:

#define TINY_MEASUREMENT_OFFLINE_SENSING_DEFAULT_FREQ_HZ 100.0f
#define TINY_MEASUREMENT_OFFLINE_SENSING_DEFAULT_DURATION_SEC 10.0f
#define TINY_MEASUREMENT_OFFLINE_SENSING_DEFAULT_ENABLE_MEMORY true
#define TINY_MEASUREMENT_OFFLINE_SENSING_DEFAULT_ENABLE_SD true
#define TINY_MEASUREMENT_OFFLINE_SENSING_DEFAULT_ENABLE_MQTT_REPORT true

Configuration Structure

typedef struct
{
    float sampling_frequency_hz;      // Sampling frequency in Hz
    float sampling_duration_sec;       // Sampling duration in seconds
    bool enable_memory;                // Enable memory buffer storage
    bool enable_sd;                    // Enable SD card storage
    bool enable_mqtt_report;          // Enable MQTT report after sampling
    const char *sd_file_path;          // SD card file path (NULL for auto-generated)
    const char *mqtt_report_topic;     // MQTT topic for report (NULL for default)
} offline_sensing_config_t;

Data Storage

Memory Buffer

  • Stores samples in PSRAM (external RAM) during sampling
  • Automatically allocated from PSRAM using heap_caps_malloc() with MALLOC_CAP_SPIRAM
  • Thread-safe access using mutex (xSemaphoreTake/xSemaphoreGive)
  • Can be retrieved after sampling completion via offline_sensing_get_memory_data()
  • Can be cleared via offline_sensing_clear_memory()
  • Buffer size = (frequency × duration) + 100 samples (with margin)

SD Card Storage

  • Saves data as CSV file
  • Automatic filename generation with timestamp and parameters
  • Format: YYYYMMDDHHMMSS_FXXXX_DXXXX.csv or BootXXXXX_FXXXX_DXXXX.csv
  • CSV format: timestamp_us,x,y,z,temp

File Naming

The module automatically generates filenames based on sampling start time, frequency, and duration. The format depends on whether system time is available:

With System Time: 

20250116120000_F0100_D0010.csv

  • 20250116120000: Date and time (YYYYMMDDHHMMSS format)

  • F0100: Frequency (4 digits, zero-padded, e.g., 100 Hz)

  • D0010: Duration (4 digits, zero-padded, e.g., 10 seconds)

Without System Time (Boot Time): 

Boot12345_F0100_D0010.csv
- Boot12345: Boot time in seconds (last 5 digits of seconds since boot)

  • F0100: Frequency (4 digits, zero-padded)

  • D0010: Duration (4 digits, zero-padded)

Note: The system automatically detects if system time is unset (1970-01-01) and falls back to boot time format.

Data Format

CSV Format

timestamp_us,x,y,z,temp
1234567890,0.012345,-0.045678,0.987654,25.50
1234567891,0.012346,-0.045679,0.987655,25.51
...
  • timestamp_us: Timestamp in microseconds since boot
  • x, y, z: Acceleration values in g (6 decimal places)
  • temp: Temperature in °C (2 decimal places)

MQTT Report Format

{
  "samples": 1000,
  "freq_hz": 100.00,
  "duration_sec": 10.00,
  "memory_ok": true,
  "sd_ok": true,
  "sd_path": "/sdcard/20250116120000_F0100_D0010.csv",
  "start_us": 1234567890,
  "end_us": 1334567890
}

Usage Workflow

  1. Initialize Sensor: Initialize ADXL355 sensor
  2. Set Sensor Handle: Call offline_sensing_set_sensor_handle()
  3. Initialize Module: Call offline_sensing_init() with configuration (allocates PSRAM buffer if memory enabled)
  4. Start Sensing: Call offline_sensing_start() (blocks until complete, auto-stops when expected samples reached)
  5. Get Report: Call offline_sensing_get_report() for results
  6. Get Data (optional): Call offline_sensing_get_memory_data() to retrieve samples
  7. Clear Memory (optional): Call offline_sensing_clear_memory() to reset buffer index
  8. Check Status (optional): Call offline_sensing_is_running() to check if sampling is active
  9. Deinitialize: Call offline_sensing_deinit() to clean up (frees PSRAM buffer)

Performance Considerations

Sampling Frequency Limits

  • Minimum: 1 Hz
  • Maximum: 4000 Hz - This limit is based on the ADXL355 sensor's maximum output data rate (ODR)
  • Recommended: 10 - 1000 Hz for most applications

Sensor-Dependent Limit

The 4000 Hz maximum sampling frequency is determined by the ADXL355 sensor's hardware capabilities, which supports a maximum output data rate of 4000 Hz. If you use a different sensor module with higher sampling capabilities, the offline sensing module architecture theoretically supports higher sampling frequencies. The ESP timer system and memory buffer design can handle frequencies beyond 4000 Hz, but the actual limit will depend on the sensor's maximum ODR (Output Data Rate).

Memory Requirements

Memory buffer size = (sampling_frequency × sampling_duration + 100) × sizeof(offline_sensing_sample_t)

  • Buffer is allocated from PSRAM (external RAM), not internal RAM
  • PSRAM typically provides 8MB of external memory
  • Example: 100 Hz × 10 sec + 100 margin = 1100 samples × 32 bytes = 35.2 KB
  • High-frequency example: 4000 Hz × 10 sec + 100 = 40100 samples × 32 bytes = 1.28 MB

PSRAM Memory Limitation

Single sampling duration is limited by available PSRAM memory size.

The maximum duration depends on the sampling frequency due to PSRAM memory capacity:

  • At 4000 Hz: Theoretical maximum duration is approximately 87 seconds (tested up to 80 seconds in practice)
  • At 100 Hz: Theoretical maximum duration is approximately 3480 seconds (about 58 minutes)

This limitation is due to the PSRAM memory capacity required to store all samples in the memory buffer before writing to SD card.

Calculation examples: - Maximum samples ≈ 8MB / 32 bytes per sample ≈ 262,144 samples - At 4000 Hz: Maximum duration ≈ 262,144 / 4000 ≈ 65.5 seconds (without margin) → ~87 seconds with margin - At 100 Hz: Maximum duration ≈ 262,144 / 100 ≈ 2,621 seconds (without margin) → ~3480 seconds (~58 minutes) with margin

For longer duration sampling at high frequencies, consider: - Using lower sampling frequencies (e.g., 100 Hz for ~58 minutes vs 4000 Hz for ~87 seconds) - Reducing the margin (not recommended) - Implementing streaming to SD card during sampling (future enhancement)

Resource Usage

  • CPU: Timer callback executes at sampling frequency
  • Memory: Buffer allocated from PSRAM based on sampling parameters (supports large buffers for high-frequency/long-duration sampling)
  • Storage: SD card write speed may limit maximum frequency for very high rates
  • Sensor: The primary limiting factor is the sensor's maximum ODR (Output Data Rate). For ADXL355, this is 4000 Hz. The module architecture itself can theoretically support higher frequencies if a faster sensor is used.
  • Blocking: offline_sensing_start() blocks until sampling completes

Error Handling

Common Errors

  • ESP_ERR_INVALID_ARG: Invalid frequency or duration (out of range)
  • ESP_ERR_INVALID_STATE: Already running or not initialized
  • ESP_ERR_NO_MEM: Memory allocation failed (PSRAM insufficient)
  • ESP_FAIL: SD card write failed

Error Recovery

  • Check sensor handle is set before starting
  • Validate frequency and duration ranges before initialization
  • Ensure SD card is mounted and writable
  • Check available PSRAM before starting high-frequency or long-duration sampling
  • At 4000 Hz, maximum duration is ~87 seconds (tested up to 80 seconds)
  • At 100 Hz, maximum duration is ~3480 seconds (~58 minutes)
  • Memory requirement: (frequency × duration + 100) × 32 bytes
  • Ensure PSRAM is enabled in project configuration

Thread Safety

  • Timer callback is executed in timer context (high priority)
  • Memory buffer access is protected by mutex
  • State checks prevent concurrent operations
  • offline_sensing_start() is blocking (runs in calling task context)

Integration Notes

SD Card Integration

  • Requires SD card to be mounted
  • Uses MOUNT_POINT constant for file paths
  • File operations use standard C file I/O
  • Errors are logged but don't stop sampling

MQTT Integration

  • Requires MQTT client to be connected
  • Checks s_is_mqtt_connected before publishing
  • Report is sent after sampling completion
  • Uses default topic /offline_sensing/report if mqtt_report_topic is NULL

Sensor Integration

  • Requires ADXL355 sensor handle
  • Must be initialized before setting handle
  • Sensor read operations are non-blocking
  • Sensor ODR Limit: The 4000 Hz maximum is constrained by ADXL355's maximum ODR. The module architecture is sensor-agnostic and can theoretically support higher frequencies with sensors that have higher ODR capabilities

Delayed and Scheduled Start

The module supports delayed and scheduled start through the command handler:

  • Delayed Start: DL=<seconds> - Start after specified delay
  • Scheduled Start: TIME=<YYMMDDHHMMSS> - Start at specified time

These features are implemented in the command handler wrapper task, not in the core offline sensing module.