Periodicity detection: method notes & corrections #1

New issue

Open

opened 2026-06-22 23:59:55 +02:00 by lz · 0 comments

lz commented

2026-06-22 23:59:55 +02:00

Owner

Context: time series in R, ~100–1000 points, has trend / non-stationary, goal is to discover an unknown period. Currently using forecast::findfrequency() + Fisher g test. Daniele suggested computing the FFT of the ACF instead.

TL;DR

The trend is the main problem, not the method choice. Fix it first.
FFT-of-ACF is redundant and inferior to what's already being done. Skip it.
Keep the Fisher g test, but as a confirmation step on prewhitened residuals — not as the discovery tool.

Why FFT-of-ACF is redundant and inferior

Redundant: by the Wiener–Khinchin theorem, the power spectral density is the Fourier transform of the autocovariance. So "FFT of the ACF" is the periodogram — the same quantity the Fisher g test already operates on. It's computing the periodogram a second way, not a different method.
Inferior: taking the FFT of the raw, un-tapered sample ACF gives the periodogram with extra variance/bias from the noisy ACF tail. Standard advice (Percival & Walden, Spectral Analysis for Physical Applications) is to taper and go directly to the periodogram, or use a smoothed/multitaper estimator — not to round-trip through the raw ACF. Without smoothing the ACF first (e.g. Blackman–Tukey), it's a step backward.
It also inherits the trend problem identically, so it does nothing for the actual blocker.

The real blocker: trend breaks both the g test and peak-picking

A trend dumps large power into the lowest frequencies. The periodogram's dominant ordinate sits near frequency ≈ 0, the Fisher g statistic (max ordinate / sum of ordinates) gets inflated, and you either "detect" a spurious period ≈ series length or the real seasonal peak gets swamped.
The Fisher g test's null is i.i.d. Gaussian noise. A trend violates this directly → p-value is not trustworthy (anticonservative).

Recommended workflow (discovery, trended, 100–1000 pts)

Detrend / prewhiten first.
- diff() — kills linear/stochastic trend but is a high-pass filter, so peak amplitudes shift (fine for locating which frequency, worse for amplitude).
- loess/spline fit + subtract — cleaner, keeps the seasonal peak interpretable (this is what findfrequency does internally).
Use a smoothed / AR spectrum for discovery, not the raw periodogram.
- The raw periodogram is an inconsistent estimator: variance does NOT shrink as n grows; each ordinate has ~100% relative error. With only hundreds of points you need variance reduction.
- spec.pgram(x, spans=...) — Daniell-smoothed periodogram (trades frequency resolution for stability).
- spec.ar(x) — AR-model spectrum, sharp peaks, good at locating a cycle, but can invent peaks if AR order (AIC-chosen) is off. This is findfrequency's engine.
Confirm the candidate with the Fisher g test on the residuals (e.g. GeneCycle / ptest). Note: g test assumes a single embedded sinusoid; with multiple harmonics it loses power.

Most robust single pipeline:
loess-detrend → spec.pgram (moderate smoothing) to find candidate freq → Fisher g test on residuals to confirm.

Gotcha: interpreting `findfrequency()`

Returning 1 means "no peak passed my hardcoded threshold" — NOT evidence of no periodicity. Don't read 1 as a negative result.

Confidence

Wiener–Khinchin equivalence, periodogram inconsistency, g-test null assumption: Certain.
Raw-ACF-FFT being the biased route, recommended pipeline ranking: Likely.

Open question

Is there a suspected periodicity source (sampling rate, diurnal/weekly structure)? Knowing the candidate frequency region would set the smoothing bandwidth.

Context: time series in R, ~100–1000 points, **has trend / non-stationary**, goal is to **discover an unknown period**. Currently using `forecast::findfrequency()` + Fisher g test. Daniele suggested computing the **FFT of the ACF** instead. ### TL;DR - The trend is the main problem, not the method choice. Fix it first. - FFT-of-ACF is **redundant and inferior** to what's already being done. Skip it. - Keep the Fisher g test, but as a **confirmation** step on prewhitened residuals — not as the discovery tool. ### Why FFT-of-ACF is redundant and inferior - **Redundant:** by the Wiener–Khinchin theorem, the power spectral density is the Fourier transform of the autocovariance. So "FFT of the ACF" *is* the periodogram — the same quantity the Fisher g test already operates on. It's computing the periodogram a second way, not a different method. - **Inferior:** taking the FFT of the raw, un-tapered sample ACF gives the periodogram with extra variance/bias from the noisy ACF tail. Standard advice (Percival & Walden, *Spectral Analysis for Physical Applications*) is to taper and go directly to the periodogram, or use a smoothed/multitaper estimator — not to round-trip through the raw ACF. Without smoothing the ACF first (e.g. Blackman–Tukey), it's a step backward. - It also inherits the trend problem identically, so it does nothing for the actual blocker. ### The real blocker: trend breaks both the g test and peak-picking - A trend dumps large power into the lowest frequencies. The periodogram's dominant ordinate sits near frequency ≈ 0, the Fisher g statistic (max ordinate / sum of ordinates) gets inflated, and you either "detect" a spurious period ≈ series length or the real seasonal peak gets swamped. - The Fisher g test's null is **i.i.d. Gaussian noise**. A trend violates this directly → p-value is not trustworthy (anticonservative). ### Recommended workflow (discovery, trended, 100–1000 pts) 1. **Detrend / prewhiten first.** - `diff()` — kills linear/stochastic trend but is a high-pass filter, so peak *amplitudes* shift (fine for locating *which* frequency, worse for amplitude). - loess/spline fit + subtract — cleaner, keeps the seasonal peak interpretable (this is what `findfrequency` does internally). 2. **Use a smoothed / AR spectrum for discovery, not the raw periodogram.** - The raw periodogram is an *inconsistent* estimator: variance does NOT shrink as n grows; each ordinate has ~100% relative error. With only hundreds of points you need variance reduction. - `spec.pgram(x, spans=...)` — Daniell-smoothed periodogram (trades frequency resolution for stability). - `spec.ar(x)` — AR-model spectrum, sharp peaks, good at *locating* a cycle, but can invent peaks if AR order (AIC-chosen) is off. This is `findfrequency`'s engine. 3. **Confirm the candidate with the Fisher g test on the residuals** (e.g. `GeneCycle` / `ptest`). Note: g test assumes a *single* embedded sinusoid; with multiple harmonics it loses power. Most robust single pipeline: **loess-detrend → `spec.pgram` (moderate smoothing) to find candidate freq → Fisher g test on residuals to confirm.** ### Gotcha: interpreting `findfrequency()` - Returning `1` means "no peak passed my hardcoded threshold" — NOT evidence of no periodicity. Don't read `1` as a negative result. ### Confidence - Wiener–Khinchin equivalence, periodogram inconsistency, g-test null assumption: **Certain**. - Raw-ACF-FFT being the biased route, recommended pipeline ranking: **Likely**. ### Open question - Is there a suspected periodicity source (sampling rate, diurnal/weekly structure)? Knowing the candidate frequency region would set the smoothing bandwidth.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

peg/journal#1

No description provided.

Rows
Columns