Detrending Methods: HP Filter vs First-Difference

Before any cycle detection algorithm can do its work, the raw price data must be prepared. Price series contain two primary components that spectral methods need to disentangle: the trend (the general directional movement over time) and the cyclical oscillations embedded within that trend. Detrending is the process of removing the trend component so that the Goertzel algorithm and other spectral analysis methods can isolate the periodic structures we actually want to study. The choice of detrending method is not trivial—it directly affects which cycles are detected, how their amplitudes are measured, and whether the resulting analysis accurately represents the underlying market structure.

Why Detrending Matters

Raw price data contains trend and cyclical components that interfere with each other during spectral analysis. The trend—the general direction of price over time—shows up in the power spectrum as a massive low-frequency component that drowns out the cycles we want to find. Before we can detect cycles reliably, we must remove the trend.

This is not optional. Skip detrending and your power spectrum will be dominated by a single peak at the lowest frequency, telling you only that price went up or down over the analysis period. The cyclical structure remains hidden beneath the overwhelming spectral energy of the trend itself. Even a modest 10% trend over 200 bars will dominate the spectrum, making it impossible to identify the 20-bar, 50-bar, or 100-bar cycles that represent the actual periodic structure of the market.

The challenge is removing the trend without distorting the cycles. Every detrending method makes tradeoffs between trend removal completeness and cycle preservation fidelity. Understanding these tradeoffs is essential for interpreting spectral results correctly.

First-Difference: Simple but Aggressive

The simplest detrending method subtracts each price from the previous one. If today's close is 105 and yesterday's was 103, the first-difference is 2. This transformation removes trend completely—even a strong uptrend becomes a series of small positive values oscillating around zero. The result is essentially a series of returns (in absolute rather than percentage terms).

Advantages of first-difference:

• Computationally trivial: A single subtraction per data point
• Trend-agnostic: Removes trend regardless of its shape (linear, exponential, polynomial, or otherwise)
• Parameter-free: No tuning required—the method is fully deterministic
• Stationarity: The resulting series is typically stationary, satisfying a key assumption of many spectral methods

The drawback: first-differencing acts as a high-pass filter, attenuating longer cycles while amplifying shorter ones. Mathematically, differencing multiplies the spectral amplitude at each frequency by a factor proportional to that frequency. A genuine 200-bar cycle may appear weaker relative to a 20-bar cycle than it actually is. For long-term analysis, this distortion matters considerably.

Hodrick-Prescott Filter: Smoother Separation

The HP filter takes a different approach. Rather than removing trend through differencing, it mathematically separates the price series into trend and cycle components by solving an optimization problem. The filter finds the smoothest possible trend line that still tracks the data reasonably, then subtracts it. The result is a cycle component that oscillates around zero while preserving the amplitude relationships between cycles of different lengths.

The key parameter is lambda (λ), which controls how smooth the trend component is. Higher lambda produces a smoother trend, preserving more of the cyclical variation in the residual. Lower lambda allows the trend to follow price more closely, which absorbs some of the longer cyclical variation into the trend estimate. Standard values depend on data frequency:

• Daily data: λ = 100 to 1,600
• Weekly data: λ = 270 to 675
• Monthly data: λ = 14,400 (the classic Hodrick-Prescott value)

In FractalCycles, we auto-select lambda based on the data timeframe using the Ravn-Uhlig adjustment: λ scales with the fourth power of the frequency ratio. This ensures that the effective smoothness of the trend component is comparable across different data frequencies, preventing lambda miscalibration from distorting cycle detection.

Linear Detrending: The Baseline Approach

A third approach, simpler than the HP filter but more nuanced than first-differencing, is linear detrending. This method fits a straight line through the data (using least-squares regression) and subtracts it. The result removes any constant upward or downward drift from the series.

Linear detrending works well when the trend is approximately linear over the analysis window. It preserves cycle amplitudes faithfully and introduces no frequency-dependent distortion. However, it fails when the trend is curved—during parabolic rallies, accelerating selloffs, or any period where the trend changes direction. In these cases, the linear approximation leaves trend residuals that contaminate the cycle component.

For markets that exhibit relatively steady trends over the analysis period, linear detrending offers a good balance between simplicity and accuracy. For markets with changing trend direction, the HP filter or first-difference will generally produce cleaner results.

Which Method When?

Our testing across various markets and timeframes suggests these guidelines:

Use first-difference when:

• Analyzing short-term cycles (under 50 bars) where high-frequency detail matters most
• Data has obvious trend changes, regime shifts, or structural breaks
• You want simplicity and parameter-free operation
• The analysis focuses on return-based cycles rather than price-deviation cycles

Use HP filter when:

• Analyzing longer-term cycles (50+ bars) where amplitude preservation matters
• You need to compare cycle amplitudes across different periods accurately
• Data covers an extended period with sustained but potentially curved trends
• The analysis will feed into composite wave construction where amplitude accuracy affects the projection

Use linear detrending when:

• The trend over the analysis window is approximately straight
• You want amplitude preservation without parameter choices
• The analysis window is short enough that trend curvature is minimal

How Detrending Affects Detected Cycles

The choice of detrending method can change which cycles appear significant in the power spectrum and how their Bartels scores are calculated. Consider a market with genuine cycles at 25 bars and 150 bars:

• First-difference will clearly detect the 25-bar cycle but may understate the 150-bar cycle due to high-pass filtering. The Bartels score for the longer cycle may be lower than its true significance warrants.
• HP filter (with appropriate lambda) will detect both cycles with more balanced amplitudes. However, if lambda is too low, the filter may absorb some of the 150-bar cycle into the trend, weakening its spectral peak.
• Linear detrending will preserve both cycles if the trend is linear, but may introduce artifacts if the trend curves significantly over the analysis window.

When in doubt, running the analysis with multiple detrending methods and comparing results provides valuable information. Cycles that appear under all methods are more robust than those that appear under only one. This cross-validation approach increases confidence in the detected structure.

Visualizing the Difference

Run the same data through both first-difference and HP filter methods and compare the resulting power spectra. You will typically see:

• First-difference emphasizes shorter cycles, with peaks skewing toward lower periods
• HP filter preserves longer cycles better, often revealing structure that first-difference misses
• Peak locations usually agree on the periods of dominant cycles, but relative heights differ
• Very long cycles (near the Nyquist limit) may appear only with the HP filter

Neither is "correct"—they answer slightly different questions. First-difference asks: what cycles exist in price changes? HP filter asks: what cycles exist in price deviations from trend? Both questions are valid, and understanding which question your analysis answers is essential for interpreting results accurately.

Common Pitfalls

Several common mistakes in detrending can compromise cycle analysis:

• Skipping detrending entirely: The most common error. The resulting spectrum shows only trend energy and no usable cycle information.
• Over-detrending: Using an HP filter with lambda too low, which absorbs genuine long-term cycles into the trend estimate and removes them from the analysis.
• Ignoring end effects: Both HP filter and differencing can produce artifacts at the beginning and end of the data series. The HP filter, in particular, can exhibit end-point bias where the trend estimate curves inappropriately at the edges of the data.
• Using percentage returns instead of price differences: Log returns or percentage returns introduce additional nonlinearity that can create spurious spectral peaks.

Implementation in FractalCycles

FractalCycles defaults to HP filter detrending with auto-calculated lambda based on the data timeframe. Users can switch to first-difference in the analysis settings under Advanced Options. For most purposes, the default works well, as the Ravn-Uhlig lambda adjustment has been calibrated against a broad range of market data.

If you are specifically interested in short-term structure or notice the HP filter producing artifacts at trend reversals, try first-difference as an alternative view. Comparing the results from both methods—noting which cycles appear under both and which appear under only one—provides additional analytical depth and helps distinguish robust cycles from method-dependent artifacts. TheHurst exponent can also inform detrending choice: strongly trending markets (high Hurst) may benefit more from HP filtering, while mean-reverting markets (low Hurst) may produce cleaner results with first-differencing.