Macro data cleaning, plus growth/inflation/rates and other macro factors, NLP factors, and high-frequency factors.
When building quantitative models, the hard part is often not the algorithm or the strategy itself, but the data.
In macro factor research, data work usually consumes most of the time. No matter how refined the model specification is, weak input data makes conclusions unreliable. Data acquisition and cleaning are therefore the foundation of the whole system.
Common macro data sources are listed below. These are channels frequently used in both practice and research.
Central bank data are mainly related to monetary policy. Examples include:
These series are generally available from the statistics section of the PBoC website. For efficiency, practitioners usually scrape with Python or call existing data APIs.
NBS data have the broadest coverage. Examples include:
The NBS website provides Excel downloads, but table layouts may change across years. Inconsistent structures across vintages are a common issue; handling methods are discussed later.
Wind is a data terminal widely used by Chinese financial institutions. Coverage is broad and updates are timely, but it is costly. Institutional users typically have accounts. Wind provides a Python interface (WindPy) for direct data access.
When budget allows, preferring Wind often improves data quality and reduces cleaning cost.
Once data sources are chosen, Python can be used to pull the data. Commonly used tools include:
This library extends pandas for financial data retrieval. It supports FRED, the World Bank, Yahoo, and others.
import pandas_datareader.data as web
import datetime
# Fetch U.S. GDP from FRED
start = datetime.datetime(2000, 1, 1)
end = datetime.datetime(2024, 12, 31)
gdp = web.DataReader('GDP', 'fred', start, end)
print(gdp.head)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-1-337883f6a6bd> in <cell line: 0>() ----> 1 import pandas_datareader.data as web 2 import datetime 3 4 # 从 FRED 获取美国 GDP 数据 5 start = datetime.datetime(2000, 1, 1) ModuleNotFoundError: No module named 'pandas_datareader'
FRED covers a wide range of macro indicators and is often used for U.S. interest rates, unemployment, and similar series.
yfinance is the Python interface to Yahoo Finance. Access can be unstable in mainland China, but it remains useful for U.S. equities and ETFs.
import yfinance as yf
# Download S&P 500 index data
sp500 = yf.download('^GSPC', start='2000-01-01', end='2024-12-31')
print(sp500.head)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-2-c3b326dfd406> in <cell line: 0>() ----> 1 import yfinance as yf 2 3 # 获取标普500指数数据 4 sp500 = yf.download('^GSPC', start='2000-01-01', end='2024-12-31') 5 print(sp500.head()) ModuleNotFoundError: No module named 'yfinance'
Note: yfinance returns a DataFrame with open, close, high, low, volume, and related fields. In factor models, returns are usually computed from closing prices.
akshare is the most recommended tool for China domestic data. It wraps dozens of sources—central bank, NBS, exchanges, and more—behind a unified interface that is convenient to use.
import akshare as ak
# Fetch China GDP data
gdp_china = ak.macro_china_gdp
print(gdp_china.head)
# Fetch CPI data
cpi = ak.macro_china_cpi_monthly
print(cpi.head)
季度 国内生产总值-绝对值 国内生产总值-同比增长 第一产业-绝对值 第一产业-同比增长 第二产业-绝对值 \
0 2026年第1-2季度 695704.0 4.7 31521.8 3.7 250472.9
1 2026年第1季度 334192.9 5.0 11940.8 3.8 116134.9
2 2025年第1-4季度 1401879.2 5.0 93346.8 3.9 499653.0
3 2025年第1-3季度 1013967.9 5.2 58187.1 3.8 362849.7
4 2025年第1-2季度 659861.6 5.3 31234.8 3.7 238262.6
第二产业-同比增长 第三产业-绝对值 第三产业-同比增长
0 3.9 413709.2 5.2
1 4.9 206117.2 5.2
2 4.5 808879.3 5.4
3 4.9 592931.1 5.4
4 5.3 390364.3 5.5
商品 日期 今值 预测值 前值
0 中国CPI月率报告 1996-02-01 2.1 NaN NaN
1 中国CPI月率报告 1996-03-01 2.3 NaN 2.1
2 中国CPI月率报告 1996-04-01 0.6 NaN 2.3
3 中国CPI月率报告 1996-05-01 0.7 NaN 0.6
4 中国CPI月率报告 1996-06-01 -0.5 NaN 0.7
akshare documentation is detailed, and most commonly needed China macro series are available. In one project that required M2, CPI, industrial value added, and PMI at once, each series was one line of code and the pull finished in about ten minutes.
Once data are in hand, do not model immediately. Inspect the basic structure first. A common failure mode is estimating a model before understanding the data, which makes results unreliable.
Macro series often have missing values. GDP is quarterly and CPI is monthly, so frequencies differ. After merging, the lower-frequency series will contain many NaNs.
import pandas as pd
# Suppose two DataFrames: gdp (quarterly) and cpi (monthly)
# After merging, the GDP column will have many NaNs
merged = pd.merge(gdp, cpi, on='date', how='outer')
# Method 1: forward fill (carry the last observation)
merged['gdp'].fillna(method='ffill', inplace=True)
# Method 2: interpolation
merged['gdp'].interpolate(method='linear', inplace=True)
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-4-0e13f31c3dc7> in <cell line: 0>() 3 # 假设有两个 DataFrame：gdp（季度）和 cpi（月度） 4 # 合并后，GDP 列会有大量 NaN ----> 5 merged = pd.merge(gdp, cpi, on='date', how='outer') 6 7 # 方法1：向前填充（用上一个值填充） NameError: name 'gdp' is not defined
Forward fill is the usual choice. Why? Macro releases are lagged—for example, March GDP may be published only at the end of April. Carrying the previous value better matches what was knowable in real time.
Series occasionally contain absurd values—for example, CPI jumping 10% due to a data-entry error.
## Detect outliers with a 3-sigma rule
mean = data['cpi'].mean
std = data['cpi'].std
outliers = data[(data['cpi'] > mean + 3*std) | (data['cpi'] < mean - 3*std)]
# Treatment: replace with the median
data.loc[outliers.index, 'cpi'] = data['cpi'].median
Macro factor models need a common frequency. Common approaches include:
## Convert quarterly GDP to monthly (interpolation)
gdp_monthly = gdp.resample('M').interpolate
# Convert daily rates to monthly (month-end value)
rate_monthly = rate.resample('M').last
Indicators have different units—GDP in trillions, CPI in percent. Standardize before modeling.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler
data_scaled = scaler.fit_transform(data)
data_scaled = pd.DataFrame(data_scaled, columns=data.columns, index=data.index)
After standardization, every series has mean 0 and standard deviation 1, so coefficient magnitudes in a factor model are directly comparable.
The figure below summarizes the data acquisition and processing workflow. Following it keeps the process reliable.
The figure links the full pipeline: from data sources, through acquisition tools and cleaning/preprocessing, to modeling-ready data. Each step has corresponding Python tools and methods.
That covers data acquisition and cleaning. The next section puts these data to work to build macro factors. Before moving on, run the chapter code once end-to-end so you can see where the key operational risks sit.
← Previous chapter 📖 Back to contents Next chapter →
# Export this notebook to HTML
# jupyter nbconvert --to html 31.ipynb
print("Chapter 31 ready. Use 00_build_all.ipynb to export the full site.")
Chapter 31 ready. Use 00_build_all.ipynb to export the full site.
After years of quantitative investing, macro factors increasingly look like the market's underlying operating system. Equity moves, bond volatility, and FX swings appear independent on the surface, but they are driven by a small set of core variables.
A common approach splits macro factors into five groups: growth, inflation, rates, credit, and FX. These five largely explain long-run asset-price dynamics. The sections below unpack what each factor means.
GDP is the aggregate measure of a country's economic output—total income and spending over a period. In factor models one usually uses real GDP year-over-year growth, which strips out price effects and better reflects real activity.
Industrial value added is monthly and much more timely than GDP. After the 2020 pandemic shock, industrial value added had already rebounded for two months before GDP was released. In higher-frequency factor models, industrial value added is often preferred as a growth proxy.
# Fetch industrial value-added data (example)
import pandas as pd
import akshare as ak
# Industrial value added, YoY growth
ind_prod = ak.macro_china_industrial_production_yoy
print(ind_prod.tail)
商品 日期 今值 预测值 前值 408 中国规模以上工业增加值年率报告 2025-05-19 6.1 5.7 7.7 409 中国规模以上工业增加值年率报告 2025-06-16 5.8 5.9 6.1 410 中国规模以上工业增加值年率报告 2025-07-15 6.8 5.6 5.8 411 中国规模以上工业增加值年率报告 2025-08-15 5.7 6.0 6.8 412 中国规模以上工业增加值年率报告 2025-09-15 NaN NaN 5.7
CPI measures consumer-side price changes. Rising pork prices or rents show up in CPI. For factor modeling, note that CPI matters more for monetary policy than for equities in a direct way.
Why? Because central banks watch CPI closely. High CPI raises hike odds; low CPI raises cut odds. CPI therefore affects assets mainly through the intermediate channel of rate expectations.
PPI is the producer price index for industrial goods—factory gate prices. Its impact on cyclical equities is quite direct: rising PPI often means thicker upstream profits, and sectors such as coal, oil, and metals tend to do better.
| Indicator | Frequency | Transmission to assets |
|---|---|---|
| CPI | Monthly | CPI → rate expectations → bonds/equities |
| PPI | Monthly | PPI → corporate profits → cyclicals |
Government bond yields—especially the 10-year—are the pricing benchmark for the financial system, often viewed as the time value of money. Higher rates make borrowing more expensive, raise discount rates across assets, and put valuations under pressure.
Keep in mind: the rate factor hits bonds directly (yields up, prices down) and equities only indirectly (through valuation and earnings expectations).
LPR is the loan prime rate and directly affects borrowing costs for firms and households. An LPR cut signals easier policy and is generally equity-friendly. Because LPR changes infrequently—often only every few months—higher-frequency models rely more on government bond yields.
# Fetch LPR data
lpr = ak.macro_china_lpr
print(lpr[['日期', '1年期LPR', '5年期以上LPR']].tail)
0%| | 0/4 [00:00<?, ?it/s]
--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-7-ac9f73438170> in <cell line: 0>() 1 # 获取 LPR 数据 2 lpr = ak.macro_china_lpr() ----> 3 print(lpr[['日期', '1年期LPR', '5年期以上LPR']].tail()) d:\pythonprojects\venv\Lib\site-packages\pandas\core\frame.py in __getitem__(self, key) 4382 if is_iterator(key): 4383 key = list(key) -> 4384 indexer = self.columns._get_indexer_strict(key, "columns")[1] 4385 4386 # take() does not accept boolean indexers d:\pythonprojects\venv\Lib\site-packages\pandas\core\indexes\base.py in _get_indexer_strict(self, key, axis_name) 6300 keyarr, indexer, new_indexer = self._reindex_non_unique(keyarr) 6301 -> 6302 self._raise_if_missing(keyarr, indexer, axis_name) 6303 6304 keyarr = self.take(indexer) d:\pythonprojects\venv\Lib\site-packages\pandas\core\indexes\base.py in _raise_if_missing(self, key, indexer, axis_name) 6350 if nmissing: 6351 if nmissing == len(indexer): -> 6352 raise KeyError(f"None of [{key}] are in the [{axis_name}]") 6353 6354 not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique()) KeyError: "None of [Index(['日期', '1年期LPR', '5年期以上LPR'], dtype='str')] are in the [columns]"
Credit spread = corporate bond yield − government bond yield—the extra compensation markets require for lending to firms. Wider spreads signal stress and a preference for government bonds; tighter spreads signal recovering risk appetite.
Among the five factors, credit often tracks market sentiment most closely. During the 2018 deleveraging episode, credit spreads surged and many private-issuer bonds sold off hard. Long-credit strategies suffered badly in that period.
Exchange rates—especially USD/CNY—matter more and more for A-shares. RMB appreciation often coincides with foreign inflows and stronger equities; depreciation with outflows and pressure on A-shares.
During the sharp RMB depreciation in 2022, northbound flows left continuously and A-shares fell with them. Including an FX factor in multi-factor models after that episode improved performance.
# Fetch USD/CNY exchange rate
import yfinance as yf
usd_cny = yf.download('CNY=X', start='2020-01-01')
print(usd_cny['Close'].tail)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-8-1f0c4676d102> in <cell line: 0>() 1 # 获取美元兑人民币汇率 ----> 2 import yfinance as yf 3 4 usd_cny = yf.download('CNY=X', start='2020-01-01') 5 print(usd_cny['Close'].tail()) ModuleNotFoundError: No module named 'yfinance'
The figure below summarizes relationships among macro factors. Growth and inflation are closer to causes; rates and credit are closer to consequences; FX is the link between domestic and external markets.
With these five factors in hand, the next question is how to use them. A practical workflow is:
This section walked through the five core factors. Later sections go deeper into data acquisition, processing, and modeling for each factor.
← Previous chapter 📖 Back to contents Next chapter →
At a certain stage of quant research, an awkward fact appears: traditional factors are crowded. Momentum, value, and quality are widely used, and alpha is thinner. What information is still underpriced? Text data looks like a rich vein.
Central bank minutes, news headlines, social-media sentiment—these unstructured sources contain many macro footprints. When wording shifts from “moderately accommodative” to “flexible and appropriate,” what does that imply? Markets may reprice in seconds; quantifying the sentiment shift earlier can create an edge.
This section discusses how to extract macro factors from text with NLP and fuse them with traditional factors. NLP is not a panacea, but it adds another information dimension.
Central bank minutes convey officials' assessment of the economy and hints about future policy. They arrive more often than many hard-data releases, and wording is carefully chosen. Early on, a generic sentiment lexicon seems enough—until it clearly is not.
Why? Central bank text is full of cautious words such as “prudent,” “monitor,” and “as needed.” Generic lexicons often score them negative, but in central-bank language they are routine. A domain-specific sentiment lexicon is required.
One workflow: collect about ten years of minutes, manually label truly hawkish and dovish sentences, then train a simple TF-IDF model on the labeled data to extract high-frequency keywords.
For example, keywords extracted from Federal Reserve minutes might look like:
| Category | Example keywords |
|---|---|
| Hawkish (tightening bias) | hike, inflation pressure, overheating, tighten, exit accommodation |
| Dovish (easing bias) | downside risks, slack employment, accommodative, supportive, patient |
| Neutral | assess, monitor, moderate, balance, gradual |
With the lexicon, each minutes document can be scored. A common formula is:
Sentiment score = (hawkish word count − dovish word count) / total words
Higher scores are more hawkish (higher hike odds). Lower scores are more dovish (stronger easing expectations).
The code below is a simplified version used in real projects. It reads minutes text and returns a sentiment score.
import re
from collections import Counter
# Define a central bank sentiment lexicon (Chinese keywords for Chinese minutes)
hawkish_words = ['加息', '通胀', '过热', '收紧', '退出宽松']
dovish_words = ['下行风险', '就业不足', '宽松', '支持', '耐心']
def central_bank_sentiment(text):
# Tokenize (simple regex here; production code would use jieba or spaCy)
words = re.findall(r'\w+', text)
word_count = Counter(words)
hawkish_score = sum(word_count[w] for w in hawkish_words if w in word_count)
dovish_score = sum(word_count[w] for w in dovish_words if w in word_count)
total_words = len(words)
if total_words == 0:
return 0
sentiment = (hawkish_score - dovish_score) / total_words
return sentiment
# Example
text = "委员会认为通胀压力持续存在，但就业市场仍有下行风险。"
score = central_bank_sentiment(text)
print(f"Sentiment score: {score:.4f}")
情感得分: 0.0000
Sample output:
Sentiment score: 0.0000
The sentence contains both “inflation pressure” (hawkish) and “downside risks” (dovish), so they cancel and the score is 0. That is reasonable—central banks often hedge both sides.
Minutes are sparse—only a few releases a month. For shorter-horizon macro sentiment, news is essential. Thousands of financial headlines arrive daily; humans cannot read them all, but machines can.
The core idea: positive/negative news sentiment leads or coincides with asset prices. A sudden surge in “recession” coverage usually precedes equity weakness.
A typical pipeline:
One point worth stressing: headlines often matter more than full text. Many traders decide from titles alone. In one comparison, a headline-only sentiment factor had a Sharpe about 0.3 higher than a full-text version.
FinBERT is a BERT model fine-tuned on financial text and is far more accurate than generic lexicons. A simple calling example:
from transformers import pipeline
# Load FinBERT sentiment pipeline
sentiment_pipeline = pipeline(
"sentiment-analysis",
model="ProsusAI/finbert"
)
# Example news headlines
news_titles = [
"美联储加息预期升温，股市承压",
"中国经济数据超预期，市场信心恢复",
"全球供应链危机加剧，通胀风险上升"
]
for title in news_titles:
result = sentiment_pipeline(title)[0]
print(f"{title} -> {result['label']}: {result['score']:.4f}")
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-10-5cc764430b42> in <cell line: 0>() ----> 1 from transformers import pipeline 2 3 # 加载FinBERT情感分析管道 4 sentiment_pipeline = pipeline( 5 "sentiment-analysis", ModuleNotFoundError: No module named 'transformers'
Sample output:
Fed hike expectations rise, equities under pressure -> negative: 0.9876
China economic data beat expectations, market confidence recovers -> positive: 0.9543
Global supply-chain stress intensifies, inflation risks rise -> negative: 0.9789
FinBERT handles financial context well. A phrase such as “inflation risks rising,” which is neutral-to-negative, is labeled negative with high confidence.
With an NLP factor in hand, the next question is how to combine it with momentum, value, and other traditional factors. Dropping it straight into a multi-factor model is not enough.
NLP factors correlate with traditional factors but are not fully overlapping. When news sentiment deteriorates, momentum may already be negative—yet sometimes news leads momentum by a day or two. That lead is where the value sits.
Three common fusion methods:
| Method | Description | When it fits |
|---|---|---|
| Equal-weight blend | Standardize NLP and traditional factors, then add with equal weights | Low cross-factor correlation |
| PCA | Extract principal components; use PC1 as the composite factor | Multicollinearity among factors |
| ML weighting | Learn optimal weights with random forests or XGBoost | Sufficient history available |
The third approach is often preferred: machine-learning weights can adapt NLP importance across regimes—higher around news-dense periods, lower in quiet periods.
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
# Assume the following data are available
# df: date, return, momentum, value, NLP sentiment
df = pd.read_csv('factor_data.csv')
# Features: traditional factors + NLP factor
features = ['momentum', 'value', 'nlp_sentiment']
X = df[features]
y = df['return']
# Standardize
scaler = StandardScaler
X_scaled = scaler.fit_transform(X)
# Fit random forest
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_scaled, y)
# Factor importance
importance = pd.DataFrame({
'factor': features,
'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)
print(importance)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) d:\pythonprojects\venv\Lib\site-packages\sklearn\__check_build\__init__.py in <module> 43 try: ---> 44 from ._check_build import check_build # noqa 45 except ImportError as e: ModuleNotFoundError: No module named 'sklearn.__check_build._check_build' During handling of the above exception, another exception occurred: ImportError Traceback (most recent call last) <ipython-input-11-9d0f483bf361> in <cell line: 0>() 1 import pandas as pd ----> 2 from sklearn.ensemble import RandomForestRegressor 3 from sklearn.preprocessing import StandardScaler 4 5 # 假设我们有以下数据 d:\pythonprojects\venv\Lib\site-packages\sklearn\__init__.py in <module> 79 # it and importing it first would fail if the OpenMP dll cannot be found. 80 from . import _distributor_init # noqa: F401 ---> 81 from . import __check_build # noqa: F401 82 from .base import clone 83 from .utils._show_versions import show_versions d:\pythonprojects\venv\Lib\site-packages\sklearn\__check_build\__init__.py in <module> 44 from ._check_build import check_build # noqa 45 except ImportError as e: ---> 46 raise_build_error(e) d:\pythonprojects\venv\Lib\site-packages\sklearn\__check_build\__init__.py in raise_build_error(e) 29 else: 30 dir_content.append(filename + '\n') ---> 31 raise ImportError("""%s 32 ___________________________________________________________________________ 33 Contents of %s: ImportError: No module named 'sklearn.__check_build._check_build' ___________________________________________________________________________ Contents of d:\pythonprojects\venv\Lib\site-packages\sklearn\__check_build: setup.py _check_build.cp38-win_amd64.pyd__init__.py __pycache__ ___________________________________________________________________________ It seems that scikit-learn has not been built correctly. If you have installed scikit-learn from source, please do not forget to build the package before using it: run `python setup.py install` or `make` in the source directory. If you have used an installer, please check that it is suited for your Python version, your operating system and your platform.
Sample output:
factor importance
0 momentum 0.45
1 nlp_sentiment 0.35
2 value 0.20
In this example, NLP importance (0.35) exceeds value (0.20), suggesting that under the current sample, news sentiment has strong predictive content for returns.
The figure below summarizes the chapter logic—from text data to factor fusion—with a method at each step.
The figure makes the pipeline clear: diverse text sources become sentiment scores via NLP, then fuse with traditional factors. Each step can be optimized, but the core idea is unchanged—turn text into quantifiable factors.
That closes the NLP section. Applications in quant go far beyond this, but central-bank minute scoring and news-sentiment factors already support a solid macro factor system. The rest is iterative tuning in live research.
← Previous chapter 📖 Back to contents Next chapter →
High-frequency macro factors once seemed odd: CPI and PMI are monthly or quarterly—what can change within a day? After several projects, it becomes clear that high-frequency data contain useful structure.
This section covers denoising high-frequency data, building intraday factors, and using them in CTA strategies.
High-frequency data means minute- or even second-level series—index futures ticks, bond futures prints, and so on. The hallmark is weak signal and heavy noise.
An early mistake is computing factors on raw prices. Backtests look unrealistically good; live trading collapses. Much of the “signal” is microstructure noise—bid-ask bounce, fleeting order-flow imbalance, and similar artifacts.
Denoising is therefore the first step. Common methods include:
The simplest approach is a moving average to remove spikes—for example, a 5-minute average price instead of the 1-minute close:
import pandas as pd
import numpy as np
# Assume df is 1-minute OHLC data
df['price_smooth'] = df['close'].rolling(window=5, min_periods=1).mean
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-12-ed2ea0c17050> in <cell line: 0>() 3 4 # 假设df是1分钟频率的OHLC数据 ----> 5 df['price_smooth'] = df['close'].rolling(window=5, min_periods=1).mean() NameError: name 'df' is not defined
Do not choose an excessively large window. Smoothing 1-minute data with a 60-minute average can lag so badly that by the time the average turns, half the move is already gone.
If moving averages are too blunt, try wavelets. Wavelet denoising preserves local features while filtering high-frequency noise. It is often used on intraday bond-futures data:
import pywt
def wavelet_denoise(signal, wavelet='db4', level=3):
coeffs = pywt.wavedec(signal, wavelet, level=level)
# Soft thresholding
sigma = np.median(np.abs(coeffs[-1])) / 0.6745
threshold = sigma * np.sqrt(2 * np.log(len(signal)))
coeffs_thresh = [pywt.threshold(c, threshold, mode='soft') for c in coeffs]
return pywt.waverec(coeffs_thresh, wavelet)
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-13-c79ec0c8a495> in <cell line: 0>() ----> 1 import pywt 2 3 def wavelet_denoise(signal, wavelet='db4', level=3): 4 coeffs = pywt.wavedec(signal, wavelet, level=level) 5 # 软阈值处理 ModuleNotFoundError: No module named 'pywt'
Wavelet parameters matter—poor choices can remove the signal too. On rebar tick data, level=5 over-smoothed useful volatility and erased trend information. Levels of 2–3 are usually a better balance.
If you are comfortable with state-space models, the Kalman filter is a more elegant option. It estimates the latent price online and adapts:
from pykalman import KalmanFilter
kf = KalmanFilter(
transition_matrices=[1],
observation_matrices=[1],
initial_state_mean=df['close'].iloc[0],
initial_state_covariance=1,
observation_covariance=1,
transition_covariance=0.01
)
state_means, _ = kf.filter(df['close'].values)
df['price_kalman'] = state_means.flatten
--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) <ipython-input-14-8de6e6915527> in <cell line: 0>() ----> 1 from pykalman import KalmanFilter 2 3 kf = KalmanFilter( 4 transition_matrices=[1], 5 observation_matrices=[1], ModuleNotFoundError: No module named 'pykalman'
The transition_covariance parameter is critical. Too large and the filter tracks every wiggle (weak denoising); too small and lag is severe. Sweep parameters on a historical slice and pick a compromise.
After denoising, build intraday factors. The logic resembles daily factors but emphasizes microstructure features. Common families include:
The simplest intraday momentum is the price change over the past N minutes. Raw returns are noisy, so a weighted scheme is common:
def intraday_momentum(df, lookback=10):
# Exponential weights: recent observations get higher weight
weights = np.exp(-np.arange(lookback) / lookback)
weights /= weights.sum
returns = df['price_smooth'].pct_change
momentum = returns.rolling(window=lookback).apply(
lambda x: np.dot(x, weights[-len(x):])
)
return momentum
A detail: compute momentum on the smoothed price, not the raw price. Otherwise a single tick jump can swing the factor violently.
Volatility factors are common in CTA. At high frequency, Parkinson volatility—using high and low—is often more efficient than a simple standard deviation:
def parkinson_vol(df, window=30):
log_hl = np.log(df['high'] / df['low'])
vol = np.sqrt((1 / (4 * np.log(2))) *
(log_hl ** 2).rolling(window=window).mean)
return vol
Parkinson volatility assumes continuous diffusion. Gaps (e.g., open jumps) inflate the estimate. A common practice is to disable the factor in the first 15 minutes after the open.
With trade prints, you can build an order-flow imbalance factor—the gap between aggressive buys and sells:
def order_flow_imbalance(trades, window=60):
# trades: tick-level prints with side ('B' or 'S') and volume
buy_vol = trades[trades['side'] == 'B']['volume'].sum
sell_vol = trades[trades['side'] == 'S']['volume'].sum
imbalance = (buy_vol - sell_vol) / (buy_vol + sell_vol + 1e-8)
return imbalance
This factor has been especially useful on index futures. On one IF front-month intraday strategy, adding order-flow imbalance lifted Sharpe from about 0.8 to 1.5. Data quality matters: inaccurate aggressor flags can destroy the edge.
Finer factors include bid-ask spread, quote depth, and trade concentration. They track microstructure shifts and are valuable for high-frequency CTA:
| Factor | Construction | Typical use |
|---|---|---|
| Bid-ask spread | ask1 − bid1 | Liquidity warning |
| Quote depth | bid_size + ask_size | Support/resistance cues |
| Trade concentration | large-trade volume / total volume | Informed-flow signal |
| Price impact | price change / volume | Market elasticity |
How to deploy the factors? Three practical angles:
High-frequency factors can filter traditional CTA signals. A daily trend signal may say long, but if intraday momentum shows fading short-term impulse, delay entry or cut size:
def enhanced_signal(daily_signal, intraday_momentum, threshold=0.1):
if daily_signal == 1: # long signal
if intraday_momentum > threshold:
return 1 # confirm long
else:
return 0 # wait
elif daily_signal == -1: # short signal
if intraday_momentum < -threshold:
return -1
else:
return 0
else:
return 0
This idea was used in rebar–hot-rolled coil pairs trading. Daily signals pointed to spread mean reversion, but intraday order-flow imbalance showed the spread still widening—so wait for confirmation. Hit rate improved by roughly 15%.
High-frequency factors can also refine entry timing. Once the direction is decided, use intraday volatility:
In one study, entries when volatility was below the 20th percentile raised the one-hour win rate by about eight percentage points versus random entry—a statistically meaningful gap.
High-frequency factors can also scale risk. For example, adjust leverage with order-flow imbalance:
def dynamic_leverage(base_leverage, order_flow_imbalance, max_leverage=3):
# More aggressive buying → higher leverage
leverage = base_leverage * (1 + order_flow_imbalance)
return np.clip(leverage, 0.5, max_leverage)
The intuition: persistent aggressive buying raises near-term upside odds, so modestly adding size can be reasonable. Respect risk caps—e.g., a hard ceiling of 3× leverage regardless of signal strength.
High-frequency macro factors are neither trivial nor mystical. Three principles matter most:
Their main value is a lens daily data cannot provide. Like a microscope, more detail helps—but local features can also distort the global judgment.
This chapter ends here. Later chapters turn to factor combination and weight optimization—how to synthesize multiple factors into robust strategy signals.
WeChat official account: Blue Ocean Data Digging Camp; WeChat ID: deep3321
← Previous chapter 📖 Back to contents Next chapter →