Whether a factor can be deployed in practice depends heavily on how clean the raw data are. Suspensions, newly listed stocks, limit-up/limit-down days, reporting lags, and look-ahead leakage can all bias factor results. The sections below follow the key steps in data processing and neutralization.
Factor data mainly come from three sources:
Obtaining factor data usually involves two steps:
Factor calculation should follow a close-then-compute-then-trade rule: after the close on day $T$, compute the factor with information available on day $T$, and use it for trading signals on day $T+1$.
Code example: fetch market data:
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
# Simulate fetching the past 20 trading days of closing prices for a stock
def get_price_data(stock_code, start_date, end_date):
# In a real project this would connect to a database or API
# Here we simulate with random data
dates = pd.date_range(start=start_date, end=end_date, freq='B')
prices = np.random.randn(len(dates)).cumsum + 100
return pd.Series(prices, index=dates, name=stock_code)
# Fetch data
price_series = get_price_data('000001.SZ', '2024-01-01', '2024-01-31')
print(price_series.head)
2024-01-01 100.620346 2024-01-02 102.121153 2024-01-03 103.701520 2024-01-04 103.175807 2024-01-05 100.646072 Freq: B, Name: 000001.SZ, dtype: float64
After data are obtained, the first task is not to compute the factor immediately, but to clean the raw series.
Outliers are observations that deviate sharply from the normal range and may reflect data errors, definitional inconsistencies, or extreme events. For example, an abnormally large one-day return or an extreme valuation reading can distort the factor distribution. Left untreated, outliers often warp statistical features and weaken the robustness of factor research.
Common outlier treatments fall into three categories:
| Method | Idea | When it fits |
|---|---|---|
| MAD (median absolute deviation) | Uses the median instead of the mean; more robust to extremes | When the factor distribution is clearly skewed |
| Percentile truncation (winsorization) | Caps values beyond chosen upper/lower percentiles | When a simple, fast treatment is preferred |
| 3σ rule | Treats observations beyond mean ± 3 standard deviations as outliers | When the factor is approximately normal |
In practice, MAD is usually more robust: it is less sensitive to extremes, so a few outliers are less likely to dominate the screen than under the 3σ rule. When the factor is skewed or heavy-tailed, MAD is typically the better default.
Outlier treatment should not be one-size-fits-all. A more reasonable approach is to treat within industry first, then across the full market. Factor distributions differ materially across industries—for example, banks and tech stocks differ in valuation levels and volatility—so mixing them can misclassify industry features as outliers.
Code example: MAD-based outlier treatment:
def mad_outlier_removal(factor_series, n=5):
"""
Identify and treat outliers with the MAD method.
n: threshold, typically 3–5
"""
median = factor_series.median
mad = np.median(np.abs(factor_series - median))
# Modified Z-score
modified_z_scores = 0.6745 * (factor_series - median) / mad
# Flag outliers
outliers = np.abs(modified_z_scores) > n
# Replace outliers with the median
factor_series_clean = factor_series.copy
factor_series_clean[outliers] = median
return factor_series_clean
# Example
raw_factor = pd.Series(np.random.randn(1000) * 10 + 50)
raw_factor[0] = 5000 # inject an extreme value
clean_factor = mad_outlier_removal(raw_factor)
print(f"Max before treatment: {raw_factor.max:.2f}")
print(f"Max after treatment: {clean_factor.max:.2f}")
处理前最大值: 5000.00 处理后最大值: 82.16
After outlier treatment, the next step is standardization.
Standardization removes the effects of inconsistent scales across factors. PE, turnover, momentum, and similar metrics have very different ranges; feeding them into a model raw can let large-scale factors dominate and hurt interpretability and robustness.
Two common standardization methods are:
Note that Z-score standardization usually assumes approximate normality, which many factors violate. Turnover, for example, is often right-skewed, so a direct Z-score is suboptimal. Prefer rank standardization, or take a log transform before standardizing.
In multi-factor construction, a common pipeline is rank standardization followed by a Z-score transform: preserve relative ordering, then push the distribution closer to normal—e.g., convert to percentile ranks and map to Z-scores via the normal inverse CDF.
Code example: rank standardization:
def rank_standardize(factor_series):
"""
Rank standardization: map factor values to [0, 1]
"""
rank = factor_series.rank
standardized = (rank - 1) / (len(rank) - 1)
return standardized
# Example
factor = pd.Series(np.random.randn(1000) * 100)
standardized_factor = rank_standardize(factor)
print(f"Raw factor range: [{factor.min:.2f}, {factor.max:.2f}]")
print(f"Standardized range: [{standardized_factor.min:.2f}, {standardized_factor.max:.2f}]")
原始因子范围: [-283.47, 352.87] 标准化后范围: [0.00, 1.00]
The next sections cover factor neutralization: after cleaning and standardization, strip industry, size, and style exposures.
This section discusses size, industry, and style neutralization: using regression or grouping methods to remove the parts of a factor that load on these known dimensions, and keeping the residual as a “purer” factor signal.
In empirical asset pricing, regressing out size/industry as a default preprocessing step is usually not the standard. More common approaches are:
Academic designs therefore often control size and industry inside the test, rather than rewriting the factor first and then pretending to test the original characteristic. Immediate neutralization can also change the question—especially when the signal is entangled with industry structure (e.g., large-cap groups heavily concentrated in banks).
In A-share quant and portfolio management, neutralization is common because product constraints and attribution goals differ:
Academia’s default question is “does this characteristic carry a premium in the full market?”; practitioner neutralization more often asks “after stripping size and industry, is there still tradable incremental information?” The two are complementary, not substitutes. Report results both before and after neutralization, and inspect industry and size exposures of the portfolios.
The core goal of neutralization is to remove industry, size, and style contamination from the factor and keep the independent information in the residual (factor ≈ style exposure + residual). The following subsections cover industry neutralization, size and style neutralization, and double neutralization.
The first step in industry neutralization is industry classification; without labels there is nothing to strip.
Mainstream industry classification schemes include:
| Scheme | Publisher | Hierarchy | Features |
|---|---|---|---|
| GICS | MSCI / S&P | 11 sectors, 24 industry groups, 69 industries | Global standard, widely used by institutions |
| ICB | FTSE Russell | 11 industries, 20 supersectors, 45 sectors | Common in Europe; similar to GICS |
| SW (Shenwan) | Shenwan Hongyuan | 31 level-1, 134 level-2 industries | Most common for A-shares; close to the domestic market |
| CITIC | CITIC Securities | 30 level-1, 110 level-2 industries | Institution-customized; updates relatively fast |
In A-share practice, Shenwan level-1 industries are the usual choice: the logic matches domestic market intuition and the data are easy to obtain. Note that Shenwan industries are revised annually—update classifications promptly.
For multi-market global strategies, prefer GICS so that comparisons across markets share one framework.
Industry neutralization is rarely done in isolation from size. In practice, both dimensions are often neutralized together—industry–size neutralization.
Size should be included because large- and small-cap stocks are unevenly distributed across industries—banks tend to be large, tech names tend to be smaller. Industry-only neutralization still leaves size effects in the factor exposure.
A concrete implementation (code example):
import pandas as pd
import statsmodels.api as sm
# Assume df has: ticker, factor value, industry dummies, log market cap
# Industry dummies: one column per industry, 0/1 encoding
def industry_market_neutralize(df, factor_col, mkt_cap_col, industry_dummies):
"""
Industry–size neutralization: regress out industry and size effects
"""
# Regressors: industry dummies + log market cap
X = df[industry_dummies].copy
X['log_mkt_cap'] = np.log(df[mkt_cap_col])
X = sm.add_constant(X) # intercept
# Dependent variable: raw factor
y = df[factor_col]
# Regression
model = sm.OLS(y, X).fit
# Residuals are the neutralized factor
df['factor_neutral'] = model.resid
return df, model
The logic is simple: regress out the part of the factor explained by industry and size; the residual is the cleaned factor.
Note: include an intercept. Without it, residual means may be nonzero, neutralization is incomplete, and backtests can show roughly a 0.5 percentage-point bias.
Regression is the most common method, but not the only one. Common stripping approaches include:
Regression can handle multiple controls at once with clear statistical properties, so it is the most widely used. For a quick single-factor check, within-group adjustment is simpler.
Code example for within-group adjustment:
def group_neutralize(df, factor_col, industry_col):
"""
Within-group adjustment: standardize within industry
"""
df['factor_neutral'] = df.groupby(industry_col)[factor_col].transform(
lambda x: (x - x.mean) / x.std
)
return df
The code is concise and fast, but it only removes industry mean differences—it does not strip within-industry size effects. For more thorough neutralization, use regression.
Several details in industry neutralization, if mishandled, push results away from what you intend. Common pitfalls:
Core takeaway: industry neutralization is about control variables—prevent industry trends from dominating factor returns by controlling for industry. Regression is the most general tool; also control size. The residual is the neutralized true factor.
In practitioner multi-factor pipelines, industry neutralization is often one of the most basic preprocessing steps. If industry effects are not stripped thoroughly, later combination and optimization tend to be dominated by industry allocation.
After industry neutralization, size and style exposures still need treatment. The following sections introduce finer-grained neutralization methods.
Take one-month turnover as an example: is the factor related to market cap?
The link is strong: small-cap turnover is usually higher than large-cap turnover. Without treatment, stock selection may tilt toward small caps, and turnover and size effects become hard to separate in factor returns.
Hard to tell apart.
Neutralization strips the part of the factor that belongs to other styles, leaving the factor’s own unique information.
Core idea: factor value = style-exposure component + residual. The target is the residual.
Size neutralization is the most common neutralization step. The approach is straightforward:
In practice, regression is common: regress the factor on market cap and take the residual as the new factor.
import pandas as pd
import statsmodels.api as sm
# Assume df has factor and market_cap
# Log market cap usually works better
df['log_mkt'] = np.log(df['market_cap'])
X = sm.add_constant(df['log_mkt'])
y = df['factor']
model = sm.OLS(y, X).fit
df['factor_neutral'] = model.resid
# After this, corr(new factor, size) should be near zero
print(df[['factor_neutral', 'log_mkt']].corr)
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-6-33aaa14fd86d> in <cell line: 0>() 4 # 假设 df 包含因子值 factor 和市值 market_cap 5 # 市值取对数，效果更好 ----> 6 df['log_mkt'] = np.log(df['market_cap']) 7 X = sm.add_constant(df['log_mkt']) 8 y = df['factor'] NameError: name 'df' is not defined
A useful alternative is to bucket by size first, then standardize within buckets to dampen extremes. Use 10 or 20 buckets depending on the number of stocks.
Size neutralization addresses only one dimension. In reality a factor may load on several styles. An earnings-growth factor, for example, may relate to size, valuation, and momentum at once.
That is when style-factor orthogonalization is needed.
The procedure mirrors size neutralization, but the regressors become multiple style factors instead of size alone.
# Assume three style factors: size, valuation, momentum
style_factors = ['log_mkt', 'pe_ratio', 'momentum_12m']
X = sm.add_constant(df[style_factors])
y = df['factor']
model = sm.OLS(y, X).fit
df['factor_orthogonal'] = model.resid
# Check correlations between the orthogonalized factor and style factors
print(df[['factor_orthogonal'] + style_factors].corr)
Caveat: more style factors are not always better—too many can strip the factor’s useful information. Typically keep 3–5 core style factors.
Caution: do not over-orthogonalize. Feeding in 10 style factors at once can drive IC toward 0.01 and nearly kill the factor, often because highly correlated variables are removed together with the signal. Orthogonalization should preserve the core signal; thoroughness is not the goal.
Sometimes orthogonalization alone is not enough. Size effects can be nonlinear—factor behavior differs materially inside large-cap versus small-cap groups. In those cases, use double neutralization.
The procedure has two steps:
Grouping first isolates size effects; within-group orthogonalization then removes other style loadings. The resulting factor is less affected by size-bucket differences and within-bucket style correlations.
def double_neutralization(df, group_col='mkt_group', style_factors=['pe_ratio', 'momentum_12m']):
"""
Double neutralization
"""
df['factor_double_neutral'] = np.nan
for group in df[group_col].unique:
mask = df[group_col] == group
sub_df = df[mask].copy
# Orthogonalize to style factors within the group
X = sm.add_constant(sub_df[style_factors])
y = sub_df['factor']
model = sm.OLS(y, X).fit
sub_df['factor_neutral'] = model.resid
# Within-group standardization
sub_df['factor_double_neutral'] = (sub_df['factor_neutral'] - sub_df['factor_neutral'].mean) / sub_df['factor_neutral'].std
df.loc[mask, 'factor_double_neutral'] = sub_df['factor_double_neutral'].values
return df
| Method | Dimensions handled | Best when | Complexity |
|---|---|---|---|
| Size neutralization | Size only | Factor highly correlated with size | Low |
| Style orthogonalization | Multiple styles | Factor loads on several styles | Medium |
| Double neutralization | Grouping + orthogonalization | Nonlinear size effects plus multi-style contamination | High |
Suggestion: start with size neutralization in early research; upgrade to style orthogonalization if the factor loads on multiple styles; reserve double neutralization for final model construction.
If IC falls after neutralization, do not abandon the factor immediately—first check whether the style controls are appropriate. Orthogonalizing a low-volatility factor to volatility, for example, can strip the signal itself because the two are essentially the same object.
Neutralization aims to remove style exposures that do not belong to the factor while preserving independent information. Done well, multi-factor models are cleaner and portfolios tend to be more stable.
After winsorization, standardization, and neutralization, the preprocessing stage of the factor production pipeline is largely complete.
Data cleaning and preprocessing may look tedious, but they are foundational to a factor-investing system. In practice, research effort often concentrates on model innovation while underinvesting in cleaning—then backtest performance fails to translate live. Give data cleaning enough weight in both factor research and portfolio construction.