GOTabPFN · Graph-guided Feature Ordering for Tabular Foundation Models

Overview

GOTabPFN targets high-dimensional tabular learning, especially HDLSS settings where the number of features can be much larger than the number of samples. The method learns a feature ordering using GO-LR, compresses local ordered feature neighborhoods with NSC-pSP, and passes compact meta-features to a frozen TabPFN-2.5 head.

Graph-guided ordering

Features are organized using graph structure and local refinement so nearby positions reflect useful dependencies.

Compact tokenization

Ordered neighborhoods are compressed into meta-features, reducing the burden of high-dimensional inputs.

Frozen predictor

The compact representation is passed to a frozen TabPFN-2.5 head for downstream prediction.

Architecture

Method Components

GO-LR: graph-guided feature ordering with local refinement.
NSC-pSP: neighborhood/segment compression through PCA-IDF-aware principal subspace projection.
Frozen TabPFN-2.5 head: downstream binary, multiclass, or regression prediction using compact meta-features.
Diagnostics: dataset-level IDF/FOE/P-success, locality gains, LES, and AUC under the cumulative explained variance-IDF curve.

GO-LR: Graph-Guided Ordering with Local Refinement

Feature graph converted into a linear feature ordering by placing strongly related features close together — Feature graph to linear ordering.

GO-LR animation

Feature graph → linear ordering

Animated graph-to-ordering transformation.

GO-LR treats feature ordering as a graph layout problem. Features are represented as nodes, pairwise feature dissimilarities define weighted edges, and the goal is to produce a linear order where strongly related features remain close in index space.

Dispersion objective

D_{G_c}(\pi)=\sum_{(i,j)\in E} w_{ij}\,|\pi(i)-\pi(j)|

Combinatorial objective The local ordering objective corresponds to weighted Minimum Linear Arrangement, making exact optimization NP-hard in the number of features.

Efficient practical solver GO-LR uses a nearest-neighbor path-style initialization and then applies direction selection plus adjacent-swap local refinement to reduce the dispersion objective.

Aligned with compression By placing related features close together, GO-LR makes contiguous ordered segments more meaningful for downstream NSC-pSP compression and frozen TabPFN-2.5 prediction.

Local refinement guarantee Direction selection and adjacent swaps are accepted only when they do not increase the dispersion cost, giving \(D_{G_c}(\pi_c)\le D_{G_c}(\pi^{(0)})\).

Ordering cost comparison against classical metaheuristics

On the Colon benchmark, GO-LR was compared against several classical ordering/metaheuristic baselines. Although some baselines achieve lower TSP-style surrogate cost, GO-LR achieves the lowest MinLA-style dispersion cost with the fastest runtime.

Ordering Method	Runtime	TSP Cost ↓	MinLA Cost ↓
GO-LR	10.07s	\(21958.78\)	\(1.4743\times10^{10}\)
Simulated Annealing	15.01s	\(11712.75\)	\(1.4803\times10^{10}\)
Genetic Algorithm	206.59s	\(11712.75\)	\(1.4803\times10^{10}\)
Ant Colony Optimization	1501.44s	\(11792.08\)	\(1.4760\times10^{10}\)
Christofides	1424.83s	\(11715.06\)	\(1.4994\times10^{10}\)

Extreme dimensionality: Cell Cycle

On the Cell Cycle RNA-seq dataset with \(n=1067\) samples and \(m=42728\) features, GO-LR achieved a lower MinLA-style dispersion cost than Simulated Annealing, supporting its alignment with the objective used before NSC-pSP compression.

Ordering Method	MinLA Cost ↓
GO-LR	\(8.14\times10^{11}\)
Simulated Annealing	\(8.51\times10^{11}\)

Interpretation: lower TSP-style surrogate cost alone does not necessarily imply a better ordering for the GOTabPFN pipeline. GO-LR directly targets the MinLA-style dispersion objective used before NSC-pSP compression.

NSC: Neuro-Inspired Subunit Compression

NSC compresses GO-LR ordered features into compact meta-features through segmentation and PCA-based segment compression — NSC converts the GO-LR ordered feature vector into contiguous segments and compresses each segment into compact meta-features.

NSC animation

Ordered features → segmentation → PCA pooling → meta-features

Animated illustration of NSC transforming GO-LR ordered features into compact meta-features.

NSC provides the representation interface between GO-LR and frozen TabPFN head. It transforms an ordered high-dimensional feature vector into a compact sequence of meta-features, making TabPFN-style inference more suitable for HDLSS data.

Segment compression

z_t(x)=\big(u_t(x)-\mu_t\big)^\top v_t, \qquad Z(x)=(z_1,\dots,z_M)

Ordered-axis segmentation NSC starts from the GO-LR-ordered feature vector and partitions it into contiguous ordered segments, so nearby features remain grouped within local neighborhoods.

Adaptive meta-feature budget The number of meta-features \(M\) is chosen from intrinsic dimensionality rather than raw ambient dimension, enabling stronger compression in redundant HDLSS settings.

Segment-wise compression Each ordered segment is compressed independently into one compact token. In GOTabPFN, the main variant uses segment-wise PCA pooling through NSC-pSP.

Final representation The compressed outputs form a structured meta-feature sequence \(Z(x)=(z_1,\dots,z_M)\), which is passed to the frozen TabPFN-2.5 predictor.

HDLSS-friendly scalability By reducing \(m\) raw features into \(M \ll m\) structured meta-features, NSC lowers memory and compute while preserving order-induced locality.

Final Prediction with Frozen TabPFN-2.5

GOTabPFN pipeline summary

GO-LR ordering → NSC compression → pretrained TabPFN-2.5 prediction

Animated summary of GOTabPFN: global feature ordering, structured compression, compact meta-features, and final prediction using a pretrained TabPFN-2.5 checkpoint.

After GO-LR and NSC, each sample is represented by compact meta-features \(Z(x)\). GOTabPFN then applies the pretrained TabPFN-2.5 checkpoint as the final predictor.

Prediction pipeline

x \xrightarrow{\mathrm{GO\mbox{-}LR}} x_{\Pi^\ast} \xrightarrow{\mathrm{NSC}} Z(x) \xrightarrow{\mathrm{TabPFN\mbox{-}2.5}} \hat{y}

Pretrained checkpoint head TabPFN-2.5 is used on compressed meta-features \(Z(x)\) with the pretrained checkpoint kept unchanged.

Fold-wise evaluation For each split, compressed train and validation representations, \(Z_{\mathrm{train}}\) and \(Z_{\mathrm{val}}\), are passed to the TabPFN-2.5 head.

Order → Compress → Predict GOTabPFN combines graph-guided ordering, NSC meta-feature compression, and a pretrained tabular foundation-model predictor.

Dataset Diagnostics: Feature Ordering - When and Why It Helps

Input Dataset CSV numeric features + target column

When? Ordering opportunity IDF ↓ · FOE ↑ · \(P_{\mathrm{success}}\) ↑

Why? Locality validation \(\Delta\)AdjCoh · \(\Delta\)HitRate · \(\Delta\)Cut

Decision Ordering useful? FOE flags opportunity; LES confirms locality

Opportunity screen

\mathrm{IDF}=\frac{\hat d}{m}, \qquad P_{\mathrm{success}}=1-\mathrm{IDF}

How to read the diagnostics Low IDF and high FOE suggest that ordering may be useful; positive locality gains and higher LES confirm that GO-LR actually created exploitable neighborhoods.

IDF/FOE estimate the opportunity for ordering, while LES measures the realized locality gain over random orderings.

GOTabPFN includes dataset-level diagnostics to estimate whether feature ordering is likely to help. The diagnostics connect when ordering may help with why it helps: useful ordering should expose local neighborhoods that NSC can pool.

When ordering may help Low IDF and high FOE indicate that the dataset has redundant or low-rank structure, leaving room for feature ordering to expose useful neighborhoods.

Why ordering helps GO-LR is useful when it improves locality over random orderings, measured by adjacency coherence, neighborhood hit-rate, and segmentation boundary cut.

Locality exploitability LES summarizes whether the learned ordering creates short-range neighborhoods that contiguity-based operators such as NSC can exploit.

Practical recommendation Treat high FOE or low IDF as the first screening signal for trying feature ordering. LES is a secondary diagnostic: positive locality gains strengthen the case, but low FOE datasets can still occasionally show useful locality.

Diagnostic	Interpretation
IDF ↓	Lower intrinsic dimension relative to ambient features; suggests redundancy or compact structure.
FOE ↑	First-pass screening signal for ordering opportunity.
LES ↑	Follow-up diagnostic showing whether GO-LR realizes locality over random ordering.
Δ Locality ↑	Positive AdjCoh, HitRate, and Cut gains indicate improved local neighborhoods over random baselines.

FOE is used as the first screening barrier for ordering opportunity; LES is a follow-up locality diagnostic that shows whether GO-LR actually realizes useful neighborhoods.

Results

GOTabPFN is evaluated on high-dimensional low-sample-size tabular benchmarks using \(5\times5\) repeated cross-validation. Across the core HDLSS benchmark suite, GOTabPFN achieves the best result on all eight datasets and the lowest average rank.

8/8 Best on core HDLSS datasets

1.00 Average rank across HDLSS tasks

7/8 Best on additional cross-domain tasks

Cross-domain extension Beyond the 8 biomedical HDLSS datasets, GOTabPFN also performs strongly on additional high-dimensional tasks spanning image-derived, biological, text-like, and camera-sensor features.

HDLSS classification performance GOTabPFN achieves the best mean accuracy on all 8 biomedical HDLSS datasets, with particularly visible gains on harder datasets such as GLI-85, SMK, and Arcene.

Gains over foundation-model heads Compared with the best original TabPFN-style heads, GOTabPFN improves accuracy on every HDLSS dataset, showing that GO-LR and NSC provide a useful ordering-aware compression interface.

Cross-domain robustness On additional high-dimensional datasets, including image-derived, biological, text-like, and camera-sensor features, GOTabPFN obtains the best average rank and top result on 7/8 tasks.

Main takeaway Ordering plus compression helps make high-dimensional tabular data more compatible with a pretrained TabPFN-2.5 predictor.

Dataset	Best Orig.	GOTabPFN	Abs. Gain	Rel. Gain
Colon	87.85	88.18	+0.33	+0.38%
Lung	96.55	97.44	+0.89	+0.92%
GLI-85	89.66	93.82	+4.16	+4.64%
SMK	71.99	74.23	+2.24	+3.11%
ALLAML	97.16	97.54	+0.38	+0.39%
Prostate	93.31	93.37	+0.06	+0.06%
Arcene	88.00	90.60	+2.60	+2.95%
TOX	93.25	93.33	+0.08	+0.09%

“Best Orig.” denotes the best original tabular foundation-model head among TabDPT, TabPFN-Wide, BETA, TuneTables, and TabICL. Values are mean accuracies under \(5\times5\) cross-validation.

Ablation insight Removing NSC, replacing GO-LR with identity/random ordering, or using weaker predictors reduces performance, indicating that both ordering-aware compression and the pretrained TabPFN-2.5 head matter.

Efficiency note On Colon, GOTabPFN reaches \(88.2\%\) accuracy in \(31.4\) seconds with modest peak GPU memory, while most memory usage remains CPU-side during graph construction and refinement.

Limitations GOTabPFN inherits TabPFN-2.5 constraints, including limits on the number of classes and samples; it is most suitable for HDLSS and related low-sample, high-dimensional regimes.

Comparative Leaderboard

GOTabPFN is compared against strong classical, deep tabular, HDLSS, and tabular foundation-model baselines. The leaderboard summarizes average rank across datasets, where lower rank is better.

50+ Compared baselines

8 Core HDLSS tasks

8 Cross-domain tasks

Core HDLSS leaderboard

Rank	Method	Avg. Rank ↓	Positioning
1	GOTabPFN	1.00 ± 0.00	Best on all 8 HDLSS datasets
2	TANDEM	3.63 ± 1.32	Strongest non-GOTabPFN average rank
3	TabPFN-Wide	3.75 ± 2.38	Strong TabPFN-style baseline
4	TabDPT	4.88 ± 1.69	Competitive tabular foundation-model baseline
5	TabICL	7.63 ± 2.29	In-context tabular baseline
6	BETA	8.13 ± 4.31	TabPFN Unleashed: another TabPFN variant but computationally slow
7	TuneTables	8.38 ± 7.70	Tuned TabPFN-style baseline

Cross-domain leaderboard

Rank	Method	Avg. Rank ↓	Positioning
1	GOTabPFN	1.25 ± 0.66	Best average rank; top result on 7/8 tasks
2	TabPFN-Wide	3.88 ± 1.62	Strongest competing average rank
3	TANDEM	4.00 ± 1.94	Competitive on cross-domain tasks
4	TabDPT	4.44 ± 1.49	Strong tabular foundation-model baseline
5	MLP	5.25 ± 2.17	Competitive classical neural baseline
6	TabICL	5.31 ± 2.46	Unsupported/OOM on some tasks
7	TuneTables	5.94 ± 1.91	Useful but less consistent across domains

Average rank is computed across datasets with lower values indicating better overall performance. OOM or unsupported runs are ranked last in the cross-domain setting.

Conclusion and Project Notes

Conclusion

GOTabPFN makes TabPFN-style small tabular foundation models more effective in HDLSS regimes by coupling MinLA-grounded feature ordering through GO-LR with NSC, a stable locality-preserving compression interface. The resulting order-to-tokenization pipeline converts high-dimensional tables into compact meta-feature sequences for a pretrained TabPFN-2.5 predictor.

Without modifying the TabPFN-2.5 backbone, GOTabPFN improves accuracy and robustness under tight feature budgets, providing a practical route to scalable in-context tabular prediction when \(m \gg n\).

Reproducibility, Impact, and Support

Acknowledgements This work was supported in part by the US National Science Foundation under Awards #1920920, #2125872, and #2223793.

Review process We thank the anonymous ICML reviewers for their valuable feedback and suggestions.

Software and data Code, notebooks, and installation instructions are available on GitHub. The package can be installed with pip install gotabpfn. Reproducing the reported results requires tabpfn==6.3.1 for TabPFN-2.5 compatibility.

Impact statement GOTabPFN is intended for scientific and biomedical settings where samples are scarce but feature spaces are large. Deployment in sensitive domains should include careful validation, bias assessment, and domain-expert oversight.

Interactive Demo

A hosted Hugging Face Spaces demo is available so users can upload a tabular dataset and run GOTabPFN without setting up the environment locally.

Demo: Upload a CSV dataset, run GOTabPFN, and view the prediction results directly in the browser.

Open Hugging Face Demo

Code and Package

The GOTabPFN repository includes the main GOTabPFN implementation, standalone GO-LR feature ordering, four NSC compression variants, diagnostics scripts, and package test notebooks.

View the GitHub repository →

Citation

Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Kumar Gyawali, Gianfranco Doretto, and Donald A. Adjeroh. GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data. International Conference on Machine Learning, 2026.

@inproceedings{habib2026gotabpfn,
  title     = {GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data},
  author    = {Habib, Al Zadid Sultan Bin and Ahamed, Md Younus and Gyawali, Prashnna Kumar and Doretto, Gianfranco and Adjeroh, Donald A.},
  booktitle = {International Conference on Machine Learning},
  year      = {2026}
}