Shikai Qiu 裘释凯

Selected Publications

* Equal contribution

1. From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

   Marc Finzi*, Shikai Qiu*, Yiding Jiang*, Pavel Izmailov, J. Zico Kolter, and Andrew Gordon Wilson

   arXiv

   Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and fail to target the useful information content. We introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. We present practical procedures to estimate epiplexity which capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization.

2. Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales

   Shikai Qiu, Zixi Chen, Hoang Phan, Qi Lei, and Andrew Gordon Wilson

   Advances in Neural Information Processing Systems (NeurIPS) 2025

   Several recently introduced deep learning optimizers utilizing matrix-level preconditioning have shown promising speedups relative to the current dominant optimizer AdamW, particularly in relatively small-scale experiments. However, efforts to validate and replicate their successes have reported mixed results. To better understand the effectiveness of these optimizers at scale, we investigate how to scale preconditioned optimizers via hyperparameter transfer. We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of optimizers, including Shampoo, SOAP, and Muon, accounting for the impact of commonly used techniques such as blocking and grafting. We find that scaling the learning rate according to μP improves transfer, but can still suffer from significant finite-width deviations. Blocking and spectral normalization help address these deviations, and scaling weight decay inversely with width performs nearly optimally across optimizers. When these scaling rules are applied correctly, Muon and Shampoo deliver approximately 1.4x and 1.3x speedups over AdamW for Llama models ranging from 190M to 1.4B parameters.

3. Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

   Shikai Qiu, Lechao Xiao, Andrew Gordon Wilson, Jeffrey Pennington, and Atish Agarwala

   International Conference on Machine Learning (ICML) 2024 (Oral)

   What scaling limits govern neural network training dynamics when model size and training time grow in tandem? We show that despite the complex interactions between architecture, training algorithms, and data, compute-optimally trained models exhibit a remarkably precise universality. Specifically, loss curves from models of varying sizes collapse onto a single universal curve when training compute and loss are normalized to unity at the end of training. With learning rate decay, the collapse becomes so tight that differences in the normalized curves across models fall below the noise floor of individual loss curves across random seeds, a phenomenon we term supercollapse. We observe supercollapse across learning rate schedules, datasets, and architectures, including transformers trained on next-token prediction, and find it breaks down when hyperparameters are scaled suboptimally, providing a precise and practical indicator of good scaling. We explain these phenomena by connecting collapse to the power-law structure in typical neural scaling laws, and analyzing a simple yet surprisingly effective model of SGD noise dynamics that accurately predicts loss curves across various learning rate schedules and quantitatively explains the origin of supercollapse.

4. Compute Better Spent: Replacing Dense Layers with Structured Matrices

   Shikai Qiu*, Andres Potapczynski*, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson

   International Conference on Machine Learning (ICML) 2024

   Dense linear layers are the dominant computational bottleneck in foundation models. Identifying more efficient alternatives to dense matrices has enormous potential for building more compute-efficient models, as exemplified by the success of convolutional networks in the image domain. In this work, we systematically explore structured matrices as replacements for dense matrices. We show that different structures often require drastically different initialization scales and learning rates, which are crucial to performance, especially as models scale. Using insights from the Maximal Update Parameterization, we determine the optimal scaling for initialization and learning rates of these unconventional layers. Finally, we measure the scaling laws of different structures to compare how quickly their performance improves with compute. We propose a novel matrix family containing Monarch matrices, the Block Tensor-Train (BTT), which we show performs better than dense matrices for the same compute on multiple tasks. On CIFAR-10/100 with augmentation, BTT achieves exponentially lower training loss than dense when training MLPs and ViTs. BTT matches dense ViT-S/32 performance on ImageNet-1k with 3.8 times less compute and is more efficient than dense for training small GPT-2 language models.

5. Large Language Models Are Zero Shot Time Series Forecasters

   Nate Gruver*, Marc Finzi*, Shikai Qiu*, and Andrew Gordon Wilson

   Advances in Neural Information Processing Systems 2023

   By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

6. Simple and Fast Group Robustness by Automatic Feature Reweighting

   Shikai Qiu*, Andres Potapczynski*, Pavel Izmailov, and Andrew Gordon Wilson

   International Conference on Machine Learning (ICML) 2023

   A major challenge to out-of-distribution generalization is reliance on spurious features – patterns that are predictive of the class label in the training data distribution, but not causally related to the target. Standard methods for reducing the reliance on spurious features typically assume that we know what the spurious feature is, which is rarely true in the real world. Methods that attempt to alleviate this limitation are complex, hard to tune, and lead to a significant computational overhead compared to standard training. In this paper, we propose Automatic Feature Reweighting (AFR), an extremely simple and fast method for updating the model to reduce the reliance on spurious features. AFR retrains the last layer of a standard ERM-trained base model with a weighted loss that emphasizes the examples where the ERM model predicts poorly, automatically upweighting the minority group without group labels. With this simple procedure, we improve upon the best reported results among competing methods trained without spurious attributes on several vision and natural language classification benchmarks, using only a fraction of their compute.

7. Parton Labeling without Matching: Unveiling Emergent Labelling Capabilities in Regression Models

   Shikai Qiu, Shuo Han, Xiangyang Ju, Benjamin Nachman, and Haichen Wang

   The European Physical Journal C 2023

   Parton labeling methods are widely used when reconstructing collider events with top quarks or other massive particles. State-of-the-art techniques are based on machine learning and require training data with events that have been matched using simulations with truth information. In nature, there is no unique matching between partons and final state objects due to the properties of the strong force and due to acceptance effects. We propose a new approach to parton labeling that circumvents these challenges by recycling regression models. The final state objects that are most relevant for a regression model to predict the properties of a particular top quark are assigned to said parent particle without having any parton-matched training data. This approach is demonstrated using simulated events with top quarks and outperforms the widely-used \(χ^2\>\)method.

8. Holistic approach to predicting top quark kinematic properties with the covariant particle transformer

   Shikai Qiu, Shuo Han, Xiangyang Ju, Benjamin Nachman, and Haichen Wang

   Physical Review D 2023

   Precise reconstruction of top quark properties is a challenging task at the Large Hadron Collider due to combinatorial backgrounds and missing information. We introduce a physics-informed neural network architecture called the Covariant Particle Transformer (CPT) for directly predicting the top quark kinematic properties from reconstructed final state objects. This approach is permutation invariant and partially Lorentz covariant and can account for a variable number of input objects. In contrast to previous machine learning-based reconstruction methods, CPT is able to predict top quark four-momenta regardless of the jet multiplicity in the event. Using simulations, we show that the CPT performs favorably compared with other machine learning top quark reconstruction approaches.

9. Model-independent search for the presence of new physics in events including \(H→γγ\> \)with \(\sqrt s\) = 13 TeV pp data recorded by the ATLAS detector at the LHC

   ATLAS Collaboration

   Journal of High Energy Physics 2023

   A model-independent search for new physics leading to final states containing \(H→γγ\>\)decays is performed with 139 fb\(^{−1}\>\)of \(\sqrt s\) = 13 TeV pp collision data recorded by the ATLAS detector at the Large Hadron Collider at CERN. This search examines 22 final states categorized by the objects that are produced in association with the Higgs boson. These objects include isolated electrons or muons, hadronically decaying \(τ\)-leptons, additional photons, missing transverse momentum, and hadronic jets, as well as jets that are tagged as containing a \(b\)-hadron. No significant excesses above Standard Model expectations are observed and limits on the production cross section at 95% confidence level are set. Detector efficiencies are reported for all 22 signal regions, which can be used to convert detector-level cross-section limits reported in this paper to particle-level cross-section constraints.