概要 - ステップ-3は、大きなものながら安価です：低コストなデコードのためのモデルシステム共同設計

ステップ-3は、大きなものながら安価です：低コストなデコードのためのモデルシステム共同設計

2025-07-25 16:53:13

{StepFun,:,"Bin Wang","Bojun Wang","Changyi Wan","Guanzhe Huang","Hanpeng Hu","Haonan Jia","Hao Nie","Mingliang Li","Nuo Chen","Siyu Chen","Song Yuan","Wuxun Xie","Xiaoniu Song","Xing Chen","Xingping Yang","Xuelin Zhang","Yanbo Yu","Yaoyu Wang","Yibo Zhu","Yimin Jiang","Yu Zhou","Yuanwei Lu","Houyi Li","Jingcheng Hu","Ka Man Lo","Ailin Huang","Binxing Jiao","Bo Li","Boyu Chen","Changxin Miao","Chang Lou","Chen Hu","Chen Xu","Chenfeng Yu","Chengyuan Yao","Daokuan Lv","Dapeng Shi","Deshan Sun","Ding Huang","Dingyuan Hu","Dongqing Pang","Enle Liu","Fajie Zhang","Fanqi Wan","Gulin Yan","Han Zhang","Han Zhou","Hanghao Wu","Hangyu Guo","Hanqi Chen","Hanshan Zhang","Hao Wu","Haocheng Zhang","Haolong Yan","Haoran Lv","Haoran Wei","Hebin Zhou","Heng Wang","Heng Wang","Hongxin Li","Hongyu Zhou","Hongyuan Wang","Huiyong Guo","Jia Wang","Jiahao Gong","Jialing Xie","Jian Zhou","Jianjian Sun","Jiaoren Wu","Jiaran Zhang","Jiayu Liu","Jie Cheng","Jie Luo","Jie Yan","Jie Yang","Jieyi Hou","Jinguang Zhang","Jinlan Cao","Jisheng Yin","Junfeng Liu","Junhao Huang","Junzhe Lin","Kaijun Tan","Kaixiang Li","Kang An","Kangheng Lin","Kenkun Liu","Lei Yang","Liang Zhao","Liangyu Chen","Lieyu Shi","Liguo Tan","Lin Lin","Lin Zhang","Lina Chen","Liwen Huang","Liying Shi","Longlong Gu","Mei Chen","Mengqiang Ren","Ming Li","Mingzhe Chen","Na Wang","Nan Wu","Qi Han","Qian Zhao","Qiang Zhang","Qianni Liu","Qiaohui Chen","Qiling Wu","Qinglin He","Qinyuan Tan","Qiufeng Wang","Qiuping Wu","Qiuyan Liang","Quan Sun","Rui Li","Ruihang Miao","Ruosi Wan","Ruyan Guo","Shangwu Zhong","Shaoliang Pang","Shengjie Fan","Shijie Shang","Shilei Jiang","Shiliang Yang","Shiming Hao","Shuli Gao","Siming Huang","Siqi Liu","Tiancheng Cao","Tianhao Cheng","Tianhao Peng","Wang You","Wei Ji","Wen Sun","Wenjin Deng","Wenqing He","Wenzhen Zheng","Xi Chen","Xiangwen Kong","Xianzhen Luo","Xiaobo Yang","Xiaojia Liu","Xiaoxiao Ren","Xin Han","Xin Li","Xin Wu","Xu Zhao","Yanan Wei","Yang Li","Yangguang Li","Yangshijie Xu","Yanming Xu","Yaqiang Shi","Yeqing Shen","Yi Yang","Yifei Yang","Yifeng Gong","Yihan Chen","Yijing Yang","Yinmin Zhang","Yizhuang Zhou","Yuanhao Ding","Yuantao Fan","Yuanzhen Yang","Yuchu Luo","Yue Peng","Yufan Lu","Yuhang Deng","Yuhe Yin","Yujie Liu","Yukun Chen","Yuling Zhao","Yun Mou","Yunlong Li","Yunzhou Ju","Yusheng Li","Yuxiang Yang","Yuxiang Zhang","Yuyang Chen","Zejia Weng","Zhe Xie","Zheng Ge","Zheng Gong","Zhenyi Lu","Zhewei Huang","Zhichao Chang","Zhiguo Huang","Zhirui Wang","Zidong Yang","Zili Wang","Ziqi Wang","Zixin Zhang","Binxing Jiao","Daxin Jiang","Heung-Yeung Shum","Xiangyu Zhang"}

{cs.LG,cs.AI}

ステップ-3は、ハードウェア意識のモデルシステム共同設計を通じてデコードコストの最適化に焦点を当てた321BパラメータのVLMです。以下の2つの主要な革新を紹介します： 1. マルチ行列分解注意力（MFA）：この革新的なメカニズムは、高い注意力の表現力を維持しつつ、KVキャッシュのサイズと計算を大幅に減少させます。 2. アテンション-FFN分離（AFD）：この分散的な推論システムは、アテンションとFFNレイヤーを専門のサブシステムに分離し、最適化されたパフォーマンスを実現します。 ### ステップ-3の主要な利点： * **大幅なコスト削減**：ステップ-3は、DeepSeek-V3やQwen3 MoE 235Bなどのモデルに比べて、特に長い文脈において大幅に低いデコードコストを実現します。 * **高い通過率**：Hopper GPU上での実装では、1GPUあたり1秒間に4,039トークンまでのデコード通過率を達成し、LLMデコードにおける新しいパレートフロンティアを設定しました。 * **ハードウェア効率**：ステップ-3は、ハードウェア統合の注意力算術強度、MoEのスパarsity、およびAFDの重要性を示しています。 ### モデルシステム共同設計： * **マルチ行列分解注意力（MFA）**：このメカニズムは、高い表現力を維持しつつ、アテンション計算に必要なパラメータの数を減少させます。 * **アテンション-FFN分離（AFD）**：このアプローチは、アテンションとFFNレイヤーを異なるサブシステムに分離し、最適化された並列性とパフォーマンスを実現します。 ### 実装と結果： * **StepMesh通信ライブラリ**：このカスタム通信ライブラリは、AFD用に特別に設計され、超低遅延とゼロSM使用を実現します。 * **パフォーマンス結果**：ステップ-3は、Hopper GPU上で1GPUあたり1秒間に4,039トークンまでのデコード通過率を達成し、同じセットアップでDeepSeek-V3を大幅に凌駕しました。 ### 結論： ステップ-3は、ハードウェア意識のモデルシステム共同設計が大規模言語モデルのデコードにおける高い効率とコスト効率を達成する可能性を示しています。そのアテンションメカニズムや分散的な推論における革新は、今後のLLM開発に貴重な洞察を提供します。