## 2014年1月12日星期日

### 群集集群

“快速混合算法（SGD和MCMC）尤其会遭受通信开销的困扰。加速通常是$n$的子线性函数$f（n）$，因为网络容量会在更大范围内减小（典型的近似值是$f（n）= n ^ \ alpha$对于某些$\ alpha<1$）。这意味着云计算的成本增加了$n / f（n）$倍，因为总工作量增加了该倍数。能源使用量类似地增加相同的因素。相比之下，单节点速度提高$k$意味着在成本和功耗上节省了简单的$k$倍。”

These HPC岛屿 do not need to stage all the data they are working on before they start doing useful work, e.g., SGD algorithms can start as soon as they receive their first mini-batch. 咖啡 and a single K20 can train on Imagenet 在 7ms per image amortized, which works out to roughly 40 megabytes per second of image data that needs to be streamed to the training node. That's not difficult to arrange if the HPC island is collocated with the HDFS cluster, and difficult otherwise, so the prediction is near the HDFS cluster is where the HPC岛屿 will be. Of course the HPC island should have a smart caching policy so that not everything has to be pulled from HDFS storage all the time. A 智能缓存策略将是任务感知的，例如，利用 主动学习 最大限度地提高HDFS和HPC岛之间的信息传输。