Want to improve this question? Add details and clarify the problem by editing this post.
I am researching the feasibility of large distributed training runs, and one question that came up is whether interdatacenter network bandwidth would be a meaningful constraint.
In particular, training a 600T parameter model might require communications taking 50ms or less, corresponding to 600T X 8 bit/ param / 50ms = 100Pbs bandwidth. My understanding is that this is unprecedentedly big, so i would want to get a sense of whether this would be physically feasible in the first place, and what would the cost be if so.
My current point of comparison is the MAREA transatlantic cable, which has 200 Tbps system capacity and cost $165m. A shorter, land-based cable should cost less, and given that a large part of this will be fixed costs, I expect a 100 Pbps, 200 miles system to be possible for $165m x 100 Pbps / 200 Tbps ~= $80 bn, just from building a system with 500x fiber pairs.
Is this very off? Is this feasible in the first place? And what other strategies could one use to increase bandwidth beyond building more cables?
