Spec-driven development is extremely useful, and that's what planning is done properly. Although, at least for now, plans have to be relatively small and digestible.
The problem with specs-as-source is that if the spec was sufficiently complete it would (almost) be the source code itself.