Duct friction depends on how much wall surface the air has to rub against for a given cross-sectional area, and that is a property of the shape.
Why round wins on paper
A circle has the smallest perimeter for a given area of any shape.
Less perimeter means less wall surface per unit of airflow, which means less friction. Round duct also uses less metal for the same capacity, is easier to seal, and is stiffer for its weight.
For a given pressure drop, round duct carries more air than any rectangular duct of the same cross-sectional area.
Equivalent diameter
Rectangular ducts are sized by converting them to the round duct that has the same pressure drop at the same airflow — the equivalent diameter.
That conversion is what duct calculators and Manual D tables provide, and it is why a rectangular duct's dimensions cannot be reasoned about directly. A 16 by 8 and a 12 by 11 differ in equivalent diameter despite similar areas.
The aspect ratio penalty
The flatter the rectangle, the worse it performs relative to its area, because a flat shape has far more perimeter per unit of area.
Design practice keeps the aspect ratio below about 4:1, and closer to square is better. Beyond that the duct needs more area, more metal and more support to carry the same air, and it is noisier.
Very flat ducts — the kind used to fit between joists — are sometimes the only option, and they should be sized knowing they are paying a penalty rather than by area.
Where rectangular is used anyway
Between floor joists, where the depth available is fixed.
In a soffit or a dropped ceiling of a given height.
As a main trunk in a basement, where a wide flat trunk fits under the joists and a round one of equivalent capacity would hang lower.
At the equipment, where plenums are almost always rectangular.
These are real constraints, and rectangular duct exists to meet them.
Round in a rectangular space
Where the depth allows it, an oval duct is a compromise: better than a flat rectangle for a given depth, and it fits where a circle does not.
Flat oval and spiral round are both available in the sizes residential systems use, and they are worth asking about when a very flat rectangle is being proposed.
Fittings matter more than shape
A round system with sharp elbows and bare stub takeoffs will perform worse than a rectangular system with long-radius elbows, turning vanes and conical takeoffs.
Turning vanes in a rectangular elbow reduce its equivalent length substantially and cost little. They are standard practice in commercial work and frequently absent in residential.
Panned joists and stud cavities
Using the space between floor joists or inside a wall cavity as a duct, closed off with a sheet metal pan, is common in older houses.
It is not a duct. Those cavities leak into the framing, into wall assemblies and often into attics and crawl spaces, and the leakage is frequently very large. On the return side, that means drawing unconditioned air directly into the system.
Where they exist and can be reached, lining them with actual duct or replacing them is one of the higher-value corrections available.
What to specify
Round where it fits, rectangular where it must, aspect ratio kept moderate, good fittings everywhere, and every joint sealed with mastic regardless of shape.
Noise
Rectangular duct has larger flat panels, and flat panels drum. A wide flat trunk can transmit blower noise into the rooms below it in a way a round trunk of the same capacity does not.
Cross-breaking the panels, adding stiffening beads, and keeping velocity within design limits all reduce it. So does a flexible connector at the equipment, which stops blower vibration entering the duct in the first place and is sometimes omitted.
Cost
Round duct uses less metal for the same capacity, and rectangular duct takes more fabrication.
In practice the cost difference on a residential job is modest and the choice is driven by the space available. Where both fit, round is the better technical answer and usually not the more expensive one.