• Why is this happening? 

Government has never properly understood either (a) the essence of local, municipal government or (b) the unique character of the property development industry.  These fault-lines are quite unconnected, but their combination has played a lethal role in the emergence of our present difficulties.  Let me consider each in turn.

Constitutional, structural factors.  Since 1919, the UK has moved steadily in the direction of London-dominated centralised nation-state, suffering a long-term decline in the quality of its city, or municipal, government.  Before WW1, the trends were different: Westminster wanted no part in the drudgeries of local government.  But from 1919 onwards, central Government was drawn more and more closely into the management of mun1cipal life - with housing provision, highway expansion with the rise of the motor-car, and finally with town-and-country planning post WW2.  Rising politicians came soon to bypass municipal government, and aim straight for Westminster.

As a society, we simply lost our focus upon the growth and development of cities.  Continental and US city authorities remained powerful players upon their national stages, and many of them enjoy the fruits of that retained status.  In the UK, municipal government simply disappeared from political sight after WW2, increasingly the unimportant creature of the Westminster/Whitehall political machine.  After all, every municipal "Development Plan" had to be confirmed by central Government before it could come into force!  It was in this period (say, 1910/2000) that our city-building skills were lost.

If our great cities had been in the care of powerful municipal authorities throughout this period, the present shortcomings (in terms of commuting systems, supplies of residential development land) would not have arisen.  For these are the meat-and-drink of municipal politics, and their management demands intimate knowledge of local city systems.  It is central Government, faced with impossible task of managing 35-or-so city systems, which got it wrong.  And Scottish and Welsh devolution offers no solution: the invisibility of Edinburgh and Glasgow, of Cardiff and Swansea, in the devolution scheme is another tell-tale sign of the awful short-sightedness that has developed.  We have simply lost the skills of city development and government, in wave upon wave of over-centralisation.

The second failing is one of commercial ignorance.  The UK has a residential development industry that is unique in Europe; it has many parallels in the US, but not on the Continent.  From 1900 onwards, boosted by the remarkable success of the building society movement, there developed in the UK a distinctive residential development industry - which became the powerhouse of urban growth for the 20th century. 

A residential development industry is much more than a house building industry.  The "construction" industry has little in common with the development industry.  Construction companies are practised at building to order, manufacturing against plan, and being paid solely for the manufacturing process.  The the business of the "developer" is quite different: he must organise the provision of suitable land, decide what houses to build, get planning permission and arrange for their design and construction - and he will ordinarily not be paid at (except for a 10% deposit) until the whole process is complete.  To different these functions, the developer was sometimes known as a "speculative housebuilder" or spec-builder, although in this sense every car-manufacturer is a spec car-builder...

The developer must focus on the saleability, and the value of the completed items of real property (site/environment/accommodation) which he creates.  And he must ensure that his total selling-price is sufficient to cover all the costs of his protracted project.  "Construction" is a low-risk, low-profit industry: "property development" (both residential and commercial) is a high-risk, high-profit business.

The problem is that Government (both civil servants and politicians) is deeply ignorant of the requirements and dynamics of the property development industry.  That is why poor John Prescott continues to flounder, with successive duff ideas in the housing field.  "Construction" is well understood within Government, and among civil servants, and links with the construction industry are well-resourced.  But the "developer" is regarded as a suspect figure, parasitic upon the body politic.

I should know.  As a housebuilder, in the 1974/77 period, I was Industrial Adviser on Construction to the then Labour Government.  Working under the great Tony Crosland, it was my job to keep the Department of the Environment informed about the state of "private house-building" (another tell-tale term...)  My Civil Service colleagues, lacking any commercial experience, simply did not grasp the nature of business systems upon which their actions impacted.  During my two years, I did my best to explain, from my position as Under-Secretary, and I certainly had a positive impact upon the configuration of the great Development Land Tax Act.  But it was ultimately to no avail.  There was no change in the permanent advisory structure, there was no development of relevant expertise in the ways of the residential development industry.  After I left, the waters closed over my desk.

And to this day, Government flounders - in getting to grips with maintaining production rates in the housing market.  The industry itself is poorly organised, with a very ineffective "Housebuilders Federation" which has never escaped the ambiguities of its title, and is fearful of attracting political and social unpopularity.  Labour politicians have proved entirely impervious to any outside influence, even from the likes of me who are on their side.  Their Civil Service advisers remain ill-informed.  And Labour politicians still retain a glowering suspicion of the "spec house-builder" as a spiv and a conman, with whom they should never do business.  There is no communication going on.

That is why Labour's housing policy in in such disarray, panicking with huge land-releases, instead of being able to rely on the ordinary local processes of urban development, regressing to post-WW1 "prefabs" in an initiative which is bound to fail.