From April families deemed to have too much living space by councils will receive a reduced payment.
Mr Miliband said this was a “bedroom tax” which would affect those with disabilities unfairly.
Mr Cameron said is was a “basic question of fairness” because those in private housing did not get benefits for unoccupied rooms.
The change affects council tenants, and those renting from housing associations, who receive housing benefit, but it does not affect claimants who rent in the private sector.
Under the government’s so-called “size criteria”, families will be assessed for the number of bedrooms they actually need.
‘Not a tax’
Tenants in social housing will have their benefit reduced by 14% if they have a spare bedroom or 25% if they have two or more extra rooms.
Mr Miliband said the policy was “not just unfair, it’s not going to work either” because of a shortage of smaller council properties for people to move into.
He accused the prime minister of misunderstanding the point of social housing, which was there to “protect the most vulnerable”, adding that two thirds of people who would be hit by the change were disabled.
He also said military families with relatives serving abroad in the armed forces might also receive less.
Mr Cameron said there was a £50m fund to help the most vulnerable tenants and it was necessary to reform welfare to bring the benefits bill down.
“This is not a tax, this is a benefit,” he said. “Let me also make the basic argument of fairness that you [Ed Miliband] seem to miss.
“If you are in private rented housing and receive no housing benefit, you don’t get money for an extra room. If you are in private housing and do get housing benefit you don’t get money for an extra room.
“So there’s a basic argument of fairness: why should we be doing more for people in social housing on housing benefit than people in private housing on housing benefit?”
He said his party was “on the side of people who work hard and want to do the right thing”.
By BBC News