Social Development Minister Paula Bennett vows lessons will be learned from the Winz privacy breach.
But opposition parties say the minister is at the heart of the problem.
A freelance journalist, Keith Ng reported in his blog how he easily accessed thousands of sensitive and highly confidential files on the Ministry of Social Development's computer network in Work and Income offices.
Ng said he was responding to a tip-off about the flaw with the computers which are offered for job-seekers to send out CVs and apply for jobs online.
Just last week, Bennett was guaranteeing security of information on a planned Government database of up to 30,000 children at risk of being abused.
"Yeah, I am going to guarantee that because I'm going to have the right group of experts that help us develop it," Bennett said on Thursday.
But thousands of Winz clients might doubt the minister after this massive privacy breach.
"I am mortified, quite frankly, that anyone's information was accessed in this way. It should not have been, full stop," Bennett said today.
It is another very public privacy blunder for the Government.
The Accident Compensation Corporation debacle led to the demotion of Cabinet minister Nick Smith and the resignation of the corporation's chairman and chief executive.
Prime Minister John Key says Government departments will have to review the security of their computer systems.
"The department heads of all agencies that have client-facing systems, we'll be asking them to take a review of those systems and the security protocols," Key said.
Opposition parties say the failures start at the top.
Labour's Social Development spokesperson Jacinda Ardern says the list of security breaches in Government departments is growing.
"The minister needs to be held accountable on this serious breach because we now not only have ACC and IRD involved in serious breaches, we have MSD to add to that list. And this is probably the most significant of all of them," Ardern said.
Green Party co-leader Metiria Turei says the problem comes back to the minister and funding cuts.
"This is about the culture of Winz and it starts with Paula Bennett," said Metiria Turei, Green Party co-leader.
"It's also about the backroom cuts that have been made, funding cuts."
This gaping hole in security did not just allow access to information held by the MSD. There were also links to the Christchurch Earthquake Recovery Authority and the Families Commission. But the ministry is playing down the significance of that information.
"It's corporate information, straight corporate information," said Brendan Boyle, Ministry of Social Development chief executive.
Bennett is apologising for this "very significant mistake" but says it is too early to point the finger.