Justice Secretary Chris Grayling has denied the prison system is in crisis as inspectors slammed the high level of violence at a Thamesmead jail.
The Cabinet minister acknowledged that the service is facing "pressures" and is having to cope with an increased number of prisoners, but insisted "we are meeting those challenges".
However, Prison Governors Association president Eoin McLennan Murray said staff shortages mean it is impossible for some jails to run a "safe, decent regime".
Last month, figures released by the Ministry of Justice painted a bleak picture of English and Welsh jails, with high levels of violence, including attacks on prison staff hitting the highest level for six years.
But Mr Grayling told the BBC: "We've actually got a prison estate where violence today is at a lower level than it was five years ago. We've got challenges from an increased population that was not expected in the last 12 months.
"We are meeting those challenges, we are recruiting more staff - but I'm absolutely clear there is not a crisis in our prisons.
"There are pressures which we are facing but there is not a crisis."
He said cost-cutting measures were based on proposals from the prison unions as an alternative to privatisation: "I accepted how we should do things, as recommended by our staff, and that is what we are doing ."
Mr Grayling's comments came as inspectors raised fears over levels of violence at Thamesmead's Isis - a jail with a high population of gang members, where a shortage of staff is highlighted as an issue.
In 2013, there were 254 fights and assaults at Isis prison, which holds young men aged 18 to 30, with 120 in the previous six months, Her Majesty's Inspectorate of Prisons (HMIP) said.
The report on Isis found that many of the incidents were serious, some were gang-related and a higher proportion than usual involved weapons.
Chief Inspector of Prisons Nick Hardwick said: "The location of the prison, the volatile population it holds and continuing staff shortages mean the risks and challenges it faces are significant.
"Opening any new prison is complicated and although Isis has now been open for three years it is still in a settling phase."
In late 2013, staff shortages led to a restricted emergency regime which, although intended to be temporary, was still in place at the time of the unannounced inspection in February.
Inspectors said the restricted regime had presented a temporary solution by keeping identified gangs apart.
However, the report added that this was "an unsophisticated plan and inadequate as a long-term solution".
However, inspectors did praise the high level of prisoners - three-quarters - who go into employment or training on release.
Michael Spurr, chief executive of the National Offender Management Service, said: " Isis manages a difficult population of young adult men - many of whom are serving sentences for violence and have links with London gangs.
"The governor and his staff have done some excellent work with the police to challenge gang affiliation and are committed to tackling and reducing violence.
"As the chief inspector makes clear, Isis is an improving prison. In particular, it is doing really good work to support resettlement. Three-quarters of prisoners go into employment or training on release - this is crucial for successful rehabilitation.
"Recruitment is taking place to address the staff shortfalls and the prison is currently receiving detached duty support from other prisons to ensure it can deliver a consistent and safe regime."