The work, which depicts a child with a crowbar next to a sandcastle and paving slab, appeared on the side of an old shop in the town in August.
Workmen were at the scene on Sunday cutting it out of the wall.
The former electrical shop was on the market for £300,00 when the work appeared, but was re-listed for £500,000 after Banksy confirmed his East Coast "spraycation".
The elusive street artist claimed responsibility for 10 pieces of work in Suffolk and Norfolk during the summer.
Taxi driver Rob Hook has been watching the builders at work in London Road North and Regent Road.
"Yesterday they put a lintel in and cut the wall above it - now they've [strengthened] both sides with wood and framing," he said.
"Now they're busy cutting the wall around it... the whole section of wall with the mural on it."
In a statement, East Suffolk Council said: "We have spoken with the landlord of this property who has confirmed the artwork is being removed."
It said it did not have any further details about the matter.
The artwork, along with three other murals near the railway station, in a park and on the beach, has drawn crowds to the town.
The council had covered it with a Perspex screen after Banksy confirmed it was genuine.
Miles Barry, chairman of Easterly Artists, said he was concerned about its removal and final destination.
"If it's being removed to be displayed in a more prominent location within the town in an artistic context then that's lovely," he said.
"If it's because the owner realised having it in location adds £200,000 to the value of the property but having it for auction… it could possibly fetch a great deal more - then that's a different situation."
"It's a great shame - it's not just about the painting on the wall... there's a social comment there about resources."
The BBC has attempted to contact the landlord of the building for comment.