Norfolk Islanders with links to Pitcairn Island are angry at claims by the administrator, Eric Hutchinson, that they tampered with the locks to historic buildings on the island.
Council of Elders president, David Buffett, said one lock was changed and the descendants of the Pitcairn Islanders have every right to access these buildings.
He said the buildings - which are part of a World Heritage area - have huge significance to Pitcairn descendants and they fear Mr Hutchinson was preparing to lock them out.
"They were the actions being taken by the Commonwealth to prospectively exclude us from the area. They were endeavouring to say, incorrectly, that we had locked out a building that is used by the courts and that is absolutely incorrect," he said.
"There has been no action whatsoever to lock out the courts. The courts have had continuity."
One of the historic buildings, the OMB, the Old Military Barracks, is the site of a tent protest that has been ongoing since the controversial takeover of the island's administration by Canberra more than five years ago.
The administrator Eric Hutchinson told the local community radio that "further actions have been taken by the Department to secure the buildings to ensure access for future works and also for the general safety and accessibility of staff in the Old Military Barrack site."
And he went on to say "I think most fair minded people would say that it is not appropriate for people, particularly without permission, without having sought approval, to be tampering with, what are world heritage listed buildings, in a world heritage listed site, that are the responsibility of, in this case, the Commonwealth, to maintain and upkeep."
But the Council of Elders maintains they have special rights to the buildings and it is not for the administrator to tell them what they can and can't do at the historic site.