Robert Donald William Farquharson, 40, will face a new trial next year after the Court of Appeal ruled that a Supreme Court judge, Philip Cummins, and the prosecution made serious mistakes about the evidence and reputation of a star witness.

But the judges said ''it was open to a jury acting reasonably to be satisfied of guilt'' beyond reasonable doubt.

Mr Farquharson was convicted of murdering his sons Jai, 10, Tyler, 7, and Bailey, 2, by driving them into a Winchelsea dam on Father's Day in 2005.

The prosecution alleged that the boys' deaths were a deliberate act of revenge by Mr Farquharson against his former wife, Cindy Gambino, over the break-up of their marriage.

Mr Farquharson denied killing his sons. He said he had suffered a coughing fit and blacked out, resulting in his car veering off the road and into the dam.

In a case that Victoria's Chief Justice, Marilyn Warren, described as containing issues that were ''complex, emotive and readily capable of being misunderstood'', the Court of Appeal ordered a retrial.

The case was largely circumstantial and key evidence came from a friend of the accused, Greg King, about threats Mr Farquharson had allegedly made at a Winchelsea fish and chip shop before his children died.

Mr King changed his story several times - each time changing the level of threats his friend had allegedly made - but eventually told police that Mr Farquharson had told him that he was ''going to pay her [Cindy] back big time'' and kill his children.

''Accident involving a dam where I survive and the kids don't,'' Mr Farquharson is alleged to have said. ''Something like Father's Day so everybody would remember.''

Police later got Mr King to wear a wire. The prosecution alleged that during these secretly taped conversations Mr Farquharson engaged in ''sustained manipulation'', trying to prevent Mr King from telling the police about his alleged threats. The jury was told it was evidence pointing to Mr Farquharson's guilty conscience.

Chief Justice Warren, Justices Geoffrey Nettle and Robert Redlich said it was essential for Justice Cummins to direct the jury that they had to be satisfied of the terms of the fish and chip shop conversation before they could infer Mr Farquharson was conscious of his alleged guilt.

The appeal judges found the prosecution had wrongfully failed to disclose that Mr King had been charged with recklessly causing injury and that police planned to provide a statement in his favour during his case.

Asked outside court if he had expected the dramatic result, Mr Farquharson's lawyer, Peter Morrissey, SC, smiled and said ''no''.

Cindy Gambino's parents said they were not surprised. ''It is just out of our hands, it is the judges' decision,'' Bob Gambino said.

The success of the appeal means Mr Farquharson will not have to pay $225,000 awarded in May to Mrs Gambino for pain and suffering.

with Kate Hagan