NEW YORK, March 1 (Reuters) - A U.S. judge has sided with a trustee's "money-in, money-out" formula to determine claims for thousands of people swindled by Bernard Madoff, a decision likely to be appealed.

Trustee Irving Picard, who is overseeing the liquidation of the now defunct Bernard L. Madoff Investment Securities LLC, says investors' claims should be based on how much money they put into the firm minus how much they took out over the years.

U.S. Bankruptcy Court Judge Burton Lifland agreed, despite what he called compelling arguments.

"Equity is achieved in this case by employing the trustee's method, which looks solely to deposits and withdrawals that in reality occurred," Lifland said in a written ruling. "To the extent possible, principal will rightly be returned to Net Losers rather than unjustly rewarded to Net Winners under the guise of profits.

"In this way, the Net Investment Method brings the greatest number of investors closest to their positions prior to Madoff's scheme in an effort to make them whole," Lifland said.

Under that formula, some investors would get nothing, or even be liable to pay the trustee. Some Madoff victims proposed in court that they receive the amounts reflected on their Nov. 30, 2008, account statements, two weeks before Madoff's arrest and the collapse of his firm.

Madoff, 71, is serving a 150-year prison term after admitting to running Wall Street's biggest investment fraud, in the tens of billions of dollars, over at least two decades.

The unraveling of the fraud shook investors and confidence in market regulators such as the U.S. Securities and Exchange Commission, which missed the Ponzi scheme despite several investigations. A Ponzi scheme is one in which early investors are paid with the money of new clients.

"The court's ruling underscores how weak and confusing our investor protection laws are today," said Tom Matzzie, president of Accountable America, a group pressing for deeper reform in the financial industry. "Very little has changed from before the Madoff frauds until today."

Picard is working under the auspices of the Securities Investor Protection Corp (SIPC), a nonprofit established by Congress in 1970 to assist investors in failed brokerages. His worldwide search for assets has so far gained only about $1.5 billion, according to court records.

Lifland said he "recognizes that the application of the Net Equity definition to the complex and unique facts of Madoff's massive Ponzi scheme is not plainly ascertainable in law.

"Indeed, the parties have advanced compelling arguments in support of both positions."

A spokeswoman for the SIPC declined immediate comment. A spokesman for Picard had no immediate comment.

The judge noted in his opinion that more than 15,000 claims had been filed. Customer money totaled $73.1 billion as of Dec. 11, 2008, the day of Madoff's arrest. He said the net of "negative" accounts was approximately $8.3 billion, with customers purportedly owed a staggering $64.8 billion.

Madoff investors who challenged the trustee have previously said they would appeal any decision made in favor of applying his method.

Lawyers for investors argued at a Feb. 2 court hearing that Madoff victims could not have known they would not be entitled to $500,000 - the maximum allowed under the Securities Investor Protection Act to a single investor.

The case is Securities Investor Protection Corp v Bernard L. Madoff Investment Securities 08-01789 in U.S. Bankruptcy Court for the Southern District of New York (Manhattan) (Reporting by Grant McCool, additional reporting by Chelsea Emery; Editing by Tim Dobbyn)