The Commodity Futures Trading Commission on Thursday sued the crypto company owned by the billionaire Winklevoss brothers, saying it misled regulators as part of an effort to gain approval for bitcoin futures in 2017.

Gemini Trust Co. provided bitcoin pricing to Cboe Global Markets Inc., which launched a bitcoin futures product in December 2017. An auction run by Gemini determined how the bitcoin contract would settle on its final day before expiration, ensuring a tight link between the futures and the market for bitcoin itself.

The CFTC’s lawsuit amounts to a broadside against one of the crypto industry’s best-known brands, accusing the company of using undisclosed incentives to goose trading during an important period of the day. The alleged efforts to boost trading volumes weren’t revealed to the CFTC, the agency said, even as Gemini executives met directly with regulators to answer questions about the exchange’s operations and whether trading during the critical window could be manipulated.

Gemini misled the CFTC in 2017 about the way the auction worked, including whether traders would have to fund their bets fully or could borrow to do more trading, the CFTC said. The regulator sought to gather such information before the contract’s launch to ensure it wouldn’t be susceptible to manipulation.

The CFTC’s lawsuit against the operator of one of the biggest U.S. crypto exchanges shows how regulators are increasingly intervening in a market that has developed without the federal guardrails that govern U.S. capital markets. It is notable for targeting a cryptocurrency company that has advertised itself as a stalwart for regulation, saying it favors formal rules for the market.

The CFTC’s lawsuit seeks to impose a derivatives-trading ban on Gemini, as well as barring the company, its employees and owners from participation in derivatives markets overseen by the federal agency. It also seeks a fine and asks Gemini to pay back any illegally earned profits. Gemini said that it plans to fight the CFTC’s allegations in court.

“Gemini has been a pioneer and proponent of thoughtful regulation since day one,” the company said. “We have an eight-year track record of asking for permission, not forgiveness, and always doing the right thing. We look forward to definitively proving this in court.”

Before the CFTC filed its lawsuit, Gemini told the regulator that it disputed many of the allegations, saying the CFTC didn’t seek the information it claims was withheld, according to a person familiar with the dispute. Gemini also said the information was immaterial, meaning it wasn’t important to the regulator’s understanding of the auction process and the proposal to launch bitcoin futures.

The CFTC didn’t claim in its lawsuit that investors were harmed by Gemini’s alleged misconduct. And while the agency said Gemini employees knew or should have known that certain statements were misleading, it didn’t accuse them of intentional misconduct.

Bitcoin futures were hotly anticipated by traders in 2017, when the price of the world’s most valuable digital coin began to skyrocket. But Chicago-based Cboe pulled the plug on its futures contract in 2019 after its volumes lagged behind those of a rival bitcoin futures contract offered by Cboe’s crosstown competitor, CME Group Inc. A spokeswoman for Cboe declined to comment.

Under U.S. law, derivatives exchanges “self-certify” new futures contracts, meaning they don’t need to get an explicit green light from the CFTC before the launch. As part of the process, the exchanges must declare they have controls to guard against price manipulation. The CFTC has emergency authority to halt trading of new contracts, but it has rarely used that power. The system gives exchanges wide latitude to engineer new products to trade.

“Making false or misleading statements to the CFTC in connection with a futures product certification undermines the CFTC’s work to ensure the financial integrity of all transactions,” said CFTC Acting Enforcement Director Gretchen Lowe in announcing the lawsuit Thursday.

As part of its explanation for why its bitcoin auction wasn’t easily manipulated, Gemini told the regulator that traders would need to fund their positions fully. The CFTC wrote in its court complaint filed in Manhattan federal court, however, that a company controlled by two Gemini insiders made unsecured loans to traders “to facilitate trading on the Gemini Exchange including in the Gemini Bitcoin Auction.” The insiders are Cameron and Tyler Winklevoss, the person familiar with the matter said.

Gemini also advanced hundreds of thousands of dollars to traders to get them to participate in the auction, the CFTC said.

Gemini’s funding efforts conflicted with what it had told regulators, the CFTC said. Requiring traders to fund their activity fully would make “improper trading conduct more expensive to malicious actors,” the agency wrote in its complaint.

Gemini believes the funding claims are wrong because traders’ positions were always fully funded, regardless of where the money came from, the person said.

Gemini also claimed to the CFTC that it had measures to prevent self-trading, in which a firm acts as both the buyer and the seller in the same transaction, when in fact self-trading could occur at Gemini on occasion, the CFTC said. Self-trading can paint a misleading picture of the trading volume in an auction and lead to manipulative conduct, the regulator said.

Gemini insiders were sometimes dismissive of the risk of occasional self-trading, according to the lawsuit. Around November 2017, a month before the futures contract launched, a Gemini customer asked about a tool that would prevent a person’s buy and sell orders from crossing during the auction.

In response, the Gemini insider wrote in an internal message that big traders could police that risk for themselves. They are grown-ups, “they can figure it out,” the person wrote. The CFTC’s lawsuit didn’t identify the insider by name.

The lawsuit says two Gemini employees approved millions of dollars in trading fee rebates for certain participants who traded in higher volumes. Gemini had told the CFTC it didn’t offer special rebates to some customers but not others, according to the lawsuit.

It also didn’t reveal to the CFTC that it had fired the two employees, one of whom had met with the CFTC about the futures contract, according to the lawsuit. Gemini insiders believed at the time that the employees weren’t trustworthy, the CFTC alleged.

Write to Dave Michaels at dave.michaels@wsj.com and Alexander Osipovich at alexander.osipovich@dowjones.com