By Don Clark and Anna Prior 

Broadcom Corp. said it is exploring strategic options for its cellular baseband business, including a potential sale or wind-down, amid stiff competition among chip makers seeking to exploit the huge smartphone market.

The chip maker said it has engaged investment bank J.P. Morgan in connection with its plans, and noted that a sale or wind-down of the segment could result in a roughly $700 million reduction in annualized research and development, and selling, general and administrative expenses. Of those savings, $100 million relates to estimated reductions in stock-based compensation, the company added.

Broadcom's shares jumped 10% on the news, recently trading at $35.13.

The company, based in Irvine, Calif., has a large portfolio of wired and wireless-communications chips. It is particularly strong in Wi-Fi chips used in smartphones and other products, which its part of what the company calls its "connectivity" business.

But it never fared as well in baseband chips, which manage a phone's radio connection to wireless networks, and faces a growing number of competitors. Qualcomm Inc., in particular, built a dominant position by beating rivals to market with chips that can use the latest cellular networks that use a technology called LTE. Intel Corp. has also begun selling LTE chips.

Broadcom sought to boost its position in the business by paying $164 million last year to buy the LTE wireless chip assets of Japan's Renesas Electronics Corp.

Scott McGregor, Broadcom's chief executive, said on a conference call that it was a "hard decision" to exit the baseband business in view of the good products the company's engineers have developed.

But Mr. McGregor said it is hard to sell technology into the profitable high end of the market, which "really depends on a couple of customers." He didn't identify them, but Apple Inc. and Samsung Electronics Co. are the largest smartphone makers.

"We just didn't see enough traction there to really cover the cost of on the order of 3,000 engineers," Mr. McGregor said.

The low and middle parts of the market, meanwhile, have attracted lots of competition. "It surprised us a little bit how many people are still in that space or even entering that space," Mr. McGregor said.

But some others have come to similar conclusions. Texas Instruments Inc., once a major supplier of baseband chips to Nokia Corp., decided to exit the market.

Broadcom said it expects to reinvest roughly $50 million of savings from exiting the baseband business into projects in the broadband, infrastructure and connectivity businesses.

The company continued to back its revenue outlook for the current quarter of $2 billion to $2.1 billion, while noting that it expects product gross margin to be at or above the high end of its previous forecast range.

Mr. McGregor noted that Broadcom's baseband business required heavy investments in research and development. The company blamed a first-quarter operating loss for its mobile and wireless segment in part on those expenses.

Revenue for the whole mobile and wireless segment for the quarter fell 10% from the prior quarter to $846 million.

The company in April said its first-quarter profit fell 14% as Broadcom's revenue and product margin slipped.

Most analysts praised Broadcom's decision. But some noted the risk that the company could lose some future sales related to W-Fi chips for smartphones, since those may gradually be integrated on the same piece of silicon as baseband functions Broadcom will no longer offer.

"We would prefer the company to exit all wireless handsets related segments," wrote Ambrish Srivastava, an analyst for BMO Capital Markets, in a research note.

Write to Don Clark at don.clark@wsj.com and Anna Prior at anna.prior@wsj.com

Subscribe to WSJ: http://online.wsj.com?mod=djnwires