If a town could have a resting heart rate, Sag Harbor’s, on this balmy mid- August evening, wouldn’t even break 45. The Fire Department Summer Carnival is in full swing—a three-piece band on the little beach opposite Long Wharf singing “What a Wonderful World” with a lilting reggae beat. The crowd of 25 or so consists of just folks, no Chloé, no Choo, no Miu Miu. On Main Street, kids are streaming out of LT Burger licking ice cream cones, and the window of the 5-and-10 holds an artless display of back-to-school supplies, a reminder that this little tableau of summer is on the wane. Sure, the shop fronts are more than speckled with boutiques, but somehow Main Street has retained a Walker Evans-and-Woolworth’s feel, a stalwart, come-as-you-are confidence. The sign on Route 114 says “East Hampton 7.” Is that miles or light years?

Either way, iconic people come from that galaxy to this one to eat at Sen Restaurant, the 19-year-old Japanese restaurant on Main Street, a few doors harbor-ward from The American Hotel. Kelly Ripa and Mark Consuelos are regulars. Alec Baldwin, another frequent guest, feels right at home at Sen. “He’ll sit down between two couples and just start a conversation,” says Jesse Matsuoka, the charismatic and articulate general manager of Sen and the adjoining Phao, which together constitute an Asian culinary archipelago on Main Street. Billy Joel is a sushi-bar habitué at lunch, and Calvin Klein usually sits wherever there is a large table because he enters with an entourage.

In keeping with Sag Harbor’s even keel, this celebrity moon dust doesn’t faze its denizens, with one exception. When Sir Paul McCartney—as Matsuoka is careful to say—has come from Amagansett, the restaurant’s pulse rate spikes. “We have to pull out all of the stops to hide him.” Which means making sure McCartney is in an out-of-the-way locale. “People go crazy. They see Billy Joel, they say, ‘It’s Billy Joel.’ But McCartney, they push through like it’s football.”

These avatars of style come to a dining room that is more sturdy than svelte. It’s a low-ceilinged space with a slightly dusky feel and a dash of rural Japan in the stout 3x5 beams that slant down above the sushi bar and kitchen. And there, beneath them, head down, face a mask of concentration, is the person who brings them here: Kazutomo Matsuoka. He’s the father of Jesse and his brother, Tora, who owns the restaurant along with Jeff Resnick.

In Japan, Kazutomo was a maegashira, a top-ranked sumo wrestler in the Makunouchi division. “Sumo wrestlers spend half of their life fighting and training and half of their life cooking and eating,” says Tora. Kazutomo cooked for sumo wrestlers on his way up the ladder, they cooked for him when he reached the summit of the sport, and when he retired, he followed a well-trodden sumo path, cooking for others in a Japanese restaurant.

The classic emphasis of the Sen menu today—tempura, teriyaki, yakitori, shishito peppers, they’re all here—is Kazutomo’s legacy. Then there are the new classics such as panko tofu, a lightly fried plinth of tofu (like a latke), crowned with eggplant dengaku and grape tomatoes and surrounded with two thin circles of sesame tofu vinaigrette; it’s a house favorite.

The fish is top quality—of course. Those wispy thin slices of fluke in the usuzukuri—usu means ‘thin’; zukuri means ‘style’—were swimming that same morning. It comes with precisely cut dashes of scallion and a paste of spices they call “red snow” (grated daikon mixed with traditional Japanese spices), which you wrap in the fluke and dip in citrus soy. The local oysters taste clean and slightly briny. Some of the fish arrives overnight by FedEx from the Honolulu Fish Company, while almost all of the rest comes from two highly regarded East End suppliers, Braun Seafood Co. and Gosman’s Fish Market. And that little puff of green on the salmon teriyaki? Those are microgreens—shiso and daikon sprouts—from Good Water Farms that have been grown to order for Sen.

But Tora offers a surprisingly candid take on the piscatorial profusion. “Everyone at our level is buying the highest quality fish,” he says. “The taste of our sushi—our secret weapon—comes from the sushi rice and the sushi rice vinegar.” The latter is known as shari-su in Japanese, and it is Kazutomo’s own recipe. “He beat it into my head as a kid,” says Tora, adding that his father turns up two or three days a week to make the sushi rice and the rice vinegar himself. (He’s also at the restaurant two or three evenings a week.)

What’s interesting on this Wednesday August evening is how fast the dining room fills up. By 7 o’clock, it’s full tilt, proving true what everyone told me about Sen: “You have to get there early.”

Tora and Jesse run the show now, but they, too, are Kazutomo’s legacy. For they didn’t just waltz into the job—in fact, they didn’t even start as busboys. They started by sweeping the basement floors and cleaning Dumpsters; they had to earn the job in the sumo manner. “He was very adamant that we started at the bottom,” says Tora, and as a result, he and Jesse “understand everything in the restaurant and how it works together.”

It’s a story of grit transmuted into character. How refreshingly old-fashioned—just like Sag Harbor itself. 23 Main St., Sag Harbor, 725-1774