Berkeley Johnson said the two-year-old girl was taken to hospital and suffered an injury to her hip.
Rescue efforts are under way in Santa Barbara County for residents trapped by mudslides that overwhelmed an area scorched by wildfires last month.
At least 15 people have died and more than two dozen others were injured.
"I don't know how the baby survived," Mr Berkeley told the Santa Barbara Independent newspaper.
"This was an hour of rain, and the house was gone," he said of his home in the community of Montecito.
Mr Johnson said he and his wife, Karen, heard a baby crying after the flooding had subsided and they managed to climb down from the roof of their swamped home.
The pair joined a fireman to dig the toddler out, scooping mud from her mouth before she was taken to hospital.
"Had we not gone over there, I don't think that kid would have [survived]", he added.
More than 50 people have been rescued already but many places are still inaccessible. Several roads are closed, including the major Highway 101.
Some 163 people have been taken to hospital. Twenty had "storm-related injuries" and four were critically hurt.
The upmarket neighbourhood includes homes owned by celebrities such as actor Rob Lowe, chat show host Ellen DeGeneres and media mogul Oprah Winfrey.
Ms Winfrey said on Instagram that her property sustained only minor damage as she toured the scene.
The first rain in months caused mudslides when it hit ground that had been burned by December's huge wildfires.
After a wildfire, burned vegetation and charred soil create a water repellent layer which blocks water absorption. Together with the loss of vegetation, this leads to an increased risk of mudslides and floods.
The Federal Emergency Management Agency (FEMA) says the risk of flooding stays "significantly higher" for up to five years after a wildfire.
"Recent burn areas will be especially vulnerable where dangerous mud and debris flows are possible," said the National Weather Service in a statement.
In some places mud was waist-deep, officials said.