This is a major loss for the last-place Ravens (1-3), who are looking to build momentum after winning their first game of the season.
Smith has "some small micro fractures in his back," according to Harbaugh. There is hope that Smith will play after the Browns game, Harbaugh said.
"I think he's going to be OK in the long run," said Harbaugh, who was interviewed at the Michigan-Maryland game where his brother Jim was coaching.
The Ravens can't afford to lose Smith for an extended period because he's their only consistent playmaker on offense. Smith is Baltimore's leading receiver with 29 catches for 373 yards and two touchdowns. The other Ravens receivers have totaled 25 receptions for 292 yards and one touchdown.
Smith was called week to week earlier Saturday but has now been already ruled out for Week 5. His prognosis seemed a lot more positive earlier Sunday but is trending down. It's good news the team thinks he may be able to play beyond Week 5 but his situation is going to have to be monitored.