"It's not like that "
The Professor isn't exactly what everyone who thinks they know him seems to believe. They see a nice old man helping little kids, all happy la la fun. Really, it's a front. Edward Oak has a past he prefers to keep hidden. At one point he was in love with two women, and he married the one who pushed for marriage sooner. His wife was Maria, the other was Agatha. When Agatha called him soon after the marriage for a 'wild night', he headed over, completely ignoring the fact that he was married.
The result? Geoffrey was born of Agatha.
Edward had been crawling with shame, but Agatha had promised to keep quiet if Ed took the kid. He did, and was surprised when Maria stayed. She left, however, soon after giving birth to their son, Giovanni.
After Maria left, he took care of his sons... to an extent. He began drinking sometime when Geoffrey was in high school, and got dangerously far into alcoholism by the time Geoffrey left.
Once he was alone, he began to try and reform. He had no one with him, and therefore had to work for himself alone. It was a long path, but he got a long way down it, trying to keep the past away.
In Ed's defense, he's just as confused as both of his sons. Especially Geoffrey...