Rep. Patrick Kennedy had decided not to run for re-election, ending the Kennedy legacy in the Senate and House after almost 60 years.
Kennedy says he wants to spend time building a family instead of building a life in politics.
The Rhode Island Democrat recorded a message for a Rhode Island television program on Sunday night.
"Now having spent two decades in politics, my life is taking a new direction, and I will not be a candidate for re-election this year," Kennedy says.
Kennedy was in a tough race against a well-financed Republican opponent, but the eight-term congressman was still expected to win. He spoke about his father's death in his remarks for Sunday.
"Illness took the life of my most-cherished mentor and confidante, my ultimate source of spirit and strength," he said, while a black-and-white photo of him as a boy sailing with his father appeared on the screen. "From the countless lives he lifted, to the American promise he helped shape, my father taught me that politics at its very core was about serving others."
Kennedy thanked the voters of his district for supporting him through episodes of alcoholic behavior and mental breakdowns.
"When I made missteps or suffered setbacks, you responded not with contempt, but compassion," he said. "Thank you for all the times you lifted me up, pushed me forward."