The 'Anger Management' star and the 35-year-old socialite have four-year-old twins Bob and Max from their marriage together - which ended in divorce over two years ago - and he is said to be worried for their safety if they are put under her care.
Brooke temporarily lost custody of her sons after she was placed on involuntary psychiatric hold following a drug overdose in May, and Charlie's second wife Denise Richards has been granted custody of until the end of the year, if his third wife can prove she can stay sober for their sake - but reports claim Charlie thinks her problems will prove too much to overcome.
An insider told RadarOnline.com: "[The twins] are so happy with Denise and Charlie is absolutely heartsick that Brooke is on track to get the boys back ... What are the odds she is going to stay sober?
"It's just not safe for the boys to be in her care. Does something tragic have to happen in order for a judge to strip Mueller of her parental rights?"
It was previously reported there will be a hearing in December in child dependency court, where a judge will decide whether Brooke should be allowed to be reunited with her sons.
A source had added: "She has impressed social workers from the Los Angeles County Department of Children & Family Services.
"Quite remarkably, given how low she has been, Brooke is now on track to get the boys back."