The 'Argo' star admits his fellow actors have now changed the way they treat him since he began working behind the camera and he isn't let in on any secrets.
He said: "I'm not one of the guys any more. I remember I was walking over to the house guests and I heard one of the guys talking about getting stoned - I think somebody had brought something into the house - and as I arrived everybody clammed up.
"I thought, 'When did I become this guy? What am I, the dad?' I'm an asshole all of a sudden."
Ben - who has three children, Violet, six, Seraphina, three, and eight-month-old Samuel, with his wife Jennifer Garner - forced the six actors who play hostages in his spy thriller 'Argo' to live like they would've done in the 70s, taking away their luxuries and he admits they didn't enjoy the experience or his demands.
He told Shortlist magazine: "I took the six and made them live on the set for a week, gave them all the newspapers and the records and the magazines and their wardrobes, took away all their phones and closed the door.
"They were having bad iPod withdrawal. There was so much bitter complaint we had to have a production assistant standing outside with the phones in case of an emergency call.
"One of the guys, Tate Donovan, really wanted to take his yoga mat in. I said, 'The yoga mat is not going in. Your character didn't have a yoga mat.'
"He said, 'Yoga started in the 70s. Don't you know that?' I said, 'I don't give a s**t. You're Bob Anders who worked in the state department in 1979 and was 48 years old. I promise you, he wasn't doing f***ing yoga.' "