All the boys in the neighborhood loved me but were afraid to express it. 

Because they were afraid of my father and not of me. 

because my father used to drink alcohol 

The boys were afraid that my father would enter their house drunk and kill them.