First of all, jobs have different weapon requirements, so you will need whatever number it lists in order to complete that job.
After that, weapons matter when you attack or are attacked by another person. In a fight, every person in your mafia can use 1 weapon, 1 armor, and 1 vehicle. If you have 501 people in your mafia, then you want to have at least 501 weapons, 501 armor, and 501 vehicles. After that, you want to get weapons that have higher attack (for when you attack someone else) or higher defense (for when someone else attacks you).