Abstract: A system and method for controlling vehicle operation based on a driver's facial expressions and physical state are provided. In particular, a system and method for differentiating between different types of emergency situations and applying an appropriate set of safety operations to ensure the safety of the driver, any other passengers, and others outside the vehicle are provided. With the system and method, facial expression recognition is used to distinguish between different types of emergency situations. For each of these emergency situations, a predetermined set of safety operations may be established. Once a particular emergency situation is detected based on the facial expression recognition, the corresponding set of safety operations are applied by sending appropriate control signals to vehicle control systems to cause the operation of the vehicle to be modified.