Abstract: A computer-implemented method for multi-modal interaction with structured web page data includes automatically injecting event monitoring code into a web page. One or more event streams generated by the injected event monitoring code are received. A request to perform an operation on data displayed on the web page is received from a user. A command is generated that has one or more parameters based at least in part on one or more mappings between the received request and the received one or more event streams. The command to fulfil the received request is executed by interacting with the web page.