Abstract: An image processing apparatus for processing a picked-up image which is output from an image pickup element in which a microlens array for obtaining the picked-up image including ray directional information of an object image formed by a photographing optical system has been arranged has: a refocus processing unit for executing a refocus processing of the obtained picked-up image on the basis of the ray directional information and generating a reconstruction image; and a defect pixel detection unit for detecting a defect pixel from the obtained picked-up image. When the picked-up image is obtained, the detection of the defect pixel of the obtained picked-up image is performed. When a refocus processing is instructed, the refocus processing is executed to the picked-up image in which the detection of the defect pixel is performed.