This article develops a human-robot cooperation to carry out treatments such as sanding, polishing, etc. on the surface of a known rigid object. For this purpose, a vision system is considered to get the object location to ensure not only the perpendicularity of the robot tool to the object surface but also a smooth approach of the tool to the surface. In order to add flexibility, the proposal includes the simultaneous combination of automatic and manual modes of operation. Thus, the human user can guide the robot tool to treat arbitrary areas (manual mode) and, when the operator releases the tool, the robot goes into the automatic mode to treat prior established areas. The method uses a task prioritization framework and three types of controllers: an admittance controller for the tool guidance; a hybrid controller to modify the tool orientation and, in the automatic mode, the tool position; and a sliding mode controller to limit the velocity at which the tool approaches the object surface. The applicability and efficacy of the proposed method is demonstrated experimentally using a conventional 6R robot arm.