The authors have proposed a deskwork support system called an “Attentive Workbench” (AWB), which uses a camera, a projector, and automatically moving trays to support the user both physically and informationally. This research intends to build an interface between the user and the AWB system based on pointing gestures. Considering the specific purposes of the AWB, a simple, reliable, and highly responsive interface is implemented. We demonstrate a deskwork support system with actual automatically moving trays that uses the proposed interface.