Most current multi-touch capable interactive user interfaces for tabletop are built from custom toolkits that are decoupled from, and on top of, the "Desktop" provided by the underlying Operating System. However, this approach requires that each individual touch system build their own suite of touch capable custom applications (such as photo browsers), usually resulting in limited functionality. In this paper, we propose a software architecture for supporting and integrating multi-touch capability on existing desktop systems, where multi-touch and multiple single pointer input can be used simultaneously to manipulate existing application windows. We present an example implementation of this architecture on the Linux Operating System, demonstrating the possibility of using touch displays in a collaborative work environment on the tabletop. The paper concludes with lessons learnt and technical challenges from our experience.