Data Intensive Infrastructures

The requirements to analyse and understand the inner workings and underlying fundamental concepts of the Web along with the elevated scalability requirements for the development of new Web infrastructures have demonstrated the necessity of Data Intensive Supercomputing (DISC) infrastructures to support researchers and developers. A prominent characteristic of DISC (as promoted by major players such as Google, Yahoo and MSN), as opposed to classic supercomputing, is the importance of very advanced data management software over high-end hardware configurations. DISC data entries are in fact known to be formed by hundreds or thousands of commodity machines connected using common commercial networking infrastructures. It is then up to the software to be able to deliver high capacity, throughput, scalability, re-configurability and fault tolerance. To be able to conduct credible research and development in the Web domain, DERI requires a DISC infrastructure. As DISC in itself is the subject of ongoing research, the goal of this work programme is to advance the research and applications of data intensive infrastructures, to create, maintain and offer to researchers a state-of-the-art DISC infrastructure, and to research novel algorithms and data structures for scalable handling of large amounts of semantic information. This work programme will draw on DERI’s previous work on large-scale semantic search engines (SWSE) and will be done in cooperation with the Irish Centre for High-End Computing (ICHEC).