Are you passionate about bridging the gap between advanced AI and robotics? Join us at the Lincoln Centre for Autonomous Systems (L-CAS) to work on an exciting project that combines state-of-the-art Large Language Models (LLMs) with real-world robotics applications.
In this project, you are going to develop an automated pipeline to generate restaurant menu recommendations using multimodal LLMs and Retrieval Augmented Generation (RAG). The final goal is to implement this pipeline into a chatbot or physical robot that can interact with customers of restaurants, giving menu recommendations based on customers’ allergies or preferences (e.g., Italian food, spicy food, lunch, breakfast, vegan, etc).
A multimodal LLM approach will be explored to extract structured data from photos of menus (i.e., Optical character recognition) and then give suggestions using a local knowledge base that is accessed through Retrieval-Augmented Generation (RAG).
You are expected to complete the project by implementing the pipeline in a chatbot (i.e., as a software-only solution). However, human-robot interaction and robotic integrations are also within the scope of this project; thus, if possible, a physical service robot (Tiago from PAL robotics) can be used for testing the solution in a physical world setting.
Required Skills
- Strong programming skills in Python
- Keen interest in robotics and Large Language Models (LLMs)
- Good understanding of Artificial Intelligence
- Familiarity with Docker and GitHub tools
- Excellent communication abilities
Desirable Skills
- Experience with ROS2
- Familiarity with Retrieval Augmented Generation (RAG)
- Familiarity with multimodal LLMs
What We Offer
- Hands-on experience with cutting-edge AI and robotics technologies
- Opportunity to develop expertise in Foundation Models and Robotics Programming
- Access to state-of-the-art robotics facilities
- Integration into the dynamic L-CAS research team and LCASTOR robotics competition team
This is an internship position suitable for students pursuing a programme of study in computer science, robotics and/or AI at the University of Lincoln. If you are interested, fill out our Expression of Interest Form, choosing Dr Leonardo Guevara (lguevara@lincoln.ac.uk) as the researcher to supervise the project.