Project available at: ttr.guide
Recorded demo available at: https://youtu.be/5jQCix0P_fE
The Tools and Technologies Research guide (TTR.guide) project aimed to provide a comprehensive guide to job market analysis and provide valuable insights for both end-users and developers by leveraging the power of natural language processing (NLP) techniques. This open-source project utilised the GPT-3.5 Turbo OpenAI API to extract tools and technologies from job postings. The project followed an Agile methodology, which allowed for continuous iteration and improvement, while careful consideration of ethical, legal, and social aspects related to data handling and user privacy were also prioritised.
The TTR platform encompasses a backend with an API and a frontend with dashboard visualisations and API documentation. The data collection component has so far amassed over 60,000 job posts, primarily focusing on programming and engineering jobs sourced from reed.co.uk API. The GPT model was used for efficient data processing, capable of processing diverse and unstructured data. The system successfully processed three bursts of data, each containing about 10,000 job posts per second.
The TTR system's adaptability and modularity are key strengths, facilitating scalability and potential expansion. The frontend features a landing page with SEO optimisations, metadata tags, and branding, an interactive API documentation page, and a search and statistics page that delivers valuable insights to users. The TTR project's open-source future is secured by licensing it under the GPL-3.0 licence, encouraging contributions from other developers and researchers while fostering growth and contributions within the broader open-source community.
The TTR project's success was due to the effectiveness of GPT-based NLP for data processing, the Agile methodology's adaptability, effective data handling and processing, scalability, open-source nature and best practices, and careful consideration of ethical, legal, and social aspects. The project provided valuable learning experiences, including NLP and GPT techniques, software development, architecture of systems and project management, and ethical, legal, and social considerations.
Future improvements to the TTR platform could include dataset expansion, enhancing NLP and GPT techniques, additional features, routine maintenance and optimization, and collaborations with job posting websites. In conclusion, the TTR project represents a significant achievement in providing a valuable and adaptable tool for job market analysis, with the experiences gained and lessons learned throughout its development serving as a strong foundation for future projects and endeavours.