Show HN: PyLLMs: – Connect and compare top AI models in Python https://news.ycombinator.com/item?id=35556796

Show HN: PyLLMs: – Connect and compare top AI models in Python Hi HN, We needed a simple way to connect to the top AI models to experiment, prototype and evaluate them. Main features: - Connect to top LLMs in few lines of code (currenly OpenAI, Anthropic and AI21 are supported) - Response meta includes tokens processed, cost and latency standardized across the models - Multi-model support: Get completitions from different models at the same time - LLM benchmark: Eevaluate models on quality, speed and cost The benchmark uses predefine questions to test AI reasoning abilities across a range of "hard" queries. The outputs are then automatically evaulauted using a powerful model (gpt-4 recommended): https://github.com/kagisearch/pyllms/blob/990855968b4bc26ab6... This helped uncover a hidden gem among models: 'claude-instant-v1' which is 4x faster, 2x cheaper and similar quality to 'crowd favorite' gpt-3.5-turbo. https://github.com/kagisearch/pyllms April 13, 2023 at 08:18AM