I just created an open source LLM observability named "TraceLM" that can track the following activities:
- token cost
- token length
- latency information etc
The following tool can be found here:
https://github.com/unnivm/tracelm-java-agent
This tools offers a very cool dashboard to track the above information. The user is required to login to access this dashboard.
This tool currently supports only Java based applications as it is basically working based on instrumentation technique. But, this tool will be extended ti support other programming languages as well.
This project is under development and am going to add more features to to so that the idea is to make it an enterprise ready product.