(NicoElNino/Shutterstock)
To say the folks at VictoriaMetrics aren’t big fans of OpenTelemetry after implementing its metrics library would be putting it mildly. “OpenTelemetery is so complicated and bloated,” said Aliaksandr Valialkin, the CTO and co-founder of the observability software company.
OpenTelemetry is a Cloud Native Computing Foundation (CNCF) project that was created in 2019 to provide an open source solution to a thorny computer problem: to create a unified standard for the three main types of observability data, including traces, metrics, and logs.
With a single standard, the original OTel thinking goes, application developers will gain access to better tools to correlate different types of observability data with specific servers or virtual machines, thereby making it easier to track down and fix software problems. As a bonus, customers would no longer be locked into proprietary IT monitoring solutions.
OTel was formed by the merger of OpenCensus and OpenTracing, so it shouldn’t be surprising that tracing is the most mature of the three-pronged project, with the library tracing delivered in 2021. That was followed in May 2022 with the initial release of the metrics capability, while the general availability of OTel logging was finally delivered in November 2023.
Most of the IT monitoring industry, as well as the three cloud giants, have gotten on board the OTel train. From established vendors like Splunk and Dynatrace to newer outfits like Elastic and Sumo Logic, supporting OTel has been popular across the industry.
When some of VictoriaMetrics’s customers requested that the company add support for OTel metrics, the company obliged. Under the guidance of CTO Valialkin, VictoriaMetric developers added the OTel metrics library to its offering, which allows users to collect, stores, and analyze metrics coming from a range of different systems.
However, the integration with OTel metrics did not go as planned. According to Valialkin, the addition of the metrics library caused the VictoriaMetrics binary to increase in size by 50%, or several hundred megabytes.
“It’s a huge price for adding support for additional ingestion format, like OpenTelemetry,” Valialkin told Datanami. “Eventually, we re-wrote this library from scratch and implemented it basically solely for VictoriaMetrics. And we are happy with the end-result because it increased the binary size by less than 1%, and that’s much better than 50% with the official library for OpenTelemetry.”
The OTel library for metrics is so bloated and inefficient because it tries to do too much, according to Valialkin. It was designed by a committee to address the various metrics needs of multiple vendors. However, the end product ended up including a large range of “edge cases” that VictoriaMetrics customers will rarely, if ever encounter.
Most of the functionality included in the OpenTelemtery standard for metrics isn’t actually used by any vendors or customers, Valialkin said. “I think that most people do not deal with the low-level details of the OpenTelemetry protocol,” he said. “That’s why they are not aware of this issue which we faced when adding support for OpenTelemetry in VictoriaMetrics.”
(Shutterstock/AI-generated)
In short, OpenTelemetery for metrics is a “car designed by a monkey,” the company said.
“Our pain point was that our attempt to adopt this standard for metric congestion caused a lot of trouble,” said Roman Khavronenko, a VictoriaMetrics engineer. “It was really painful to do. It was expensive in terms of engineering hours. It was expensive in terms of quality of code.”
Others have filed similar compliants. A user on Hacker News last year said working with OTel was not easy.
“I find myself having to think orthogonally to common sense whenever I try to use one of its SDKs,” wrote user iofiiiiiiiii. “Nothing works the way you expect it to, everything has 3 layers of unnecessary abstraction and needs to be approached via the back door. Many features have caveats about when it works, where it works, how much it works, during what phase of the moon it works and how long your strings can be when Jupiter is visible in the sky.”
However, despite the pain, OTel users still see the value of the approach. Hacker News user iofiiiiiiiii stated that the open source project is “a partial win.” “While the developer experience is pretty unpleasant and I am also disappointed with the actual daily usage, from an architectural perspective it opens up new opportunities that did not exist before.”
Even though Valialkin and the VictoriaMetrics team have soured on OTel as a whole, the core idea–to standardize observability data–remains a good one.
“I actually like the idea of OpenTelemtery to unify observability into a single place in order to be able to collect and transform and process observability data in one place,” Valialkin said. “The idea is good. But the implementation is not very good. We’re not very happy with it.”
A better choice for metrics already exists, Valialkin said. Prometheus has a much smaller library, and its text exposition protocol is efficient, straightforward, and widely used. It’s already a de-facto standard, and would make a better official standard than what the OTel team came up with, he said.
“It’s widely used by many software companies,” Valialkin said. “It has good coverage for metrics. It’s easy to debug. It’s easy to implement. It’s easy to parse compared to OpenTelemetry. And it is not too complex. It is not bloated compared to OpenTelemetry.”
Alas, Prometheus is only used for metrics, and doesn’t support OTel standards for labeling, which is important for correlating with other observability data types. “It may be hard to switch between logs and traces and metrics when using Prometheus text exposition format for metrics collection,” Valialkin said.
VictoriaMetrics is moving forward with its support for the OTel metrics data standard, albeit through its own custom library. Even with the slimmed down custom library, ingesting OTel metrics data is slower and consumes more resources than any of the other metric data types supported by VictoriaMetrics.
“We have support of 10 protocols, or something like that, for ingestion,” Khavronenko said. “None of them is so complex.”
Meanwhile, VictoriaMetrics is moving forward with a new product: VictoriaLogs. The company is also adopting OTel’s log protocol for the product, and is girding for more of the same.
“I think that it shares the same complexity as the standard for metrics,” Valialkin said,” so that may be the main stumbling point.”
Related Items:
OpenTelemetry Gains Momentum as Observability Standard
Why Roblox Picked VictoriaMetrics for Observability Data Overhaul
Open Source Times Series Database VictoriaMetrics Sees Significant Growth