a new way to interact with your data in golang
Zenq is an internal DSL (Domain Specific Language) for golang. trying to provide a new way to work with various data-sources in an integrated environment. datasources can be of any kind, such as golang internal data-types (channels, slices) and async datasources like (csv,json,postgres db, mysql db).
zenq is inspired by C# linq and Java streams. for collection processing we have a dedicated query engine named thor, which uses operator fusion pattern to process the query-chain in a single execution unit. this approach can be very performant. our benchmarks shows that thor engine filtered a slice of 50,000,000 items in just 2 seconds.
an expressive query on collections using zenq:
Group[bool, ComplexObjectToSearch](
From(items).Where(func(search ComplexObjectToSearch) bool {
return search.Age > 20
}),
func(item ComplexObjectToSearch) bool {
return item.Flag
}
).Collect()
Polymorphic queries
but what about async datasources? After all the idea is that we provide A Polymorphic Query Language. the key to process async data-sources is to use the zenq stream api's. besides when we have a large data-set is that really practical to load all the data inside the memory and process them? and how can we query our data in different places and infrastructures? sometimes the data can be in-memory, other times its in a files (csv, json) and most of times its in a relational database.
luckily we provided a very capable Domain-Specific-Language for golang to be compatible with all these environments in an integrated way. the idea is that we use different kinds of adapters and zenq processing the data in a unified way.
1 - FromData to initiate stream from slices.
2 - FromChannel to initiate stream Channels.
3 - FromSqlRows to initiate stream an RDBMS (mysql,postgres).
4 - FromJsonArr to initiate stream from a json file.
5 - FromCsvFile to initiate stream from a csv file.
after initiating the stream, we can use zenq stream Pipelines to process your data.
A Real‑World Example of MySql Streams
Imagine a scenario with a large user base where you need to process users individually, such as validating each one against an external web service. Loading all records into memory is neither efficient nor scalable. Conversely, repeatedly opening and closing a database connection for every single row creates a significant performance bottleneck. With the new Zenq Streams API, you can initiate a stream using a single database connection to process rows iteratively, just like a cursor. This approach significantly reduces memory consumption and optimizes performance by eliminating unnecessary database round-trips.
the answer is in my github. study the readme.md file and search for
'A Real‑World Example of MySql Streams'.
visit the Repository and be surprised!