Rendered at 16:10:45 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
rtpg 16 hours ago [-]
I'm a bit worried this is intro'ing the classic problem we have in Bazel land, where everyone is having to show up with their own sort of packaging scripts etc instead of using upstream tooling one way or another.
I had the impression `zig` already has stuff like `zig cc`. Would... would `zig make` be an impossible proposition? Maybe that makes no sense.
kingstnap 16 hours ago [-]
zig make is zig build.
I haven't kept my zig skills sharp enough but zig build and the build.zig file basically lets you create a dag of steps and create various targets. It's basically make but the programming language is zig itself and the associated code happens to be in the standard library.
There are just too many C/C++ build systems out there. If you wanted to cover a majority of projects in the wild, you'd also need `zig cmake`, `zig automake`, `zig ninja`, `zig msbuild` and more. Not worth it imho.
pornel 1 hours ago [-]
And it's not even a matter of supporting a build system. Each build script for each build system is its own snowflake with its own custom method of finding or building its dependencies, its own method of configuring optional features, etc.
Sorting that out and unifying it for every project is basically the tedious job that Linux distros do.
Defletter 15 hours ago [-]
This is something that bothers me about languages that proclaim C/C++ compat because it's almost exclusively just talking about the ABI. I will say that Zig's build system is definitely one of the better ones, just look at the sqlite3 repo: it just points to the amalgamation zip and compiles it statically, meaning you can compile it with whichever flags you want, knowing it's guaranteed to work exactly how you want. Even though I personally don't use sqlite, I cannot help but appreciate the simplicity here, whereas other systems languages with C/C++ compat (eg: D) cannot do this.
With that said, Zig falls well short when dependencies have more involved build systems that are too obnoxious to recreate, eg: libevent. I have a bit of a sentimental attachment to libevent and would love to use it within my Zig projects, but the only viable options are: 1) dynamically link; 2) have two [or more] separate build steps; or 3) wait until allyourcodebase does the obnoxious work for me. None of which are appealing so I end up using different libraries just for the build-system integration. And by their nature of being Zig libraries, they're usually less known, less battle-tested, and riskier to use because it's more likely to be some random guy's side project.
Putting aside the StackOverflow tier "why would you ever do/want that?" troll argument, the issue is that I'm being obstructed from using the libraries I want to use over build system incompatibilities. It feels somewhat similar to the complaint about having two [or more] languages within the same codebase.
nvme0n1p1 14 hours ago [-]
Yeah, I tend to bail out and just use @cImport and linkSystemLibrary, for exactly this reason.
skywal_l 11 hours ago [-]
RIP @cImport.
ghthor 4 hours ago [-]
I think Nix can help you here; as you can delegate all the make/cmake/ninja handling to Nix package derivations and you final package can consume a bunch of .a files from them.
peesem 15 hours ago [-]
you _can_ just tell zig build to call make (std.Build.addSystemCommand, https://ziglang.org/documentation/0.16.0/std/#std.Build.addS...). the only benefits of an inbuilt make would be not having to install it and that it could maybe intercept calls to external compilers and replace them with `zig cc` or something similar but i get the impression that supporting nontrivial make scripts would be very hard
rtpg 14 hours ago [-]
so then my glib question: what is the purpose of this project?
Efforts to pull out information from existing build systems (if possible, ofc) means that you don't have to spend your time _rewriting build scripts_.
I don't know if it's really possible. Maybe this is step .. 4 of 34 to getting there. Just wondering if there's an alternative here that doesn't involve rewriting build scripts
Maybe a part of the pitch here is to just upstream these build scripts so that ones makefile becomes "just" `zig build`? In that case I'd actually like to see (for example) a fork of oyacc using what is figured out in https://github.com/allyourcodebase/yacc to "just" use the build script even in the higher level project.
flohofwoe 9 hours ago [-]
> so then my glib question: what is the purpose of this project?
The whole point is to massively simplify integrating those libraries into Zig projects by explicitly not having to deal with external build systems or separate C/C++ compiler toolchains.
> Is the grpc makefile no bueno?
Does it work out-of-the-box on Windows? That's usually the first question when encountering a Makefile (and just looking at the first lines of the Makefile, it doesn't, instead it needs to run inside mings: https://github.com/grpc/grpc/blob/a63b3a4d949a7057a0e5443e7a...).
With Zig you usually only need the Zig toolchain, which runs just fine in a regular Windows cmd.exe.
gravypod 4 hours ago [-]
> With Zig you usually only need the Zig toolchain, which runs just fine in a regular Windows cmd.exe.
It's amazing how complicated we made things with bad software engineering and how good we can make things with good software engineering.
peesem 14 hours ago [-]
i do like zig and its build system, but i agree that i see little point to the allyourcodebase repositories. you can probably learn about what you _can_ do with the build system through them, but i don't think they're best practice. they just redefine the build graph either statically (harder to update) or through path generation with python or something of the sort (what's the point, it's not contained in the zig build system anymore). i also remember some repositories getting quite outdated, though it seems there's been more effort to keep them updated these days.
the one thing i will blame zig for here is the refusal to make some system to just include all of the files in a directory (hence allyourcodebase's python scripts). it makes sense with zig's import & module system but i think it would probably be a big blocker for some projects that they'd have to manually list out every single C file in their source.
you can currently use the filesystem apis to manually walk through files in your build script, though this is discouraged and will go away with upcoming change of isolating the build script
forrestthewoods 15 hours ago [-]
> classic problem we have in Bazel land
Can you tell me more about this? I’ve never used Bazel. I have used Buck extensively. But have never used either in an open source context.
rtpg 15 hours ago [-]
there's a wonderful blog post about this that I wish I could find and can't. I feel like it was a github gist.
The gist of it is something like:
- Bazel community hits a problem with something like how npm behaves around paths
- Bazel's solution is to create a patching system around paths to resolve them
- rinse and repeat for every piece of tooling us "normies" are using
The end result is that instead of Bazel's efforts showing up in the base tooling, they build up a bunch of their own tooling and patches, and the underlying tools don't get really much if any benefit. Moving to Bazel still is a PITA, and there's a lack of improving onramps
In an alternate universe when people in Bazel land hit issues they would show up with patches and recommendations _to the underlying tools_ to get things merged in so that the workflows can be better supported across the board.
I don't think anyone working on Bazel is against upstreaming, but I think they discount the value of it because, hey, _they_ can get patches and hacks working for their flow
forrestthewoods 14 hours ago [-]
Ah interesting, thanks.
My intuition is that npm, cargo, zig build etc are all “wrong”. And that Bazel/Buck are architecturally correct. Build systems should be polyglot by default! All these build systems and package managers that are per language are wrong.
But unfortunately Bazel and Buck have a decade plus of tech debt and baggage from their corporate overlords. Internal buck is actually mostly nice. I don’t know why anyone ever use it externally.
rkangel 7 hours ago [-]
> My intuition is that npm, cargo, zig build etc are all “wrong”. And that Bazel/Buck are architecturally correct.
Arguably yes. The fundamental issue is that Bazel etc. are "hermetic" build systems - every build step they run is in a sandbox with only the inputs (source files and tools) that you have explicitly specified - nothing implicit everything explicit. And relies on the output you get from those inputs being the same every time.
This gives a wonderful property of knowing exactly which steps need performing and being able to cache everything - your whole build DAG is a merkle tree.
The downside is that many build steps do not fit into that mould - they make arbitrary network calls, or want to be able to access corners of your filesystem, or produce slightly different outputs each time due to multithreading etc. So you need to work around this in some way to make the larger system work.
(FYI, this is exactly the same as the Nix and Guix build systems except that for them the granularity of a step is "build one package" rather than "build one C file")
rtpg 14 hours ago [-]
It's not about the package managers not being polyglot. It's about, for example, witnessing that `pip install` installs binaries with absolute paths and then writing a patch _for pip_ that Bazel uses, instead of figuring out how to change upstream pip to make that a configuration bit that Bazel uses.
Bazel uses npm/cargo/pip in most people's usage of these tools! People still use these packages managers, with random patches and tweaks (because the overall ecosystem assumes you're using them). But Bazel often includes random fixups in their own tools.
For example [0] is some patches Bazel does to node to get it to play nicely witH Bazel. In an (IMO) better world Bazel people figure out what they need, and figure out what they can get upstreamed into node to get things working.
Instead we have patches that (if you jump through the history) have been around for 6+ years
This comment gives off the impression that bazel just wraps cargo/pip/npm invocations, which is incorrect (at least in the commonly used scenarios)
First of all, bazel does not handle any of these languages/ecosystems itself. Such logic is delegated to language-specific rulesets, of which there are multiple implementations with different tradeoffs.
While it's true that bazel allows to patch all dependencies on the fly, in my experience many Bazel users do try to contribute back, because managing stacks of hundreds of patches across all your dependencies isn't particularly fun.
forrestthewoods 13 hours ago [-]
> Bazel uses npm/cargo/pip in most people's usage of these tools!
Oh interesting. Buck totally replaces cargo / pip. I think the JS story is a little messier. Not sure as I try my best to not touch JS.
dzbarsky 4 hours ago [-]
See my comment above - the common language rulesets work in similar ways as Buck(2), AIUI. (Except bazel has features to consume external dependencies without vendoring, as opposed to Reindeer approach for example). Another difference is that since Bazel has much more adoption across the OSS world, there are more viable choices in language implementation rulesets, as opposed to Buck's prelude.
imtringued 8 hours ago [-]
The problem has less to do with the fact that they are language specific build systems and more to do with the fact that build systems in general have refused to standardize.
If every build system came with sandboxing and programmable APIs that are all mostly the same, someone could have built a wrapper on top that exposes a standardized interface.
LoganDark 16 hours ago [-]
When I build software to use on my personal machine, I usually swap the build system out for tup. Maintainers never see this, but tup is just better for me. (Exception: Rust projects that use cargo, and JS projects that use bundling)
pjmlp 10 hours ago [-]
This one is precious, they remove the dependency on clang, by shipping clang.
> Clang: Zig is a full compiler toolchain and happens to also bundle all of clang.
flohofwoe 9 hours ago [-]
Eventually Clang will become an external dependency though and "somehow" integrated into the build system (so hopefully not much will change convenience-wise - because the tight integration of the Zig toolchain with C and C++ is indeed one of its most precious features).
pjmlp 9 hours ago [-]
While reducing the LLVM influence on compiler tools is welcomed, I wonder how long that eventually will take.
One thing I appreciate in Go, despite everything, is that they eventually bootstrapped it, with all the plus and minus it entails.
Still that line could have been written differently, maybe in the way you mentioned.
jeremyjh 16 hours ago [-]
I just looked at one example...Wayland's meson.build is 142 LOC, but build.zig is 581.
The build.zig doesn't build anything from the doc or tests folders. Otherwise, yeah.
jeremyjh 15 hours ago [-]
Well then, nevermind.
gregdaniels421 16 hours ago [-]
The Zig is far more explicit about what it is doing with lots of very long dotted commands for specifiy version headers and such, so maybe not super fair unless you like magic.
jeremyjh 16 hours ago [-]
Whats an example of magic in the meson.build that you don't understand? I think this is the first meson.build I've ever looked at and it seems very straightforward.
gregdaniels421 15 hours ago [-]
Didn't say I didn't understand it, I just said it was less explicit. Read through the Zig and it is super clear what every step is doing, and it sets things much more explicitly than the meson build
which is doing some config automake work. Which Meson does implicitly(magically).
pmarreck 15 hours ago [-]
[flagged]
0x69420 8 hours ago [-]
2. Fork the upstream project (optionally remove other -- now useless :^) -- build scripts),
oh god, it's meson wrapdb all over again. my heart goes out to all the library maintainers who'll have zig users go into their issue tracker like "and if you don't want this patch, don't worry, i'll just submit it to allyourcodebase" implicitly threatening upstream with a perpetual drip-feed of issues pertaining to a questionably-maintained fork they don't control.
Make / GNUMake / CMake / autoconf / bash scripts / batch scripts / powershell scripts: Zig is a complete build system that works on all supported platforms and can do everything those other tools do.
this bodes well.
i am begging people to please stop doing this genre of "compatibility" initiative.
gracefulliberty 17 hours ago [-]
Both an excellent showcase of the Zig build system and a convenient way to use C libraries within Zig.
Panzerschrek 11 hours ago [-]
Yet another attempt to fix the problem with 14 competing standards by introducing one more standard.
Now a C++ developer can face a problem, when he needs some thirdparty dependency and it requires Zig to be built.
pjmlp 10 hours ago [-]
Except there is hardly anything worthwhile in Zig for C++ developers to care about.
It is more the other way around, Zig devs need to interface with the existing C++ ecosystem, and industry standards.
fuhsnn 15 hours ago [-]
These look more like configure snapshots for Zig's bundled Clang than full ports of build system.
The HAVE_/WITH_ defines are supposed to be dynamically probed to adapt to different toolchain environments, setting them manually like that[1][2][3] could only work well for specific targets and for specific versions.
> supposed to be dynamically probed to adapt to different toolchain environments
that's a bygone practice from before compilers were widely capable.
wahern 10 hours ago [-]
You still can't test for the existence of functions, non-macro constants, typedefs, structure members, or other arbitrary interfaces. Newer compiler introspection facilities like __has_include or __has_builtin generally only help with compiler portability.
musl libc specifically recommends using external feature detection a la autoconf, and is so opinionated about this that it refuses to add any identifying macros, like __GLIBC__.
I had the impression `zig` already has stuff like `zig cc`. Would... would `zig make` be an impossible proposition? Maybe that makes no sense.
I haven't kept my zig skills sharp enough but zig build and the build.zig file basically lets you create a dag of steps and create various targets. It's basically make but the programming language is zig itself and the associated code happens to be in the standard library.
https://ziglang.org/documentation/0.16.0/std/#std.Build
https://ziglang.org/learn/build-system/
There are just too many C/C++ build systems out there. If you wanted to cover a majority of projects in the wild, you'd also need `zig cmake`, `zig automake`, `zig ninja`, `zig msbuild` and more. Not worth it imho.
Sorting that out and unifying it for every project is basically the tedious job that Linux distros do.
With that said, Zig falls well short when dependencies have more involved build systems that are too obnoxious to recreate, eg: libevent. I have a bit of a sentimental attachment to libevent and would love to use it within my Zig projects, but the only viable options are: 1) dynamically link; 2) have two [or more] separate build steps; or 3) wait until allyourcodebase does the obnoxious work for me. None of which are appealing so I end up using different libraries just for the build-system integration. And by their nature of being Zig libraries, they're usually less known, less battle-tested, and riskier to use because it's more likely to be some random guy's side project.
Putting aside the StackOverflow tier "why would you ever do/want that?" troll argument, the issue is that I'm being obstructed from using the libraries I want to use over build system incompatibilities. It feels somewhat similar to the complaint about having two [or more] languages within the same codebase.
It does look like for some projects it's almost entirely "just declare the dep tree" (https://github.com/allyourcodebase/boringssl), + the tiniest patch.
But what's going on in grpc for example? Like this build file https://github.com/allyourcodebase/grpc/blob/master/build.zi...... why is this build file in this repo? Is the grpc makefile no bueno?
Efforts to pull out information from existing build systems (if possible, ofc) means that you don't have to spend your time _rewriting build scripts_.
I don't know if it's really possible. Maybe this is step .. 4 of 34 to getting there. Just wondering if there's an alternative here that doesn't involve rewriting build scripts
Maybe a part of the pitch here is to just upstream these build scripts so that ones makefile becomes "just" `zig build`? In that case I'd actually like to see (for example) a fork of oyacc using what is figured out in https://github.com/allyourcodebase/yacc to "just" use the build script even in the higher level project.
The whole point is to massively simplify integrating those libraries into Zig projects by explicitly not having to deal with external build systems or separate C/C++ compiler toolchains.
> Is the grpc makefile no bueno?
Does it work out-of-the-box on Windows? That's usually the first question when encountering a Makefile (and just looking at the first lines of the Makefile, it doesn't, instead it needs to run inside mings: https://github.com/grpc/grpc/blob/a63b3a4d949a7057a0e5443e7a...).
With Zig you usually only need the Zig toolchain, which runs just fine in a regular Windows cmd.exe.
It's amazing how complicated we made things with bad software engineering and how good we can make things with good software engineering.
the one thing i will blame zig for here is the refusal to make some system to just include all of the files in a directory (hence allyourcodebase's python scripts). it makes sense with zig's import & module system but i think it would probably be a big blocker for some projects that they'd have to manually list out every single C file in their source.
you can currently use the filesystem apis to manually walk through files in your build script, though this is discouraged and will go away with upcoming change of isolating the build script
Can you tell me more about this? I’ve never used Bazel. I have used Buck extensively. But have never used either in an open source context.
The gist of it is something like:
- Bazel community hits a problem with something like how npm behaves around paths
- Bazel's solution is to create a patching system around paths to resolve them
- rinse and repeat for every piece of tooling us "normies" are using
The end result is that instead of Bazel's efforts showing up in the base tooling, they build up a bunch of their own tooling and patches, and the underlying tools don't get really much if any benefit. Moving to Bazel still is a PITA, and there's a lack of improving onramps
In an alternate universe when people in Bazel land hit issues they would show up with patches and recommendations _to the underlying tools_ to get things merged in so that the workflows can be better supported across the board.
I don't think anyone working on Bazel is against upstreaming, but I think they discount the value of it because, hey, _they_ can get patches and hacks working for their flow
My intuition is that npm, cargo, zig build etc are all “wrong”. And that Bazel/Buck are architecturally correct. Build systems should be polyglot by default! All these build systems and package managers that are per language are wrong.
But unfortunately Bazel and Buck have a decade plus of tech debt and baggage from their corporate overlords. Internal buck is actually mostly nice. I don’t know why anyone ever use it externally.
Arguably yes. The fundamental issue is that Bazel etc. are "hermetic" build systems - every build step they run is in a sandbox with only the inputs (source files and tools) that you have explicitly specified - nothing implicit everything explicit. And relies on the output you get from those inputs being the same every time.
This gives a wonderful property of knowing exactly which steps need performing and being able to cache everything - your whole build DAG is a merkle tree.
The downside is that many build steps do not fit into that mould - they make arbitrary network calls, or want to be able to access corners of your filesystem, or produce slightly different outputs each time due to multithreading etc. So you need to work around this in some way to make the larger system work.
(FYI, this is exactly the same as the Nix and Guix build systems except that for them the granularity of a step is "build one package" rather than "build one C file")
Bazel uses npm/cargo/pip in most people's usage of these tools! People still use these packages managers, with random patches and tweaks (because the overall ecosystem assumes you're using them). But Bazel often includes random fixups in their own tools.
For example [0] is some patches Bazel does to node to get it to play nicely witH Bazel. In an (IMO) better world Bazel people figure out what they need, and figure out what they can get upstreamed into node to get things working.
Instead we have patches that (if you jump through the history) have been around for 6+ years
[0]: https://github.com/aspect-build/rules_js/tree/main/js/privat...
First of all, bazel does not handle any of these languages/ecosystems itself. Such logic is delegated to language-specific rulesets, of which there are multiple implementations with different tradeoffs.
For python - https://github.com/aspect-build/rules_py parses a UV lockfile and does not use either UV or pip at build time.
For rust - https://github.com/hermeticbuild/rules_rs parses the cargo lockfiles but does not invoke cargo for compilation.
For js, https://github.com/aspect-build/rules_js parses the pnpm lockfile but does not invoke npm or pnpm at build time in the commonly used configurations.
While it's true that bazel allows to patch all dependencies on the fly, in my experience many Bazel users do try to contribute back, because managing stacks of hundreds of patches across all your dependencies isn't particularly fun.
Oh interesting. Buck totally replaces cargo / pip. I think the JS story is a little messier. Not sure as I try my best to not touch JS.
If every build system came with sandboxing and programmable APIs that are all mostly the same, someone could have built a wrapper on top that exposes a standardized interface.
> Clang: Zig is a full compiler toolchain and happens to also bundle all of clang.
One thing I appreciate in Go, despite everything, is that they eventually bootstrapped it, with all the plus and minus it entails.
Still that line could have been written differently, maybe in the way you mentioned.
https://github.com/allyourcodebase/wayland/blob/master/build... https://github.com/wayland-mirror/wayland/blob/main/meson.bu...
``` const wayland_version_header = b.addConfigHeader(.{ .style = .{ .cmake = upstream.path("src/wayland-version.h.in") }, }, .{ .WAYLAND_VERSION_MAJOR = @as(i64, @intCast(version.major)), .WAYLAND_VERSION_MINOR = @as(i64, @intCast(version.minor)), .WAYLAND_VERSION_MICRO = @as(i64, @intCast(version.patch)), .WAYLAND_VERSION = b.fmt("{f}", .{version}), });
```
which is doing some config automake work. Which Meson does implicitly(magically).
i am begging people to please stop doing this genre of "compatibility" initiative.
Now a C++ developer can face a problem, when he needs some thirdparty dependency and it requires Zig to be built.
It is more the other way around, Zig devs need to interface with the existing C++ ecosystem, and industry standards.
The HAVE_/WITH_ defines are supposed to be dynamically probed to adapt to different toolchain environments, setting them manually like that[1][2][3] could only work well for specific targets and for specific versions.
[1] https://github.com/allyourcodebase/libxml2/blob/38fb69d375bc...
[2] https://github.com/allyourcodebase/rnnoise/blob/47db9c212d7e...
[3] https://github.com/allyourcodebase/wayland/blob/f992cd71e199...
that's a bygone practice from before compilers were widely capable.
musl libc specifically recommends using external feature detection a la autoconf, and is so opinionated about this that it refuses to add any identifying macros, like __GLIBC__.