[CDRIVER-6142] Refactor dependency/environment setup in Earthfile #2165
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change rewrites much of Earthfile's dependency-installation handling code to be simpler to modify, and more flexible to use. Previously, each environment had its own target that needed to know how to install all of our dependencies for the specific platform. This refactor inverts the relationship: Instead, each dependency has an Earthly function that knows how to install itself based on whatever platform it happens to be running within. This allows the FROM line to be a plain image, rather that one of our handcrafted `env.foo` targets. This will make introducing and testing additional environments far more simple in the future. This change also allows us to remove the tools/build.earth utility, since we can now fit everything in the top-level file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This changeset modifies how Earthly environments are specified and set up for the build.
While doing refactors to the
amongocCI plan, I found that it is far simpler to have an Earthly function for each dependency that knows how to "do the right thing" to install the relevant packages for whatever host platform it happens to be running within, especially by adding a new__can_installutility. The following significant changes are made:env.fooEarthly targets are dropped. Instead, environments are specified using--frominstead of--env, which just executes a regularFROMcommand to set the base container image.ADD_FOOEarthly function is defined, and it will just "do the right thing" to obtain the relevant package for the build (or do nothing, if that dependency is disabled).__toolscript, which explodes itself using__aliasto reproduce the same set of tools that were available beforehand.__can_install <pkg>, which determines whether a package<pkg>is available for installation. This makes dep management far simpler since we don't need to do fancy platform/tool detection when we can just say "Ifgcc-c++is available, install it." without needing to know which package manager we're going to use.--from(formerly--env) has been modified to take a plain container image specifier, the logic for switching to ECR for the base platform container images is moved toearthly.py.+init,+build-environment,+configure, and+build. The main purpose of this split is to support subdividing the EVG build into the four steps so that we get separate timing/logging information for each.All of the above also has the added benefit of being able to test any Linux distro/version on-the-fly by just passing a new
--fromrather than needing to modifyEarthfileto support a new platform.