Fixed properties propagation from cluster policies
When working with cluster policies, sometimes it’s quirky to specify all the fixed fields in the deployment configuration.
To make this simpler, a new functionality is introduced - this functionality is called “property propagation”.
How this works?
Specify
policy_name
property in thenew_cluster
sectiondbx
will automatically fetch the policy definition using Cluster Policies APIJob cluster definition will be updated accordingly with all fixed properties provided from the policy definition during the
dbx deploy
stepThere is no need to specify
policy_id
manually, it will be automatically added to the cluster definition
Caveats
policy_name
shall exist in your Cluster PoliciesIf there are any conflicting configurations provided (i.e. deployment configuration has value provided, and this value is not the same as in the policy) - deployment will fail
Job properties adjustment happens only during the
dbx deploy
step, and it’s applied only when there is anew_cluster
withpolicy_name
property specifiedOnly the fixed properties will be automatically propagated