Fix upcoming confusing versioning of jackson-annotations

[sdeleuze]

As we progress in adding Jackson 3 support accross the Spring portfolio, I would like to raise what is probably my main remaining big concern. JSTEP-1 does a fairly good job in describing why the decision has been made to keep the Jackson 2 package and groupId of jackson-annotations, I understand the intent even if it introduces challenges.

My concern is about the choice that has been made to have distinct Jackson 2 and Jackson 3 jackson-annotations which are almost the same, but not exactly, and to publish different versions for these 2 variants. I think that's creates uncessary confusion when there is a need to mix Jackson 2 and Jackson 3 "engines" in Frameworks (Spring Boot for example), in the community ecosystem, in end-users projects or at the scale of organizations that are using jackson-annotations to annotate the domain model and where the various microservices or applications will not migrate to Jackson 3 at the same time.

In practice, this creates a very uncomfortable and IMO uncessary choice dilemma: when using side by side Jackson 2 and 3, which variant of jackson-annotations should be used (2.x or 3.x)? How close are those 2 variants today? How close will they be in a couple of years? What are the differences (mainly deprecations and Java 17 baseline I think)? How can we be sure that at the scale of the Jackson ecosystem (Jackson core projects, Jackson modules, community extensions and end users project) that both variants are and remain compatible with Jackson 2 and 3.

Even worse, the fact there are 2.x and 3.x variants of Jackson jackson-annotations makes it impossible to use side by side Jackson 2 and Jackson 3 BOMS as they will define different versions for this dependency. And even for projects not using BOMs, it is not obvious which variants they should use.

Even JTEP-1 ackownledge this confusion by saying "essentially, difference in versioning may seem confusing to users"!

My proposal to solve this serious flaw is to fully embrace the fact that there will be a single jackson-annotations shared between Jackson 2 and Jackson 3 instead of the current situation:

• jackson-annotations would keep using versions 2.x in addition to keep its groupId and package
• As proposed in https://github.com/FasterXML/jackson-future-ideas/wiki/JSTEP-1#annotations-versioning-drop-patch-version, the patch version would be dropped the upcoming releases for jackson-annotations would be 2.20, 2.21, 2.22, etc.
• A release of jackson-annotations should be done only when there are some annotation changes, otherwise new releases of jackson-databind keep using the same version of jackson-annotations
• If a new release of jackson-annotations is needed for either Jackson 2.x or 3.x need, that would trigger a coordinated release of both Jackson 2 and 3 in order to avoid version mismatches in the BOMS and ensure proper testing.

IMO the end results would be very satisfying, as it would make the compatibility story between Jackson 2 and 3 even stronger, allow first class side by side usages of Jackson 2 and 3 (at the scale of an organization, it will happen very frequently), it fixes the currently broken dependency management story, and even conceptually I think it is much cleaner to fully embrace the fact there is effectively a single dependency version shared by Jackson 2 and 3.

Any thoughts?

[cowtowncoder]

Thank you @sdeleuze !

Ok, one thing I am not sure wrt above is this: would there ONLY be 2.x releases of jackson-annotations OR 2.x AND 3.x versions (but with well-defined relationship like 2.20[.0]/3.0, 2.21[.0]/3.1.
(I think our earlier discussions suggested latter but this sounds more like former)

Second: dropping of patch from 2.x versions is in itself quite disruptive: it will break common use case of defining just a single Maven/Gradle jackson.version used for dependencies, as it would now be 2.20 for jackson-annotations but 2.20.0 for, say, jackson-databind.
This is NOT a problem if one uses jackson-bom, however, so hard to gauge how common a problem it is.
This concern is the main reason I was planning to only drop patch level from 3.0 series.

Third: if we only release new jackson-annotations if there are actual changes (if I understand it correctly) -- instead of as part of Jackson minor release like now -- wouldn't this leave "holes" or (possibly worse) make versions across annotations and the rest out of sync.
That is: assuming we get the first synchronized release of Jackson 3.0, with jackson-databind 3.0.0, jackson-annotations 3.0 (... assuming we do 3.x ones), but with 3.1 there are no changes to annotations, databind 3.1.0 would then depend on annotations 3.0. More interestingly, assuming Jackson 3.2 does add something to annotations, would we get annotations 3.2 (keep in-sync for versions that do exist, but now a "hole", 3.1 missing), or 3.1 (out-of-sync, but no "holes").
My thinking otherwise would be that we always release jackson-annotations and keep versions in-sync, as is done now.
... actually, if and when we'd only release jackson-annotations 2.x, above is moot point -- and I think what you meant was that if Jackson 3.x databind requires additions to jackson-annotations, then we always need a full Jackson 2.x minor release. To keep 2.x versioning in place.

So maybe an example of planned/suggested release flow would make sense?

EDIT: I think I got it, after re-reading proposal by @sdeleuze, will summarize my understanding in a new comment.

[cowtowncoder]

Ok, so, if I understand proposal correctly, suggestion is to:

1. DROP jackson-annotations 3.x, keeping 2.x going on (with its existing com.fasterxml.jackson group id) -- this avoids confusion over group id for 3.x (where all other 3.x components have tools.jackson as group id)
2. Make other Jackson 3.x components rely on matching/compatible jackson-annotations 2.x version
   • This is planned to be mostly hidden by use of jackson-bom by users, but is visible to those that do not.
3. Drop patch level releases from jackson-annotations 2.x, starting with 2.20
4. No need to otherwise sync 2.x and 3.x releases (so we do NOT need to always release new 2.x AND 3.x minor versions), except that in (rare?) case of 3.x databind (etc) needing additions to jackson-annotations, it may be necessary to release new 2.x minor version (because jackson-annotations is still always part of 2.x minor release)

This adds minor complexity of "why does jackson-databind 3.0.0 rely on jackson-annotations 2.20?" but eliminates problems like:

1. Possibility of parallel 2.x/3.x jackson-annotations having subtle incompatibilities
2. Possible conflicts for Gradle or Maven dependency resolution
3. Requiring elaborate testing, planning to guard against (1)
4. Confusion by users on whether and how 2.x/3.x jackson-annotations can co-exist or be substituted.

[sdeleuze]

Looks like indeed to match with my proposal.

[sdeleuze]

Notice that from my POV the complexity added is IMO none because users already wonder why there is a dependency from jackson-databind 3.0.0 to the com.fasterxml groupId in current arrangement. So by using 2.20 instead of 3.0.0 for jackson-annotations version, you make it almost explicit that the reason is that it is a shared dependency on Jackson 2 annotations. So from my POV it brings more clarity.

[cowtowncoder]

Right, I tried to mention that aspect ("only 2.x has 'com.fasterxml.jackson', only 3.x 'tools.jackson'").

[cowtowncoder]

@pjfanning @JooHyukKim @k163377 @yawkat WDYT?

[JooHyukKim]

+1 for keeping single jackson-annotations version. Seems clear to me which option would hurt less considering annotations' nature being annotated on POJO'.

I may wrong but I believe the Problem we need to solve for are...

• Case 1 : Two versions annotations, 2.x and 3.x
  • things actually breaking.
• Case 2 : Single version 2.x only
  • confusion/awkward-ness

[cowtowncoder]

(1) would not necessarily break things -- plan was not to break usage -- but it would require very high level of diligence to effectively keep, say, jackson-annotations 2.20 and 3.0 identical.
If that failed, things could indeed break.
And if so... why even have 2 supposedly different versions if they are not actually different.

[JooHyukKim]

Ah yes, i also meant break-if-not-maintained. Good point on why even have 2 separate versions part

[pjfanning]

sounds good to me - can we make the next 3.0.0-rcx release use jackson-annotations 2.19.1 and then aim to do a jackson-annotations 2.20 (no patch version) release with jackson-databind 2.20.0 release?

[cowtowncoder]

@pjfanning Yes, I think that makes sense.

[wilkinsona]

Thanks for considering this, @cowtowncoder

I've been discussing this with @sdeleuze from a Spring Boot perspective as we're trying to figure out how to handle our Jackson 2 -> Jackson 3 migration. We're anticipating users having apps that need both versions of Jackson while the ecosystem completes its migration, hence the interest in being able to use both versions at the same time without disagreement over the version of jackson-annotations. The points that you've summarised will achieve that for us.

We use Jackson's bom and would continue to do so (using both Jackson 2's and Jackson 3's boms) so dropping the patch level from the version of jackson-annotations should have minimal consequences for us.

[cowtowncoder]

Ok: I think this is enough consensus for me, even if this is a huge chance. I will go ahead and update https://github.com/FasterXML/jackson-future-ideas/wiki/JSTEP-1 to reflect change; file an issues to:

1. Drop patch version from jackson-annotations 2.20
2. Change jackson-bom 3.x to use jackson-annotations 2.19.1

and go from there.

[JooHyukKim]

👌 Lemme know if anything I can take over some load off u

[sdeleuze]

Thanks a lot for taking in account our feedback, much appreciated. With that change, I think Jackson 3 is almost ready for GA!

[cowtowncoder]

This feedback has been great -- much rather get it before release. It's quite common to only learn about issues post-release when things are much more difficult to resolve. So great to avoid that in this case.

[cowtowncoder]

Ok turns out there were indeed couple of incompatibilities (things added in 3.0 but not in 2.x). So I had not maintained parity diligently. Or perhaps more accurately, compatibility was one-way: Jackson 2.x could use 3.x jackson-annotations but not the other way around.
Now backporting those things (minor but necessary), to get 3.x jackson-databind compile against 2.x jackson-annotations (2.19.2-SNAPSHOT).

What this means in practical terms is that we need 2.19.2 release before 3.0.0-rc6, to get compatible jackson-annotations (given 2.20 is not really ready for RCs).

[cowtowncoder]

@jjohannes I realized this might be challenge to Gradle version alignment and in particular for gmm (gradle module metadata). WDYT?

Basically changes are:

• 2.x will drop patch level from jackson-annotations for 2.20.x
• 3.x will use 2.x jackson-annotations versions: 3.0.x will use 2.20, and so on (no separate 3.x versions released for jackson-annotations)

I am planning to get 3.0.0-rc6 released quite soon, but hope to resolve Gradle issues, if any, before that.

[jjohannes]

There should be no issue with the alignment that breaks something.

IIUC, it does probably not make sense anymore to put the "BOM dependency" into jackson-annotations as it is part of both the 2.x and 3.x BOM now.

I think the cleanest would be to remove the "BOM dependency" from jackson-annotations, by removing this piece of the plugin configuration:

https://github.com/FasterXML/jackson-annotations/blob/2.x/pom.xml#L185-L193

[cowtowncoder]

Ah. Yes, that makes sense now that 2.20+ will be used by 3.x, cannot assume strict 1-to-1 mapping b/w annotation package, other components.

We will see how things work with 2.19.2 + 3.0.0-rc6 release (combo to switch annotation versioning over), see if there are unexpected issues to resolve before eventual 2.20.0 / 3.0.0 (final) combo.

[wilkinsona]

Here's a concrete example of a problem that occurs with jackson-databind 3.0.0-rc-5 and jackson-annotations 2.19.1:

java.lang.NoSuchFieldError: Class com.fasterxml.jackson.annotation.JsonFormat$Shape does not have member field 'com.fasterxml.jackson.annotation.JsonFormat$Shape POJO'
	at tools.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:399)
	at tools.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:361)
	at tools.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265)
	at tools.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
	at tools.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:158)
	at tools.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:732)
	at tools.jackson.databind.deser.jdk.UntypedObjectDeserializer._findCustomDeser(UntypedObjectDeserializer.java:179)
	at tools.jackson.databind.deser.jdk.UntypedObjectDeserializer.resolve(UntypedObjectDeserializer.java:152)
	at tools.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)
	at tools.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
	at tools.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:158)
	at tools.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:741)
	at tools.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:2871)
	at tools.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2689)
	at tools.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1705)

I guess this is one of the incompatibilities that you mentioned above, @cowtowncoder.

[cowtowncoder]

@wilkinsona Yes; for now 2.19.2-SNAPSHOT needs to be used (until I release 2.19.2 and update jackson-bom etc)

[cowtowncoder]

Changed, 2.20-rc1 released, closing.