Change referenced assemblies processing to be on demand in MarkStep

For historical reasons ILLinker assembly resolution logic was very explicit and done as a single step in the pipeline LoadReferencesStep this was easy to implement but it does not work well with the more advanced features which are have been and will keep be adding to the linker.

There are several drawbacks which would be addressed if the linker was able to resolve the assembly reference on demand during MarkStep (support for on-demand assembly resolution in any step is not required at this point) as well as potential speed improvement for loading and analyzing fewer assemblies. It'd also get ILLinker closer to csc behaviour and require only necessary references for processing to be specified and not all possible references as it's required today.

Related issues which would be addressed by this

@vitek-karas

/cc @vitek-karas

Delete GetClassHierarchy/GetDerivedInterfacesForInterface

Delete GetClassHierarchy/GetDerivedInterfacesForInterface (#1186)

I plan to break the fix up into the following phases:

Phase 1
- Lazily TypeMap on assembly load. Allows linking lazy assemblies
- Lazy load assemblies from reflection, DynamicDependency, DynamicallyAccessedMembers
  Will fix recursive DynamicDependency
  Fixes Reflection patterns can not resolve assemblies outside of the reference closure #943 (lazy load reflection assemblies)
  Fixes [feature/dataflow] Add a pre-pass to scan type name strings in the app #1079 (lazy load DynamicallyAccessedMembers assemblies)
XML still not processed lazily, only from static reference closure
- Doesn't solve problem where XML is not processed from DynamicDependency attribute
  (Embedded XML not processed for assemblies referenced only via PreserveDependency attribute #843)
- Inlining still happens entirely before markstep, by getting inline candidates from static reference closure
Phase 2
- 2.1
  - Eliminate global pre-pass for DynamicDependency
    Will break inlining into DynamicDependency assemblies
  - Support lazily processing XML for additional dependencies
    (But not substitutions - as those could potentially modify already-marked IL)
  - Ensure that facade XML is processed if there are marked typerefs to the facade
    Fixes (Embedded XML not processed for assemblies referenced only via PreserveDependency attribute #843)
- 2.2
  - Support inlining into lazily loaded assemblies
    Gives back inlining behavior for DynamicDependency assemblies,
    also for DynamicallyAccessedMembers and reflection
Check in 2.1 and 2.2 together to prevent breaking change
Phase 3 (optional)
- Support substitutions XML from lazy assemblies
  Limit to substitutions within the assembly (not its dependencies, which may have already been marked)
  Will make substitutions XML behavior more consistent with dependency XML

Looks good!

(But not substitutions - as those could potentially modify already-marked IL)

Since we basically just remove code it should be fine. We can theoretically overmark in this case, but it's very very unlikely. Also - there should be basically no dependencies from the already marked code into the newly loaded assembly, since otherwise we would have loaded the dependency earlier.

Few topics/questions which I think are worth thinking before diving into implementation

How are we going to handle missing requested assemblies in the code to the developers?
What will be processed for descriptors/attributes/substitutions from already loaded assemblies which point to delay loaded assemblies?
How will the custom steps pipeline be triggered for delay loaded assemblies?
How do we ensure no assembly is loaded after MarkStep or is the expectation to trigger the re-processing anytime?
Will be facades handled differently considering they will always trigger another load?

How are we going to handle missing requested assemblies in the code to the developers?

Do you mean unresolved refs that are part of the static assembly reference closure? I think these should be handled the same as they are now - either skipped, or treated as an error (depending on --skip-unresolved).

What will be processed for descriptors/attributes/substitutions from already loaded assemblies which point to delay loaded assemblies?

Nothing would change in Phase 1 - such XML would be processed up-front, and it could load new assemblies, but those will not have their own XML processed.

In Phase 2 delay loaded assemblies would have their dependencies XML processed recursively (if marked). This XML can pull in new delay loaded assemblies, etc. It will only support XML that adds new dependencies - not substitutions which would modify already loaded assemblies, same goes for attributes XML. In this phase I suggest not to process delay loaded substitutions/attributes. Substitutions from the static reference closure are still processed as before.

In Phase 3 we could add partial support for delay loaded substitutions/attributes. We could limit them to modifying the containing assembly, or we could do as Vitek suggests and allow modifying already loaded assemblies, but without re-processing them - so it could lead to over-marking. I would prefer limiting the modifications since I find this easier to reason about.

How will the custom steps pipeline be triggered for delay loaded assemblies?

I think we will need a similar extensibility mechanism for the delay-loaded per-assembly logic. I am imagining that there is a list of per-assembly steps where custom steps can be added. Maybe we could add something like:

interface IAssemblyStep {
  void ProcessAssembly (AssemblyDefinition assembly);
}

Alternatively, maybe we could use ISubStep for this. This part needs more thought - I think it will be easier to design a good API once the implementation is a bit further along and we know exactly how these steps need to interact with the rest of the linker.

How do we ensure no assembly is loaded after MarkStep or is the expectation to trigger the re-processing anytime?

I think the latter - we want MarkStep to be able to add assemblies, for example when they are referenced by reflection, and have most of the processing logic run for these delay loaded assemblies (but not logic that would require re-analyzing already loaded assemblies).

Will be facades handled differently considering they will always trigger another load?

I don't think they would need special considerations except that some steps need to traverse exported types. Do you foresee issues here?

How do we ensure no assembly is loaded after MarkStep or is the expectation to trigger the re-processing anytime?

I think this is an interesting question - where exactly in the pipeline is not that interesting, but there will HAVE to be a point after which modifications are not allowed. Maybe a better way to put this would be "Once Sweep step starts, that's it". It's not that we could not handle it on the Cecil side, but we need to be able to freeze the annotations, so that sweep and similar steps can reason about them as a whole. For example the "Sealer" step is like that - we would only want to run it once "Everything" is annotated.

That kind of partially answers the problem with custom steps - we probably need two kinds of custom steps:

Per-assembly, can modify annotations, ...
Global-analysis, runs on everything - and after certain point can't modify annotations

We could limit them to modifying the containing assembly, or we could do as Vitek suggests and allow modifying already loaded assemblies, but without re-processing them - so it could lead to over-marking

I would vote for only allowing substitutions within the assembly that defines the thing being substituted. Otherwise we're opening ourselves interesting problems (linker will do substitutions based on value X returned from a method and eliminate branches at all callsites, then assembly Foo walks in and changes X to Y, invalidating linker's original assumptions). It can be dealt with I guess, but do we really need to?

I agree that substitutions should only work on the same assembly (tricky for those on command line, but those will "preload" all the necessary code, so it should be doable). The potential problem is that linker then does constant propagation across methods (and assemblies) - my original point was about the fact that constant propagation will basically never go "back" into the already processed (loaded) code. So if we limit substitutions like Michal suggests, it should be OK to also limit the constant propagation to only the newly loaded assembly and any future loaded assemblies, but no need to revisit already loaded assemblies.

I think we should write down the flow - the order of things which should happen - ignore the "step" design, just write down the actions we know need to happen. Something like:

Request to load assembly -> load it's XMLs and process them -> apply substitutions -> apply constant propagation -> marking -> ....

And then put together a couple of evil cases (where new assemblies are requested in the middle of the above) and how the system should handle those to get the correct result. THEN we can figure out how to actually implement it (steps, or something else, and so on).

What behavior do we want for a dynamically loaded assembly that's unused? For example, GetType("MissingType, MyAssembly") would load MyAssembly without requiring code from it. This won't run module initializers (at least on coreclr), but my reading of the spec is that implementations could run module initializers as soon as the module is loaded.

The current behavior allows the linker to remove MyAssembly. Assuming this is what we want, what should happen if the default action is copy? Normally, copy will keep unused statically referenced assemblies - but it's not clear to me what should happen for unused dynamic references.

GetType("MissingType, MyAssembly", throwOnError: true) throws different exceptions based on whether the file, or the type doesn't exist. Arguably this is very niche, but I would just keep the assembly so that we get this variation right, unless it's too much work (more than 5 minutes :)) to do that (I assume it really doesn't need much work - if we have trouble keeping assemblies with nothing in it marked, we can just mark the "<Module>" type so that we have something marked for sure and keeping the assembly will probably just fall out).

The plan above was still controversial so I've created a doc with a new design that avoids loading references up-front in most cases: #1626 Hopefully iterating on the design doc will be a better way to get to an agreement.

Enable processing of dynamically referenced assemblies

Closing as implemented in 6.0

Delete GetClassHierarchy/GetDerivedInterfacesForInterface (dotnet/lin…

Navigation Menu

Repository navigation

Change referenced assemblies processing to be on demand in MarkStep #1164

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Change referenced assemblies processing to be on demand in MarkStep #1164

Description

Activity

marek-safar commented on May 4, 2020

sbomer commented on Oct 22, 2020

vitek-karas commented on Oct 22, 2020

marek-safar commented on Oct 23, 2020

sbomer commented on Oct 23, 2020

vitek-karas commented on Oct 23, 2020

MichalStrehovsky commented on Oct 23, 2020

vitek-karas commented on Oct 23, 2020

23 remaining items

sbomer commented on Nov 7, 2020

MichalStrehovsky commented on Nov 9, 2020

sbomer commented on Nov 12, 2020

marek-safar commented on Feb 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions