This tutorial extends the prior article on Docker Compose debugging in .Net. If you’re unfamiliar with debugging Dot Net Core Docker applications, please refer to it and the prior articles for more information before starting.
A wise Code Sloth learns just enough to kick start their practical learning journey. While the journey to this point may seem long, we must study hard. Failure to do so and the consequences may be, severe!
The Unfortunate Gap in Dockerfile Consumption Between Development and “Deployed Environments”
It is really awesome that Visual Studio provides us with the debugging tools that it does. It is also really amazing that we are able to use these tools to their fullest while working in Docker containers.
Unfortunately however, it’s not amazing that the steps that enable us to use these tools bypass the majority of our Docker assets. This divergence ultimately causes drift between our local development environment and any subsequent CICD pipeline that builds the entire Dockerfile to deploy our code elsewhere.
Let’s take another look at the default Dockerfile that was generated in the prior tutorial for the RedisReader
application:
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging. FROM mcr.microsoft.com/dotnet/runtime:6.0 AS base WORKDIR /app FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build WORKDIR /src COPY ["RedisReader/RedisReader.csproj", "RedisReader/"] COPY ["SharedGoodness/SharedGoodness.csproj", "SharedGoodness/"] RUN dotnet restore "RedisReader/RedisReader.csproj" COPY . . WORKDIR "/src/RedisReader" RUN dotnet build "RedisReader.csproj" -c Release -o /app/build FROM build AS publish RUN dotnet publish "RedisReader.csproj" -c Release -o /app/publish /p:UseAppHost=false FROM base AS final WORKDIR /app COPY --from=publish /app/publish . ENTRYPOINT ["dotnet", "RedisReader.dll"]
The design of this file has one issue that is almost guaranteed to be hit at some point during the extension of your application. An issue that you will not observe during your local debugging, and may not even realise as you continue you leverage your Dockerfile to productionise images of your code.
Extending Our Existing Dot Net Core Docker Use Case
As is the case for most applications, our demo app has a new requirement that has caused it to change. We now have have an additional Class Library defined in the solution. This project contains references to additional NuGet dependencies:
- NewtonSoft.Json for Json serialisation/deserialisation
- Bogus for generating random data
Let’s take a closer look:
We can see that:
- We have a new project called
AnotherDependency
- This project has the two additional NuGet package references
- It is referenced by our
RedisReader
application
Let’s take a quick look at what this simple new dependency does:
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; namespace AnotherDependency { public interface IAnotherDependency { void DoSomethingWithJson(); } }
Here we have a very simple interface that declares a single function that will do something with json.
using Bogus; using Microsoft.Extensions.Logging; using Newtonsoft.Json; namespace AnotherDependency { public class AnotherConcreteDependency : IAnotherDependency { private ILogger<AnotherConcreteDependency> _logger; public AnotherConcreteDependency(ILogger<AnotherConcreteDependency> logger) { _logger = logger ?? throw new ArgumentNullException(nameof(logger)); } public void DoSomethingWithJson() { var faker= new Faker(); var randomObject = new { TestProperty = 1, AnotherTestProperty = faker.Random.String(), RandomProperty = faker.Random.Int(0, 10) }; var redundantVariable = JsonConvert.SerializeObject(randomObject); _logger.LogInformation($"Some log message: {redundantVariable}"); } } }
AnotherConcreteDependency
simply converts an anonymous type that is populated with some random data into a json string. It then writes this string value to the console.
using AnotherDependency; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using SharedGoodness.Redis; using SharedGoodness.SharedUtilities; namespace RedisReader { public class BackgroundRedisReader : BackgroundService { ILogger<BackgroundRedisReader> _logger; IRedisRepository _redisRepository; IAnotherDependency _anotherDependency; public BackgroundRedisReader(ILogger<BackgroundRedisReader> logger, IRedisRepository redisRepository, IAnotherDependency anotherDependency) { _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _redisRepository = redisRepository ?? throw new ArgumentNullException(nameof(redisRepository)); _anotherDependency = anotherDependency ?? throw new ArgumentNullException(nameof(anotherDependency)); } protected override async Task ExecuteAsync(CancellationToken stoppingToken) { var random = new Random(); await Task.Delay(TimeSpan.FromSeconds(1)); while (!stoppingToken.IsCancellationRequested) { _logger.LogInformation($"Executing again at {DateTime.Now.PrettifyDate()}"); var currentValue = await _redisRepository.ReadFromRedis(RedisConstants.RedisKey); _logger.LogInformation($"Fetched {currentValue}"); _anotherDependency.DoSomethingWithJson(); await Task.Delay(TimeSpan.FromSeconds(2)); } } } }
Finally, our existing RedisReader
BackgroundRedisReader
class is extended to aggregate this new interface, which is bound to its concrete implementation at startup time. The DoSomethingWithJson
method is then invoked once per loop.
Cache Miss, Oh No!
At this point we have a new dependency in the mix. What does this mean for our Docker build – will it now fail because the new project is not referenced in the Dockerfile? Let’s take a look.
docker builder prune
To make the next steps completely transparent, we’ll start by pruning our build cache.
Then we’ll run our build command:
PS C:\_dev\CodeSloth\multi project docker debugging\DebuggingMultipleProjectsInDocker> docker build -t test-image --progress=plain -f .\RedisReader\Dockerfile .
This time opting to use the --progress=plain
parameter to more clearly display all of the output that is written to the terminal. We see the following:
#1 [internal] load build definition from Dockerfile #1 sha256:25d2bf51062b06ed9c71c79046942585c44b04d810917dd6dd2ec41059ecbeb3 #1 transferring dockerfile: 860B done #1 DONE 0.1s #2 [internal] load .dockerignore #2 sha256:ecf480d7e6b97b12c70cb7a91adc7f28f07ac631685cda244648cd5865c1784b #2 transferring context: 382B done #2 DONE 0.1s #4 [internal] load metadata for mcr.microsoft.com/dotnet/runtime:6.0 #4 sha256:0128d5218b10a50bf55970db7c113f09503205e2b2ada4931bb5d0c6628fdd2a #4 DONE 0.0s #3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0 #3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b #3 DONE 1.0s #10 [internal] load build context #10 sha256:5be938b20133973036e197437281e9b35e1fc50c03ca71938d7fa1000ddf4c20 #10 DONE 0.0s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 resolve mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 0.0s done #8 ... #5 [base 1/2] FROM mcr.microsoft.com/dotnet/runtime:6.0 #5 sha256:2f0c599a662b466f9a09402b483721cd4263038964a3bba095f42135663b4c67 #5 DONE 0.1s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 ... #10 [internal] load build context #10 sha256:5be938b20133973036e197437281e9b35e1fc50c03ca71938d7fa1000ddf4c20 #10 transferring context: 16.83kB 0.0s done #10 DONE 0.2s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 1.82kB / 1.82kB done #8 sha256:3f5873abb5240a10f3abee05c6f89933d2da0b06037a0532aeb7ddd7959f8252 2.01kB / 2.01kB done #8 sha256:05057078be7d5b0fdc8424f965a11d416639373f9388ecaeb4e2af2ce5bbc1c4 7.17kB / 7.17kB done #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 0B / 9.46MB 0.1s #8 ... #6 [base 2/2] WORKDIR /app #6 sha256:a01dd2d516cd2b9d43c2f898ff1f32ca74f47f1c9c7cc91189d15ebdd66a7adb #6 DONE 0.2s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 0B / 25.37MB 0.2s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 0B / 148.14MB 0.2s #8 ... #7 [final 1/2] WORKDIR /app #7 sha256:fee5a31b081ff7b359025150181826c014703920ff1bfcc43f240ef44f62d6b9 #7 DONE 0.1s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 1.05MB / 9.46MB 0.5s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 2.10MB / 9.46MB 0.7s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 3.15MB / 9.46MB 1.2s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 2.10MB / 25.37MB 1.4s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 4.19MB / 9.46MB 1.5s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 5.24MB / 9.46MB 2.1s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 4.19MB / 25.37MB 2.2s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 6.29MB / 9.46MB 2.5s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 6.29MB / 25.37MB 2.9s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 7.34MB / 9.46MB 3.0s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 8.39MB / 9.46MB 3.4s #8 sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 9.46MB / 9.46MB 3.8s done #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 8.39MB / 25.37MB 3.9s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 0B / 12.89MB 3.9s #8 extracting sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd #8 extracting sha256:871ef3419da3410a47aa97b7655d8543add053e27cac5c5922ff3ee1f75793cd 0.3s done #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 8.39MB / 148.14MB 4.5s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 1.05MB / 12.89MB 4.5s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 2.10MB / 12.89MB 4.9s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 10.49MB / 25.37MB 5.5s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 3.15MB / 12.89MB 5.5s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 4.19MB / 12.89MB 6.2s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 16.78MB / 148.14MB 6.8s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 12.58MB / 25.37MB 6.9s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 5.24MB / 12.89MB 6.9s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 6.29MB / 12.89MB 7.6s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 7.34MB / 12.89MB 7.9s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 8.39MB / 12.89MB 8.3s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 9.44MB / 12.89MB 8.7s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 10.49MB / 12.89MB 9.2s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 14.68MB / 25.37MB 9.3s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 25.17MB / 148.14MB 9.3s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 11.53MB / 12.89MB 9.5s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 12.89MB / 12.89MB 9.8s #8 sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 12.89MB / 12.89MB 9.8s done #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 16.78MB / 25.37MB 10.4s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 18.87MB / 25.37MB 11.4s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 33.55MB / 148.14MB 11.5s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 20.97MB / 25.37MB 12.2s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 23.07MB / 25.37MB 12.9s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 41.94MB / 148.14MB 13.5s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 25.17MB / 25.37MB 13.6s #8 sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 25.37MB / 25.37MB 13.8s done #8 extracting sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 0.1s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 51.38MB / 148.14MB 15.0s #8 extracting sha256:c3514d10142f3a43d3037bc770248d6093c76d46a47ebe8ac4232c8b29d9eaab 1.2s done #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 59.77MB / 148.14MB 16.3s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 68.16MB / 148.14MB 17.5s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 76.55MB / 148.14MB 18.8s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 84.93MB / 148.14MB 20.1s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 93.32MB / 148.14MB 21.3s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 101.71MB / 148.14MB 22.6s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 110.10MB / 148.14MB 23.9s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 118.49MB / 148.14MB 25.1s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 126.88MB / 148.14MB 26.3s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 135.27MB / 148.14MB 27.6s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 143.65MB / 148.14MB 28.8s #8 sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 148.14MB / 148.14MB 30.2s done #8 extracting sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 #8 extracting sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 5.1s #8 extracting sha256:c65769fdd163d4fcba401982b5b50f0f78ec1970e68c45fb6671a8864c977683 6.0s done #8 extracting sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 0.1s #8 extracting sha256:8b2829492cd27a90e4bde8169f4c4e3d2e6c17be2354f230684b05d40ea6df90 0.6s done #8 DONE 37.6s #9 [build 2/8] WORKDIR /src #9 sha256:fd721b9ab8612450c39dfab43831d16b893f71b294b4d92a8c1d6fdaf0c47a22 #9 DONE 2.8s #11 [build 3/8] COPY [RedisReader/RedisReader.csproj, RedisReader/] #11 sha256:51b64f2a7d2289edb94b254e4c74a85a99840117e108bc665deee434cf2f101a #11 DONE 0.2s #12 [build 4/8] COPY [SharedGoodness/SharedGoodness.csproj, SharedGoodness/] #12 sha256:441d4a7a2fcad1fd09f43d86a81d18ef08cfecd80eaab27ac49a004609b2f51d #12 DONE 0.2s #13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:47cf23add107fc9255f473b375fbe9bb569c56e603ce42c5a66d3a9361082d32 #13 1.013 Determining projects to restore... #13 1.015 Skipping project "/src/AnotherDependency/AnotherDependency.csproj" because it was not found. #13 1.040 Skipping project "/src/AnotherDependency/AnotherDependency.csproj" because it was not found. #13 6.468 Restored /src/SharedGoodness/SharedGoodness.csproj (in 5.18 sec). #13 6.469 Restored /src/RedisReader/RedisReader.csproj (in 5.18 sec). #13 DONE 6.7s #14 [build 6/8] COPY . . #14 sha256:92b6001e3c92b4263309115d8cb1380c2910b1aacf61a47f1c859f3374f029ce #14 DONE 0.1s #15 [build 7/8] WORKDIR /src/RedisReader #15 sha256:7c20bddb57e072c0613a18d85897df4803a1395b03f9e4a63de95e6af5da143c #15 DONE 0.1s #16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:062b1fc36c62900fc68b22c7608b046a2d83b99abf639ba0ca65319b2a6fe938 #16 0.429 MSBuild version 17.3.1+2badb37d1 for .NET #16 1.001 Determining projects to restore... #16 3.599 Restored /src/AnotherDependency/AnotherDependency.csproj (in 1.76 sec). #16 3.599 Restored /src/RedisReader/RedisReader.csproj (in 1.77 sec). #16 3.604 1 of 3 projects are up-to-date for restore. #16 5.895 AnotherDependency -> /app/build/AnotherDependency.dll #16 5.906 SharedGoodness -> /app/build/SharedGoodness.dll #16 6.232 RedisReader -> /app/build/RedisReader.dll #16 6.258 #16 6.258 Build succeeded. #16 6.258 0 Warning(s) #16 6.258 0 Error(s) #16 6.258 #16 6.258 Time Elapsed 00:00:05.74 #16 DONE 6.5s #17 [publish 1/4] WORKDIR /src #17 sha256:46af958bb6f5841870eb47e735c5664827cb815bc1f34eab87d07567ff6bd7bf #17 DONE 0.1s #18 [publish 2/4] COPY . . #18 sha256:e3993f41dcd090c9186b175aa9889f1736aebeeb1feb16192682bd60b355e8fd #18 DONE 0.1s #19 [publish 3/4] WORKDIR /src/RedisReader #19 sha256:f3cded69eb37649d1b2020106bbf7f4d83a6610304decb6721d027fc713c70d7 #19 DONE 0.1s #20 [publish 4/4] RUN dotnet publish "RedisReader.csproj" -c Release -o /app/publish /p:UseAppHost=false #20 sha256:03fd8a39945e8afc205fbeae63825b5446accb8b2246a5f1a99a98a2123475b0 #20 0.494 MSBuild version 17.3.1+2badb37d1 for .NET #20 0.941 Determining projects to restore... #20 1.736 All projects are up-to-date for restore. #20 2.228 AnotherDependency -> /src/AnotherDependency/bin/Release/net6.0/AnotherDependency.dll #20 2.335 SharedGoodness -> /src/SharedGoodness/bin/Release/net6.0/SharedGoodness.dll #20 4.067 RedisReader -> /src/RedisReader/bin/Release/net6.0/RedisReader.dll #20 4.119 RedisReader -> /app/publish/ #20 DONE 4.2s #21 [final 2/2] COPY --from=publish /app/publish . #21 sha256:f286a74f752c514163e833665d0c80301be015a8feffe29838570395333421fe #21 DONE 0.2s #22 exporting to image #22 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 #22 exporting layers #22 exporting layers 0.1s done #22 writing image sha256:5f5396a8d3bf7028ec1fd16cdc33360421014f44b5d4e7d9c65563482143aa3a #22 writing image sha256:5f5396a8d3bf7028ec1fd16cdc33360421014f44b5d4e7d9c65563482143aa3a done #22 naming to docker.io/library/test-image done #22 DONE 0.2s
That was a lot of information. However, the moral of the story is that the build passed. We successfully created an image for our RedisReader
.
However, there’s a more subtle issue that has been introduced that works against the build time optimisation that we receive from Docker layer caching (a topic that we discussed here).
Let’s laser focus on the specific problematic areas, starting with build step #13.:
#13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:47cf23add107fc9255f473b375fbe9bb569c56e603ce42c5a66d3a9361082d32 #13 1.013 Determining projects to restore... #13 1.015 Skipping project "/src/AnotherDependency/AnotherDependency.csproj" because it was not found. #13 1.040 Skipping project "/src/AnotherDependency/AnotherDependency.csproj" because it was not found. #13 6.468 Restored /src/SharedGoodness/SharedGoodness.csproj (in 5.18 sec). #13 6.469 Restored /src/RedisReader/RedisReader.csproj (in 5.18 sec). #13 DONE 6.7s
Build step #13 demonstrates that we did not restore AnotherDependency
during the dotnet restore
build step because it could not be found. We know that this project did not exist in the container at dotnet restore
time, because our Dockerfile explicitly lists the projects that it will copy for this phase. AnotherDependency
was not added to this list.
This has a flow on impact into step #16 below:
#16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:062b1fc36c62900fc68b22c7608b046a2d83b99abf639ba0ca65319b2a6fe938 #16 0.429 MSBuild version 17.3.1+2badb37d1 for .NET #16 1.001 Determining projects to restore... #16 3.599 Restored /src/AnotherDependency/AnotherDependency.csproj (in 1.76 sec). #16 3.599 Restored /src/RedisReader/RedisReader.csproj (in 1.77 sec). #16 3.604 1 of 3 projects are up-to-date for restore. #16 5.895 AnotherDependency -> /app/build/AnotherDependency.dll #16 5.906 SharedGoodness -> /app/build/SharedGoodness.dll #16 6.232 RedisReader -> /app/build/RedisReader.dll #16 6.258 #16 6.258 Build succeeded. #16 6.258 0 Warning(s) #16 6.258 0 Error(s) #16 6.258 #16 6.258 Time Elapsed 00:00:05.74 #16 DONE 6.5s
Here we can see that step #16 has had to restore the missing dependencies from AnotherDependency
. This step is able to do so, as we have since copied all of the source code into the container for the build step, along with the project and solution files.
While the dotnet build
and dotnet publish
steps will restore any missing dependencies at the time of their execution, it is critical to understand that a Docker layer will only be generated for the end result of a given command that modifies the filesystem. I.e. one command will, at most, generate a single cacheable layer.
This means, that despite the fact that the dotnet build
command above restored the required packages, the resulting image will constitute both the restore step and the subsequent build of the code. This makes the cacheability of this layer low.
Any time the code of this project changes the layer will need to be recreated, as the contents of the filesystem to build will be different. In recreating this layer, we will once again need to restore AnotherDependency
by downloading packages from the internet, even if none of the dependencies themselves have changed.
Unfavourable Outcomes Either Way
While the implications of layer caching may not seem like a big deal for a single project, we will see a gradual increase in build times as the dependency tree of a given application grows over time. Couple this with the additional time required to run unit/integration tests and pushing changes through your CICD pipeline will become sloth. But not in a Code Sloth way.
Maintaining the Dockerfile isn’t exactly a cakewalk either. Engineers need to remember to add new dependencies into the Dockerfile and/or update/maintain existing references if the structure of the solution changes over time. This then becomes a maintenance overhead and increases the perceived complexity of maintaining the solution.
There has to be a better way!
Dynamic Dockerfiles for Dotnet Restore
With a small revision to the Dockerfile of our RedisReader
(and other application’s Dockerfiles) we can ensure that we do not have to continue to maintain them as dependencies or folder structures change.
Instead of explicitly copying each project, we can use these steps (thanks to this gist on GitHub), to first copy the *.csproj files into the container (which will flatten their structure), and then move them into newly created subfolders based on each of the project’s names.
Note: this strategy assumes that:
- Your projects exist in a folder that matches their name
- Each project has its own folder, and that all targeted projects sit in the same directory
By default, Visual Studio conforms to this convention. Based on the structure of your repo, you may have a src
subfolder, which contains your application logic. You may also have a test
subfolder at the same level as src
which contains all of your test code. The gist code sample reflects this style (although only makes reference to src
– you can copy and paste this in your Dockerfile, and adjust the command to copy over test projects in a similar way).
We can see the relative folder structure in the RedisReader
csproj
file below in the section containing ProjectReference
tags. The relative reference traverses one folder above, and then into another subfolder with a name that matches the csproj
:
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> <TargetFramework>net6.0</TargetFramework> <ImplicitUsings>enable</ImplicitUsings> <Nullable>enable</Nullable> <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS> </PropertyGroup> <ItemGroup> <PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" /> <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.17.0" /> </ItemGroup> <ItemGroup> <ProjectReference Include="..\AnotherDependency\AnotherDependency.csproj" /> <ProjectReference Include="..\SharedGoodness\SharedGoodness.csproj" /> </ItemGroup> </Project>
The example Dockerfile content from the gist above can be seen below:
COPY src/*/*.csproj ./ RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
This has been modified for our tutorial to reflect that our source code does not exist within a src
subfolder in the root of the repo. Instead, we use the following to simply scoop up all csproj
files and put them into their respective folders:
COPY ./*/*.csproj ./ RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
The revised commands can be seen in the complete Dockerfile of the RedisReader
below:
#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging. FROM mcr.microsoft.com/dotnet/runtime:6.0 AS base WORKDIR /app FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build WORKDIR /src COPY ./*/*.csproj ./ RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done RUN dotnet restore "RedisReader/RedisReader.csproj" COPY . . WORKDIR "/src/RedisReader" RUN dotnet build "RedisReader.csproj" -c Release -o /app/build FROM build AS publish WORKDIR /src COPY . . WORKDIR "/src/RedisReader" RUN dotnet publish "RedisReader.csproj" -c Release -o /app/publish /p:UseAppHost=false FROM base AS final WORKDIR /app COPY --from=publish /app/publish . ENTRYPOINT ["dotnet", "RedisReader.dll"]
Running our build command a second time (after purging the cache) produces a much better output:
#1 [internal] load build definition from Dockerfile #1 sha256:4556a9543dadc2988c0f33a521eb1061dc9f8cdce95167d7893fed78f1d23d43 #1 transferring dockerfile: 846B done #1 DONE 0.1s #2 [internal] load .dockerignore #2 sha256:a181dd9aa3c1f248d0dd41f87f37c8c7a2f56c56b4d2d374d77be9d161e5ed05 #2 transferring context: 382B 0.0s done #2 DONE 0.1s #4 [internal] load metadata for mcr.microsoft.com/dotnet/runtime:6.0 #4 sha256:0128d5218b10a50bf55970db7c113f09503205e2b2ada4931bb5d0c6628fdd2a #4 DONE 0.0s #3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0 #3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b #3 DONE 0.2s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 DONE 0.0s #5 [base 1/2] FROM mcr.microsoft.com/dotnet/runtime:6.0 #5 sha256:2f0c599a662b466f9a09402b483721cd4263038964a3bba095f42135663b4c67 #5 DONE 0.0s #9 [build 2/8] WORKDIR /src #9 sha256:fd721b9ab8612450c39dfab43831d16b893f71b294b4d92a8c1d6fdaf0c47a22 #9 CACHED #10 [internal] load build context #10 sha256:4c952441568471b2f864b7dd40c6859c221cfe897899785c9dbaf30e9e28d370 #10 transferring context: 16.81kB 0.0s done #10 DONE 0.1s #11 [build 3/8] COPY ./*/*.csproj ./ #11 sha256:27e70badf8511adff03a66b2ae139bbd47bd9938a5684a29853373dd83cfb4aa #11 DONE 0.1s #12 [build 4/8] RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done #12 sha256:40318d37bd1d1b715a4d5af9f877a71f93939da70e9925308f968f7b94e7b96a #12 DONE 0.4s #13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:74e2154ed89f134032690bc64b9b8e54ea90f523b5cec48002e876399210ed50 #13 1.005 Determining projects to restore... #13 5.020 Restored /src/AnotherDependency/AnotherDependency.csproj (in 3.38 sec). #13 6.836 Restored /src/SharedGoodness/SharedGoodness.csproj (in 5.21 sec). #13 6.841 Restored /src/RedisReader/RedisReader.csproj (in 5.21 sec). #13 DONE 7.4s #14 [build 6/8] COPY . . #14 sha256:8b8cecf187d06e67f98c9061db8e1a54931cbb09000b4f6962cf977b6dab8aa7 #14 DONE 0.4s #15 [build 7/8] WORKDIR /src/RedisReader #15 sha256:a05ef2492283c68a687df34639b2ee1a07b55d9a662e175f8296382451c784ba #15 DONE 0.3s #16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:899127ee350d9e363af7c68268f5ea6f5d2cf999839fdbd4a36448054b5dc83d #16 0.456 MSBuild version 17.3.1+2badb37d1 for .NET #16 0.953 Determining projects to restore... #16 1.700 All projects are up-to-date for restore. #16 3.872 AnotherDependency -> /app/build/AnotherDependency.dll #16 3.875 SharedGoodness -> /app/build/SharedGoodness.dll #16 4.119 RedisReader -> /app/build/RedisReader.dll #16 4.131 #16 4.131 Build succeeded. #16 4.131 0 Warning(s) #16 4.131 0 Error(s) #16 4.131 #16 4.131 Time Elapsed 00:00:03.60 #16 DONE 4.3s #17 [publish 1/4] WORKDIR /src #17 sha256:6c53ec50466d552b50b5e214929eadd20aeb9610e9aa573d52b659d56bddf152 #17 DONE 0.1s #18 [publish 2/4] COPY . . #18 sha256:71386241b3b038b5898cead797a7dc35e19f51fe97163fdcfdf20bfedf214f90 #18 DONE 0.1s #19 [publish 3/4] WORKDIR /src/RedisReader #19 sha256:0f95aae72bd46c1d5e1057dc7ecaa337a14c26aa59f328acc39871158ddb5150 #19 DONE 0.1s #20 [publish 4/4] RUN dotnet publish "RedisReader.csproj" -c Release -o /app/publish /p:UseAppHost=false #20 sha256:f0a49dc87d5f73577dfb0af91a39d8c66c94f1633e6318fad8505dce79353767 #20 0.429 MSBuild version 17.3.1+2badb37d1 for .NET #20 0.870 Determining projects to restore... #20 1.573 All projects are up-to-date for restore. #20 2.036 AnotherDependency -> /src/AnotherDependency/bin/Release/net6.0/AnotherDependency.dll #20 2.200 SharedGoodness -> /src/SharedGoodness/bin/Release/net6.0/SharedGoodness.dll #20 3.775 RedisReader -> /src/RedisReader/bin/Release/net6.0/RedisReader.dll #20 3.826 RedisReader -> /app/publish/ #20 DONE 4.0s #6 [base 2/2] WORKDIR /app #6 sha256:a01dd2d516cd2b9d43c2f898ff1f32ca74f47f1c9c7cc91189d15ebdd66a7adb #6 CACHED #7 [final 1/2] WORKDIR /app #7 sha256:fee5a31b081ff7b359025150181826c014703920ff1bfcc43f240ef44f62d6b9 #7 CACHED #21 [final 2/2] COPY --from=publish /app/publish . #21 sha256:6540cdec86f5f551ef54286c71b4345080c53042f6319879532deac1715d0869 #21 CACHED #22 exporting to image #22 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 #22 exporting layers done #22 writing image sha256:5f5396a8d3bf7028ec1fd16cdc33360421014f44b5d4e7d9c65563482143aa3a done #22 naming to docker.io/library/test-image #22 naming to docker.io/library/test-image done #22 DONE 0.1s
Here we can see that build step #13 successfully dotnet restore
s each project in our solution:
#13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:74e2154ed89f134032690bc64b9b8e54ea90f523b5cec48002e876399210ed50 #13 1.005 Determining projects to restore... #13 5.020 Restored /src/AnotherDependency/AnotherDependency.csproj (in 3.38 sec). #13 6.836 Restored /src/SharedGoodness/SharedGoodness.csproj (in 5.21 sec). #13 6.841 Restored /src/RedisReader/RedisReader.csproj (in 5.21 sec). #13 DONE 7.4s
Subsequent build steps that would otherwise invoke dotnet restore
now leverage the packages that have been restored already. This is first seen in build step #16
#16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:899127ee350d9e363af7c68268f5ea6f5d2cf999839fdbd4a36448054b5dc83d #16 0.456 MSBuild version 17.3.1+2badb37d1 for .NET #16 0.953 Determining projects to restore... #16 1.700 All projects are up-to-date for restore. #16 3.872 AnotherDependency -> /app/build/AnotherDependency.dll #16 3.875 SharedGoodness -> /app/build/SharedGoodness.dll #16 4.119 RedisReader -> /app/build/RedisReader.dll #16 4.131 #16 4.131 Build succeeded. #16 4.131 0 Warning(s) #16 4.131 0 Error(s) #16 4.131 #16 4.131 Time Elapsed 00:00:03.60 #16 DONE 4.3s
Leveraging the Cached Layer
After making a simple code change to BackgroundRedisReader.cs
in the RedisReader
project, and running a build command again, we receive the following output:
PS C:\_dev\CodeSloth\multi project docker debugging\DebuggingMultipleProjectsInDocker> docker build -t test-image --progress=plain -f .\RedisReader\Dockerfile . #1 [internal] load build definition from Dockerfile #1 sha256:5bc9a12452a9787ba4dda54f1c741f0d0f3b64c8c8f869f0f7942fd2cf146305 #1 transferring dockerfile: 32B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 sha256:0366d71e6ae57c56256c1eb5bf77078da3d629b5abb3cfa7943eb5c2543dc776 #2 transferring context: 35B done #2 DONE 0.1s #3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0 #3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b #3 ... #4 [internal] load metadata for mcr.microsoft.com/dotnet/runtime:6.0 #4 sha256:0128d5218b10a50bf55970db7c113f09503205e2b2ada4931bb5d0c6628fdd2a #4 DONE 0.0s #3 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0 #3 sha256:9eb4f6c3944cfcbfe18b9f1a753c769fc35341309a8d4a21f8937f47e94c712b #3 DONE 0.2s #8 [build 1/8] FROM mcr.microsoft.com/dotnet/sdk:6.0@sha256:a788c58ec0604889912697286ce7d6a28a12ec28d375250a7cd547b619f19b37 #8 sha256:fff7c57bbc14150de4574cecfd040bdf8a628dc4f5265c2e038bd3fd64bdd55a #8 DONE 0.0s #5 [base 1/2] FROM mcr.microsoft.com/dotnet/runtime:6.0 #5 sha256:2f0c599a662b466f9a09402b483721cd4263038964a3bba095f42135663b4c67 #5 DONE 0.0s #10 [internal] load build context #10 sha256:812dfbb1e7158c786fcc3f3878a25dd3a7525858b86c0b76b1a36c6e941626b9 #10 transferring context: 3.15kB 0.0s done #10 DONE 0.1s #11 [build 3/8] COPY ./*/*.csproj ./ #11 sha256:0c298d9854c5401db42d9517d165c850f63c767b970cc6636b6fa52da6695451 #11 CACHED #12 [build 4/8] RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done #12 sha256:36df46de93b50c64406cf42393646bf8fd5ad108dbaca02464ea772c64ca3127 #12 CACHED #9 [build 2/8] WORKDIR /src #9 sha256:fd721b9ab8612450c39dfab43831d16b893f71b294b4d92a8c1d6fdaf0c47a22 #9 CACHED #13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:29a09ab1157be002879b9cfa5bbb58a3dd794c4dd21db5ea33c17d6b6982ba74 #13 CACHED #14 [build 6/8] COPY . . #14 sha256:fc35031a353474a1174095b124326c5d9a19419ef990793fe6f018ef4a65a44f #14 DONE 0.1s #15 [build 7/8] WORKDIR /src/RedisReader #15 sha256:b967d1eb3f6dd38e9ff93ab87ae44ed8fd654edf58053cfee895e73f2b14c2b7 #15 DONE 0.1s #16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:c71c93c23a2f69737dd90713c6ceede86b0191b13143e9eea77a681ad9f1e958 #16 1.225 MSBuild version 17.3.1+2badb37d1 for .NET #16 1.865 Determining projects to restore... #16 2.829 All projects are up-to-date for restore. #16 5.360 AnotherDependency -> /app/build/AnotherDependency.dll #16 5.373 SharedGoodness -> /app/build/SharedGoodness.dll #16 5.651 RedisReader -> /app/build/RedisReader.dll #16 5.663 #16 5.663 Build succeeded. #16 5.663 0 Warning(s) #16 5.663 0 Error(s) #16 5.664 #16 5.664 Time Elapsed 00:00:04.32 #16 DONE 5.8s #17 [publish 1/4] WORKDIR /src #17 sha256:a39e4782dccf2c9775e2dde673421bed4682681d54a2a6cd47bf080e697844f8 #17 DONE 0.1s #18 [publish 2/4] COPY . . #18 sha256:4fdfaa9e0052b50d7d5a7694fb3bd320aaebf0bc8bda073264a743b4d905a34f #18 DONE 0.1s #19 [publish 3/4] WORKDIR /src/RedisReader #19 sha256:145dc9886e8a9a6dce9dc7b5b20d4fc12dd6e773a4e9af845d28cfaf28b42d0e #19 DONE 0.2s #20 [publish 4/4] RUN dotnet publish "RedisReader.csproj" -c Release -o /app/publish /p:UseAppHost=false #20 sha256:4f1eb3086dd2b35a5281125bdfbe0dfc6d636aa872b17778fe645908b1e9a516 #20 0.473 MSBuild version 17.3.1+2badb37d1 for .NET #20 0.914 Determining projects to restore... #20 1.622 All projects are up-to-date for restore. #20 2.104 AnotherDependency -> /src/AnotherDependency/bin/Release/net6.0/AnotherDependency.dll #20 2.221 SharedGoodness -> /src/SharedGoodness/bin/Release/net6.0/SharedGoodness.dll #20 3.945 RedisReader -> /src/RedisReader/bin/Release/net6.0/RedisReader.dll #20 3.998 RedisReader -> /app/publish/ #20 DONE 4.1s #6 [base 2/2] WORKDIR /app #6 sha256:a01dd2d516cd2b9d43c2f898ff1f32ca74f47f1c9c7cc91189d15ebdd66a7adb #6 CACHED #7 [final 1/2] WORKDIR /app #7 sha256:fee5a31b081ff7b359025150181826c014703920ff1bfcc43f240ef44f62d6b9 #7 CACHED #21 [final 2/2] COPY --from=publish /app/publish . #21 sha256:02847984d1a53f3adc90064209dbdbea869665f894d7ba4934e3b259b2604cd7 #21 DONE 0.2s #22 exporting to image #22 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00 #22 exporting layers #22 exporting layers 0.1s done #22 writing image sha256:90174d429011fb9d3ea682a1072acab363bdc9f6b8a0048e357328ff6af39327 done #22 naming to docker.io/library/test-image done #22 DONE 0.2s
Once again we will zoom into the important parts of the above output.
Below we can see that build step #13 has leveraged the cached layer for dotnet restore
. It has been able to do this, because the csproj
files copied into the container were unmodified. This means that the step has not needed to download packages from the internet into the container and instead has used the local Docker cached layer, which accurately represents the restored NuGet package dependencies of the project:
#13 [build 5/8] RUN dotnet restore "RedisReader/RedisReader.csproj" #13 sha256:29a09ab1157be002879b9cfa5bbb58a3dd794c4dd21db5ea33c17d6b6982ba74 #13 CACHED
We can also see in build step #16 that dotnet build
has been re-run against the modified source code. This is because a cs
file was changed, and meant that build step #14 and onwards could not used cached layers. This time instead of leveraging the freshly restored NuGet packages from dotnet restore
within the build
stage, it has used the cached layer’s packages from step #13 when rebuilding the project:
#16 [build 8/8] RUN dotnet build "RedisReader.csproj" -c Release -o /app/build #16 sha256:c71c93c23a2f69737dd90713c6ceede86b0191b13143e9eea77a681ad9f1e958 #16 1.225 MSBuild version 17.3.1+2badb37d1 for .NET #16 1.865 Determining projects to restore... #16 2.829 All projects are up-to-date for restore. #16 5.360 AnotherDependency -> /app/build/AnotherDependency.dll #16 5.373 SharedGoodness -> /app/build/SharedGoodness.dll #16 5.651 RedisReader -> /app/build/RedisReader.dll #16 5.663 #16 5.663 Build succeeded. #16 5.663 0 Warning(s) #16 5.663 0 Error(s) #16 5.664 #16 5.664 Time Elapsed 00:00:04.32 #16 DONE 5.8s
Sloth Summary
The default Dockerfile that Visual Studio generates for us is great. However, we’ve seen in this tutorial how we can make a couple of small adjustments to it that will reduce the maintenance burden that would otherwise be associated with maintaining its default form alongside your evolving solution.
By dynamically finding all csproj
files in your repository (or specific subfolders of your repository) and copying them into a folder structure that matches your local disk, the referential integrity of project dependencies is kept in tact. This allows us to create highly cacheable Docker layers from the bare minimum dependencies for dotnet restore
that will serve to decrease our Docker build times during future runs.