Error when running methods in parallel (using Parallel.ForEach)

Hello,

We recently migrated the project to the ASP.NET Core. After this, we faced a bug with code that runs several methods in parallel.

We have a product mapper class that gets prepared variant models using this code:

var variants = new ConcurrentDictionary<Variant, VariantDto>();
Parallel.ForEach(
    activeVariants,
    new ParallelOptions
    {
        MaxDegreeOfParallelism =
            Convert.ToInt32(Environment.ProcessorCount * 0.8), // max 80% CPU
    },
    v => variants.TryAdd(v, _variantMapper.Map(v)));

Inside "_variantMapper.Map" we use "PromotionEngine.Run()"method and "CustomerContext.Current.GetContactById(customerId)" method. And when loading the product page, one of these methods throws an error every time. Here are examples of these errors:

• CustomerContext.Current.GetContactById(customerId) - throws "Object reference not set to an instance of an object" exception.
  • Stack trace:

       at Mediachase.BusinessFoundation.Data.Business.BusinessManager.Execute(Request request)
       at Mediachase.BusinessFoundation.Data.Business.BusinessManager.Load(String metaClassName, PrimaryKeyId primaryKeyId)
       at Mediachase.Commerce.Customers.CustomerContext.InnerGetContactById(Guid contactId)
       at Mediachase.Commerce.Customers.CustomerContext.<>c__DisplayClass32_0.<GetContactById>b__0()
       at Mediachase.Commerce.Customers.CustomersCache.<>c__DisplayClass6_0`1.<ReadThrough>g__WrapperAction|0()
       at EPiServer.Framework.Cache.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, String key, Func`1 readValue, Func`2 evictionPolicy)
       at Mediachase.Commerce.Extensions.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, Boolean useCache, String cacheKey, IEnumerable`1 masterKeys, TimeSpan duration, Func`1 load)
       at Mediachase.Commerce.Extensions.ObjectInstanceCacheExtensions.ReadThrough[T](IObjectInstanceCache cache, String cacheKey, IEnumerable`1 masterKeys, TimeSpan duration, Func`1 load)
       at Mediachase.Commerce.Customers.CustomersCache.ReadThrough[T](String key, IEnumerable`1 masterKeys, TimeSpan timeout, Func`1 load)
       at Mediachase.Commerce.Customers.CustomerContext.GetContactById(Guid contactId)
       at Commerce.Carts.Extensions.IOrderRepositoryExtensions.GetCart(IOrderRepository orderRepository, Guid customerId, String cartName) in ...\Extensions\IOrderRepositoryExtensions.cs:line 15
       at Commerce.Carts.CartService.Get(Guid customerId) in ...CartService.cs:line 66
       at Commerce.Carts.CartService.Get() in ...CartService.cs:line 61
       at Commerce.Variants.VariantMapper.GetAllowedQuantity(Variant variant, Boolean& isInCart, Decimal qtyPerBaseUnitOfMeasure) in ...VariantMapper.cs:line 645
       at Commerce.Variants.VariantMapper.Map(Variant variant) in ...VariantMapper.cs:line 358
       at Commerce.Products.ProductMapper.<>c__DisplayClass14_0`1.<Map>b__5() in ...ProductMapper.cs:line 117

    
    
    
• PromotionEngine.Run() - throws "Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct" exception.
  • Stack trace: 

       at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported()
       at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
       at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
       at EPiServer.Personalization.VisitorGroups.VisitorGroupRole.IsMatchCached(IPrincipal principal, HttpContext httpContext)
       at EPiServer.Personalization.VisitorGroups.VisitorGroupRole.IsMatch(IPrincipal principal, HttpContext httpContext)
       at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.IsInVisitorGroup(Guid visitorGroupId, HttpContext httpContext, String campaignName)
       at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.ShouldFilter(ContentReference campaignLink, IDictionary`2 cachedVisitorGroupLookup, PromotionFilterContext filterContext)
       at EPiServer.Commerce.Marketing.CampaignVisitorGroupFilter.Filter(PromotionFilterContext filterContext)
       at EPiServer.Commerce.Marketing.PromotionFilters.Filter(IEnumerable`1 allPromotions, IEnumerable`1 couponCodes, RequestFulfillmentStatus requestedStatus)
       at EPiServer.Commerce.Marketing.PromotionEngine.Run(IOrderGroup orderGroup, PromotionEngineSettings settings)
       at Commerce.Discounts.CustomPromotionEngine.Run(IOrderGroup orderGroup, PromotionEngineSettings settings) in ...CustomPromotionEngine.cs:line 42
       at Commerce.Marketing.IPromotionEngineExtensions.GetDiscountedPrice(IPromotionEngine promotionEngine, VariationContent variant, IMarket market, RequestFulfillmentStatus requestFulfillmentStatus, IPriceValue price) in ...IPromotionEngineExtensions.cs:line 65
       at Commerce.Variants.VariantMapper.GetSaleAndDefaultPrices(Variant variant, ProductRelationshipDto outletParent, IMarket market, Nullable`1 ctoCode, Boolean isAvailableForCurrentCustomer, Boolean procaseEnabled, IReadOnlyList`1& promotions) in ...VariantMapper.cs:line 421
       at Commerce.Variants.VariantMapper.Map(Variant variant) in ...VariantMapper.cs:line 367
       at Commerce.Products.ProductMapper.<>c__DisplayClass14_0`1.<Map>b__5() in ...ProductMapper.cs:line 117

I found this article https://world.optimizely.com/blogs/Johan-Bjornfot/Dates1/2023/8/parallel-tasks-and-backgroundcontext/ and tried to use "IBackgroundContextFactory.Create()" inside the lambda method in "Parallel.ForEach". Here's the updated code:

Parallel.ForEach(
    activeVariants,
    new ParallelOptions
    {
        MaxDegreeOfParallelism =
            Convert.ToInt32(Environment.ProcessorCount * 0.8), // max 80% CPU
    },
    v =>
    {
        using (_backgroundContextFactory.Create())
        {
            variants.TryAdd(v, _variantMapper.Map(v));
        }
    });

Thanks to this code, the number of errors has decreased. Only one type of error occurs: "Object reference not set to an instance of an object" inside the "CustomerContext.Current.GetContactById(Guid contactId)" (it seems that "PromotionEngine.Run()" use this method too). And I only got one of these errors the first time I loaded the page, and the next time I loaded the product page it opened (at least the error didn't occur every time I loaded the page). But if I reload the page in the browser several times right after it loads, I can still catch one of these errors.

I'm not sure what we can do from our code side to get rid of this error. Can you help me how to properly run several methods in parallel so that a this exception doesn't occur?

Thanks!