Does the illustrations affect the conversion rate compared to the product screenshot?

Honestly, whatever tells your story is fine.

I'd spend way way way more effort on identifying :

1. who you're talking to
2. what you're solving for them
3. communicating that to them.

This is what drives conversions.

Couple of examples:

https://www.mintmachine.xyz/ < A site for managing NFT drops, each section addresses something very intrinsic to running an NFT drop at scale.

https://smallandroidphone.com/ < my current project... total long form content, no design whatsoever (we updated the fonts recently, but original site used system fonts 😂) -- 10 to 40% of visitors convert to community members (depending on inbound source).

It's what you're saying that matters -- connect with your visitor and their needs and they will become a user / customer.

Totally agree with @bhbryant, landing page must tell the story, and describe what is problem, and how your product solve that.

I highly recommend to have:

• Demo of product, it can be even with dummy content.
• Easy way to signup without entering card details + trial period (14 or 7 days)

There are too many variables to state an ultimatum either way.
However, as long it looks good, it's not your biggest worry early on.

I don't find product screenshots that useful when I visit websites because you miss a lot of context on your landing page.

They are useful on features pages or tutorials, but not in the first 30 seconds.

Personally, I use an interactive chart to help sell the idea as the visitor will see the value provided by using my tool. It doesn't matter so much what the tool looks like.

Ideally this is what your illustration will do, showcase the value you provide!

It depends a lot on the illustrations, most landing pages nowadays use the same stock cartoonish illustrations, which make the page look generic and cheap.