Having looked into a few options, I think I am settled on the following approach for unit testing FireStarter:
Using Firebase or Firebase Emulator in tests would allow end to end tests to prove certain functions are working. However this comes with some issues.
Firstly testing like this is slow, and so you can only test a small fraction of what you want if you want your tests to run reasonably.
Secondly testing like this can be flaky, as emulator memory leaks or network errors connecting to Firebase can cause test failures.
Thirdly, it is time consuming for the developer to maintain tests like this in my experience, the setup/teardown becomes complex. Testing even simple scenarios gets hairy!
Jest seems a reasonable choice, works well with NextJS, and has a good mocking library. Other options might be just as good. I haven't checked them out though.
With the Version 9 Firebase API that lets you import functions directly, it is fairly easy to mock those out using Jest. There isn't much alternative anyway, as NPM packages that do this seem to have gone out of date.
I am using a facade called IBackend.ts. This means that the forms interact with IBackend, and can mocked that out in UI tests. I then separately test the backed implementation.
Reasons for this:
Simplify front end code.
Possible to create different back ends in the future, for example a Parse backend.
Simplify tests. More tests, but each test has to test less.
Changes to Firebase API doesn't affect components and UI code.
That's it for now. The main message here is that when working on a framework I can think more about these design decisions as there is not really a deadline.
These testing decisions should give the magic of great test coverage, tests that run quickly and reliably while being able to discover real bugs and prove that we are handling all the API error codes that might come back.