First off I would want to minimise any unwanted coupling to the toroidal output coil, so what you posted earlier today with red and yellow traces needs looking at.
Yes, we should determine what the ratio of this coupling is. e.g. the ratio of the current amplitude fed into the pancake coils vs. received by the toroidal coil.
For the calculation of this ratio the I/O impedance should be known.
A plot of the insertion loss vs. frequency (via two-port S21 measurement), would also give us information whether the crosstalk is capacitive or inductive in nature, because the impedance of acoustic and capacitive crosstalk decreases with frequency, and the impedance of inductive crosstalk increases with frequency.
Ideally with no water present and both coils tuned to the NMR frequency we want zero coupling.
That would be ideal, indeed.
With the toroidal coil's field at right angles to the pancake coil's field there should be zero magnetic coupling, so it is probably down to capacitive coupling.
No. You forgot that Itsu's toroidal winding has only one layer ...and that means that this coil has non-negligible sensitivity to the circumferential currents present in the pancake coils.
Another way to look at it is noticing that the wires in the pancake coil are not perpendicular to the wires in the toroidal coil. The latter has a winding pitch, which results in 1 turn circumferential component.
Of course, this can be cancelled with properly connected bifilar coil or even-layered coil.
Although you correctly have the pancake coils in series opposing there are two ways this can be set up, either the outer connection of one pancake is connected to the inner connection of the other, or the connection is at both outers or both inners (the opposing field determined by which way round the pancakes are placed).
Winding pancake coils in opposite directions constitutes another option. This can also avoid the plexi coil-formers being on different sides of the coils.
Also, an effort should be made to cancel their collective radial current, which arises as the electric charges move from the outer to the inner radius of the pancake coil (or vice versa).
With an unbalanced drive from the signal source I can't see any way to eliminate capacitive coupling for either configuration. Ideally it requires a balanced drive, then one configuration is better than the other. If the drive is at both outers while the series connection is at both inners (or vice versa), then the capacitive coupling is minimised. So I am sorry to say that you need a wide bandwidth balun, something you have already played with on another thread.
I agree that balanced drive and Rx will help a little.
One you achieve minimum unwanted coupling, then you can add water and look for the NMR signal which by its gyratory action couples the 90 degree input and output fields together.
The unique property of this signal is that it persists after the stimulus from the pancake coils ceases, so temporal separation is another method to minimize the crosstalk.
P.S.
At the risk of sounding nauseating. Litz wire provides clear benefits besides presenting lower AC impedance at RF.
For example, I just noticed that an
unlooped piece of 1mm solid copper wire gets too hot to touch in the proximity of a 75W 100kHz inductive heater (w/ solenoidal coil).
Notice, that this is a straight piece of wire, which is
not connected in a loop, so circular eddy currents in it, can have the maximum diameter of 1mm.
Yet, significant energy transfer* happens anyway. It happens
regardless of this piece of this wire being oriented perpendicularly to the heater's coil or parallel to it. It also does not matter whether this piece of wire is
not inside the heater's solenoid - 5mm outside of heater's coil is enough (with perpendicular and parallel wire orientations, too).
An unlooped piece of a Litz wire (of the same diameter) does not experience any noticeable heating in any orientation and position.
* I'll let Smudge calculate whether eddy currents in this straight unlooped solid wire, which can raise its temperature by +25ºC, can affect the conductivity along this wire or induce electric potentials along this wire ...and how much quantitatively.