Recently, we were working on applying lookup filtering using JS on a custom entity form. While testing it, we got an error on each key press on the lookup field “0x80040203 Invalid Argument”.
We were constructing the record filter criteria based on certain condition. While debugging the code, we found that the filter criteria(FetchXML filter tag) we were constructing was not in the correct format for filtering the records.
Putting appropriate condition in place/modifying the existing condition, we managed to consturct the filter criteria properly and the issue got fixed.
Hope it helps !!
Ajit Patra 