Aside from the reasons already mentioned for why this is a terrible plan; it strikes me that trying to use chatbots to catch people is putting yourself on the wrong side of an asymmetric struggle.
If you induce a person to suspend disbelief, some surprisingly rudimentary systems can keep them busy for quite a while(eg. Tamagotchi, furbies, ELIZA, whatever is behind that attractive stock photo that allegedly wants to hook up with me and lives conveniently nearby, etc.). Sometimes they just don't care that the bot isn't human; sometimes a bit of wishful thinking or interests confined to a very specific subject keeps them from probing too hard and allows a relatively small library of well tuned responses to do the job.
If, however, someone is inclined to be a bit suspicious(as is reasonable, if you are chatting with an unknown party about doing something notably illegal), the problem is vastly harder. At a minimum, you'd need to be able to pass a somewhat adversarial Turing test quite reliably; and/or build rapport so effectively that the person you are talking to will like/trust/otherwise affectively glom-on to you hard enough to let their guard down. Unless somebody is keeping really quiet about it; we don't have anything nearly that sophisticated on the table. Team Computational Linguistics is having a good day if they can handle my use of a vague pronoun reference or about a zillion other sloppy and ambiguous things that somehow mostly don't keep us from understanding each other; much less doing all that and being my cyber-jihad buddy.
So, your 'terrorist detector' bot would need downright uncanny abilities just to get to the point where we can start talking about the dubious wisdom of mass-scale incitement and entrapment. On the other hand, y'know what would be amenable to a much simpler(and vastly more plausible, quite possibly even 'off the shelf' with a bit of retraining for something that isn't advertising or customer 'service')? A 'terrorist' bot.
Dubious grammar? Sorry, English not his first language. Inability to talk plausibly about general-knowledge topics/random human chitchat? He's just here because he wants to learn more about jihad; and he's cautious about talking about his personal life and not interested in talking about the weather.
That seems like starting a fight you really, really, aren't going to have a good time winning. Emulating a wannabe terrorist looking to do a little self-radicalizing at least well enough to waste a nonzero amount of some relatively expensive spook's time looks like a problem amenable to the sorts of 'agents' that we are already seeing in commercial use for selling stuff, spying on consumers, and providing bad customer service. Emulating the mentor/co-conspirator that a wannabe terrorist is looking for, on the other hand, is going to need something considerably more impressive to even reach the point where it is effective enough for the "Are you sure this isn't a terrible idea?" level criticism becomes relevant.