Bound cofactor in EC_GROUP_set_generator()

Instead of bounding only bounding the group order, also bound the
cofactor using Hasse's theorem. This could probably be made a lot
tighter since all curves of cryptographic interest have small
cofactors,  but for now this is good enough.

A timeout found by oss-fuzz creates a "group" with insane parameters
over a 40-bit field: the order is 14464, and the cofactor has 4196223
bits (which is obviously impossible by Hasse's theorem). These led to
running an expensive loop in ec_GFp_simple_mul_ct() millions of times.

Fixes oss-fuzz #46056

Diagnosed and fix joint with jsing

ok inoguchi jsing (previous version)

0 files changed, 0 insertions, 0 deletions