rsa: fix bn_reduce_once_in_place call for rsaz_mod_exp_avx512_x2

bn_reduce_once_in_place expects the number of BN_ULONG, but factor_size is moduli bit size. Fixes #18625. Signed-off-by: Xi Ruoyao <xry111@xry111.site> Reviewed-by: Tomas Mraz <tomas@openssl.org> Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from #18626)
openssl · Jun 23, 2022 · 4d8a88c · 4d8a88c
1 parent eea820f
commit 4d8a88c
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/crypto/bn/rsaz_exp_x2.c b/crypto/bn/rsaz_exp_x2.c
@@ -257,6 +257,9 @@ int ossl_rsaz_mod_exp_avx512_x2(BN_ULONG *res1,
     from_words52(res1, factor_size, rr1_red);
     from_words52(res2, factor_size, rr2_red);
 
+    /* bn_reduce_once_in_place expects number of BN_ULONG, not bit size */
+    factor_size /= sizeof(BN_ULONG) * 8;
+
     bn_reduce_once_in_place(res1, /*carry=*/0, m1, storage, factor_size);
     bn_reduce_once_in_place(res2, /*carry=*/0, m2, storage, factor_size);