Training LR update Formula

Let B0 = 124 and LR0 = 0.025 (your tuned baseline).

For a new effective batch size B:

LR_linear = 0.025 * (B / 124)

LR_sqrt   = 0.025 * sqrt(B / 124)

Use linear if B is within 2-4x of 124.

Use sqrt if B is much larger (e.g., >1000) or if linear diverges. 

Training LR update/change formulla based on batch_size & number of gpus: 

Let η_old, B_old, G_old be some previous setup, and we want η_new for B_new, G_new. Use both linear and √ rules. For illustration, assume momentum SGD. (We denote total global BS = B×G here.)

1. Example A: Old: BS=256, GPUs=1, η_old=0.10. New: BS=194, GPUs=1.

• Global scale k = (194×1)/(256×1) = 0.758.

• Linear: η_new = 0.10×0.758 = 0.0758.

• √-rule: η_new = 0.10×√0.758 ≈ 0.0869.

2. Example B: Old: BS=512 total (64 on 8 GPUs), η_old=0.10 (as used in Partial-FC). New: BS=194 total (on 1 GPU).

• k = 194/512 ≈ 0.379.

• Linear: η_new = 0.10×0.379 = 0.0379.

• √-rule: η_new = 0.10×√0.379 ≈ 0.0616.

3. Example C: Old: BS=128 (per GPU) on 4 GPUs (total 512), η_old=0.05. New: BS=194 on 1 GPU (total 194).

• k = 194/512 ≈ 0.379 (same ratio as B).

• Linear: η_new = 0.05×0.379 ≈ 0.01895.

• √-rule: η_new = 0.05×√0.379 ≈ 0.0308.

4. Example D (AdamW): Old: BS=64 on 4 GPUs (total 256), η_old=0.001 (AdamW). New: BS=194 on 1 GPU (total 194).

• k = 194/256 ≈ 0.758.

• Linear (for AdamW, often too aggressive): η_new_lin = 0.001×0.758 = 0.000758.

• √-rule (more typical for Adam): η_new = 0.001×√0.758 ≈ 0.000872.
  Here one might conservatively use η≈9e-4.