Training LR update Formula

Let B0 = 124 and LR0 = 0.025 (your tuned baseline).

For a new effective batch size B:

LR_linear = 0.025 * (B / 124)

LR_sqrt   = 0.025 * sqrt(B / 124)

Use linear if B is within 2-4x of 124.

Use sqrt if B is much larger (e.g., >1000) or if linear diverges. 

Training LR update/change formulla based on batch_size & number of gpus: 

Let η_old, B_old, G_old be some previous setup, and we want η_new for B_new, G_new. Use both linear and √ rules. For illustration, assume momentum SGD. (We denote total global BS = B×G here.)

1. Example A: Old: BS=256, GPUs=1, η_old=0.10. New: BS=194, GPUs=1.

• Global scale k = (194×1)/(256×1) = 0.758.

• Linear: η_new = 0.10×0.758 = 0.0758.

• √-rule: η_new = 0.10×√0.758 ≈ 0.0869.

2. Example B: Old: BS=512 total (64 on 8 GPUs), η_old=0.10 (as used in Partial-FC). New: BS=194 total (on 1 GPU).

• k = 194/512 ≈ 0.379.

• Linear: η_new = 0.10×0.379 = 0.0379.

• √-rule: η_new = 0.10×√0.379 ≈ 0.0616.

3. Example C: Old: BS=128 (per GPU) on 4 GPUs (total 512), η_old=0.05. New: BS=194 on 1 GPU (total 194).

• k = 194/512 ≈ 0.379 (same ratio as B).

• Linear: η_new = 0.05×0.379 ≈ 0.01895.

• √-rule: η_new = 0.05×√0.379 ≈ 0.0308.

4. Example D (AdamW): Old: BS=64 on 4 GPUs (total 256), η_old=0.001 (AdamW). New: BS=194 on 1 GPU (total 194).

• k = 194/256 ≈ 0.758.

• Linear (for AdamW, often too aggressive): η_new_lin = 0.001×0.758 = 0.000758.

• √-rule (more typical for Adam): η_new = 0.001×√0.758 ≈ 0.000872.
  Here one might conservatively use η≈9e-4.

c++ 20 features std:views, transform, filter

#include <algorithm>

#include <cctype>

#include <functional>

#include <iostream>

#include <string>

#include <vector>

#include <ranges>

using namespace std;

struct student

{

    string name;

    string roll;

    string dob;

    string departmnt;

    

};

int main()

{

    auto show_student = [](const student& s)

                            { 

                                cout <<"Name:       "<< s.name << endl; 

                                cout <<"Roll:       "<< s.roll << endl; 

                                cout <<"DOB:        "<< s.dob << endl; 

                                cout <<"Department: "<< s.departmnt << endl; 

                            };

    

    auto show_data = [](const string& s)

                            { 

                                cout << s << endl; 

                            };

                            

    auto uppercase = [](student s) -> student {

        

        std::transform(s.name.begin(), s.name.end(), s.name.begin(), ::toupper);

        return s;

    };

    

    auto formatname = [](const string& s) -> string {

        

        std::string name = "";

        bool upper = true;

        for(int i=0;i<s.size();i++)

        {

            if(s[i] >= 'a' && s[i] <= 'z')

            {

                if(upper)

                {

                    name.push_back( (s[i] - 'a') + 'A');

                    upper = false;

                }

                else

                    name.push_back( s[i] );

            }

            else if(name.size() > 0 && (s[i]=='.' || s[i]==' '))

            {

                if(name[name.size()-1]=='.' || name[name.size()-1]==' ')

                {

                    

                }

                else

                {

                    name.push_back( s[i] );

                    if(s[i]=='.')

                    name.push_back(' ');

                }

                upper = true;

            }

            else if(s[i] >= 'A' && s[i] <= 'Z')

            {

                if(upper)

                {

                    name.push_back( s[i]);

                    upper = false;

                }

                else

                    name.push_back( (s[i]-'A' + 'a') );

            }

        }

        return name;

    };

    

    vector<student> students;

    

    students.push_back({.name="MD. NAzmul KIBRia", .roll="0304015", .dob="01-01-1985", .departmnt="CSE"});

    students.push_back({.name="Md. Shakil Ahmed",  .roll="0303013", .dob="01-01-1986", .departmnt="CIVIL"});

    students.push_back({.name="Md. Rezwan Salam",  .roll="0302025", .dob="01-01-1984", .departmnt="ELECTRICAL"});

    students.push_back({.name="md.omar faruque",   .roll="0304020", .dob="01-01-1983", .departmnt="CSE"});

    std::ranges::sort(students, [](auto&l, auto&r){return l.roll < r.roll; });

    std::ranges::for_each(students, show_student);

    

    std::cout <<std::endl<< "CSE students: " << std::endl;

    auto cse_students = students | std::views::filter([](const student&s){ return s.departmnt == "CSE";});

    std::ranges::for_each(cse_students, show_student);

    

    std::cout <<std::endl<< "CSE Student Names: " << std::endl;

    std::ranges::for_each(students | std::views::filter([](const student&s){ return s.departmnt == "CSE";})

                                   | std::views::transform([](const student&s) -> std::string {return s.name;})

                                   | std::views::transform(formatname)

                                   ,show_data);

    

    std::cout <<std::endl<< "CSE Student Rolls: " << std::endl;

    std::ranges::for_each(students | std::views::filter([](const student&s){ return s.departmnt == "CSE";})

                                   | std::views::transform([](const student&s) -> std::string {return s.roll;})

                                   ,show_data);

    std::cout << std::endl<<"============" << std::endl;

    std::ranges::for_each(students, show_student);

    

    return 0;

}

Outputs:

Name:       Md. Rezwan Salam

Roll:       0302025

DOB:        01-01-1984

Department: ELECTRICAL

Name:       Md. Shakil Ahmed

Roll:       0303013

DOB:        01-01-1986

Department: CIVIL

Name:       MD. NAzmul KIBRia

Roll:       0304015

DOB:        01-01-1985

Department: CSE

Name:       md.omar faruque

Roll:       0304020

DOB:        01-01-1983

Department: CSE

CSE students: 

Name:       MD. NAzmul KIBRia

Roll:       0304015

DOB:        01-01-1985

Department: CSE

Name:       md.omar faruque

Roll:       0304020

DOB:        01-01-1983

Department: CSE

CSE Student Names: 

Md. Nazmul Kibria

Md. Omar Faruque

CSE Student Rolls: 

0304015

0304020

============

Name:       Md. Rezwan Salam

Roll:       0302025

DOB:        01-01-1984

Department: ELECTRICAL

Name:       Md. Shakil Ahmed

Roll:       0303013

DOB:        01-01-1986

Department: CIVIL

Name:       MD. NAzmul KIBRia

Roll:       0304015

DOB:        01-01-1985

Department: CSE

Name:       md.omar faruque

Roll:       0304020

DOB:        01-01-1983

Department: CSE 

Designated Initializer in C vs CPP

 C supports a powerful designated initializes specially for Array & Struct in any order. Doesn't depend on declaration order. It was introduced since 1999. Here is an example: 

The output of the above program is:

C supports nested, mixed designated initialization: 

C++ initiated this designated initialization in c++20 in year 2020 and we can say it as a limited version. It depends on declaration order of the struct variables. We can not use any order like C language. Also we can not use it for Array :(

Only we can skip some middle variables which will be initialized by zero but we must need to follow it's original declaration order.

Here is an example of c++20:

C++ supports mixed type to designated initialization but doesn't support nested type :(

Rules for Designated Initializers (CPP):

1. Each data member can have only one designator.  

2. Designators are applicable only for aggregate initialization.  

3. Nested designators are not allowed.  

4. Regular initialization cannot be combined with designators in the same expression.  

5. It is not mandatory to specify all data members in the initialization expression.  

6. Designators can reference only non-static data members.  

7. The order of designators in the initialization expression must match the order of data members in the class declaration.  

Advantages of Designated Initialization

    Readability: By specifying the exact data member being initialized, the designator ensures clarity and eliminates the possibility of errors.

   Flexibility: It allows you to skip initializing certain data members and depend on their default values instead.

   Compatibility with C: A similar initialization syntax is widely used in C99 (with even more relaxed rules). The C++20 feature enables writing nearly identical code, facilitating code sharing between C and C++.

   Standardization: While compilers like GCC and Clang already provided extensions for this feature, its inclusion in the standard ensures uniform support across all compilers.

Ref: C Struct Initialization & CPP Designated initialization & Article

Modify a private const variable from outside of class in c++

 Is it possible to modify a private const variable from outside the class in cpp? No it doesn't and it should not supposed to happen.

Unfortunately we can do this with pointer. How? Let's see the example:

#include <iostream>

class Test
{

    private:
    const int cval1 = 1;
    const int cval2 = 2;

    public: 
    void print()
    { 
        std::cout << "const value1 = " << cval1 << std::endl;
        std::cout << "const value2 = " << cval2 << std::endl;

    }    

};

int main()
{

    Test t;
    t.print();

    int * p = reinterpret_cast<int*>(&t);
   *p = 10; //modifying 1st integer
   *(p+1) = 20; //modifying 2nd integer

    std::cout << "After modification: " << std::endl;
    t.print();

    return 0;
}