Regtest Mode development for Feathercoin 0.9.6.2

[wrapper]

https://github.com/bitcoin/bitcoin/issues/4242

If you run setgenerate true too rapidly in regtest mode, bitcoind can crash with

/home/ubuntu/install/include/boost/thread/pthread/condition_variable_fwd.hpp:81: boost::condition_variable::~condition_variable(): Assertion `!ret' failed.

This was reproduced with the Linux 64bit gitian binary. 

Sometimes it kills bitcoind without a visible asert failure. Sometimes bitcoind just misses the RPC entirely without any visible error and without killing bitcoind.

[wrapper]

[FAQ] Where are coins generated, in software terms?

Regtest and Testnet are special cases of normal mining.

Each has a set of parameters, ports and directories to work with but otherwise goes through the same code.

Testnet set-ups are done to test an event, like a fork, regression testing is similar but does not connect to the internet.

Instead of a miner regtest uses ./feathercoin-cli and passed commands to feathercoind to mine some coins.

 ./feathercoin-cli -regtest setgenerate true 1

setgenerate is deprecated and the new command will be generate for later (than 0.9.6.2) versions.

miner.cpp

 void GenerateBitcoins(bool fGenerate, CWallet* pwallet, int nThreads) 
 {
     static boost::thread_group* minerThreads = NULL;
     printf("Here in coin generation\n"); // RegTest Debug
     if (nThreads < 0) {
         if (Params().NetworkID() == CChainParams::REGTEST)
             nThreads = 1;
         else
             nThreads = boost::thread::hardware_concurrency();
    }

     if (minerThreads != NULL)
     {
        minerThreads->interrupt_all();
         delete minerThreads;
         minerThreads = NULL;
     }

     if (nThreads == 0 || !fGenerate)
         return;

     minerThreads = new boost::thread_group();
     for (int i = 0; i < nThreads; i++)
        minerThreads->create_thread(boost::bind(&FeathercoinMiner, pwallet));
 }

[wrapper]

Current status getting regtest running :

Various issues are affecting how complex it is to set up regtest. Whilst some are addressed there is still an unknown issue.

Creating Genesis Block and regtest directory

./feathercoind -regtest -listen=0 -debug -daemon
Searching for genesis block...
genesis.nTime = 1396255061 
genesis.nNonce = 3250989159 
genesis.nVersion = 3 
genesis.hashMerkleRoot = 97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9 
genesis.GetHash = 29d5f68acf1fb95303120bb51c35a32d270bd736235f928de49b94d87a3782a8
Feathercoin server starting

ReserverKeyFromPool 1.1 - wallet.cpp
ReserverKeyFromPool 1.2 - wallet.cpp
ReserverKeyFromPool 1.3 - wallet.cpp
ReserverKeyFromPool 2.1 - wallet.cpp
ReserverKeyFromPool 2.2 - wallet.cpp 
ReserverKeyFromPool 2.3 - wallet.cpp
ReserverKeyFromPool 3.1 - wallet.cpp
ReserverKeyFromPool 3.2 - wallet.cpp
ReserverKeyFromPool 3.3 - wallet.cpp
ReserverKeyFromPool 3.4 - wallet.cpp
ReserverKeyFromPool 4.1 - wallet.cpp

Getting Blockchain info :

 ./feathercoin-cli -regtest getblockchaininfo
 {
 "chain" : "regtest",
 "blocks" : 0,
 "bestblockhash" : "29d5f68acf1fb95303120bb51c35a32d270bd736235f928de49b94d87a3782a8",
 "difficulty" : 0.00000000,
 "verificationprogress" : 1.00000000,
 "chainwork" : "0000000000000000000000000000000000000000000000000000000000000002"
  }

generatebitcoins 1.1 - rpcmining.cpp
FeathercoinMiner
FeathercoinMiner 2
FeathercoinMiner 2.1
FeathercoinMiner 2.2
In CreateNewBlockWithKey 1 miner.cpp
InGetReservedKey 1.1-wallet.cpp
InGetReservedKey 1.1.1-wallet.cpp
ReserverKeyFromPool 1.1 - wallet.cpp
ReserverKeyFromPool 1.2 - wallet.cpp
ReserverKeyFromPool 1.3 - wallet.cpp

Where it is hanging on creating a block, whilst getting a public key in wallet.dat :

void CWallet::ReserveKeyFromKeyPool(int64_t& nIndex, CKeyPool& keypool)
{
    printf("ReserverKeyFromPool 1.1 - wallet.cpp\n"); // RegTest Debug
    nIndex = -1;
    printf("ReserverKeyFromPool 1.2 - wallet.cpp\n", keypool.vchPubKey); // RegTest Debug
    keypool.vchPubKey = CPubKey();
    printf("ReserverKeyFromPool 1.3 - wallet.cpp\n"); // RegTest Debug
    {
        LOCK(cs_wallet);
        printf("ReserverKeyFromPool 2.1 - wallet.cpp\n"); // RegTest Debug
        if (!IsLocked()) {
        printf("ReserverKeyFromPool 2.2 - wallet.cpp\n"); // RegTest Debug          
            TopUpKeyPool();
        printf("ReserverKeyFromPool 2.3 - wallet.cpp\n"); // RegTest Debug
        }
        // Get the oldest key
        if(setKeyPool.empty())
            return;
        printf("ReserverKeyFromPool 3.1 - wallet.cpp\n"); // RegTest Debug
        CWalletDB walletdb(strWalletFile);
        printf("ReserverKeyFromPool 3.2 - wallet.cpp\n"); // RegTest Debug
        nIndex = *(setKeyPool.begin());
        printf("ReserverKeyFromPool 3.3 - wallet.cpp\n"); // RegTest Debug
        setKeyPool.erase(setKeyPool.begin());
        printf("ReserverKeyFromPool 3.4 - wallet.cpp\n"); // RegTest Debug
        if (!walletdb.ReadPool(nIndex, keypool))
        throw runtime_error("ReserveKeyFromKeyPool() : read failed");
        if (!HaveKey(keypool.vchPubKey.GetID()))
        throw runtime_error("ReserveKeyFromKeyPool() : unknown key in key pool");
        assert(keypool.vchPubKey.IsValid());
        printf("ReserverKeyFromPool 4.1 - wallet.cpp\n"); // RegTest Debug
        // if (fDebug) LogPrintf("keypool reserve %d\n", nIndex);

Which is called from miner.cpp :

CBlockTemplate* CreateNewBlockWithKey(CReserveKey& reservekey)  
{
    CPubKey pubkey;
    printf("In CreateNewBlockWithKey 1 miner.cpp\n"); //RegTest Debug
    if (!reservekey.GetReservedKey(pubkey)) {
        printf("In CreateNewBlockWithKey !reservekey.GetReservedKey(pubkey)) 1.1 miner.cpp\n"); //RegTest Debug
        return NULL;
    }
    printf("In CreateNewBlockWithKey 2 miner.cpp\n"); //RegTest Debug
    CScript scriptPubKey = CScript() << pubkey << OP_CHECKSIG;
    printf("In CreateNewBlockWithKey 3 miner.cpp\n"); //RegTest Debug
    return CreateNewBlock(scriptPubKey);
}

[wrapper]

rpcmining.cpp ~L176

./feathercoin-cli -regtest setgenerate true 1

// -regtest mode: don't return until nGenProcLimit blocks are generated
if (fGenerate && Params().NetworkID() == CChainParams::REGTEST)
{
    printf("generate   5.1 rpcmining.cpp\n"); //RegTest Debug
    int nHeightStart = 0;
    int nHeightEnd = 0;
    int nHeight = 0;
    int nGenerate = (nGenProcLimit > 0 ? nGenProcLimit : 1);
    {   // Don't keep cs_main locked
        printf("generate   5.2 lock rpcmining.cpp\n"); //RegTest Debug
        LOCK(cs_main);
        nHeightStart = chainActive.Height();
        nHeight = nHeightStart;
        nHeightEnd = nHeightStart+nGenerate;
    }
    int nHeightLast = -1;
    while (nHeight < nHeightEnd)
    {
        if (nHeightLast != nHeight)
        {
            nHeightLast = nHeight;
            printf("generate   5.3 Generate in rpcmining.cpp\n"); //RegTest Debug
            GenerateBitcoins(fGenerate, pwalletMain, 1);
            
        }
        MilliSleep(1);
        {   // Don't keep cs_main locked
            LOCK(cs_main);
            nHeight = chainActive.Height();
        }
        printf("generate   5.4 Generate out rpcmining.cpp: nheight %d Start %d End %d Last %d \n", nHeight, nHeightStart, nHeightEnd, nHeightLast ); //RegTest Debug
    }
    printf("generate   5.5 out rpcmining.cpp\n"); //RegTest Debug
}
else // Not -regtest: start generate thread, return immediately
{
    mapArgs["-gen"] = (fGenerate ? "1" : "0");
    mapArgs ["-genproclimit"] = itostr(nGenProcLimit);
    GenerateBitcoins(fGenerate, pwalletMain, nGenProcLimit);
}

return Value::null;
}

Output

 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0

[wrapper]

miner.cpp L683

GenerateBitcoins called from rpcmining.cpp

void GenerateBitcoins(bool fGenerate, CWallet* pwallet, int nThreads)
{
    printf("In GenerateBitcoins - 1.1  L664 miner.cpp \n"); // RegTest Debug
    static boost::thread_group* minerThreads = NULL;
    printf("In GenerateBitcoins - 1.2  - miner.cpp \n"); // RegTest Debug
    if (nThreads < 0) {
        if (Params().NetworkID() == CChainParams::REGTEST)
            nThreads = 1;
        else
            nThreads = boost::thread::hardware_concurrency();
    }
    printf("In GenerateBitcoins - 1.3  miner threads miner.cpp \n"); // RegTest Debug
    if (minerThreads != NULL)
    {
        minerThreads->interrupt_all();
        delete minerThreads;
        minerThreads = NULL;
    }

    if (nThreads == 0 || !fGenerate)
        return;
    printf("In GenerateBitcoins - 1.4  create thread miner.cpp \n"); // RegTest Debug
    minerThreads = new boost::thread_group();
    printf("In GenerateBitcoins - 1.5  create miners miner.cpp \n"); // RegTest Debug
    for (int i = 0; i < nThreads; i++)
        minerThreads->create_thread(boost::bind(&FeathercoinMiner, pwallet));
       printf("In GenerateBitcoins - 1.6  exiting GenerateBitcoins \n"); // RegTest Debug
    }

[wrapper]

RegTest is now working in the test version, Scrypt only. It need setting up to do that, so I’ll try resetting it up with the fixes @Bushstar has done to the 0.9.6.x series - Scrypt / Neoscrypt interface.

 {
"chain" : "regtest",
"blocks" : 222,
"bestblockhash" : "95ce08426d17cfb6bc6bf519d479ec0c2f29827c7165fd71d0a005d8877c8768",
"difficulty" : 0.00000000,
"verificationprogress" : 1.00000000,
"chainwork" : "00000000000000000000000000000000000000000000000000000000000000e0"
}

[wrapper]

RegTest is now working with scrypt, just clearing up a few anomalies with the halving calculation and testing it.

The calculation should include nSubsidyHalvingInterval = 256; from chainparams.cpp …

 int64_t GetBlockValue(int nHeight, int64_t nFees)
 {    
     int64_t nSubsidy = 200 * COIN;

	    if(nHeight >= nForkThree || (TestNet() || RegTest()) )
		     nSubsidy = 80 * COIN;

     // Halving subsidy happens every 2,100,000 blocks. The code below takes account for the
     // fact that the first 204,639 blocks took 2.5 minutes and after changed to 1 minute.
     nSubsidy >>= (nHeight + 306960) / 2100000;
     return nSubsidy + nFees;  
 }

Trying this :

 if (!RegTest()) {
     nSubsidy >>= (nHeight + 306960) / 2100000;
 }
     else {
         int halvings = nHeight / Params().SubsidyHalvingInterval();
         if (halvings >= 64)
                return nFees;
         nSubsidy >>= halvings;
     }

[wrapper]

[FAQ] What is RegTest mode?

RegTest, or Regression Testing mode loads up a compact set of initiating parameters and creates a genesis block in the .feathercoin/regtest directory. Coins can created by running setgenerate (soon to be generate) in the command line interface.

This branch (0.9.6.2 dev) contains a working RegTest, sync speed up and Bush’s latest fixes.

In Regtest mode you create a test wallet and generate coins to experiment with.

https://github.com/FeatherCoin/Feathercoin/tree/0.9.6.2

RegTest now uses noescrypt. setgenerate -1 (seems to) works with the daemon, but not the wallet. The halving is set at 256

Pretty cool watching your generated coins come in to the wallet, although they do slow down as the difficulty increases. I’ve been through 2 halvenings.

After compiling : https://github.com/FeatherCoin/Feathercoin/tree/master-0.9/doc

cd ~/Feathercoin/src
./feathercoind -regtest -listen=0 -debug -daemon

./feathercoin-cli -regtest setgenerate true 300

./feathercoin-cli -regtest getinfo

To start up a test wallet in Regression Test mode

cd ~/Feathercoin/src/qt

./feathercoin-qt -regtest -listen=0 -debug

[Cookieboy]

Latest version in development 0.9.6.2 has the fixes in it to make this function! Thank you @wrapper and @Bushstar for your hard work on it.

[Wellenreiter]

Well done Gentlemen ::thumbsup:

:D