Regtest Mode development for Feathercoin 0.9.6.2

wrapper

@Cookieboy - I’ve uploaded a bug fix version to test, I can’t run it moment, about 6:00…

https://github.com/wrapperband/Feathercoin/tree/0.9.6.2-0.10-BackportsTest

Cookieboy

@Bushstar - Do you have a few minutes you could look at the regtest method, and see why I can’t get a genesis block generated in that mode automatically?

Cookieboy

OK, I am still digging, but I am getting closer. I was able to enable logging on the genesis block and got this returned

root@d3af4a9ce3a1:/data# feathercoin-cli -datadir=1 start
Searching for genesis block…
genesis.nTime = 1296688602
genesis.nNonce = 0
genesis.nVersion = 1
genesis.hashMerkleRoot = 97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9
genesis.GetHash = 530827f38f93b43ed12af0b3ad25a288dc02ed74d6d7857862df51fc56c416f9
error: couldn’t connect to server

I then took the GetHash value and replaced the value on Line 197 https://github.com/FeatherCoin/Feathercoin/blob/0.9.6.2/src/chainparams.cpp#L197

Prefacing it with 0x - then 530827f38f93b43ed12af0b3ad25a288dc02ed74d6d7857862df51fc56c416f9

After that, I am able to generate a block with the command

feathercoin-cli -datadir=1 -regtest setgenerate true 1

But then it hangs when I try and generate any others. I kill the process and can get it to generate 1 or two more. Then hangs… Any ideas?

wrapper

RegTest Bug Fix version available to Test

This fix makes feathercoin’s -regtest basically operational. It stalls creating blocks after 2 or 4 blocks.

https://github.com/wrapperband/Feathercoin/tree/0.9.6.2-RegTest-SetUp

**Create the regtest directory**
 ./feathercoind -regtest -listen=0 -debug -daemon

  Generate some blocks.
 ./feathercoin-cli -regtest setgenerate true 2

How to generate a new RegTest Genesis block.

chainparams.cpp

Comments out the following variables :

 genesis.nTime = 1396255061;
 genesis.nBits = 0x207fffff;
 genesis.nNonce = 3251853103;
 genesis.nVersion = 2;

   assert(hashGenesisBlock == uint256("0x3f8c076e2290f8d126be6afd574cb5070ac13bd5e117c564b6a1c51c1c4101f2"));

Leave nVersion as 2 if required.

Uncomment log request genesis block loop chainparams.cpp ~Line 207. Rebuild the feathercoin daemon.

run :

 ./feathercoind -regtest -listen=0 -debug -daemon

It will return these values :

Searching for genesis block...
genesis.nTime = 1396255061 
genesis.nNonce = 3251853103 
genesis.nVersion = 2 
genesis.hashMerkleRoot = 97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9 
genesis.GetHash = 3f8c076e2290f8d126be6afd574cb5070ac13bd5e117c564b6a1c51c1c4101f2

Copy the created parameters to the commented variables. Recompile with the correct variable and comment out the genesis creation loop.

Then run :

 ./feathercoind -regtest -listen=0 -debug -daemon

and create blocks with :

./feathercoin-cli -regtest setgenerate true 101

Blocks and regtest logs are created in ~/.feathercoin/regtest/

This code was added to main.cpp ~ Line 1,337, to be able to set a lower block time for tests.

if (RegTest()) {
    nTargetTimespan = 5; // 1 minute timespan
    nTargetSpacing = 5; // 1 minute block
}

Find information :

./feathercoin-cli -regtest getinfo

{
"version" : 90602,
"protocolversion" : 70006,
"walletversion" : 60000,
"balance" : 0.00000000,
"blocks" : 2,
"timeoffset" : 0,
"connections" : 0,
"proxy" : "",
"difficulty" : 0.00000000,
"testnet" : false,
"keypoololdest" : 1503669680,
"keypoolsize" : 101,
"paytxfee" : 0.00000000,
"relayfee" : 0.01000000,
"errors" : "This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"
}

./feathercoin-cli -regtest getnetworkinfo

 {
"version" : 90602,
"protocolversion" : 70006,
"timeoffset" : 0,
"connections" : 0,
"proxy" : "",
"relayfee" : 0.01000000,
"localaddresses" : [
]
 }

wrapper

RegTest version now building blocks, 1 by one. So, made some progress :

 ~/Feathercoin/src$ ./feathercoin-cli -regtest getblockchaininfo
 {
    "chain" : "regtest",
    "blocks" : 5,
    "bestblockhash" : "84b44ec0bf52a1349c79809d64c512ea64de5c873828fa2ac39148293cde6b98",
    "difficulty" : 0.00000000,
    "verificationprogress" : 1.00000000,
    "chainwork" :  "000000000000000000000000000000000000000000000000000000000000001e"
 }

error message

 terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
 what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument

wrapper

RegTest debugging

Regtest script setgenerate is run from :

/Feathercoin/contrib/bitrpc/bitrpc.py

 elif cmd == "setgenerate":
     try:
         gen= raw_input("Generate? (true/false): ")
         cpus = raw_input("Max processors/cores (-1 for unlimited, optional):")
         try:
             print access.setgenerate(gen, cpus)
         except:
             print access.setgenerate(gen)
     except:

However get error :

./feathercoin-cli -regtest setgenerate true 101 1

error: {"code":-1,"message":"setgenerate generate ( genproclimit )\n\nSet 'generate' true or false to turn generation on or off.\nGeneration is limited to 'genproclimit' processors, -1 is unlimited.\nSee the getgenerate call for the current setting.\n\nArguments:\n1. generate         (boolean, required) Set to true to turn on generation, off to turn off.\n2. genproclimit     (numeric, optional) Set the processor limit for when generation is on. Can be -1 for unlimited.\n                    Note: in -regtest mode, genproclimit controls how many blocks are generated immediately.\n\nExamples:\n\nSet the generation on with a limit of one processor\n> feathercoin-cli setgenerate true 1\n\nCheck the setting\n> feathercoin-cli getgenerate \n\nTurn off generation\n> feathercoin-cli setgenerate false\n\nUsing json rpc\n> curl --user myusername --data-binary '{\"jsonrpc\": \"1.0\", \"id\":\"curltest\", \"method\": \"setgenerate\", \"params\": [true, 1] }' -H 'content-type: text/plain;' http://127.0.0.1:9337/\n"}

So try ./feathercoin-cli -regtest setgenerate true 4

i.e. generate = true
number of cpu cores = 4

 ./feathercoin-cli -regtest getblockchaininfo
 {
     "chain" : "regtest", 
     "blocks" : 11,
     "bestblockhash" : "91750ef4a6f96500a2b76f00e38823f1f230919f58a2051184e17a44f7bdaf8d",
     "difficulty" : 0.00000000,
     "verificationprogress" : 1.00000000,
     "chainwork" : "0000000000000000000000000000000000000000000000000000000000000031"
 }

wrapper

https://github.com/bitcoin/bitcoin/search?utf8=✓&q=getgenerate(&type=

doc/release-notes/release-notes-0.13.0.md
Showing the top match Last indexed on 18 Oct 2016
createwitnessaddress, addwitnessaddress.

Removed RPC commands: setgenerate, getgenerate.
New options were added to fundrawtransaction: includeWatching, changeAddress, changePosition and feeRate.

wrapper

https://github.com/bitcoin/bitcoin/issues/4242

If you run setgenerate true too rapidly in regtest mode, bitcoind can crash with

/home/ubuntu/install/include/boost/thread/pthread/condition_variable_fwd.hpp:81: boost::condition_variable::~condition_variable(): Assertion `!ret' failed.

This was reproduced with the Linux 64bit gitian binary. 

Sometimes it kills bitcoind without a visible asert failure. Sometimes bitcoind just misses the RPC entirely without any visible error and without killing bitcoind.

wrapper

[FAQ] Where are coins generated, in software terms?

Regtest and Testnet are special cases of normal mining.

Each has a set of parameters, ports and directories to work with but otherwise goes through the same code.

Testnet set-ups are done to test an event, like a fork, regression testing is similar but does not connect to the internet.

Instead of a miner regtest uses ./feathercoin-cli and passed commands to feathercoind to mine some coins.

 ./feathercoin-cli -regtest setgenerate true 1

setgenerate is deprecated and the new command will be generate for later (than 0.9.6.2) versions.

miner.cpp

 void GenerateBitcoins(bool fGenerate, CWallet* pwallet, int nThreads) 
 {
     static boost::thread_group* minerThreads = NULL;
     printf("Here in coin generation\n"); // RegTest Debug
     if (nThreads < 0) {
         if (Params().NetworkID() == CChainParams::REGTEST)
             nThreads = 1;
         else
             nThreads = boost::thread::hardware_concurrency();
    }

     if (minerThreads != NULL)
     {
        minerThreads->interrupt_all();
         delete minerThreads;
         minerThreads = NULL;
     }

     if (nThreads == 0 || !fGenerate)
         return;

     minerThreads = new boost::thread_group();
     for (int i = 0; i < nThreads; i++)
        minerThreads->create_thread(boost::bind(&FeathercoinMiner, pwallet));
 }

wrapper

Current status getting regtest running :

Various issues are affecting how complex it is to set up regtest. Whilst some are addressed there is still an unknown issue.

Creating Genesis Block and regtest directory

./feathercoind -regtest -listen=0 -debug -daemon
Searching for genesis block...
genesis.nTime = 1396255061 
genesis.nNonce = 3250989159 
genesis.nVersion = 3 
genesis.hashMerkleRoot = 97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9 
genesis.GetHash = 29d5f68acf1fb95303120bb51c35a32d270bd736235f928de49b94d87a3782a8
Feathercoin server starting

ReserverKeyFromPool 1.1 - wallet.cpp
ReserverKeyFromPool 1.2 - wallet.cpp
ReserverKeyFromPool 1.3 - wallet.cpp
ReserverKeyFromPool 2.1 - wallet.cpp
ReserverKeyFromPool 2.2 - wallet.cpp 
ReserverKeyFromPool 2.3 - wallet.cpp
ReserverKeyFromPool 3.1 - wallet.cpp
ReserverKeyFromPool 3.2 - wallet.cpp
ReserverKeyFromPool 3.3 - wallet.cpp
ReserverKeyFromPool 3.4 - wallet.cpp
ReserverKeyFromPool 4.1 - wallet.cpp

Getting Blockchain info :

 ./feathercoin-cli -regtest getblockchaininfo
 {
 "chain" : "regtest",
 "blocks" : 0,
 "bestblockhash" : "29d5f68acf1fb95303120bb51c35a32d270bd736235f928de49b94d87a3782a8",
 "difficulty" : 0.00000000,
 "verificationprogress" : 1.00000000,
 "chainwork" : "0000000000000000000000000000000000000000000000000000000000000002"
  }

generatebitcoins 1.1 - rpcmining.cpp
FeathercoinMiner
FeathercoinMiner 2
FeathercoinMiner 2.1
FeathercoinMiner 2.2
In CreateNewBlockWithKey 1 miner.cpp
InGetReservedKey 1.1-wallet.cpp
InGetReservedKey 1.1.1-wallet.cpp
ReserverKeyFromPool 1.1 - wallet.cpp
ReserverKeyFromPool 1.2 - wallet.cpp
ReserverKeyFromPool 1.3 - wallet.cpp

Where it is hanging on creating a block, whilst getting a public key in wallet.dat :

void CWallet::ReserveKeyFromKeyPool(int64_t& nIndex, CKeyPool& keypool)
{
    printf("ReserverKeyFromPool 1.1 - wallet.cpp\n"); // RegTest Debug
    nIndex = -1;
    printf("ReserverKeyFromPool 1.2 - wallet.cpp\n", keypool.vchPubKey); // RegTest Debug
    keypool.vchPubKey = CPubKey();
    printf("ReserverKeyFromPool 1.3 - wallet.cpp\n"); // RegTest Debug
    {
        LOCK(cs_wallet);
        printf("ReserverKeyFromPool 2.1 - wallet.cpp\n"); // RegTest Debug
        if (!IsLocked()) {
        printf("ReserverKeyFromPool 2.2 - wallet.cpp\n"); // RegTest Debug          
            TopUpKeyPool();
        printf("ReserverKeyFromPool 2.3 - wallet.cpp\n"); // RegTest Debug
        }
        // Get the oldest key
        if(setKeyPool.empty())
            return;
        printf("ReserverKeyFromPool 3.1 - wallet.cpp\n"); // RegTest Debug
        CWalletDB walletdb(strWalletFile);
        printf("ReserverKeyFromPool 3.2 - wallet.cpp\n"); // RegTest Debug
        nIndex = *(setKeyPool.begin());
        printf("ReserverKeyFromPool 3.3 - wallet.cpp\n"); // RegTest Debug
        setKeyPool.erase(setKeyPool.begin());
        printf("ReserverKeyFromPool 3.4 - wallet.cpp\n"); // RegTest Debug
        if (!walletdb.ReadPool(nIndex, keypool))
        throw runtime_error("ReserveKeyFromKeyPool() : read failed");
        if (!HaveKey(keypool.vchPubKey.GetID()))
        throw runtime_error("ReserveKeyFromKeyPool() : unknown key in key pool");
        assert(keypool.vchPubKey.IsValid());
        printf("ReserverKeyFromPool 4.1 - wallet.cpp\n"); // RegTest Debug
        // if (fDebug) LogPrintf("keypool reserve %d\n", nIndex);

Which is called from miner.cpp :

CBlockTemplate* CreateNewBlockWithKey(CReserveKey& reservekey)  
{
    CPubKey pubkey;
    printf("In CreateNewBlockWithKey 1 miner.cpp\n"); //RegTest Debug
    if (!reservekey.GetReservedKey(pubkey)) {
        printf("In CreateNewBlockWithKey !reservekey.GetReservedKey(pubkey)) 1.1 miner.cpp\n"); //RegTest Debug
        return NULL;
    }
    printf("In CreateNewBlockWithKey 2 miner.cpp\n"); //RegTest Debug
    CScript scriptPubKey = CScript() << pubkey << OP_CHECKSIG;
    printf("In CreateNewBlockWithKey 3 miner.cpp\n"); //RegTest Debug
    return CreateNewBlock(scriptPubKey);
}

wrapper

rpcmining.cpp ~L176

./feathercoin-cli -regtest setgenerate true 1

// -regtest mode: don't return until nGenProcLimit blocks are generated
if (fGenerate && Params().NetworkID() == CChainParams::REGTEST)
{
    printf("generate   5.1 rpcmining.cpp\n"); //RegTest Debug
    int nHeightStart = 0;
    int nHeightEnd = 0;
    int nHeight = 0;
    int nGenerate = (nGenProcLimit > 0 ? nGenProcLimit : 1);
    {   // Don't keep cs_main locked
        printf("generate   5.2 lock rpcmining.cpp\n"); //RegTest Debug
        LOCK(cs_main);
        nHeightStart = chainActive.Height();
        nHeight = nHeightStart;
        nHeightEnd = nHeightStart+nGenerate;
    }
    int nHeightLast = -1;
    while (nHeight < nHeightEnd)
    {
        if (nHeightLast != nHeight)
        {
            nHeightLast = nHeight;
            printf("generate   5.3 Generate in rpcmining.cpp\n"); //RegTest Debug
            GenerateBitcoins(fGenerate, pwalletMain, 1);
            
        }
        MilliSleep(1);
        {   // Don't keep cs_main locked
            LOCK(cs_main);
            nHeight = chainActive.Height();
        }
        printf("generate   5.4 Generate out rpcmining.cpp: nheight %d Start %d End %d Last %d \n", nHeight, nHeightStart, nHeightEnd, nHeightLast ); //RegTest Debug
    }
    printf("generate   5.5 out rpcmining.cpp\n"); //RegTest Debug
}
else // Not -regtest: start generate thread, return immediately
{
    mapArgs["-gen"] = (fGenerate ? "1" : "0");
    mapArgs ["-genproclimit"] = itostr(nGenProcLimit);
    GenerateBitcoins(fGenerate, pwalletMain, nGenProcLimit);
}

return Value::null;
}

Output

 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0 
 generate   5.4 Generate out rpcmining.cpp: nheight 0 Start 0 End 1 Last 0

wrapper

miner.cpp L683

GenerateBitcoins called from rpcmining.cpp

void GenerateBitcoins(bool fGenerate, CWallet* pwallet, int nThreads)
{
    printf("In GenerateBitcoins - 1.1  L664 miner.cpp \n"); // RegTest Debug
    static boost::thread_group* minerThreads = NULL;
    printf("In GenerateBitcoins - 1.2  - miner.cpp \n"); // RegTest Debug
    if (nThreads < 0) {
        if (Params().NetworkID() == CChainParams::REGTEST)
            nThreads = 1;
        else
            nThreads = boost::thread::hardware_concurrency();
    }
    printf("In GenerateBitcoins - 1.3  miner threads miner.cpp \n"); // RegTest Debug
    if (minerThreads != NULL)
    {
        minerThreads->interrupt_all();
        delete minerThreads;
        minerThreads = NULL;
    }

    if (nThreads == 0 || !fGenerate)
        return;
    printf("In GenerateBitcoins - 1.4  create thread miner.cpp \n"); // RegTest Debug
    minerThreads = new boost::thread_group();
    printf("In GenerateBitcoins - 1.5  create miners miner.cpp \n"); // RegTest Debug
    for (int i = 0; i < nThreads; i++)
        minerThreads->create_thread(boost::bind(&FeathercoinMiner, pwallet));
       printf("In GenerateBitcoins - 1.6  exiting GenerateBitcoins \n"); // RegTest Debug
    }

wrapper

RegTest is now working in the test version, Scrypt only. It need setting up to do that, so I’ll try resetting it up with the fixes @Bushstar has done to the 0.9.6.x series - Scrypt / Neoscrypt interface.

 {
"chain" : "regtest",
"blocks" : 222,
"bestblockhash" : "95ce08426d17cfb6bc6bf519d479ec0c2f29827c7165fd71d0a005d8877c8768",
"difficulty" : 0.00000000,
"verificationprogress" : 1.00000000,
"chainwork" : "00000000000000000000000000000000000000000000000000000000000000e0"
}

wrapper

RegTest is now working with scrypt, just clearing up a few anomalies with the halving calculation and testing it.

The calculation should include nSubsidyHalvingInterval = 256; from chainparams.cpp …

 int64_t GetBlockValue(int nHeight, int64_t nFees)
 {    
     int64_t nSubsidy = 200 * COIN;

	    if(nHeight >= nForkThree || (TestNet() || RegTest()) )
		     nSubsidy = 80 * COIN;

     // Halving subsidy happens every 2,100,000 blocks. The code below takes account for the
     // fact that the first 204,639 blocks took 2.5 minutes and after changed to 1 minute.
     nSubsidy >>= (nHeight + 306960) / 2100000;
     return nSubsidy + nFees;  
 }

Trying this :

 if (!RegTest()) {
     nSubsidy >>= (nHeight + 306960) / 2100000;
 }
     else {
         int halvings = nHeight / Params().SubsidyHalvingInterval();
         if (halvings >= 64)
                return nFees;
         nSubsidy >>= halvings;
     }

wrapper

[FAQ] What is RegTest mode?

RegTest, or Regression Testing mode loads up a compact set of initiating parameters and creates a genesis block in the .feathercoin/regtest directory. Coins can created by running setgenerate (soon to be generate) in the command line interface.

This branch (0.9.6.2 dev) contains a working RegTest, sync speed up and Bush’s latest fixes.

In Regtest mode you create a test wallet and generate coins to experiment with.

https://github.com/FeatherCoin/Feathercoin/tree/0.9.6.2

RegTest now uses noescrypt. setgenerate -1 (seems to) works with the daemon, but not the wallet. The halving is set at 256

Pretty cool watching your generated coins come in to the wallet, although they do slow down as the difficulty increases. I’ve been through 2 halvenings.

After compiling : https://github.com/FeatherCoin/Feathercoin/tree/master-0.9/doc

cd ~/Feathercoin/src
./feathercoind -regtest -listen=0 -debug -daemon

./feathercoin-cli -regtest setgenerate true 300

./feathercoin-cli -regtest getinfo

To start up a test wallet in Regression Test mode

cd ~/Feathercoin/src/qt

./feathercoin-qt -regtest -listen=0 -debug

Cookieboy

Latest version in development 0.9.6.2 has the fixes in it to make this function! Thank you @wrapper and @Bushstar for your hard work on it.

Wellenreiter

Well done Gentlemen ::thumbsup:

:D