Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node v1.3.4 not syncing BAD BLOCK #2049

Closed
roflganker opened this issue Dec 7, 2023 · 14 comments
Closed

Node v1.3.4 not syncing BAD BLOCK #2049

roflganker opened this issue Dec 7, 2023 · 14 comments

Comments

@roflganker
Copy link

System information

Geth version: 1.3.4
OS & Version: Linux 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64 GNU/Linux
Commit hash : -

Expected behaviour

Node syncing correctly, serving eth_ RPC

Actual behaviour

Doing absolutely nothing, not syncing, serving stale data

Steps to reproduce the behaviour

Run bnbchain/bsc v1.3.4 inside docker compose

Backtrace

WARN [12-07|12:36:32.071] Synchronisation failed, dropping peer    peer=fc48b76dc01d76d5e31ab7edd1da50d4d091c37b2bc46fbb76bd88dd6b071773 err="retrieved hash chain is invalid: could not apply tx 0 [0x4acc62af010802b685d0b5a0b5882f6343ba31e758c6dd11ae778b4ae811f88c]: nonce too high: address 0x0a9a1A5C71aE3a8eF617f4E3A189F20291b4dF51, tx: 10797 state: 0"
INFO [12-07|12:36:34.176] Skip duplicated bad block                number=34,141,848 hash=28be3f..e3445a
ERROR[12-07|12:36:34.176] 
########## BAD BLOCK #########
Block: 34141848 (0x28be3f2269276de3ad03db81f73818e240874859cd33059ac1ccedeb6de3445a)
Error: could not apply tx 0 [0x4acc62af010802b685d0b5a0b5882f6343ba31e758c6dd11ae778b4ae811f88c]: nonce too high: address 0x0a9a1A5C71aE3a8eF617f4E3A189F20291b4dF51, tx: 10797 state: 0
Platform: geth (devel) go1.20.11 amd64 linux
VCS: 885de2c1-
Chain config: &params.ChainConfig{ChainID:56, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:0, BerlinBlock:31302048, YoloV3Block:<nil>, CatalystBlock:<nil>, LondonBlock:31302048, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, RamanujanBlock:0, NielsBlock:0, MirrorSyncBlock:5184000, BrunoBlock:13082000, EulerBlock:18907621, GibbsBlock:23846001, NanoBlock:21962149, MoranBlock:22107423, PlanckBlock:27281024, LubanBlock:29020050, PlatoBlock:30720096, HertzBlock:31302048, HertzfixBlock:34140700, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(nil), Parlia:(*params.ParliaConfig)(0x4680890), IsDevMode:false}
Receipts: 
##############################
 
WARN [12-07|12:36:34.177] Synchronisation failed, dropping peer    peer=283f7add3e91dfade9243447db265eec332df77582ac8b9470119acba9b34f88 err="retrieved hash chain is invalid: could not apply tx 0 [0x4acc62af010802b685d0b5a0b5882f6343ba31e758c6dd11ae778b4ae811f88c]: nonce too high: address 0x0a9a1A5C71aE3a8eF617f4E3A189F20291b4dF51, tx: 10797 state: 0"
WARN [12-07|12:36:34.177] Synchronisation failed, retrying         peer=283f7add3e91dfade9243447db265eec332df77582ac8b9470119acba9b34f88 err="peer is unknown or unhealthy"
WARN [12-07|12:36:40.537] Synchronisation failed, dropping peer    peer=f9256eee1c5652dd2c1d5d52265d8986f77bb9306a315e5663862ba16f157bc9 err="retrieved hash chain is invalid: syncing canceled (requested)"
WARN [12-07|12:36:40.537] Rewinding blockchain to block            target=34,140,848
INFO [12-07|12:37:30.112] Current status                           table_size=133 pending_size=70 db_size=36,586,535
ERROR[12-07|12:37:56.376] Expired request does not exist           peer=4d7b68c1da221b2bbe78734ff6af4636eb5c20f86645c05f534d6db6298ba126
ERROR[12-07|12:37:56.723] Expired request does not exist           peer=607641f9e119ac65106495ed9436483c7bd2d1a0451c942e9f9361d200767767
ERROR[12-07|12:38:16.706] Expired request does not exist           peer=57b4b6751fde268f408c1288ad4f07d9a8f88ee090d578acf5222eab5c412ae8
ERROR[12-07|12:38:16.930] Expired request does not exist           peer=53731f875040ab832c373d14030bc074664c8a5e8229c429f5172a4bb4a8ee60
ERROR[12-07|12:38:17.217] Expired request does not exist           peer=574f36176d0ebc654ba638b8a4eb39f962adaacb3477643b4ce28f3413c05bb4
ERROR[12-07|12:38:17.227] Expired request does not exist           peer=4a1c8346f12182f2bbc76def176ed3d0fb5230a8d8157eaea545201289938464
WARN [12-07|12:38:19.762] Snapshot extension registration failed   peer=5778b638 err="peer connected on snap without compatible eth support"
INFO [12-07|12:38:30.113] Current status                           table_size=137 pending_size=4  db_size=36,586,535
WARN [12-07|12:39:18.417] Served eth_coinbase                      reqid=3 t="26.107µs" err="etherbase must be explicitly specified" X-Forwarded-For=nil
WARN [12-07|12:39:18.417] Served eth_coinbase                      reqid=3 duration="25.345µs" err="etherbase must be explicitly specified"
INFO [12-07|12:39:30.114] Current status                           table_size=139 pending_size=5  db_size=36,586,535
WARN [12-07|12:39:56.915] Peer stalling, dropping                  peer=603f39a1 waited=2m0.385s
WARN [12-07|12:39:58.148] Served eth_coinbase                      reqid=16 t="64.159µs" err="etherbase must be explicitly specified" X-Forwarded-For=nil
WARN [12-07|12:39:58.148] Served eth_coinbase                      reqid=16 duration="62.302µs" err="etherbase must be explicitly specified"
WARN [12-07|12:39:58.168] Served eth_protocolVersion               reqid=23 t="13.236µs" err="the method eth_protocolVersion does not exist/is not available" X-Forwarded-For=nil
WARN [12-07|12:39:58.168] Served eth_protocolVersion               reqid=23 duration="11.9µs"   err="the method eth_protocolVersion does not exist/is not available"
WARN [12-07|12:39:58.171] Served eth_coinbase                      reqid=28 t="26.554µs" err="etherbase must be explicitly specified" X-Forwarded-For=nil
WARN [12-07|12:39:58.171] Served eth_coinbase                      reqid=28 duration="24.794µs" err="etherbase must be explicitly specified"
WARN [12-07|12:39:58.175] Served eth_protocolVersion               reqid=34 t="5.495µs"  err="the method eth_protocolVersion does not exist/is not available" X-Forwarded-For=nil
WARN [12-07|12:39:58.175] Served eth_protocolVersion               reqid=34 duration="4.609µs"  err="the method eth_protocolVersion does not exist/is not available"
INFO [12-07|12:40:30.114] Current status                           table_size=137 pending_size=28 db_size=36,586,535
INFO [12-07|12:41:30.115] Current status                           table_size=139 pending_size=2  db_size=36,586,535
INFO [12-07|12:42:30.115] Current status                           table_size=141 pending_size=4  db_size=36,586,535
INFO [12-07|12:43:30.114] Current status                           table_size=141 pending_size=7  db_size=36,586,535
INFO [12-07|12:44:30.116] Current status                           table_size=142 pending_size=16 db_size=36,586,535
INFO [12-07|12:45:30.116] Current status                           table_size=143 pending_size=29 db_size=36,586,535
INFO [12-07|12:46:30.117] Current status                           table_size=142 pending_size=4  db_size=36,586,535
INFO [12-07|12:47:30.114] Current status                           table_size=142 pending_size=3  db_size=36,586,535
INFO [12-07|12:48:30.117] Current status                           table_size=143 pending_size=8  db_size=36,586,535
INFO [12-07|12:49:30.117] Current status                           table_size=144 pending_size=2  db_size=36,586,535
INFO [12-07|12:50:30.118] Current status                           table_size=145 pending_size=5  db_size=36,586,535
INFO [12-07|12:51:30.117] Current status                           table_size=144 pending_size=19 db_size=36,586,535
INFO [12-07|12:52:30.115] Current status                           table_size=145 pending_size=2  db_size=36,645,098
INFO [12-07|12:53:30.117] Current status                           table_size=149 pending_size=3  db_size=36,645,098
INFO [12-07|12:54:30.117] Current status                           table_size=150 pending_size=8  db_size=36,645,098
INFO [12-07|12:55:30.115] Current status                           table_size=150 pending_size=20 db_size=36,645,098

Additional info

The node was working correctly on 1.2.x for several months. Recently I migrated to 1.3.4 and... It was working correcly for a short time (several hours), and stuck at block 34141848.
What did I try in order to fix this:

  • Rewinding for 1000 blocks backwards
  • Applying configuration from assets of release 1.3.4
  • Restarting the node and waiting for ~15 mins
@roflganker
Copy link
Author

My docker-compose service:

  geth:
    image: ghcr.io/bnb-chain/bsc:1.3.4
    user: 0:0
    command:
    - --config=/bsc/config/config.toml
    - --datadir=/node
    - --rpc.allow-unprotected-txs
    - --tries-verify-mode=none
    - --cache=32000
    - --http
    - --http.corsdomain=*
    - --nat=extip:89.105.216.188
    - --v5disc
    - --syncmode=full
    - --verbosity=3
    - --txpool.rejournal=1h
    - --txpool.pricelimit=1100000000
    - --txpool.pricebump=10
    - --txpool.accountslots=512
    - --txpool.globalslots=4096
    - --txpool.accountqueue=64
    - --txpool.globalqueue=1024
    - --txpool.lifetime=30m
    - --txpool.reannouncetime=1m
    ports:
    - 0.0.0.0:30311:30311/tcp
    - 0.0.0.0:30311:30311/udp
    volumes:
    - ./genesis.json:/bsc/config/genesis.json:ro
    - ./config.toml:/bsc/config/config.toml:ro
    - /var/lib/bnbchain/:/node/:rw

My config.toml:

[Eth]
NetworkId = 56
NoPruning = false
NoPrefetch = false
LightPeers = 50
TrieTimeout = 100000000000
EnablePreimageRecording = false
EWASMInterpreter = ""
EVMInterpreter = ""

[Eth.Miner]
GasFloor = 30000000
GasCeil = 40000000
GasPrice = 1000000000
Recommit = 10000000000

[Eth.TxPool]
Locals = []
NoLocals = false
Journal = "transactions.rlp"

[Eth.GPO]
Blocks = 20
Percentile = 60
OracleThreshold = 20

[Node]
IPCPath = "geth.ipc"
HTTPHost = "0.0.0.0"
NoUSB = true
InsecureUnlockAllowed = false
HTTPPort = 8545
HTTPVirtualHosts = ["*"]
HTTPModules = ["eth", "net", "web3", "txpool", "parlia"]
#WSPort = 8546
#WSModules = ["net", "web3", "eth"]

[Node.P2P]
MaxPeers = 200
NoDiscovery = false
BootstrapNodes = ["enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311","enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311","enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311"]
StaticNodes = ["enode://7ac95307b00962c6257574892fd4b53b38d6b6814aa1dec044539854ca507f2d2c8fbabfde98c5cbb3d2e86d6e7aaf417cc28ecaf403d24f69a9bccc3f818002@54.248.214.5:30311","enode://e8a97e90dae05940a04a37ddad445b1d1f270e1a0ec1caf31f77ecd9a664db3c0f4750914afbb6f017d214ecef4a5f991b2f16987a76ec38ceb06f7f8f2f9f76@52.17.96.222:30311","enode://1841077024720c251f58e6eeb10c2a3846db3610b2f4e8210e7035d0623f4ab6caef94c3bf215cb548e7c7e41d2755da33b63685de425e07aeb5cef017ea8cb5@52.51.36.24:30311","enode://09a0e9335b71f9269dcf099bc8ec071dbf04ac7ef67d95561b958621808bbdc91d5364e2d9769793afc63267d164d1ede6d6c84dc301c83363e038162a13dbaf@54.228.153.191:30311","enode://76545d473172288ccc65e3cce0d48c07389e84266be4cf1770f31541c1b012394890ef87c54e1552dbeb675bb23d49b0cc884c1f1fd4b40bbaf2a8f84e416ca4@18.178.12.40:30311","enode://20d04257749893d7193b8e3ed619d46384d28b350508bef163b52ee9dc60efc4f562aee00c7fde5cfa83e4e9723b0e90d6422d9031b6069734bd7e24a9ed8e73@107.21.209.99:30311","enode://f49aae223b44de74a202d43976e7f7b0394fccef478cb0682f9d5a8ef9174ebbf5e4738cd3ccf83158c18c84f0973563e6b4d66fa5bd107dfb7a06fc6206011a@34.255.168.176:30311","enode://e421c47edc33d0d929e755196e92c8e7c26f24a38cc76dd17b0e78c304426e7e6d34e3f4e39f5dae1397c3cfac8238eb45533a62e118b183afd66b652fd1d03a@54.249.78.93:30311","enode://22455eaa4eaf4c12dc2386840a590fdc632ef669f86a8e1adf8d1421ca2c2f69d6cef6ecc8b976f945d1fcbc3c8181edf75494b1ecb28eaf15f8b21413644385@54.217.65.254:30311","enode://91f5b18b0285da4983f7c91d2b19b6404e5400313cfdea6b218355a8bafa34a529f9fe5af7a42d95ba61ccb16d0bd67be049c447e5611b27f8b59ffa7967fdf7@54.178.30.104:30311","enode://03f1b885d42edd4492fd87f05d4d720fef675079476eeb07959fd3dd1f8774ccf5690d8c52c8b91474f1bb0c95e973a3dbe6831d8375aca502a18aefb3acab87@54.227.131.21:30311","enode://38144f02adbe8817b2b0f241f921491bc612dd8cd603fa9cd3faa31c76697ad5e00297cba38d0a8f594752c56c9a7e84f53d7f71c0b37b5680cea5e732b6b946@34.243.164.24:30311","enode://0a5ace121aa41b7a4df80ba5f487d3e0f6cd6284223bd6a19227bd680d595be6b6f464a4ca34003155f7764cc600eb3263094179f07f6f383d44299c866901bd@35.76.22.152:30311","enode://514334110fb750cf9c09d048815f2a39e2d869e4040c89f9a4c74dc08dd61556e596477ca3edd82ab5b4e41d8b23eb5c0da9fc521edb81ec0e062c57c4ce9700@34.248.99.52:30311"]
ListenAddr = ":30311"
EnableMsgEvents = false

[Node.HTTPTimeouts]
ReadTimeout = 30000000000
WriteTimeout = 30000000000
IdleTimeout = 120000000000

@roflganker
Copy link
Author

My eth.syncing:

{
  currentBlock: 34141847,
  healedBytecodeBytes: 0,
  healedBytecodes: 0,
  healedTrienodeBytes: 0,
  healedTrienodes: 0,
  healingBytecode: 0,
  healingTrienodes: 0,
  highestBlock: 34146592,
  startingBlock: 34141847,
  syncedAccountBytes: 0,
  syncedAccounts: 0,
  syncedBytecodeBytes: 0,
  syncedBytecodes: 0,
  syncedStorage: 0,
  syncedStorageBytes: 0
}

My geth.net

{
  listening: true,
  peerCount: 197,
  version: "56"
}

@roflganker roflganker changed the title Node v1.3.4 does not syncing BAD BLOCK Node v1.3.4 not syncing BAD BLOCK Dec 7, 2023
@roflganker
Copy link
Author

It completely died

INFO [12-07|14:08:30.118] Current status                           table_size=161 pending_size=4  db_size=36,569,195
INFO [12-07|14:09:30.116] Current status                           table_size=162 pending_size=2  db_size=36,569,195
INFO [12-07|14:10:30.118] Current status                           table_size=160 pending_size=48 db_size=36,569,195
INFO [12-07|14:11:30.118] Current status                           table_size=161 pending_size=18 db_size=36,569,195
INFO [12-07|14:12:30.116] Current status                           table_size=164 pending_size=26 db_size=36,569,195
INFO [12-07|14:13:30.116] Current status                           table_size=162 pending_size=3  db_size=36,569,195
INFO [12-07|14:14:30.118] Current status                           table_size=161 pending_size=48 db_size=36,569,195
INFO [12-07|14:15:30.119] Current status                           table_size=161 pending_size=4  db_size=36,569,195
INFO [12-07|14:16:30.118] Current status                           table_size=162 pending_size=31 db_size=36,569,195
INFO [12-07|14:17:30.116] Current status                           table_size=163 pending_size=3  db_size=36,612,958
INFO [12-07|14:18:30.118] Current status                           table_size=163 pending_size=26 db_size=36,612,958
INFO [12-07|14:19:30.112] Current status                           table_size=162 pending_size=1  db_size=36,612,958
INFO [12-07|14:20:30.117] Current status                           table_size=161 pending_size=3  db_size=36,612,958
INFO [12-07|14:21:30.116] Current status                           table_size=162 pending_size=1  db_size=36,612,958
INFO [12-07|14:22:30.115] Current status                           table_size=161 pending_size=0  db_size=36,612,958
ERROR[12-07|14:22:39.243] Missing block in the middle, aiming genesis number=25,952,856 hash=ec0ca2..0d6e1d
INFO [12-07|14:22:39.243] Reset to block with genesis state        number=0          hash=0d2184..d57b5b
CRIT [12-07|14:22:39.244] Failed to commit genesis state           err="not found"

Help plz

@icculp
Copy link

icculp commented Dec 8, 2023

Got the same behavior on 1.3.4, reverting to 1.3.3 it moves again

@weiihann
Copy link
Contributor

weiihann commented Dec 8, 2023 •
edited
Loading

@roflganker could you help change --verbosity=3 to --verbosity=4, rerun your node for 5 minutes and share the debugging logs?

@zzzckck
Copy link
Collaborator

zzzckck commented Dec 8, 2023 •
edited
Loading

--tries-verify-mode=none will be fast node, it does not support long range unwind.
Likely you didn't upgrade to v1.3.4 or v1.2.15 before the Hertzfix hard fork, which is around 07th Dec, 8am UTC.

Then your nodes will be broken, can not sync anymore, need to reset your environment.

@zzzckck
Copy link
Collaborator

zzzckck commented Dec 8, 2023

Got the same behavior on 1.3.4, reverting to 1.3.3 it moves again

yea, but you will have problem next time, if you upgrade to v1.3.4+

@zzzckck
Copy link
Collaborator

zzzckck commented Dec 8, 2023

You may need to download the fastnode snapshot and sync again: https://github.com/48Club/bsc-snapshots

@gituser
Copy link

gituser commented Dec 10, 2023 •
edited
Loading

There is same issue with v1.3.5 on testnet when trying to sync from scratch I get an error
I'm running with pebble new engine activated and --tries-verify-mode=none:

t=2023-12-10T19:17:11+0300 lvl=info msg="Skip duplicated bad block"              number=19,249,200 hash=0xa680b2ac156470bdb8f3a2bf7b33fe5c375060d8a3f71a2a41d667fb786c8dc0
t=2023-12-10T19:17:11+0300 lvl=eror msg="\n########## BAD BLOCK #########\nBlock: 19249200 (0xa680b2ac156470bdb8f3a2bf7b33fe5c375060d8a3f71a2a41d667fb786c8dc0)\nError: mismatching validator list on epoch block\nPlatform: geth (devel) go1.20 amd64 linux\nVCS: fe928d47-\nChain config: &params.ChainConfig{ChainID:97, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:0, BerlinBlock:31103030, YoloV3Block:<nil>, CatalystBlock:<nil>, LondonBlock:31103030, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc0000564f8), KeplerTime:(*uint64)(0xc000056500), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, RamanujanBlock:1010000, NielsBlock:1014369, MirrorSyncBlock:5582500, BrunoBlock:13837000, EulerBlock:19203503, GibbsBlock:22800220, NanoBlock:23482428, MoranBlock:23603940, PlanckBlock:28196022, LubanBlock:29295050, PlatoBlock:29861024, HertzBlock:31103030, HertzfixBlock:35682300, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(nil), Parlia:(*params.ParliaConfig)(0x48e7050), IsDevMode:false}\nReceipts: \n  0: cumulative: 793712 gas: 793712 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xb5dd5e91c9d25baf7578be22e9c7e19b271da2aabccfe3f6b3ffccbf41dcdb05 logs: [0xc0c151a2c0 0xc0c151a420 0xc0c151a6e0 0xc0c151a790] bloom: 00020000080000040000200000002000000000000000000000000000000000200000000000000010000000000000000000000000000000000000001000100000000000001000000000000008000000000000000000000000000000000000000000000000000004000000000000000000000200000000000000000010000000000000000000000000000000000000000000000000400000000200000000000000000000010000000000000000010000400000000020000000000000000000000000000212000000000000000001000000000000000000001000010000000200080000000000000040000000000000000000000800000000000000000000000000 state: \n  1: cumulative: 821835 gas: 28123 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6e9821124ac17afdb8fb67e830cb988f39549a811c0ea3763bd82bc97ec62310 logs: [0xc0c151ac60] bloom: 00000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000010240000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000040000000800000000000000000020000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000100004000000000000000000000000000000000000000000000000000000000000000000000000000040000010000000000400000000000008000000000000000000000000000000000000 state: \n  2: cumulative: 1139664 gas: 317829 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x72b1f3927b2b175ef01cbd8d2806e8f345f8913117a22d002376d81532dcf956 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: \n  3: cumulative: 1208663 gas: 68999 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc5e2110a0e21f25a55651d8dba17885a50b6cbfd932f970ccd4e812adbd8e600 logs: [0xc0c151ae70 0xc0c151b290 0xc0c151b810] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000100000000000000000000000000000400000000000000108000000000000000000000000000000000000000000000000000010000001000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000080000000000000000002000000000000000000000020000000000000000000000000000000000000000000000000080000000000000000000000400000000000000000000200 state: \n  4: cumulative: 1245546 gas: 36883 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x824ed14256b56ef227ab5f1adad76923a9707792122f22d162aabc6debc1ee18 logs: [0xc0c151b970] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000020000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000080000000000000000010000000000000000000000000000010000000000000000000000000000000800000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000400000002000080000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: \n##############################\n"
t=2023-12-10T19:17:11+0300 lvl=warn msg="Synchronisation failed, dropping peer"  peer=477bb318d8bc6b2a040b5de9313b4edc8e978a5ecbf8fd75f5590e2a07fa21fc err="retrieved hash chain is invalid: mismatching validator list on epoch block"
t=2023-12-10T19:17:11+0300 lvl=warn msg="Synchronisation failed, retrying"       peer=477bb318d8bc6b2a040b5de9313b4edc8e978a5ecbf8fd75f5590e2a07fa21fc err="peer is unknown or unhealthy"
t=2023-12-10T19:17:14+0300 lvl=info msg="Skip duplicated bad block"              number=19,249,200 hash=0xa680b2ac156470bdb8f3a2bf7b33fe5c375060d8a3f71a2a41d667fb786c8dc0
t=2023-12-10T19:17:14+0300 lvl=eror msg="\n########## BAD BLOCK #########\nBlock: 19249200 (0xa680b2ac156470bdb8f3a2bf7b33fe5c375060d8a3f71a2a41d667fb786c8dc0)\nError: mismatching validator list on epoch block\nPlatform: geth (devel) go1.20 amd64 linux\nVCS: fe928d47-\nChain config: &params.ChainConfig{ChainID:97, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:0, BerlinBlock:31103030, YoloV3Block:<nil>, CatalystBlock:<nil>, LondonBlock:31103030, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc0000564f8), KeplerTime:(*uint64)(0xc000056500), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, RamanujanBlock:1010000, NielsBlock:1014369, MirrorSyncBlock:5582500, BrunoBlock:13837000, EulerBlock:19203503, GibbsBlock:22800220, NanoBlock:23482428, MoranBlock:23603940, PlanckBlock:28196022, LubanBlock:29295050, PlatoBlock:29861024, HertzBlock:31103030, HertzfixBlock:35682300, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(nil), Parlia:(*params.ParliaConfig)(0x48e7050), IsDevMode:false}\nReceipts: \n  0: cumulative: 793712 gas: 793712 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xb5dd5e91c9d25baf7578be22e9c7e19b271da2aabccfe3f6b3ffccbf41dcdb05 logs: [0xc0ce0342c0 0xc0ce034370 0xc0ce0344d0 0xc0ce034630] bloom: 00020000080000040000200000002000000000000000000000000000000000200000000000000010000000000000000000000000000000000000001000100000000000001000000000000008000000000000000000000000000000000000000000000000000004000000000000000000000200000000000000000010000000000000000000000000000000000000000000000000400000000200000000000000000000010000000000000000010000400000000020000000000000000000000000000212000000000000000001000000000000000000001000010000000200080000000000000040000000000000000000000800000000000000000000000000 state: \n  1: cumulative: 821835 gas: 28123 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6e9821124ac17afdb8fb67e830cb988f39549a811c0ea3763bd82bc97ec62310 logs: [0xc0ce034b00] bloom: 00000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000010240000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000040000000800000000000000000020000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000100004000000000000000000000000000000000000000000000000000000000000000000000000000040000010000000000400000000000008000000000000000000000000000000000000 state: \n  2: cumulative: 1139664 gas: 317829 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x72b1f3927b2b175ef01cbd8d2806e8f345f8913117a22d002376d81532dcf956 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: \n  3: cumulative: 1208663 gas: 68999 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc5e2110a0e21f25a55651d8dba17885a50b6cbfd932f970ccd4e812adbd8e600 logs: [0xc0ce034d10 0xc0ce034dc0 0xc0ce034e70] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000100000000000000000000000000000400000000000000108000000000000000000000000000000000000000000000000000010000001000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000080000000000000000002000000000000000000000020000000000000000000000000000000000000000000000000080000000000000000000000400000000000000000000200 state: \n  4: cumulative: 1245546 gas: 36883 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x824ed14256b56ef227ab5f1adad76923a9707792122f22d162aabc6debc1ee18 logs: [0xc0ce034fd0] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000020000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000080000000000000000010000000000000000000000000000010000000000000000000000000000000800000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000400000002000080000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: \n##############################\n"
t=2023-12-10T19:17:14+0300 lvl=warn msg="Synchronisation failed, dropping peer"  peer=594b6d28bf31b2336f3fdff7853efb54decf48031560d396574ed42494f03219 err="retrieved hash chain is invalid: mismatching validator list on epoch block"
t=2023-12-10T19:17:14+0300 lvl=warn msg="Synchronisation failed, retrying"       peer=594b6d28bf31b2336f3fdff7853efb54decf48031560d396574ed42494f03219 err="peer is unknown or unhealthy"

after restart there is another error:

Dec 10 19:29:55 bsc-testnet-dev geth[31691]: WARN [12-10|19:29:55.452] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
Dec 10 19:29:56 bsc-testnet-dev geth[31691]: Chain metadata
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   databaseVersion: 8 (0x8)
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   headBlockHash: 0x3ede2da72be77364642b5c7bd7899a3c97a8448c3575e2540c39ddeb4bb7a027
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   headFastBlockHash: 0x3ede2da72be77364642b5c7bd7899a3c97a8448c3575e2540c39ddeb4bb7a027
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   headHeaderHash: 0x3ede2da72be77364642b5c7bd7899a3c97a8448c3575e2540c39ddeb4bb7a027
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   lastPivotNumber: <nil>
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   len(snapshotSyncStatus): 0 bytes
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   snapshotDisabled: false
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   snapshotJournal: 859116 bytes
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   snapshotRecoveryNumber: <nil>
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   snapshotRoot: 0xda86085462f0b1d2721cafc98a4ab3164797679f431d8f96261ab32075762af8
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   txIndexTail: 14990482 (0xe4bc92)
Dec 10 19:29:56 bsc-testnet-dev geth[31691]:   fastTxLookupLimit: <nil>
Dec 10 19:29:56 bsc-testnet-dev geth[31691]: Fatal: Could not open database: open /home/bsc/.bsc_testnet/geth/chaindata/ancient/headers.cidx: no such file or directory

@weiihann
Copy link
Contributor

@gituser I think that's a different issue. If you're still facing the problem, please open a new issue

@MatusKysel
Copy link
Contributor

@gituser you need to add flags --pruneancient=true --syncmode=full

@gituser
Copy link

gituser commented Dec 12, 2023 •
edited
Loading

@gituser you need to add flags --pruneancient=true --syncmode=full

I've added those in my config.toml it seems the bug that config isn't respected somehow.

@weiihann will do.

@MatusKysel
Copy link
Contributor

@gituser you need to add flags --pruneancient=true --syncmode=full

I've added those in my config.toml it seems the bug that config isn't respected somehow.

@weiihann will do.

you should try the from cli not in config

@zzzckck
Copy link
Collaborator

zzzckck commented Dec 13, 2023

@roflganker will

--tries-verify-mode=none will be fast node, it does not support long range unwind. Likely you didn't upgrade to v1.3.4 or v1.2.15 before the Hertzfix hard fork, which is around 07th Dec, 8am UTC.

Then your nodes will be broken, can not sync anymore, need to reset your environment.

@roflganker likely, your node is in this case, need to recover your node from scratch.
Or you can provide the node start up log, when you switched to v1.3.4. I think there could be some hint.
Here is the similar log, we observed in our node:

t=2023-12-07T08:49:47+0000 lvl=warn msg="Rewinding chain to upgrade configuration" err="mismatching hertzfix fork block in database (have timestamp 0, want timestamp 0, rewindto timestamp 0)"
t=2023-12-07T08:49:47+0000 lvl=warn msg="Rewinding blockchain to block"         target=34,140,699

I will close the issue now.

@zzzckck zzzckck closed this as completed Dec 13, 2023
@HarshitJoshi HarshitJoshi mentioned this issue Jan 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@gituser @MatusKysel @weiihann @icculp @roflganker @zzzckck