More fearless features #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

etschannen merged 11 commits into apple:master from etschannen:feature-remote-logs

Mar 14, 2018

Contributor

etschannen commented Mar 9, 2018

No description provided.

etschannen added 10 commits

February 26, 2018 17:09


          changed remoteDcId to a vector of ids, to support future configuratio…

470f5c0

…ns where there are multiple remote databases


          added region-based configuration to support a large variety of fearle…

1194e3a

…ss setups. Currently only 1 primary 1 remote setups are allowed.


          fix: we must commit to the number of log routers we are going to use …

8c88041

…when recruiting the primary, because it determines the number of log router tags that will be attached to mutations


          fix: sim2 logic for when a kill is safe was incorrect

68606c7


          fix: inactive cursors are still useful if their version is larger tha…

9d4cdc8

…n the current version


          fix: shards affected by team failure did not properly handle separate…

fa7eaea

… teams for the remote and primary data centers


          Merge branch 'master' into feature-remote-logs

3abf4d7


          fix: merge problem

c74211b


          fix: do not serialize an a tlog interface without a unique id

72d56a7


          fix: the recovery actor was holding a copy of the tlogInterface after…

f6a22c1

… the tlog was removed

alexmiller-apple suggested changes

View reviewed changes

fdbserver/DatabaseConfiguration.cpp Outdated

    
              	}

              	if(satelliteTLogReplicationFactor > 0 && !satelliteTLogPolicy) {

              		satelliteTLogPolicy = IRepPolicyRef(new PolicyAcross(satelliteTLogReplicationFactor, "zoneid", IRepPolicyRef(new PolicyOne())));

              	for(auto r : regions) {

Contributor

alexmiller-apple Mar 13, 2018

I think you're missing a reference here? Your satellite policy might thus disappearing into the ether?

fdbserver/DatabaseConfiguration.cpp

    
              			return false;

              		}

              		dcIds.insert(r.dcId);

              		priorities.insert(r.priority);

Contributor

alexmiller-apple Mar 13, 2018

priorities doesn't then get used anywhere?

fdbserver/DatabaseConfiguration.cpp Outdated

    
              	std::set<Key> dcIds;

              	std::set<int> priorities;

              	dcIds.insert(Key());

              	for(auto r : regions) {

Contributor

alexmiller-apple Mar 13, 2018

I don't think r and s are a correctness problem here, but a const & would do no harm.

fdbserver/DatabaseConfiguration.cpp Outdated

    
              			result["logs"] = format("%d", desiredTLogCount);

              		if(regions.size()) {

              			StatusArray regionArr;

              			for( auto r : regions) {

Contributor

alexmiller-apple Mar 13, 2018

RegionInfo copy

fdbserver/DatabaseConfiguration.cpp Outdated

    
              			result["satellite_logs"] = format("%d", satelliteDesiredTLogCount);

              				if(r.satellites.size()) {

              					StatusArray satellitesArr;

              					for(auto s : r.satellites) {

Contributor

alexmiller-apple Mar 13, 2018

SatelliteInfo copy

fdbserver/DatabaseConfiguration.h Outdated

    
              		if(!primaryDcId.present()) return 1;

              		return 2 + primarySatelliteDcIds.size() + remoteSatelliteDcIds.size();

              		int minRequired = 0;

              		for(auto r : regions) {

Contributor

alexmiller-apple Mar 13, 2018

RegionInfo copy. And many more, so I'm going to stop commenting on them one by one.

fdbclient/ManagementAPI.actor.cpp

    
              			out[p+key] = value;

              		if( key == "regions" ) {

              			json_spirit::mValue mv;

              			json_spirit::read_string( value, mv );

Contributor

alexmiller-apple Mar 13, 2018

Is JSON the permanent plan here...?

fdbserver/DataDistribution.actor.cpp

    
              	state Transaction tr( cx );

              	state std::map<UID, Optional<Key>> server_dc;

              	state std::map<vector<UID>, std::pair<vector<UID>, vector<UID>>> team_cache;

Contributor

alexmiller-apple Mar 13, 2018

Just to double check, the possible number of distinct vector keys that team_cache can have is pretty constrained, so the fact that we never remove entries isn't a memory concern?


          use references instead of copying regionInfo

2e74105

etschannen merged commit 8045b73 into apple:master

etschannen added a commit to etschannen/foundationdb that referenced this pull request


          Merge pull request apple#63 from etschannen/feature-remote-logs

ce5b908

More fearless features

atn34 added a commit to atn34/foundationdb that referenced this pull request


          Fix UBSAN report

47de9d9

/Users/anoyes/workspace/foundationdb/fdbserver/TLogInterface.h:149:8: runtime error: load of value 232, which is not a valid value for type 'bool'
    #0 0xc608fb in TLogPeekReply::TLogPeekReply(TLogPeekReply const&) /Users/anoyes/workspace/foundationdb/fdbserver/TLogInterface.h:149
    #1 0x242bf87 in ILogSystem::ServerPeekCursor::ServerPeekCursor(TLogPeekReply const&, LogMessageVersion const&, LogMessageVersion const&, int, int, bool, long, Tag) /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:35
    #2 0x242da77 in ILogSystem::ServerPeekCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:47
    #3 0x24362d5 in ILogSystem::MergedPeekCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:325
    apple#4 0x244bf45 in ILogSystem::MultiCursor::cloneNoMore() /Users/anoyes/workspace/foundationdb/fdbserver/LogSystemPeekCursor.actor.cpp:838
    apple#5 0x36b5a36 in a_body1cont5loopBody1 /Users/anoyes/workspace/foundationdb/fdbserver/storageserver.actor.cpp:2621
    apple#6 0x36b3110 in a_body1cont5loopHead1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8664
    apple#7 0x36b07fe in a_body1cont5 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8576
    apple#8 0x36abda8 in a_body1cont4when1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8582
    apple#9 0x36a8dc2 in a_body1cont4 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8454
    apple#10 0x36a4bf6 in a_body1cont3break1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8489
    apple#11 0x36a2c01 in a_body1cont3loopBody1cont1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8505
    apple#12 0x369fd36 in a_body1cont3loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8513
    apple#13 0x3700dcb in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/storageserver.actor.g.cpp:8528
    apple#14 0x36e5210 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#15 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#16 0x2484b07 in a_body1loopBody1cont1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1526
    apple#17 0x24822cf in a_body1loopBody1cont2 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1535
    apple#18 0x247e228 in a_body1loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1541
    apple#19 0x249be87 in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:1556
    apple#20 0x249668f in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#21 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#22 0x80557e in a_body1when1 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11591
    apple#23 0x8916ef in a_callback_fire /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11620
    apple#24 0x8735f5 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#25 0x4dfb2a in SAV<Void>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#26 0x24820f8 in a_body1cont1loopBody1when1 /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:860
    apple#27 0x249c852 in a_callback_fire /Users/anoyes/build/foundationdb/fdbserver/LogSystemPeekCursor.actor.g.cpp:886
    apple#28 0x249786c in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#29 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#30 0x248b39f in a_body1cont2 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11858
    apple#31 0x2489d02 in a_body1when1 /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11865
    apple#32 0x249a150 in a_callback_fire /Users/anoyes/build/foundationdb/flow/genericactors.actor.g.h:11880
    apple#33 0x2492a4f in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#34 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#35 0x248df9b in a_body1cont2 /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2762
    apple#36 0x248b7da in a_body1when1 /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2769
    apple#37 0x2499c88 in a_callback_fire /Users/anoyes/build/foundationdb/fdbrpc/genericactors.actor.g.h:2784
    apple#38 0x2492371 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#39 0xc9d2dc in SAV<TLogPeekReply>::finishSendAndDelPromiseRef() /Users/anoyes/workspace/foundationdb/flow/flow.h:479
    apple#40 0xc60fb3 in void SAV<TLogPeekReply>::sendAndDelPromiseRef<TLogPeekReply&>(TLogPeekReply&) /Users/anoyes/workspace/foundationdb/flow/flow.h:472
    apple#41 0xc1137a in NetSAV<TLogPeekReply>::receive(ArenaObjectReader&) /Users/anoyes/workspace/foundationdb/fdbrpc/fdbrpc.h:111
    apple#42 0x78eda75 in a_body1cont1 /Users/anoyes/workspace/foundationdb/fdbrpc/FlowTransport.actor.cpp:652
    apple#43 0x78f7967 in a_body1cont2 /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2369
    apple#44 0x78ed4d8 in a_body1when1 /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2375
    apple#45 0x791af45 in a_callback_fire /Users/anoyes/build/foundationdb/fdbrpc/FlowTransport.actor.g.cpp:2390
    apple#46 0x7914670 in fire /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#47 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /Users/anoyes/workspace/foundationdb/flow/flow.h:447
    apple#48 0x959891 in void Promise<Void>::send<Void>(Void&&) const /Users/anoyes/workspace/foundationdb/flow/flow.h:778
    apple#49 0x7b4b022 in Sim2::execTask(Sim2::Task&) (/Users/anoyes/build/foundationdb/bin/fdbserver+0x7b4b022)
    apple#50 0x7bf9172 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1cont1(Void const&, int) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:979
    apple#51 0x7be7b72 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_body1loopBody1when1(Void const&, int) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5391
    apple#52 0x7c32a09 in Sim2::RunLoopActorState<Sim2::RunLoopActor>::a_callback_fire(ActorCallback<Sim2::RunLoopActor, 0, Void>*, Void) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5406
    apple#53 0x7c1fc7d in ActorCallback<Sim2::RunLoopActor, 0, Void>::fire(Void const&) /Users/anoyes/workspace/foundationdb/flow/flow.h:998
    apple#54 0x4df0dc in void SAV<Void>::send<Void>(Void&&) /Users/anoyes/workspace/foundationdb/flow/flow.h:447
    apple#55 0x959891 in void Promise<Void>::send<Void>(Void&&) const /Users/anoyes/workspace/foundationdb/flow/flow.h:778
    apple#56 0x7fe74ae in N2::PromiseTask::operator()() /Users/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:481
    apple#57 0x7fb7001 in N2::Net2::run() /Users/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:657
    apple#58 0x7b71bdd in Sim2::_runActorState<Sim2::_runActor>::a_body1(int) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:989
    apple#59 0x7b2ee5b in Sim2::_runActor::_runActor(Sim2* const&) /Users/anoyes/build/foundationdb/fdbrpc/sim2.actor.g.cpp:5608
    apple#60 0x7b2f272 in Sim2::_run(Sim2* const&) /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:987
    apple#61 0x7b2f2d2 in Sim2::run() /Users/anoyes/workspace/foundationdb/fdbrpc/sim2.actor.cpp:996
    apple#62 0x2104064 in main /Users/anoyes/workspace/foundationdb/fdbserver/fdbserver.actor.cpp:1793
    apple#63 0x7fb7c6561504 in __libc_start_main (/lib64/libc.so.6+0x22504)
    apple#64 0x464914  (/Users/anoyes/build/foundationdb/bin/fdbserver+0x464914)

atn34 pushed a commit to atn34/foundationdb that referenced this pull request


          Merge pull request apple#63 from apple/release-6.2

da618c4

Merge release-6.2

1inker referenced this pull request in owtech/foundationdb


          Fixed testcases from 'restarting' group

Test  #63: restarting/from_7.0.0/ConfigureTestRestart-1
Test  #64: restarting/from_5.0.0/CycleTestRestart-1
Test  #65: restarting/from_5.0.0/StorefrontTestRestart-1
Test  #66: restarting/from_6.2.33/SnapTestAttrition-1
Test  #67: restarting/from_6.2.33/SnapTestSimpleRestart-1
Test  #68: restarting/from_6.2.33/SnapTestRestart-1
Test  #69: restarting/from_6.2.33/SnapCycleRestart-1
Test  #70: restarting/from_5.1.7/DrUpgradeRestart-1
Test  #71: restarting/from_5.2.0/ClientTransactionProfilingCorrectness-1
Test  #72: restarting/to_6.2.33/CycleTestRestart-1

sfc-gh-ljoswiak pushed a commit to sfc-gh-ljoswiak/foundationdb that referenced this pull request


          Merge pull request apple#63 from sfc-gh-mpilman/vgasiunas/flow-serial…

355493d

…izer

Fix flow buffer test generator to support all types

vishesh pushed a commit to vishesh/foundationdb that referenced this pull request


          Merge pull request apple#63 from FoundationDB/eng/gen-cdb

33165d1

[interop] generate swift compile commands

gxglass mentioned this pull request

Updates to Platform.cpp relating to format_backtrace #12342

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet