Disable flaky filecoin test for now

[hannahhoward]

Goals

currently, seed.run PRs fail 90%+ of the time because of the Filecoin test.

until we have a more reliable way to test the filecoin flow, the PR disables it.

the inserted comment suggests a few solutions -- either spinning up custom infra for w3filecoin pipeline on each pr to get more reliable results, or mocking out the pipeline

either way, this is the temporary fix -- I don't want to put in extra retries or anything -- we just need to find a better way to test, or investigate if there are in fact underlying issues, but in the meantime, other PRs need to get merged.

[seed-deploy]

View stack outputs

• pr419-w3infra-BillingDbStack

  Name	Value
  customerTableName	pr419-w3infra-customer
  spaceDiffTableName	pr419-w3infra-space-diff
  spaceSnapshotTableName	pr419-w3infra-space-snapshot
  usageTable	pr419-w3infra-usage

• pr419-w3infra-BillingStack

  Name	Value
  ApiEndpoint	https://a4l2dy0108.execute-api.us-east-2.amazonaws.com
  billingCronHandlerURL	https://fv2uhnr3nht6vltfghout4n5xe0gdnew.lambda-url.us-east-2.on.aws/
  CustomDomain	https://pr419.billing.web3.storage

• pr419-w3infra-CarparkStack

  Name	Value
  BucketName	carpark-pr419-0
  Region	us-east-2

• pr419-w3infra-RoundaboutStack

  Name	Value
  ApiEndpoint	https://a4xifqw4b7.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr419.roundabout.web3.storage

• pr419-w3infra-UcanInvocationStack

  Name	Value
  invocationBucketName	invocation-store-pr419-0
  taskBucketName	task-store-pr419-0
  workflowBucketName	workflow-store-pr419-0

• pr419-w3infra-UploadApiStack

  Name	Value
  ApiEndpoint	https://t0j8siog0f.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr419.up.web3.storage

• pr419-w3infra-BusStack

• pr419-w3infra-FilecoinStack

• pr419-w3infra-IndexerStack

• pr419-w3infra-ReplicatorStack

• pr419-w3infra-UcanFirehoseStack

• pr419-w3infra-UploadDbStack

[alanshaw]

It loathes me to do this. The staging environment is the closest thing we have to production and the last place we can test end to end interactions across the whole system.

The flakiness needs to be fixed, not ignored. Oftentimes it is the test code that is to blame but sometimes it's an actual bug.

(FWIW I did not write these tests and it frustrates me that not enough care and attention was made to ensure they are robust.)

If the staging environment itself has caused the tests to fail (changes to the system that are not under the control of this repo) then it needs to be fixed. If this is the case and we're disabling these tests because of it then we're effectively saying our staging environment is broken and we cannot use it and we are completely blind when deploying changes to know if the entire system works together.

I understand there is an inherent problem here that we cannot deploy the entire system from a single repo. Unfortunately it's an artefact of history (and also an artefact of deployment across multiple cloud providers to optimise for cost) that I'd like to change in the future.

[Peeja]

I'm with @alanshaw's sentiments. I think if we have a test failure, that should be a blocker: either we confirm that the test is unnecessary, or we fix it. To do otherwise degrades our confidence in our continuous delivery, and the whole thing falls apart.

That said, it doesn't need to block everyone. We can operate with slightly less confidence for a limited time.

I recommend we merge this and escalate the card that resolves, so that one person is blocked on it while everyone else can move forward with slightly more caution.

[Peeja]

• Lint was failing due to unused things. I've removed them.
• Hilariously, Seed was failing with a different error. Hopefully it passes this time.

[alanshaw]

That said, it doesn't need to block everyone. We can operate with slightly less confidence for a limited time.

I recommend we merge this and escalate the card that resolves, so that one person is blocked on it while everyone else can move forward with slightly more caution.

+1 I should have cross posted here but I did suggest this same resolution in Discord.

[Peeja]

@alanshaw Added to top of sprint as storacha/project-tracking#129. Good to merge this PR?