Disable large payload stages #14023
Disable large payload stages #14023
Conversation
github-actions
bot
added
area: build
|
It seems the large payload tests are removed not disabled. Have we identified why they are running for more than a week or do we have a plan to fix them? Will these be re-enabled later? |
@agarwal-navin good questions. Yes, they are removed. I used the term 'disabled' because the code making it happen is still going to be in the repo (the code allowing for large custom payloads). They will be re-enabled sort of, by #14000, which would allow the regular test to send a large payload every N ops (right now 500 ops) but I'm still testing it privately with https://dev.azure.com/fluidframework/internal/_build/results?buildId=127797&view=results. Timeline is EOD today if stuff goes as planned. There were multiple problems with the original change and leaving them on was mostly done to expose them. Most notable being the chunk size was too small causing the test to send 1000 times more ops (in some cases). The other issue is that these stages use a lock, so while they run, other odsp/frs stages cannot. If this stage takes too long, it is going to slow down the other CI runs. |
That makes sense. Thanks! Can you please add this to the description? Having this context will be helpful. |
## Description The *:large CI stages create lots of traffic and take a long time to run. This could cause pipeline delays, as other ODSP/FRS stages are waiting on the `stress-(frs|odsp)-lock`. This has been running for almost a week, the results have been captured and configs have been adjusted to get to a point where they can be integrated with the regular stress tests run. Re-enabling chunking and compression in the stress tests will be done by microsoft#14000 after testing. There were multiple problems with the original change and leaving them on was mostly done to expose them. Most notable being the chunk size was too small causing the test to send 1000 times more ops (in some cases). The other issue is that these stages use a lock, so while they run, other odsp/frs stages cannot. If this stage takes too long, it is going to slow down the other CI runs.
Description
The *:large CI stages create lots of traffic and take a long time to run. This could cause pipeline delays, as other ODSP/FRS stages are waiting on the
stress-(frs|odsp)-lock. This has been running for almost a week, the results have been captured and configs have been adjusted to get to a point where they can be integrated with the regular stress tests run.Re-enabling chunking and compression in the stress tests will be done by #14000 after testing.
There were multiple problems with the original change and leaving them on was mostly done to expose them. Most notable being the chunk size was too small causing the test to send 1000 times more ops (in some cases).
The other issue is that these stages use a lock, so while they run, other odsp/frs stages cannot. If this stage takes too long, it is going to slow down the other CI runs.