goworker: Added 'MaxAgeRetries' option to the Goworker #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xescugc
force-pushed
the
fg-retries-clean
branch
2 times, most recently
from
7d1458b
to
616f9b2
Compare
xlr-8
suggested changes
Jul 1, 2021
| } | ||
|
|
||
| func cleanExpiredRetries() { | ||
| ok, err := client.SetNX(fmt.Sprintf("%s%s", workerSettings.Namespace, keyForCleaningExpiredRetries), os.Getpid(), cleaningExpiredRetriesInterval/2).Result() |
There was a problem hiding this comment.
What exactly does this do? And why not log something on !ok result?
Comment on lines
+250
to
+254
| client.LSet(fmt.Sprintf("%sfailed", workerSettings.Namespace), int64(i), hopefullyUniqueValueWeCanUseToDeleteJob) | ||
| client.LRem(fmt.Sprintf("%sfailed", workerSettings.Namespace), 1, hopefullyUniqueValueWeCanUseToDeleteJob) |
There was a problem hiding this comment.
The logic behind this is not obvious, why setting and then removing?
| @@ -39,6 +39,21 @@ func newWorker(id string, queues []string) (*worker, error) { | |||
| func (w *worker) MarshalJSON() ([]byte, error) { | |||
| return json.Marshal(w.String()) | |||
| } | |||
| func (w *worker) UnmarshalJSON(b []byte) error { | |||
| RetriedAt string `json:"retried_at"` | ||
| } | ||
|
|
||
| func (f *failure) GetRetriedAtTime() (time.Time, error) { |
|
Added comments to the missing/unclear parts. I will not add tests yet (tested it manually) as the full lib is without any tests and I want to make an internal refactor to facilitate testing and add testing for everything. |
xlr-8
approved these changes
Jul 19, 2021
This option is useful to automatically remove retried failed jobs from the 'failed' queue that exceede that duration, this check will be done every 1m. Also changed the 'failed.FailedAt' and added 'failed.RetriedAt' and switched them to type string. The main reason is that the Ruby lib is setting those values in an specific format and Ruby can read multiple formats into one, but GO cannot and we need to actually use the same ones or the unmarshaler does not work so I decided to switch them to 'string' and add helpers to set/get the values that will directly convert them. All the logic has more or less been ported from the Ruby version, on how to remove failed jobs and how the data is stored, as the 'MaxAgeRetries' is something unique from this GO version
xescugc
force-pushed
the
fg-retries-clean
branch
from
3debefc
to
f213a0e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This option is useful to automatically remove retried failed jobs from the 'failed' queue that exceede that duration, this
check will be done every 1m.
Also changed the 'failed.FailedAt' and added 'failed.RetriedAt' and switched them to type string. The main reason is
that the Ruby lib is setting those values in an specific format and Ruby can read multiple formats into one, but GO
cannot and we need to actually use the same ones or the unmarshaler does not work so I decided to switch them to
'string' and add helpers to set/get the values that will directly convert them.
All the logic has more or less been ported from the Ruby version, on how to remove failed jobs and how the data
is stored, as the 'MaxAgeRetries' is something unique from this GO version