Expose Sidekiq dead job queue configuration options

Set the default limit of dead jobs to 5k, which should equal to about 50M memory
usage in the worst case. Sidekiq's default is 10k. Set the expiration
timeout to 6 weeks, matching our minor release schedule. Sidekiq's
default is 6 months.

Given the new federation code stores the XML payload in the job
parameters, fine-tuning here can give more significant memory savings.

Changelog.md

@@ -161,6 +161,7 @@ before.
 * Add a note for people with disabled JavaScript [#6777](https://github.com/diaspora/diaspora/pull/6777)
 * Do not include conversation subject in notification mail [#6910](https://github.com/diaspora/diaspora/pull/6910)
 * Add 'Be excellent to each other!' to the sidebar [#6914](https://github.com/diaspora/diaspora/pull/6910)
+* Expose Sidekiq dead queue configuration options
 
 # 0.5.11.0
 

config/defaults.yml

@@ -16,6 +16,8 @@ defaults:
       concurrency: 5
       retry: 10
       backtrace: 15
+      dead_jobs_limit: 5000
+      dead_jobs_timeout: 3628800 # 6 weeks
       log: 'log/sidekiq.log'
     s3:
       enable: false

config/diaspora.yml.example

@@ -86,10 +86,24 @@ configuration: ## Section
       #retry: 10
 
       ## Lines of backtrace that are stored on failure (default=15).
-      ## Set n to the required value. Set this to false to reduce memory
+      ## Set n to the required value. Set this to false to reduce Redis memory
       ## usage (and log size) if you're not interested in this data.
       #backtrace: 15
 
+      ## Number of jobs to keep in the dead queue (default=5000).
+      ## Jobs get into the dead queue after they failed and exhausted all retries.
+      ## Increasing this setting will increase the memory usage of Redis.
+      ## Once gone from the dead queue, a failed job is permanently lost and
+      ## cannot be retried manually.
+      # dead_jobs_limit: 1000
+
+      ## Number of seconds a job remains in the dead queue (default=3628800 (one week)).
+      ## Jobs get into the dead queue after they failed and exhausted all retries.
+      ## Increasing this setting will increase the memory usage of Redis.
+      ## Once gone from the dead queue, a failed job is permanently lost and
+      ## cannot be retried manually.
+      # dead_jobs_timeout: 15552000 # 6 months
+
       ## Log file for Sidekiq (default="log/sidekiq.log")
       #log: "log/sidekiq.log"
 

config/sidekiq.yml

@@ -5,6 +5,8 @@
 :logfile: "<%= AppConfig.sidekiq_log %>"
 <% end %>
 :concurrency: <%= AppConfig.environment.sidekiq.concurrency.to_i %>
+:dead_max_jobs: <%= AppConfig.environment.sidekiq.dead_jobs_limit.to_i %>
+:dead_timeout_in_seconds: <%= AppConfig.environment.sidekiq.dead_jobs_timeout.to_i %>
 :queues:
   - socket_webfinger
   - photos